Image processing method and apparatus, and electronic device

Abstract

An image processing method, comprising: detecting a zoom operation, the zoom operation being an operation of adjusting a first focal length to a second focal length; acquiring a first image collected by means of a lens after the zoom operation, and zoom information related to the first image, the zoom information comprising: the type of the lens before and after zooming and / or the zoom ratio of the first image; and on the basis of the zoom information, performing denoising and demosaicing on the first image by means of a neural network model. In this way, in an image processing process, zoom information in a zoom process is introduced into an input of a neural network model, so that the neural network model can obtain a capability of sensing the zoom process. Thus, one neural network model can be used to perform denoising and demosaicing processing on images taken using different lenses and different sensor modes, and computing resources, storage resources and model training costs are reduced.

Description

An image processing method, apparatus and electronic device

[0001] This application claims priority to Chinese Patent Application No. 202411434493.X, filed on October 14, 2024, entitled “An Image Processing Method, Apparatus and Electronic Device”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of artificial intelligence (AI) technology, and more particularly to an image processing method, apparatus, and electronic device. Background Technology

[0003] Joint-denoise-demosaicing (JDD) is an image processing technique applicable to fields such as digital photography. In digital photography (e.g., taking photos or videos), camera sensors capture images using arrays of color filters, most commonly Bayer color filters. This design allows each pixel to capture only one of the three colors: red, green, and blue. Therefore, a demosaicing process is needed to estimate the other two colors at each pixel location, generating a full-color image. Furthermore, the raw data output by image sensors often contains various types of noise, especially random noise under low-light conditions. Traditional image processing workflows typically perform denoising first, followed by demosaicing. However, this order can lead to over-smoothing, loss of detail, or checkerboard effects. Therefore, JDD technology was developed to combine denoising and demosaicing processes, achieving more efficient and higher-quality image reconstruction.

[0004] Currently, due to the size limitations of mobile devices and other edge devices, their imaging systems typically do not support continuous zoom mechanisms. Therefore, they often use lenses with fixed focal lengths, combined with digital zoom technology to achieve focal length variations. Simultaneously, to improve image quality within limited space, their image sensors employ various imaging modes, such as pixel binning. However, using lenses with different focal lengths and multiple imaging modes leads to changes in image data distribution and field of view (FOV), thus affecting image quality, such as causing detail loss or increased noise, posing additional challenges to image processing.

[0005] One effective solution is to apply JDD (Judges-Distributed Modeling) techniques in the RAW domain. However, traditional JDD methods are suitable for processing RAW images at a fixed resolution, maintaining the same output resolution. This requires significant computational and storage resources when processing high-resolution images, which is impractical for resource-constrained mobile devices. Therefore, in practice, JDD models with different downsampling magnifications are typically used to reduce resource requirements. During zooming, switching between JDD models with different downsampling magnifications can meet diverse imaging needs. However, since each downsampling magnification model requires independent training, this results in high training costs. Summary of the Invention

[0006] This application provides an image processing method, apparatus, electronic device, computer storage medium, and computer product that can simultaneously reduce computing resources, storage resources, and model training costs.

[0007] In a first aspect, this application provides an image processing method, comprising: detecting a zoom operation, wherein the zoom operation is an operation of adjusting a first focal length to a second focal length; acquiring a first image captured by a lens after the zoom operation and zoom information related to the first image, wherein the zoom information includes: the lens type before and after zoom and / or the zoom ratio of the first image; and, based on the zoom information, performing denoising and de-mosaicing on the first image using a neural network model.

[0008] In this way, by introducing zoom information during the zoom process into the input of the neural network model during image processing, the neural network model can gain the ability to perceive the zoom process. Thus, a single neural network model can perform denoising and de-mosaic processing on images from different lenses and sensor modes, while reducing computing resources, storage resources, and model training costs.

[0009] In one possible implementation, based on zoom information, the first image is denoised and de-mosaiced using a neural network model, including: processing the zoom information using the neural network model to obtain a scale control signal, the scale control signal including: a first input scale of the first image, the first input scale corresponding to the sampling scale of the model data processing; extracting features from the first image through an input branch in the neural network model associated with the first input scale to obtain a first feature; and denoising and de-mosaicing of the first image based on the first feature in the neural network model.

[0010] Therefore, by controlling the input scale through zoom information, different input scales can be selected for the corresponding images under different zoom information. Since different input scales correspond to different input branches, and different input branches have different downsampling magnifications, different downsampling magnifications can be achieved through a neural network model during the zoom process. Thus, a neural network model can reduce the computational and storage resources as needed during the zoom process, thereby simultaneously reducing computational resources, storage resources, and model training costs.

[0011] In one possible implementation, the scale control signal further includes a first output scale, which corresponds to the sampling scale of the model data processing. In this case, based on the first feature, denoising and de-mosaicing of the first image are performed, including: feature extraction of the first feature to obtain the backbone feature; and processing of the backbone feature through the output branch of the neural network model associated with the first output scale to achieve denoising and de-mosaicing of the first image. Therefore, by controlling the output scale through zoom information, different output scales can be selected for the corresponding image under different zoom information to ensure maximum preservation of input information, thereby avoiding premature loss of details and improving image processing quality.

[0012] In one possible implementation, the zoom information further includes: the zoom magnification of the second image and the position of the first image in the image sequence acquired after the focal length switch, wherein the second image was acquired earlier than the first image, and the acquisition times of the first and second images are consecutive; the scale control signal further includes: the second input scale of the previous frame image, which corresponds to the sampling scale of the model data processing. At this point, based on the first feature, denoising and de-mosaicing are performed on the first image, including: extracting features from the second image through the input branch in the neural network model associated with the second input scale to obtain the second feature; fusing the first feature and the second feature based on the spatial embedding feature and the field-of-view mask to obtain the fused feature, wherein the spatial embedding feature is obtained by processing the zoom information through the neural network model and is used to indicate the zoom process, and the field-of-view mask is obtained based on the registration information between the first and second images and is used to indicate the overlapping area of ​​the field of view between the first and second images; and denoising and de-mosaicing are performed on the first image based on the fused feature.

[0013] Therefore, by introducing historical data, the model can acquire the ability to perceive and fuse temporal information. Simultaneously, by using a field-of-view mask, the difficulty of model fusion learning can be reduced, and spatial scale alignment fusion between the second and first images can be achieved. Furthermore, by introducing zoom information, the model can identify changes in the zoom process during temporal fusion, achieving a smooth transition in zoom switching effects.

[0014] In one possible implementation, based on spatial embedding features and a field-of-view mask, the first and second features are fused to obtain a fused feature. This includes: combining the spatial embedding features, the field-of-view mask, the first feature, and the second feature to obtain a third feature; processing the third feature to obtain a history transformation matrix and a first transformation matrix; transforming the second feature using the history transformation matrix to obtain a history alignment feature; combining the history alignment feature and the first feature to obtain a fourth feature; and transforming the fourth feature using the first transformation matrix to obtain the fused feature. Thus, the features of the historical image and the features of the first image can be realized.

[0015] In one possible implementation, the scale control signal further includes a first output scale, which corresponds to the sampling scale of the model data processing. In this case, based on the fused features, denoising and de-mosaicing are performed on the first image, including: feature extraction from the fused features to obtain the backbone features; and processing the backbone features through the output branch in the neural network model associated with the first output scale to achieve denoising and de-mosaicing of the first image. Therefore, by controlling the output scale through zoom information, different output scales can be selected for the corresponding images under different zoom levels to ensure maximum preservation of input information, thereby avoiding premature loss of details and improving image processing quality.

[0016] In one possible implementation, after obtaining the backbone features, the process further includes: processing the backbone features based on style embedding features. These style embedding features are obtained by processing zoom information using a neural network model. The style embedding features are used to indicate how to unify the feature distribution from data from different lenses to the same lens standard. Thus, by performing style normalization on the features, the data can be scaled to a specific range or distribution, achieving uniformity in adjustment parameters such as color and brightness across different image sensors. This allows data from different sources to be compared and processed under the same standard.

[0017] In one possible implementation, based on style embedding features, the backbone features are processed, including: extracting features from the data distribution in the backbone features to obtain style features, which are used to characterize the data distribution in the backbone features; combining the style features and style embedding features to obtain a fifth feature; and processing the backbone features using the standard deviation and mean obtained through the processing of the fifth feature. This achieves style normalization of the backbone features.

[0018] Secondly, this application provides an image processing apparatus comprising a detection module and a processing module. The detection module is used to detect a zoom operation, wherein the zoom operation is an operation of adjusting a first focal length to a second focal length. The processing module is used to acquire a first image captured by the lens after the zoom operation and zoom information related to the first image, the zoom information including: the lens type before and after zooming and / or the zoom ratio of the first image. The processing module is further used to perform denoising and de-pixelation on the first image based on the zoom information using a neural network model.

[0019] In one possible implementation, when the processing module performs denoising and de-mosaicing on the first image based on zoom information using a neural network model, it is also used to: process the zoom information using the neural network model to obtain a scale control signal, the scale control signal including: a first input scale of the first image, the first input scale corresponding to the sampling scale of the model data processing; extract features from the first image through the input branch in the neural network model associated with the first input scale to obtain a first feature; and perform denoising and de-mosaicing on the first image based on the first feature in the neural network model.

[0020] In one possible implementation, the scale control signal further includes a first output scale, which corresponds to the sampling scale of the model data processing. In this case, when the processing module performs denoising and de-mosaicing on the first image based on the first feature, it is also used to: extract features from the first feature to obtain the backbone features; and process the backbone features through the output branch in the neural network model associated with the first output scale to achieve denoising and de-mosaicing of the first image.

[0021] In one possible implementation, the zoom information further includes: the zoom magnification of the second image and the position of the first image in the image sequence acquired after the focal length switch, wherein the second image was acquired earlier than the first image, and the acquisition times of the first and second images are continuous; the scale control signal further includes: the second input scale of the second image, which corresponds to the sampling scale of the model data processing. At this time, when the processing module performs denoising and de-mosaicing on the first image based on the first feature, it is also used to: extract features from the second image through the input branch in the neural network model associated with the second input scale to obtain the second feature; fuse the first feature and the second feature based on the spatial embedding feature and the field-of-view mask to obtain the fused feature, wherein the spatial embedding feature is obtained by processing the zoom information through the neural network model and is used to indicate the zoom process, and the field-of-view mask is obtained based on the registration information between the first and second images and is used to indicate the overlapping area of ​​the field of view between the first and second images; and perform denoising and de-mosaicing on the first image based on the fused feature.

[0022] In one possible implementation, when the processing module fuses the first feature and the second feature based on the spatial embedding feature and the field of view mask to obtain the fused feature, it specifically performs the following steps: combining the spatial embedding feature, the field of view mask, the first feature, and the second feature to obtain the third feature; processing the third feature to obtain the history transformation matrix and the first transformation matrix; transforming the second feature using the history transformation matrix to obtain the history alignment feature; combining the history alignment feature and the first feature to obtain the fourth feature; and transforming the fourth feature using the first transformation matrix to obtain the fused feature.

[0023] In one possible implementation, the scale control signal further includes a first output scale, which corresponds to the sampling scale of the model data processing. In this case, when the processing module performs denoising and de-mosaicing on the first image based on the fused features, it specifically performs: feature extraction on the fused features to obtain the backbone features; and processes the backbone features through the output branch in the neural network model associated with the first output scale to achieve denoising and de-mosaicing of the first image.

[0024] In one possible implementation, after obtaining the backbone features, the processing module is further used to: process the backbone features based on style embedding features, wherein the style embedding features are obtained by processing zoom information through a neural network model, and the style embedding features are used to indicate the unification of feature distributions from different lens data to the same lens standard.

[0025] In one possible implementation, when the processing module performs normalization processing on the backbone features based on the style embedding features, it specifically performs the following: extracts features from the data distribution in the backbone features to obtain style features, which are used to characterize the data distribution in the backbone features; combines the style features and style embedding features to obtain the fifth feature; and processes the backbone features using the standard deviation and mean obtained through the processing of the fifth feature.

[0026] Thirdly, this application provides an image processing apparatus, including at least one processor and an interface; the at least one processor acquires program instructions or data through the interface; the at least one processor is configured to execute program line instructions to implement the method described in the first aspect. For example, the image processing apparatus may be a chip.

[0027] Fourthly, this application provides an electronic device, comprising: one or more processors; and a memory; wherein the memory stores one or more computer programs, the one or more computer programs including instructions that, when executed by the one or more processors, cause the electronic device to perform the method described in the first aspect.

[0028] Fifthly, this application provides a computer-readable storage medium storing a computer program that, when run on an electronic device, causes the electronic device to perform the method described in the first aspect.

[0029] In a sixth aspect, this application provides a computer program product that, when run on an electronic device, causes the electronic device to perform the method described in the first aspect.

[0030] It is understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0031] Figure 1 is a schematic diagram of the structure of a neural network model provided in an embodiment of this application;

[0032] Figure 2 is a schematic diagram of another neural network model provided in an embodiment of this application;

[0033] Figure 3 is a schematic diagram of the structure of another neural network model provided in an embodiment of this application;

[0034] Figure 4 is a schematic diagram of one structure of the feature fusion network shown in Figure 3;

[0035] Figure 5 is a schematic diagram of another neural network model provided in an embodiment of this application;

[0036] Figure 6 is a schematic diagram of the structure of another neural network model provided in an embodiment of this application;

[0037] Figure 7 is a structural schematic diagram of the style normalization module shown in Figure 6;

[0038] Figure 8 is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0039] Figure 9 is a flowchart illustrating an image processing method provided in an embodiment of this application;

[0040] Figure 10 is a schematic diagram of a process for denoising and de-mosaicing an image of the current frame provided in an embodiment of this application;

[0041] Figure 11 is a schematic diagram of another process for denoising and de-mosaicing the current frame image provided in an embodiment of this application;

[0042] Figure 12 is a schematic diagram of another process for denoising and de-mosaicing an image of the current frame provided in an embodiment of this application;

[0043] Figure 13 is a schematic diagram of another process for denoising and de-mosaicing an image of the current frame provided in an embodiment of this application;

[0044] Figure 14 is a schematic diagram of the structure of an image processing device provided in an embodiment of this application;

[0045] Figure 15 is a schematic diagram of another image processing device provided in an embodiment of this application;

[0046] Figure 16 is a schematic diagram of the system architecture corresponding to a neural network model provided in an embodiment of this application;

[0047] Figure 17 is a detailed schematic diagram of the modules in the system shown in Figure 16;

[0048] Figure 18 is a schematic diagram of a process for obtaining fusion features according to an embodiment of this application;

[0049] Figure 19 is a schematic diagram of a process for obtaining style-normalized backbone features according to an embodiment of this application. Detailed Implementation

[0050] In this article, the term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A exists alone, A and B exist simultaneously, and B exists alone. The symbol " / " in this article indicates that the related objects are in an "or" relationship; for example, A / B means A or B.

[0051] The terms "first" and "second," etc., used in the specification and claims herein are used to distinguish different objects, not to describe a specific order of objects. For example, "first response message" and "second response message," etc., are used to distinguish different response messages, not to describe a specific order of response messages.

[0052] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0053] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, multiple elements means two or more elements, etc.

[0054] As can be seen from the foregoing description, configuring a JDD model within an electronic device to achieve zoom functionality in smartphones and other electronic devices inevitably leads to a waste of computational and storage resources. Furthermore, using multiple JDD models with different sampling rates results in issues with the smoothness of rate switching and high training costs. If a neural network model could be used on electronic devices to automatically adjust the sampling rate based on the zoom level, it would reduce both computational and storage resources and model training costs, while also achieving a smooth transition.

[0055] Based on the above concept, this application provides a neural network model that can take zoom information during the zoom process as input to gain the ability to perceive the zoom process. Thus, a single neural network model can perform denoising and de-mosaic processing on RAW images from different lenses and sensor modes to achieve smooth zoom functionality. The zoom information may include one or more of the following: a lens switching flag (i.e., the lens type before and after zooming), the zoom ratio of the image captured before zooming, the zoom ratio of the image captured after zooming, and the position of the image captured after zooming after the switching point (e.g., the lens or sensor mode switching point, or the focal length switching point) after the focal length switching (i.e., the position of the image captured after zooming within the image sequence captured after the focal length switching). The lens switching flag can be used to indicate the lens used before and after the lens switching. In some embodiments, the position of the image captured after zooming within the image sequence captured after the focal length switching can also be understood as the frame number of the image captured after the focal length switching.

[0056] For example, Figure 1 shows a schematic diagram of the architecture of a neural network model provided in an embodiment of this application. As shown in Figure 1, the neural network model 100 may include: a zoom information processing module 110, a multi-scale input network 120, a backbone network 130, and a cropping & scaling module 140.

[0057] The zoom information processing module 110 is mainly used to process zoom information to obtain a scale control signal. The scale control signal includes the input scale of the current frame RAW image. The scale in the scale control signal corresponds to the sampling scale of the model data processing. At this time, the zoom information may include: a lens switching flag and / or the zoom magnification of the current frame RAW image. For example, the current frame RAW image can be understood, but is not limited to, as the image captured by the lens after zooming. In some embodiments, the current frame RAW image may be the latest image captured by the lens after a focal length switch, such as the latest image captured by the lens during photography or video recording, or the latest image to be processed after a focal length switch, such as the latest image to be processed in a recorded video, etc. The specific choice depends on the actual situation and is not limited here. Additionally, the current frame RAW image can also be replaced with "current image" or any other image. For ease of description, the image will continue to be described as a "frame" below.

[0058] The multi-scale input network 120 is mainly used to select the input branch corresponding to the current frame RAW image based on the input scale of the current frame RAW image in the scale control signal, and to extract features from the current frame RAW image to obtain its features. The multi-scale input network 120 includes multiple input branches of different scales. Different input branches have different downsampling ratios. For example, one input branch may have a downsampling ratio of 1.5x, another input branch may have a downsampling ratio of 1x, and so on. Exemplarily, the multi-scale input network 120 can be, but is not limited to, a U-Net network.

[0059] The backbone network 130 is mainly used to process the features of the current frame RAW image extracted by the multi-scale input network 120 to obtain the backbone features. At this point, the backbone features output by the backbone network 130 can be understood as the original JDD result. The original JDD result refers to the image after JDD processing. The original JDD result usually has a high resolution, possibly a full-resolution image.

[0060] The cropping & scaling module 140 is primarily used to crop and reduce the original JDD result to obtain the final JDD result. Cropping refers to extracting a sub-region from the original JDD result and removing unnecessary parts to reduce redundant information and improve stability. For example, if the original image resolution is 4000x3000, a 2000x1500 sub-region can be cropped. For instance, reducing redundant information can mean removing unimportant parts of the image and retaining the core area; improving stability means that in zoom or image stabilization processes, cropping can remove unstable areas at the edges, improving image stability. Reducing the resolution refers to converting a high-resolution image to a low-resolution image to reduce computational and storage resources and improve output efficiency. For example, reducing a 2000x1500 image to 1920x1080. For instance, reduction methods can use algorithms such as bilinear interpolation and bicubic interpolation to maintain image quality.

[0061] In neural network model 100, by introducing zoom information to control the input scale, different input scales can be selected for the corresponding RAW images under different zoom information. Different input scales correspond to different input branches, and different input branches have different downsampling magnifications. In this way, different magnification downsampling can be achieved through a neural network model during the zoom process, thereby realizing the function of reducing computing and storage resources as needed during the zoom process through a neural network model.

[0062] As can be seen from the above introduction of neural network model 100, this model can select different input scales for RAW images at different focal lengths. Furthermore, to ensure maximum preservation of input information and improve image processing quality, the scale of the model's output can also be controlled. Based on this concept, this application provides another neural network model. The following describes this other neural network model.

[0063] For example, Figure 2 shows a schematic diagram of another neural network model provided in an embodiment of this application. As shown in Figure 2, the neural network model 200 may include: a zoom information processing module 210, a multi-scale input network 120, a backbone network 130, a multi-scale output network 220, and a cropping & scaling module 140. For the multi-scale input network 120, the backbone network 130, and the cropping & scaling module 140, please refer to the relevant description in Figure 1 above, and it will not be repeated here.

[0064] In Figure 2, the zoom information processing module 210 is mainly used to process zoom information to obtain a scale control signal. This scale control signal includes the output scale and the input scale of the current frame RAW image. The zoom information can include one or more of the following: a lens switching flag, the zoom ratio of the current frame RAW image, the zoom ratio of historical frame data, and the frame number of the current frame RAW image after the focal length switch. The scale control signal indicates the input scale of the current frame RAW image and the output scale of the original JDD result. The multi-scale output network 210 is mainly used to select the output branch corresponding to the appropriate output scale based on the output scale in the scale control signal, and process the features output by the backbone network 130 to obtain the original JDD result. The multi-scale output network 220 includes multiple output branches with different scales. The upsampling ratios of different output branches are different. For example, one output branch has an upsampling ratio of 1.5x, another output branch has an upsampling ratio of 1x, and so on. For example, the multi-scale output network 220 can be, but is not limited to, a U-Net network.

[0065] In neural network model 200, by introducing zoom information to control the input scale, different input scales can be selected for the corresponding RAW images under different zoom levels. Different input scales correspond to different input branches, and different input branches have different downsampling rates. This allows a single neural network model to achieve downsampling at different magnifications during zooming, thus enabling the reduction of computational and storage resources as needed during zooming. Furthermore, by introducing zoom information to control the output scale, scale control can also be applied to the model's output to ensure maximum preservation of input information, thereby avoiding premature loss of details and preventing sharpness jumps from high to low resolution.

[0066] In the aforementioned neural network models 100 and 200, to achieve a smooth transition between images before and after zooming during the zooming process and to avoid abrupt changes in FOV, sharpness, noise, color, brightness, etc., between zoomed images, historical data such as historical frame RAW images or historical frame JDD results can be added during the model processing. Based on this concept, this application provides another neural network model. The following describes this other neural network model provided by this application embodiment.

[0067] For example, Figure 3 shows a schematic diagram of another neural network model provided in an embodiment of this application. As shown in Figure 3, the neural network model 300 may include: a zoom information processing module 310, a multi-scale input network 320, an FOV mask generator 330, a feature fusion network 340, a backbone network 130, and a cropping & scaling module 140. For the backbone network 130 and the cropping & scaling module 140, please refer to the relevant description in Figure 1 above, and it will not be repeated here.

[0068] In Figure 3, the zoom information processing module 310 is mainly used to process zoom information to obtain scale control signals and spatial embedding features. The scale control signal includes the input scale of the current frame RAW image and the input scale of the historical frame data. The spatial embedding features are used to indicate the zoom process. The historical frame data are images acquired before the current frame RAW image; they can be images acquired before or after zooming, without limitation here.

[0069] The multi-scale input network 320 is primarily used to select the corresponding input branch based on the input scale of the current frame RAW image in the scale control signal, and extract features from the current frame RAW image to obtain its features; and to select the corresponding input branch based on the input scale of historical frame data in the scale control signal, and extract features from the historical frame data to obtain its features. For example, the scale of the features of the current frame and the features of the historical frames output by the multi-scale input network 320 can be the same. For example, the multi-scale input network 320 includes multiple input branches with different scales. The downsampling ratios of the different input branches are different. For example, one input branch has a downsampling ratio of 1.5x, another input branch has a downsampling ratio of 1x, and so on. For example, the multi-scale input network 320 can be, but is not limited to, a U-Net network.

[0070] The FOV Mask generator 330 is primarily used to generate a mask (i.e., an FOV mask) of the overlapping FOV regions of the current RAW image and historical frame data, based on the registration information of the current frame RAW image and historical frame data. The FOV mask indicates the overlapping FOV regions of the current frame RAW image and historical frame data. The registration information of the current frame RAW image and historical frame data characterizes their spatial correspondence, ensuring geometric alignment of images at different time points or from different viewpoints. For example, the registration information of the current frame RAW image and historical frame data can be obtained by detecting and matching feature points, calculating the transformation matrix, and applying the matrix to achieve precise alignment between the two.

[0071] The feature fusion network 340 is used to fuse the features of the current frame RAW image and historical frame data output by the multi-scale input network 320, conditioned on the spatial embedding features output by the zoom information processing module 310 and the FOV mask output by the FOV mask generator 330, to obtain fused features. Through the spatial embedding features, the feature fusion network 340 can perceive the zoom process, thereby achieving a smooth transition of spatial detail features during the fusion process. Through the FOV mask, the feature fusion network 340 can identify the regions of interest during the fusion process. Regions where the current frame RAW image and historical frame data overlap require attention, while non-overlapping regions, since they are not involved in the fusion, can be ignored or have reduced attention. In some embodiments, as shown in FIG4, the feature fusion network 340 may include: feature processing modules 341, 342, and 343, and a spatial transformation estimation module 344. The feature processing module 341 is used to process the features of the current frame RAW image output by the multi-scale input network 320 to obtain feature Fc. Feature processing module 342 processes the features of historical frame data output by multi-scale input network 320 to obtain feature Fh. Feature processing module 343 processes the FOV mask output by FOV Mask generator 330 to obtain feature Ff. Feature processing modules 341, 342, and 343 can all be constructed using base layers of deep models such as convolutional layers. The feature fusion network 340 can concatenate the features obtained by feature processing modules 341, 342, and 343, and the spatial embedding features output by zoom information processing module 310, to obtain feature Fs. For example, features Fc, Fh, Ff, and spatial embedding features can be stretched into 1-dimensional features and stacked together to construct feature Fs. Spatial transformation estimation module 344 processes feature Fs to obtain historical frame transformation matrix Hh and current frame transformation matrix Hc. The historical frame transformation matrix Hh is used to align the features of historical frames to the features of the current frame, achieving spatial alignment between the two. The current frame transformation matrix Hc is used to align the features of the current frame to features in a preset space to achieve a smooth transition. During the fusion process, the feature fusion network 340 can use the historical frame transformation matrix Hh to transform the features of historical frames to obtain historical aligned features Ft. Then, the historical aligned features Ft are concatenated with the features of the current frame (e.g., stacked), and then transformed using the current frame transformation matrix Hc to obtain the fused features. For example, the spatial transformation estimation module 344 can be constructed using a linear layer or a base layer of a similar depth model.

[0072] In neural network model 300, by introducing historical frame data, the model gains the ability to perceive and fuse temporal information. Simultaneously, guided by FOV masks, the difficulty of model fusion learning is reduced, and spatial scale alignment between historical and current frames is achieved. By introducing zoom information, the model can identify changes in the zoom process during temporal fusion, achieving a smooth transition between zoom angles. Furthermore, by controlling the input scale using zoom information, different input scales can be selected for the corresponding RAW images under different zoom conditions. Since different input scales correspond to different input branches with different downsampling rates, a single neural network model can implement downsampling at different rates during zooming, thus achieving smooth zoom functionality.

[0073] In the aforementioned neural network model 300, to ensure maximum retention of input information, the scale of the model output can also be controlled. Based on this concept, this application provides another neural network model. The following describes this other neural network model provided by this application embodiment.

[0074] For example, Figure 5 shows a schematic diagram of another neural network model provided in an embodiment of this application. As shown in Figure 5, the neural network model 500 may include: a zoom information processing module 510, a multi-scale input network 320, an FOV Mask generator 330, a feature fusion network 340, a backbone network 130, a multi-scale output network 220, and a cropping & scaling module 140. The zoom information processing module 510 is mainly used to process zoom information to obtain input scale control signals, spatial embedding features, and output scale control signals. For the multi-scale input network 320, FOV Mask generator 330, feature fusion network 340, backbone network 130, multi-scale output network 220, and cropping & scaling module 140, please refer to the relevant descriptions in Figures 1, 2, and 3 above, and they will not be repeated here.

[0075] Since the model in Figure 5 is a superposition of the three models mentioned above, it possesses the functions of all three. In other words, the model in Figure 5 can achieve both a smooth transition during zoom switching and a balance between power consumption and performance.

[0076] In the four models described above, to reduce the differences caused by varying responses of different image sensors to color, brightness, etc., the style of the features output by the backbone network 130 in the four models can be uniformly processed. This involves scaling the data to a specific range or distribution to unify the adjustment parameters for color, brightness, etc., of different image sensors, thereby enabling data from different sources to be compared and processed under the same standard. Based on this concept, this application provides another neural network model. The following describes this additional neural network model.

[0077] For example, Figure 6 shows a schematic diagram of another neural network model provided in an embodiment of this application. As shown in Figure 6, the neural network model 600 may include: a zoom information processing module 610, a multi-scale input network 320, an FOV Mask generator 330, a feature fusion network 340, a backbone network 130, a stylization normalization module 620, a multi-scale output network 220, and a cropping & scaling module 140. For the multi-scale input network 320, FOV Mask generator 330, feature fusion network 340, backbone network 130, multi-scale output network 220, and cropping & scaling module 140, please refer to the relevant descriptions in Figures 1, 2, 3, and 4 above, which will not be repeated here. In some embodiments, the system architecture corresponding to the neural network model shown in Figure 6 may be as shown in Figures 16 and 17. Among them, Figure 17 is a refinement of the system architecture shown in Figure 16. Figure 16 includes: a zoom information processing module, a multi-scale input module based on zoom information, an FOV mask generator, an FOV-based spatial fusion module, a unified backbone module, a style normalization module, and a multi-scale output module based on zoom information. Combining Figures 6 and 17, it can be seen that the zoom information processing module in Figure 16 is equivalent to the zoom information processing module 610 in Figure 6; the multi-scale input module based on zoom information in Figure 16 is equivalent to the multi-scale input network 320 in Figure 6; the FOV mask generator in Figure 16 is equivalent to the FOV mask generator 330 in Figure 6; the FOV-based spatial fusion module in Figure 16 is equivalent to the feature fusion network 340 in Figure 6; the unified backbone module in Figure 16 is equivalent to the backbone network 130 in Figure 6; the style normalization module in Figure 16 is equivalent to the style normalization module 620 in Figure 6; and the multi-scale output module based on zoom information in Figure 16 is equivalent to the multi-scale output network 220 and the cropping & scaling module 140 in Figure 6. In Figure 17, the zoom information processing module includes: a scale switching control module, a spatial embedding extraction module, and a style embedding extraction module. The scale switching control module outputs a scale control signal, the spatial embedding extraction module outputs spatial embedding features, and the style embedding extraction module outputs style embedding features.

[0078] The zoom information processing module 610 is mainly used to process zoom information to obtain scale control signals, spatial embedding features, and style embedding features. The scale control signal includes the input scale of the current frame RAW image, the input scale of historical frame data, and the output scale. The style embedding features are used to indicate the standard for unifying the feature distribution of data from different lenses to the same lens (e.g., the data distribution of the main camera lens), so that subsequent adjustments to image sensor parameters such as color and brightness can be unified.

[0079] The stylization normalization module 620 is mainly used to output style-normalized features by taking the style embedding features output by the zoom information processing module 610 as conditions and the features output by the backbone network 130 as input. In some embodiments, as shown in FIG7, the stylization normalization module 620 may include: a backbone style extraction module 621 and a style normalization feature extraction module 622. The backbone style extraction module 621 is mainly used to process the backbone features Fmaster output by the backbone network 130 to obtain style features Fs that characterize the data distribution in the backbone features Fmaster. The stylization normalization module 620 can concatenate the style features Fs and the style embedding features output by the zoom information processing module 610 to obtain concatenated features Fp. For example, the style features Fs and the style embedding features can be stretched into 1-dimensional features and stacked together to construct the concatenated features Fp. The style normalization feature extraction module 622 is mainly used to process the concatenated features Fp to obtain the normalized style [sn, bn]. Where sn is the standard deviation and bn is the mean. The stylization normalization module 620 can use the normalized style [sn, bn] to process the backbone feature Fmaster output by the backbone network 130, that is, Fmaster*sn+bn, to obtain the style-normalized feature.

[0080] It should be understood that the style normalization module 620 can also be configured in the aforementioned neural network models 100, 200, 300, and 500. In this case, the corresponding zoom information processing modules in neural network models 100, 200, 300, and 500 can all generate style embedding features. The configuration of the style normalization module 620 in neural network models 100, 200, 300, and 500 is the same as that in neural network model 600, and will not be described in detail here. In addition, the RAW images described above can also be replaced with other images, such as images processed by beautification, blurring, etc., and the replaced scheme is still within the protection scope of this application.

[0081] To facilitate understanding of the working process of the above model, the following description will use a video recording scenario and the working process of the neural network model 600 shown in Figure 6 as an example.

[0082] For example, taking the main camera's FHD zoom switching threshold as an example, the model needs to switch from 1.5x downsampling to 1x resolution. The current frame information is: RAW image, BGGR sensor mode, size 3920x2204, channels split before input, data rearranged to 1960x1102x4. The historical frame information is: the JDD result of the previous frame, channels arranged according to BGR, size 1920x1080, spatially PixelShuffled, deformed to 960x540x4. Zoom information is: both before and after the switch are main camera (marked with 0), zoom ratio of the previous and current frames (1.35, 1.36), and the first frame after the switching point. Therefore, the zoom information is represented by a vector as: [0 (lens type before switching), 0 (lens type after switching), 1.35, 1.36, 1 (frame number after switching point)]. The registration information between the current frame and historical frames is the registration matrix H from the previous frame to the next frame. Switching from 1.5x downsampling to 1x transient: The resolution output by the 1.5x model is smaller than the final target resolution (1920x1080), so it is necessary to switch to the 1x model. In this case, the result of the 1x model needs to be cropped and then reduced to the target resolution.

[0083] Under the above conditions, the working process of neural network model 600 is as follows:

[0084] 1) The zoom information is input into the zoom information processing module 610 to obtain the scale control signal, spatial embedding feature, and style embedding feature. Specifically, based on the zoom information [0,0,1.35,1.36,1], the model downsampling rate is switched. The historical frame selects the 1.5 downsampling branch input, and the current frame selects the 1x branch input, using the 1x branch output to obtain the scale control signal: [1.5,1,1]. In addition, with the zoom information [0,0,1.35,1.36,1] as input, after passing through two linear layers, the spatial embedding feature Fsp and the style embedding feature Fst are output using sigmoid activation.

[0085] 2) Based on the scale control signal, the RAW images of the historical frame and the current frame are input into the input branches of different scale levels in the multi-scale input network 320 to obtain the features of the historical frame and the features of the current frame, respectively. Specifically, the multi-scale input network 320 receives the scale control signal [1.5,1,1]. Based on the value of the first bit, 1.5, the historical frame is input into the 1.5 downsampling branch to obtain the historical frame feature Fh; and based on the value of the second bit, 1, the current frame is input into the 1x branch to obtain the current frame feature Fc.

[0086] 3) Input the registration matrix H into the FOVMask generator 330 to obtain the mask of the FOV overlapping area between the historical frame and the current frame. The mask marks the FOV overlapping area as 1 and the non-overlapping area as 0.

[0087] 4) In the feature fusion network 340, the features of the historical frame and the current frame are fused using the FOV mask and spatial embedding feature Fsp as conditions to obtain fused features. See the relevant description in Figure 4 above; it will not be repeated here. Alternatively, see Figure 18. As shown in Figure 18, the specific steps are as follows: Step 4-1: The current frame input feature (feature 1) is processed by feature processing 1 (convolutional layer, etc.) to obtain feature Fc_d; Step 4-2: The historical frame input feature (feature 2) is processed by feature processing 2 (convolutional layer, etc.) to obtain feature Fh_d; Step 4-3: The FOVMask is processed by feature processing 3 (convolutional layer, etc.) to obtain feature Ffov_d; Step 4-4: Features Fc_d, Fh_d, Ffov_d, and Fsp are... The features are stretched into 1D features and stacked together to construct feature Fsp_d; Step 4-5: Input Fsp_d into the spatial transformation estimation submodule of the linear layer and the base layer of the same depth model to obtain the historical frame transformation matrix Hh and the current frame transformation matrix Hc; Step 4-6: Use the historical frame transformation matrix Hh to transform the historical frame input features to obtain the historical aligned features Fh_t; Step 4-7: Stack the historical aligned features Fh_t and the current frame input features, and use the current frame transformation matrix Hc to transform them to obtain the fused output features Ffuse.

[0088] 5) The fused features are used as input to the backbone network 130. After processing by the backbone network 130, the backbone features are output.

[0089] 6) In the style normalization module 620, the style embedding feature is used as a condition and the backbone feature is used as input to output the style-normalized backbone feature. See the relevant description in Figure 7 above, which will not be repeated here. Alternatively, see Figure 19. As shown in Figure 19, the specific steps are as follows: Step 6-1: Using the backbone feature Fmaster as input, the backbone style extraction module, constructed through the convolutional layer and other deep model base layers, obtains the backbone style feature Fst_m; Step 6-2: The backbone style feature Fst_m and the style embedding feature Fst are stretched into 1-dimensional features and stacked together to construct the feature Fst_d; Step 6-3: The feature Fst_d is input into the style normalization feature extraction module to obtain the normalized style [sn,bn]; Step 6-4: The normalized style is used to process the backbone feature Fmaster (Fmaster*sn+bn) to obtain the style-normalized backbone feature Fm_st.

[0090] 7) In the multi-scale output network 220, the style-normalized backbone features are used as input, and the output scale is selected according to the multi-scale control signal to obtain the original JDD result. Specifically, the multi-scale output network 220 receives the scale control signal [1.5,1,1], and according to the value of the third bit 1, the style-normalized backbone features are input into the 1x branch to obtain the original JDD result.

[0091] 8) The cropping & scaling module 140 receives the scale control signal [1.5,1,1], judges the value of the first and second bits, performs cropping and scaling, and obtains the final JDD result.

[0092] The above is a description of the neural network model provided in the embodiments of this application. As can be seen from the above description, by introducing zoom information into the model input, the model gains the ability to perceive the zoom process. Furthermore, by selecting different input and output scales based on the zoom information, both power consumption and performance can be considered. In addition, by introducing zoom information and a FOV mask, a smooth transition of the zoom effect can also be achieved.

[0093] Furthermore, by deploying the neural network model described above on mobile phones and other electronic devices, a user-unobtrusive zoom function can be achieved on the electronic device. This allows users to experience virtually no zoom operation in scenarios involving focus switching, such as video recording or photography, resulting in a smoother and more natural viewing experience and significantly enhancing the user's visual experience. The following describes an electronic device provided by an embodiment of this application. This electronic device can be a mobile phone, tablet computer, desktop computer, laptop computer, handheld computer, notebook computer, ultra-mobile personal computer (UMPC), netbook, as well as cellular phones, personal digital assistants (PDAs), augmented reality (AR) devices, virtual reality (VR) devices, artificial intelligence (AI) devices, wearable devices, in-vehicle devices, smart home devices, and / or smart city devices, etc. This application embodiment does not impose any special limitations on the specific type of this electronic device.

[0094] As shown in Figure 8, the electronic device 800 may include: a processor 810, a universal serial bus (USB) interface 820, a display screen 830, a camera 840, an indicator 850, buttons 860, a sensor module 870, a charging management module 880, a power management module 881, a battery 882, and an AI accelerator 890. The sensor module 870 may include: a proximity sensor 870A, an ambient light sensor 870B, a touch sensor 870C, an accelerometer sensor 870D, and a gyroscope sensor 870E, etc.

[0095] The processor 810 may include one or more processing units, such as an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), and / or a baseband processor. These different processing units may be independent devices or integrated into one or more processors. The controller can generate operation control signals based on the instruction opcode and timing signals to control instruction fetching and execution.

[0096] The processor 810 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 810 is a cache memory. This memory can store instructions or data that the processor 810 has just used or that are used repeatedly. If the processor 810 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 810, and thus improves the efficiency of the system.

[0097] In some embodiments, the processor 810 may include one or more interfaces. Interfaces may include inter-integrated circuit (I2C) interfaces, mobile industry processor interfaces (MIPI), general-purpose input / output (GPIO) interfaces, and / or universal serial bus (USB) interfaces, etc.

[0098] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 810 may include multiple I2C buses. The processor 810 can couple to the touch sensor 870C, charger, flash, camera 840, etc., through different I2C bus interfaces. For example, the processor 810 can couple to the touch sensor 870C through the I2C interface, enabling the processor 810 and the touch sensor 870C to communicate through the I2C bus interface, thereby realizing the touch function of the electronic device 800.

[0099] The MIPI interface can be used to connect the processor 810 to peripheral devices such as the display screen 830 and the camera 840. The MIPI interface includes a camera serial interface (CSI) and a display serial interface (DSI). In some embodiments, the processor 810 and the camera 840 communicate via the CSI interface to enable the electronic device 800 to capture images. The processor 810 and the display screen 830 communicate via the DSI interface to enable the electronic device 800 to display images.

[0100] The GPIO interface is configurable via software. It can be configured as a control signal or a data signal. In some embodiments, the GPIO interface can be used to connect the processor 810 to the camera 840, display 830, sensor module 870, etc. The GPIO interface can also be configured as an I2C interface, MIPI interface, etc.

[0101] The USB interface 820 is a USB standard compliant interface, which can be a Mini USB interface, Micro USB interface, USB Type-C interface, etc. The USB interface 820 can be used to connect a charger to charge electronic device 800, and can also be used for data transfer between electronic device 800 and peripheral devices. It can also be used to connect other electronic devices, such as AR devices.

[0102] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the electronic device 800. In other embodiments of this application, the electronic device 800 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0103] The charging management module 880 receives charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 880 receives charging input from the wired charger via a USB interface 820. In some wireless charging embodiments, the charging management module 880 receives wireless charging input via the wireless charging coil of the electronic device 800. While charging the battery 882, the charging management module 880 can also supply power to the electronic device 800 via the power management module 881.

[0104] The power management module 881 connects the battery 882, the charging management module 880, and the processor 810. The power management module 881 receives input from the battery 882 and / or the charging management module 880, providing power to the processor 810, display screen 830, and camera 840, etc. The power management module 881 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In some other embodiments, the power management module 881 may also be located within the processor 810. In still other embodiments, the power management module 881 and the charging management module 880 may be housed in the same device.

[0105] Electronic device 800 implements display functions through a GPU, a display screen 830, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 830 and the application processor. The GPU performs mathematical and geometric calculations and is used for graphics rendering. Processor 810 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0106] The display screen 830 is used to display images, videos, etc. In some embodiments, the electronic device 800 may include one or M display screens 830, where M is a positive integer greater than 1.

[0107] Electronic device 800 can perform shooting functions through ISP, camera 840, GPU, display 830 and application processor.

[0108] The ISP (Image Signal Processor) is used to process data fed back from the camera 840. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimization of image noise, brightness, and color. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 840.

[0109] Camera 840 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, electronic device 800 may include one or N cameras 840, where N is a positive integer greater than 1.

[0110] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when electronic device 800 is selecting a frequency, the DSP is used to perform Fourier transforms on the frequency energy.

[0111] Indicator 850 can be an indicator light, used to indicate charging status, power changes, or to indicate messages, notifications, etc.

[0112] Buttons 860 include a power button, volume buttons, zoom buttons, etc. Buttons 860 can be mechanical buttons, touch buttons, or virtual buttons, etc. Electronic device 800 can receive button input and generate key signal inputs related to user settings and function control of electronic device 800, such as zoom signals.

[0113] The gyroscope sensor 870E can be used to determine the motion attitude of the electronic device 800. In some embodiments, the gyroscope sensor 870E can determine the angular velocity of the electronic device 800 around three axes (i.e., the x, y, and z axes). The gyroscope sensor 870E can be used for image stabilization. For example, when the shutter is pressed, the gyroscope sensor 870E detects the angle of the electronic device 800's shake, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to counteract the shake of the electronic device 800 through reverse movement, thus achieving image stabilization. The gyroscope sensor 870E can also be used in navigation and motion-sensing gaming scenarios.

[0114] The 870D accelerometer sensor can detect the magnitude of acceleration of an electronic device 800 in various directions (typically three axes). When the electronic device 800 is stationary, it can detect the magnitude and direction of gravity. It can also be used to identify the posture of the electronic device, and can be applied to applications such as screen orientation switching and pedometers.

[0115] A distance sensor 870A is used to measure distance. Electronic device 800 can measure distance via infrared or laser. In some embodiments, during a shooting scene, electronic device 800 can utilize the distance sensor 870A to measure distance for rapid focusing.

[0116] The ambient light sensor 870B is used to sense the brightness of ambient light. The electronic device 800 can adaptively adjust the brightness of the display screen 830 based on the sensed ambient light brightness. The ambient light sensor 870B can also be used to automatically adjust the white balance when taking a picture.

[0117] Touch sensor 870C, also known as a "touch device," can be located on display screen 830. The touch sensor 870C and display screen 830 together form a touchscreen, also known as a "touchscreen." Touch sensor 870C detects touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 830. In other embodiments, touch sensor 870C may also be located on the surface of electronic device 800, in a different position than display screen 830.

[0118] AI accelerator 890 is hardware specifically designed to perform artificial intelligence and machine learning tasks, and can be used to support any of the neural network models described in Figures 1 to 7 above. AI accelerator 890 can be, but is not limited to, a tensor processing unit, a field-programmable gate array, an application-specific integrated circuit (ASIC), or a neural processing unit. In some embodiments, after processing the scale control signal, AI accelerator 890 can control which circuits on the hardware circuitry associated with the neural network model are activated or deactivated to determine the input branches processing data in the multi-scale input network of the neural network model, and / or, to determine the output branches processing data in the multi-scale output network of the neural network model. For example, during zooming, the enabled and disabled states of certain hardware circuitry logic units on electronic device 800 can change with zooming.

[0119] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 800. In other embodiments of this application, the electronic device 800 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0120] Next, based on the content described above, an image processing method provided by an embodiment of this application will be introduced. It is understood that this method is proposed based on the content described above, and some or all of the content in this method can be found in the relevant descriptions above.

[0121] Please refer to Figure 9, which shows a flowchart of an image processing method provided in an embodiment of this application. It is understood that this method can be executed by any device, apparatus, platform, or cluster of devices with computing and processing capabilities. For example, the method can be executed by an image processing device, which can be implemented by software and / or hardware, and can be, but is not limited to, configured in an electronic device, typically in a mobile phone. For ease of description, the following description will use an electronic device as the execution subject. As shown in Figure 9, the image processing method includes:

[0122] S901. A zoom operation was detected. The zoom operation is the operation of adjusting the first focal length to the second focal length.

[0123] In this embodiment, the zoom operation can be triggered by the user or automatically by the system; no limitation is made here. For example, a user can manually adjust the focus while taking an image, or the electronic device can automatically adjust the focus while taking an image (i.e., taking a picture) or recording a video (i.e., recording a video). When a focus change occurs on the electronic device, the electronic device can detect the zoom operation.

[0124] S902. Obtain the first image captured by the lens after the zoom operation and the zoom information related to the first image. The zoom information includes: the lens type before and after zoom and / or the zoom ratio of the first image.

[0125] In this embodiment, the electronic device can capture images through its lens. After the focus is switched, the electronic device can acquire the first image captured by its lens, as well as zoom information related to the first image. This zoom information may include: the lens type before and after zooming and / or the zoom ratio of the first image. It should be understood that the first image can be a frame from a video or an image from a photographic scene, depending on the actual situation, and is not limited here. In some embodiments, the image captured by the lens on the electronic device at the current moment or the latest image captured by the lens can be referred to as the "first image". Additionally, the first image can be the first image in the image sequence captured after the focus is switched, or it can be any other image, and is not limited here. In the video captured by the electronic device, the first image can be a frame captured after the focus is switched. It should be understood that when the electronic device is replaced by another device, the lens can be a lens associated with that other device. Furthermore, the first image can be the original image captured by the lens, or it can be an image processed with beautification, blurring, etc., depending on the actual situation, and is not limited here.

[0126] S903. Based on zoom information, the first image is denoised and de-pixelated using a neural network model.

[0127] In this embodiment, after acquiring the first image and corresponding zoom information, the electronic device can guide a neural network model to denoise and remove mosaic effects from the first image using the zoom information. Specifically, the electronic device can input the zoom information and the first image into the neural network model, which will then process the zoom information and the first image to achieve denoising and mosaic removal. For example, the neural network model can be any of the neural network models described in Figures 1, 2, 3, 5, and 6.

[0128] In this way, by introducing zoom information during the zoom process into the input of the neural network model during image processing, the neural network model can gain the ability to perceive the zoom process. Thus, a single neural network model can perform denoising and de-mosaic processing on images from different lenses and sensor modes, while reducing computing resources, storage resources, and model training costs.

[0129] In some embodiments, in S903, when the neural network model is the neural network model 100 described in FIG1, as shown in FIG10, the process of denoising and de-mosaicing the first image may include: in S1001, processing the zoom information through the neural network model to obtain a scale control signal, the scale control signal including: a first input scale of the first image, which corresponds to the sampling scale of the model data processing. At this time, the zoom information may include: lens identification before and after focus and / or the zoom magnification of the first image. For example, the scale control signal can be obtained by processing the zoom information through the zoom information processing module 110 described in FIG1.

[0130] In S1002, feature extraction is performed on the first image through the input branch in the neural network model associated with the first input scale to obtain the first feature. For example, the first feature can be obtained by extracting features from the first image through the input branch in the multi-scale input network 120 described in Figure 1 associated with the first input scale.

[0131] In S1003, within the neural network model, denoising and de-mosaicing are performed on the first image based on the first feature. For example, the first feature can be processed using the backbone network 130 described in Figure 1 to achieve denoising and de-mosaicing of the first image.

[0132] For a detailed description of the implementation process of each step in Figure 10, please refer to the aforementioned description of the neural network model 100 shown in Figure 1, which will not be repeated here.

[0133] In Figure 10, since the multi-scale input network 120 in the neural network model 100 can perform sampling at different scales during model data processing, the neural network model 100 can process images acquired at different focal lengths. By controlling the input scale through zoom information, different input scales can be selected for the corresponding images under different zoom information; and different input scales correspond to different input branches, and different input branches have different downsampling magnifications. In this way, different downsampling magnifications can be achieved through a neural network model during the zoom process, thereby realizing the function of reducing computing and storage resources as needed during the zoom process through a single neural network model.

[0134] Furthermore, when the neural network model 100 is configured with the style normalization module 620 described in Figure 6, style embedding features can be obtained by processing zoom information through the neural network. These style embedding features are used to indicate the standard for unifying the feature portions of data from different lenses to the same lens standard. At this point, when denoising and de-mosaicing the first image based on the first feature, feature extraction can be performed on the data distribution in the first feature to obtain style features. These style features are used to characterize the data distribution in the first feature. Then, the style features and style embedding features are combined to obtain corresponding combined features. Next, the standard deviation and mean obtained through combined feature processing are used to process the first feature. Finally, denoising and de-mosaicing are performed on the first image based on the processed first feature. In this way, the data can be scaled to a specific range or distribution to achieve the unification of adjustment parameters such as color and brightness from different image sensors, thereby enabling data from different sources to be compared and processed under the same standard.

[0135] In some embodiments, in S903, when the neural network model is the neural network model 200 described in FIG2, as shown in FIG11, the process of denoising and de-mosaicing the first image may include: in S1101, processing the zoom information through the neural network model to obtain a scale control signal; the scale control signal includes: a first input scale of the first image, and a first output scale. The first output scale corresponds to the sampling scale of the model data processing. At this time, the zoom information may include: the lens type before and after zooming and / or the zoom ratio of the first image. For example, the scale control signal can be obtained by processing the zoom information through the zoom information processing module 210 described in FIG2.

[0136] In S1102, feature extraction is performed on the first image through the input branch in the neural network model associated with the first input scale to obtain the first feature. For example, the first feature can be obtained by extracting features from the first image through the input branch in the multi-scale input network 120 described in Figure 2 associated with the first input scale.

[0137] S1103, feature extraction is performed on the first feature to obtain the backbone feature. For example, the backbone feature can be obtained by extracting features from the first feature through the backbone network 130 described in Figure 2.

[0138] S1104, the backbone features are processed through the output branch associated with the first output scale in the neural network model to achieve denoising and de-mosaicing of the first image. For example, the backbone features can be processed through the output branch associated with the first output scale in the multi-scale output network 220 described in Figure 2 to achieve denoising and de-mosaicing of the first image.

[0139] For a detailed description of the implementation process of each step in Figure 11, please refer to the aforementioned description of the neural network model 200 shown in Figure 2, which will not be repeated here.

[0140] In Figure 11, during the data processing of the neural network model, the input and output scales are controlled by zoom information. This allows for the selection of different input and output scales for the corresponding images under different zoom levels. In this way, during the zooming process, a neural network model can reduce computational and storage resources as needed while ensuring maximum retention of input information, thereby avoiding premature loss of details and improving image processing quality.

[0141] Furthermore, when the neural network model 200 is configured with the style normalization module 620 described in Figure 6, style embedding features can be obtained by processing zoom information through the neural network. These style embedding features are used to indicate the standard for unifying the feature distribution of data from different lenses to the same lens standard. At this point, after extracting the backbone features from the first feature, feature extraction can be performed on the data distribution within the backbone features to obtain style features. These style features are used to characterize the data distribution within the backbone features. Then, the style features and style embedding features are combined to obtain corresponding combined features. Next, the backbone features are processed using the standard deviation and mean obtained through the combined feature processing. Finally, the normalized backbone features are processed through the output branch associated with the first output scale in the neural network model, and the first image is denoised and de-mosaiced. In this way, the data can be scaled to a specific range or distribution to achieve the unification of adjustment parameters such as color and brightness from different image sensors, thereby enabling data from different sources to be compared and processed under the same standard.

[0142] In some embodiments, in S903, when the neural network model is the neural network model 300 described in FIG3, as shown in FIG12, the process of denoising and de-mosaicing the first image may include: in S1201, processing the zoom information through the neural network model to obtain a scale control signal, the scale control signal including: a first input scale of the first image and a second input scale of the second image. The first input scale and the second input scale both correspond to the sampling scale of the model data processing. The second image was acquired earlier than the first image, and the acquisition times of the first image and the second image are consecutive. That is, the second image is a historical image. At this time, the zoom information may include: the lens type before and after zooming and / or the zoom ratio of the first image, and the zoom ratio of the second image and the position of the first image in the image sequence acquired after the focal length switch. Additionally, processing the zoom information through the neural network model can also obtain spatial embedding features that indicate the zoom process. For example, both the scale control signal and the spatial embedding features can be obtained by processing the zoom information through the zoom information processing module 310 described in FIG3.

[0143] In S1202, feature extraction is performed on the first image through the input branch in the neural network model associated with the first input scale to obtain the first feature. For example, the first feature can be obtained by extracting features from the first image through the input branch in the multi-scale input network 320 described in Figure 3 associated with the first input scale.

[0144] In S1203, feature extraction is performed on the second image through the input branch in the neural network model associated with the second input scale to obtain the second feature. For example, the second feature can be obtained by extracting features from the second image through the input branch in the multi-scale input network 320 described in Figure 3 associated with the second input scale.

[0145] In S1204, a field-of-view (FOV) mask is obtained based on the registration information between the first image and the second image. The FOV mask is used to indicate the overlapping area of ​​the FOVs between the first image and the second image. For example, the FOV mask can be obtained by processing the registration information between the first image and the second image using the FOV mask generator 330 described in FIG3.

[0146] In step S1205, the first feature and the second feature are fused based on the spatial embedding feature and the field-of-view mask to obtain the fused feature. For example, the fused feature can be obtained by processing the spatial embedding feature, the field-of-view mask, the first feature, and the second feature using the feature fusion network 340 described in Figure 3. For example, during fusion, the spatial embedding feature, the field-of-view mask, the first feature, and the second feature can be combined first, for example, by splicing and stacking, to obtain the third feature. Next, the third feature is processed to obtain the history transformation matrix and the first transformation matrix. Then, the second feature is transformed using the history transformation matrix to obtain the history alignment feature. Next, the history alignment feature and the first feature are combined to obtain the fourth feature. Finally, the fourth feature is transformed using the first transformation matrix to obtain the fused feature. The fusion process can be referred to the description of the feature fusion network 340 in Figure 3 above, and will not be repeated here.

[0147] In S1206, based on the fusion features, denoising and de-mosaic are performed on the first image. For example, the fusion features can be processed by the backbone network 130 described in Figure 3 to achieve denoising and de-mosaic on the first image.

[0148] For a detailed description of the implementation process of each step in Figure 12, please refer to the aforementioned description of the neural network model 300 shown in Figure 3, which will not be repeated here.

[0149] In Figure 12, during the data processing of the neural network model, by introducing historical data, zoom information, and field-of-view mask, the model can acquire the ability to perceive and fuse temporal information, identify changes in the zoom process, and achieve spatial scale alignment and fusion of the first and second images. Thus, a smooth zoom function can be achieved through a neural network model, while simultaneously reducing computational and storage resources as needed during the zoom process.

[0150] Furthermore, when the neural network model 300 is equipped with the style normalization module 620 described in Figure 6, style embedding features can be obtained by processing zoom information through the neural network. These style embedding features are used to indicate the standard for unifying the feature distribution of data from different lenses to the same lens standard. At this point, after extracting the backbone features from the fused features, feature extraction can be performed on the data distribution within the backbone features to obtain style features. These style features characterize the data distribution within the backbone features. Then, the style features and style embedding features are combined to obtain corresponding combined features. Next, the standard deviation and mean obtained through the combined feature processing are used to perform style normalization on the fused features. Finally, the style-normalized backbone features are processed through the output branch associated with the first output scale in the neural network model to denoise and de-mosaic the first image. In this way, the data can be scaled to a specific range or distribution to achieve uniformity of adjustment parameters such as color and brightness from different image sensors, allowing data from different sources to be compared and processed under the same standard.

[0151] In some embodiments, in S903, when the neural network model is the neural network model 500 described in FIG5, as shown in FIG13, the process of denoising and de-mosaicing the first image may include: in S1301, processing the zoom information through the neural network model to obtain a scale control signal. The scale control signal includes: a first input scale of the first image and a second input scale of the second image, and an output scale. The first input scale, the second input scale, and the output scale all correspond to the sampling scale of the model data processing. The second image is acquired earlier than the first image, and the acquisition times of the first image and the second image are consecutive. At this time, the zoom information may include: the lens type before and after zooming and / or the zoom ratio of the first image, and the zoom ratio of the second image and the position of the first image in the image sequence acquired after the focal length switch. Additionally, processing the zoom information through the neural network model can also obtain spatial embedding features that indicate the zoom process. For example, both the scale control signal and the spatial embedding features can be obtained by processing the zoom information through the zoom information processing module 510 described in FIG5.

[0152] In S1302, feature extraction is performed on the first image through the input branch in the neural network model associated with the first input scale to obtain the first feature. For example, the first feature can be obtained by extracting features from the first image through the input branch in the multi-scale input network 320 described in Figure 5 associated with the first input scale.

[0153] In S1303, feature extraction is performed on the second image through the input branch in the neural network model associated with the second input scale to obtain the second feature. For example, the second feature can be obtained by extracting features from the second image through the input branch in the multi-scale input network 320 described in Figure 5 associated with the second input scale.

[0154] In S1304, a field-of-view (FOV) mask is obtained based on the registration information between the first image and the second image. The FOV mask is used to indicate the overlapping area of ​​the FOVs between the first image and the second image. For example, the FOV mask can be obtained by processing the registration information between the first image and the second image using the FOV mask generator 330 described in FIG5.

[0155] In S1305, the first feature and the second feature are fused based on the spatial embedding feature and the field of view mask to obtain the fused feature. For example, the fused feature can be obtained by processing the spatial embedding feature, the field of view mask, the first feature, and the second feature through the feature fusion network 340 described in Figure 5.

[0156] In S1306, feature extraction is performed on the fused features to obtain the backbone features. For example, the backbone features can be obtained by extracting features from the fused features using the backbone network 130 described in Figure 5.

[0157] In S1307, the backbone features are processed through the output branch associated with the first output scale in the neural network model to achieve denoising and de-mosaicing of the first image. For example, the backbone features can be processed through the output branch associated with the first output scale in the multi-scale output network 220 described in Figure 5 to achieve denoising and de-mosaicing of the first image.

[0158] For a detailed description of the implementation process of each step in Figure 11, please refer to the aforementioned description of the neural network model 500 shown in Figure 5. It will not be repeated here.

[0159] In Figure 11, the neural network model used is the same as that in Figure 5. The model in Figure 5 is a superposition of the three models described in Figures 1, 2, and 3, thus possessing the functions of all three models. Therefore, by performing denoising and de-mosaic processing as shown in Figure 11, a smooth transition during zoom switching can be achieved, while also balancing power consumption and performance.

[0160] Furthermore, when the neural network model 500 is equipped with the style normalization module 620 described in Figure 6, style embedding features can be obtained by processing zoom information through the neural network. These style embedding features are used to indicate the standard for unifying the feature distribution of data from different lenses to the same lens standard. At this point, after extracting the backbone features from the fused features, feature extraction can be performed on the data distribution within the backbone features to obtain style features. These style features characterize the data distribution within the backbone features. Then, the style features and style embedding features are combined to obtain the corresponding combined features. Next, the standard deviation and mean obtained through the combined feature processing are used to perform style normalization on the fused features. Finally, the style-normalized backbone features are processed through the output branch associated with the first output scale in the neural network model, resulting in denoising and de-mosaicing of the first image. In this way, the data can be scaled to a specific range or distribution to achieve uniformity of adjustment parameters such as color and brightness from different image sensors, allowing data from different sources to be compared and processed under the same standard.

[0161] It is understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. In addition, the various embodiments or technical features involved in the embodiments described above can be combined according to the actual situation, and the combined solution is still within the protection scope of this application.

[0162] Based on the methods in the above embodiments, this application also provides an image processing apparatus.

[0163] For example, Figure 14 shows a schematic diagram of an image processing apparatus provided in an embodiment of this application. As shown in Figure 14, the image processing apparatus 1400 includes a detection module 1401 and a processing module 1402. The detection module 1401 is used to detect a zoom operation, which is an operation to adjust a first focal length to a second focal length. The processing module 1402 is used to acquire a first image captured by the lens after the zoom operation and zoom information related to the first image, including the lens type before and after zooming and / or the zoom ratio of the first image. The processing module 1402 is also used to perform noise reduction and de-pixelation on the first image based on the zoom information using a neural network model.

[0164] In some embodiments, when the processing module 1402 performs denoising and de-mosaicing on the first image based on zoom information using a neural network model, it is further configured to: process the zoom information using a neural network model to obtain a scale control signal, the scale control signal including: a first input scale of the first image, the first input scale corresponding to the sampling scale of the model data processing; extract features from the first image through an input branch in the neural network model associated with the first input scale to obtain a first feature; and perform denoising and de-mosaicing on the first image based on the first feature in the neural network model.

[0165] In some embodiments, the scale control signal further includes a first output scale, which corresponds to the sampling scale of the model data processing. At this time, when the processing module 1402 performs denoising and de-mosaicing on the first image based on the first feature, it is also used to: extract features from the first feature to obtain backbone features; and process the backbone features through the output branch in the neural network model associated with the first output scale to achieve denoising and de-mosaicing of the first image.

[0166] In some embodiments, the zoom information further includes: the zoom magnification of the second image and the position of the first image in the image sequence acquired after the focal length switch, wherein the acquisition time of the second image is earlier than that of the first image, and the acquisition times of the first image and the second image are continuous; the scale control signal further includes: the second input scale of the second image, which corresponds to the sampling scale of the model data processing. At this time, when the processing module 1402 performs denoising and de-mosaicing on the first image based on the first feature, it is also used to: extract features from the second image through the input branch in the neural network model associated with the second input scale to obtain the second feature; fuse the first feature and the second feature based on the spatial embedding feature and the field of view mask to obtain the fused feature, wherein the spatial embedding feature is obtained by processing the zoom information through the neural network model, and the spatial embedding feature is used to indicate the zoom process, and the field of view mask is obtained based on the registration information between the first image and the second image, and the field of view mask is used to indicate the overlapping area of ​​the field of view between the first image and the second image; and perform denoising and de-mosaicing on the first image based on the fused feature.

[0167] In some embodiments, when the processing module 1402 fuses the first feature and the second feature based on the spatial embedding feature and the field of view mask to obtain the fused feature, it is specifically configured to: combine the spatial embedding feature, the field of view mask, the first feature and the second feature to obtain the third feature; process the third feature to obtain the history transformation matrix and the first transformation matrix; transform the second feature using the history transformation matrix to obtain the history alignment feature; combine the history alignment feature and the first feature to obtain the fourth feature; and transform the fourth feature using the first transformation matrix to obtain the fused feature.

[0168] In some embodiments, the scale control signal further includes a first output scale, which corresponds to the sampling scale of the model data processing. At this time, when the processing module 1402 performs denoising and de-mosaicing on the first image based on the fusion features, it specifically performs: feature extraction on the fusion features to obtain backbone features; and processes the backbone features through the output branch in the neural network model associated with the first output scale to achieve denoising and de-mosaicing of the first image.

[0169] In some embodiments, after obtaining the backbone features, the processing module 1402 is further configured to: process the backbone features based on the style embedding features, wherein the style embedding features are obtained by processing the zoom information through a neural network model, and the style embedding features are used to indicate the unification of feature distributions from different lens data to the same lens standard.

[0170] In some embodiments, when the processing module 1402 processes the backbone features based on style embedding features, it is specifically used to: extract features from the data distribution in the backbone features to obtain style features, which are used to characterize the data distribution in the backbone features; combine the style features and style embedding features to obtain a fifth feature; and process the backbone features using the standard deviation and mean obtained through the processing of the fifth feature.

[0171] It should be understood that the above-described device is used to execute the methods in the above embodiments. The implementation principle and technical effect of the corresponding program modules in the device are similar to those described in the above methods. The working process of the device can be referred to the corresponding process in the above methods, and will not be repeated here.

[0172] Based on the methods in the above embodiments, this application also provides an image processing apparatus. Please refer to FIG15, which is a schematic diagram of another image processing apparatus provided in this application. As shown in FIG15, the image processing apparatus 1500 includes one or more processors 1501 and interface circuits 1502. Optionally, the image processing apparatus 1500 may further include a bus 1503. Wherein:

[0173] Processor 1501 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed through integrated logic circuits in the hardware of processor 1501 or through software instructions. Processor 1501 may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods and steps disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor.

[0174] The interface circuit 1502 can be used to send or receive data, instructions or information. The processor 1501 can use the data, instructions or other information received by the interface circuit 1502 to process the data, instructions or other information, and can send the processed information out through the interface circuit 1502.

[0175] Optionally, the image processing apparatus 1500 also includes a memory, which may include read-only memory and random access memory, and provides operation instructions and data to the processor. A portion of the memory may also include non-volatile random access memory (NVRAM).

[0176] Optionally, the memory stores executable software modules or data structures, and the processor can execute corresponding operations by calling the operation instructions stored in the memory (which may be stored in the operating system).

[0177] Optionally, the interface circuit 1502 can be used to output the execution results of the processor 1501.

[0178] It should be noted that the functions of processor 1501 and interface circuit 1502 can be implemented through hardware design, software design, or a combination of hardware and software; no restrictions are imposed here.

[0179] It should be understood that each step of the above method embodiments can be completed by hardware logic circuits or software instructions in a processor.

[0180] Based on the methods in the above embodiments, this application provides a computer-readable storage medium storing a computer program that, when run on an electronic device, causes the electronic device to perform the methods described in the above embodiments. Exemplarily, the computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives).

[0181] Based on the methods in the above embodiments, this application provides a computer program product that, when run on an electronic device, causes the electronic device to execute the methods described in the above embodiments.

[0182] It is understood that the processor in the embodiments of this application may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor may be a microprocessor or any conventional processor.

[0183] The method steps in the embodiments of this application can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an ASIC.

[0184] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, cloud computing platform, or data center to another website, computer, cloud computing platform, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a cloud computing platform or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0185] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application.

[0186] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this application.

Claims

1. An image processing method, characterized by, The method includes: A zoom operation was detected, which is an operation of adjusting the first focal length to the second focal length. Acquire a first image captured by the lens after the zoom operation and zoom information related to the first image, wherein the zoom information includes: the lens type before and after zoom and / or the zoom ratio of the first image; Based on the zoom information, the first image is denoised and depixelated using a neural network model.

2. The method of claim 1, wherein, The step of denoising and de-mosaicing the first image based on the zoom information using a neural network model includes: The zoom information is processed by the neural network model to obtain a scale control signal, which includes: a first input scale of the first image, and the first input scale corresponds to the sampling scale of the model data processing; The first feature is obtained by extracting features from the first image through the input branch in the neural network model that is associated with the first input scale. In the neural network model, the first image is denoised and depixelated based on the first feature.

3. The method of claim 2, wherein, The scale control signal further includes: a first output scale, which corresponds to the sampling scale of model data processing; The step of denoising and de-mosaicing the first image based on the first feature includes: Feature extraction is performed on the first feature to obtain the backbone feature; The backbone features are processed through the output branch in the neural network model that is associated with the first output scale to achieve denoising and de-mosaicing of the first image.

4. The method of claim 2, wherein, The zoom information also includes: the zoom ratio of the second image and the position of the first image in the image sequence acquired after the focal length switch, wherein the acquisition time of the second image is earlier than that of the first image, and the acquisition times of the first image and the second image are consecutive; the scale control signal also includes: the second input scale of the second image, wherein the second input scale corresponds to the sampling scale of the model data processing; The step of denoising and de-mosaicing the first image based on the first feature includes: The second image is used to extract features through the input branch in the neural network model that is associated with the second input scale, so as to obtain the second features; Based on spatial embedding features and field-of-view masks, the first feature and the second feature are fused to obtain fused features. The spatial embedding features are obtained by processing the zoom information through the neural network model and are used to indicate the zoom process. The field-of-view mask is obtained based on the registration information between the first image and the second image and is used to indicate the overlapping area of ​​the field of view between the first image and the second image. Based on the fusion features, the first image is denoised and depixelated.

5. The method of claim 4, wherein, The method of fusing the first feature and the second feature based on spatial embedding features and field-of-view mask to obtain fused features includes: The spatial embedding feature, the field of view mask, the first feature, and the second feature are combined to obtain the third feature; The third feature is processed to obtain the historical transformation matrix and the first transformation matrix; The second feature is transformed using the historical transformation matrix to obtain the historical alignment feature; The historical alignment feature and the first feature are combined to obtain the fourth feature; The fourth feature is transformed using the first transformation matrix to obtain the fused feature.

6. The method according to claim 4 or 5, characterized in that, The scale control signal further includes: a first output scale, which corresponds to the sampling scale of model data processing; The step of denoising and de-mosaicing the first image based on the fusion features includes: Feature extraction is performed on the fused features to obtain the backbone features; The backbone features are processed through the output branch in the neural network model that is associated with the first output scale to achieve denoising and de-mosaicing of the first image.

7. The method according to claim 3 or 6, characterized in that, After obtaining the main features, the process also includes: Based on style embedding features, the backbone features are processed, wherein the style embedding features are obtained by processing the zoom information through the neural network model, and the style embedding features are used to indicate the unification of feature distribution from different lens data to the same lens standard.

8. The method according to claim 7, characterized in that, The processing of the backbone features based on style embedding features includes: Feature extraction is performed on the data distribution in the main features to obtain style features, which are used to characterize the data distribution in the main features; The style feature and the style embedding feature are combined to obtain the fifth feature; The main features are processed using the standard deviation and mean obtained through the fifth feature processing.

9. An image processing apparatus, characterized in that, include: The detection module is used to detect zoom operation, which is the operation of adjusting the first focal length to the second focal length; The processing module is used to acquire a first image captured by the lens after the zoom operation and zoom information related to the first image, wherein the zoom information includes: the lens type before and after zoom and / or the zoom ratio of the first image. The processing module is further configured to perform noise reduction and depixelation on the first image based on the zoom information using a neural network model.

10. The apparatus according to claim 8, characterized in that, When the processing module performs denoising and depixelation on the first image based on the zoom information using a neural network model, it is also used for: The zoom information is processed by the neural network model to obtain a scale control signal, which includes: a first input scale of the first image, and the first input scale corresponds to the sampling scale of the model data processing; The first feature is obtained by extracting features from the first image through the input branch in the neural network model that is associated with the first input scale. In the neural network model, the first image is denoised and depixelated based on the first feature.

11. The apparatus according to claim 10, characterized in that, The scale control signal further includes: a first output scale, which corresponds to the sampling scale of model data processing; When the processing module performs denoising and depixelation on the first image based on the first feature, it is further configured to: Feature extraction is performed on the first feature to obtain the backbone feature; The backbone features are processed through the output branch in the neural network model that is associated with the first output scale to achieve denoising and de-mosaicing of the first image.

12. The apparatus according to claim 10, characterized in that, The zoom information also includes: the zoom ratio of the second image and the position of the first image in the image sequence acquired after the focal length switch, wherein the acquisition time of the second image is earlier than that of the first image, and the acquisition times of the first image and the second image are consecutive; the scale control signal also includes: the second input scale of the second image, wherein the second input scale corresponds to the sampling scale of the model data processing; When the processing module performs denoising and depixelation on the first image based on the first feature, it is further configured to: The second image is used to extract features through the input branch in the neural network model that is associated with the second input scale, so as to obtain the second features; Based on spatial embedding features and field-of-view masks, the first feature and the second feature are fused to obtain fused features. The spatial embedding features are obtained by processing the zoom information through the neural network model and are used to indicate the zoom process. The field-of-view mask is obtained based on the registration information between the first image and the second image and is used to indicate the overlapping area of ​​the field of view between the first image and the second image. Based on the fusion features, the first image is denoised and depixelated.

13. The apparatus according to claim 12, characterized in that, When the processing module fuses the first feature and the second feature based on spatial embedding features and a field-of-view mask to obtain fused features, it is specifically used for: The spatial embedding feature, the field of view mask, the first feature, and the second feature are combined to obtain the third feature; The third feature is processed to obtain the historical transformation matrix and the first transformation matrix; The second feature is transformed using the historical transformation matrix to obtain the historical alignment feature; The historical alignment feature and the first feature are combined to obtain the fourth feature; The fourth feature is transformed using the first transformation matrix to obtain the fused feature.

14. The apparatus according to claim 12 or 13, characterized in that, The scale control signal further includes: a first output scale, which corresponds to the sampling scale of model data processing; When the processing module performs denoising and demosaicing on the first image based on the fusion features, it is specifically used for: Feature extraction is performed on the fused features to obtain the backbone features; The backbone features are processed through the output branch in the neural network model that is associated with the first output scale to achieve denoising and de-mosaicing of the first image.

15. The apparatus according to claim 11 or 14, characterized in that, After obtaining the backbone features, the processing module is further used to: Based on style embedding features, the backbone features are processed, wherein the style embedding features are obtained by processing the zoom information through the neural network model, and the style embedding features are used to indicate the unification of feature distribution from different lens data to the same lens standard.

16. The apparatus according to claim 15, characterized in that, When processing the backbone features based on style embedding features, the processing module is specifically used for: Feature extraction is performed on the data distribution in the main features to obtain style features, which are used to characterize the data distribution in the main features; The style feature and the style embedding feature are combined to obtain the fifth feature; The main features are processed using the standard deviation and mean obtained through the fifth feature processing.

17. An image processing apparatus, characterized in that, Includes at least one processor and interface; The at least one processor obtains program instructions or data through the interface; The at least one processor is configured to execute the program line instructions to implement the method as described in any one of claims 1-8.

18. An electronic device, characterized in that, include: One or more processors; Memory; The memory stores one or more computer programs, the one or more computer programs including instructions that, when executed by the one or more processors, cause the electronic device to perform the method as described in any one of claims 1-8.

19. A computer-readable storage medium storing a computer program that, when run on an electronic device, causes the electronic device to perform the method as described in any one of claims 1-8.

20. A computer program product, characterized in that, When the computer program product is run on an electronic device, it causes the electronic device to perform the method as described in any one of claims 1-8.

Images