Image processing method and device and electronic equipment

Abstract

The invention discloses an image processing method and device and electronic equipment, and belongs to the technical field of artificial intelligence. The image processing method comprises the following steps: respectively acquiring high-definition feature information and blurring feature information of a first image; performing fusion processing on the high-definition feature information and the blurring feature information to obtain first fusion feature information; and obtaining a blurred image based on the first fusion feature information.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of artificial intelligence, and particularly relates to an image processing method and device and electronic equipment. BACKGROUND

[0002] Blurring rendering refers to aesthetic blur rendering on an out-of-focus area in an image, and is widely applied to photography and image processing to imitate a natural blur effect produced by a large-aperture lens when a photo is taken.

[0003] At present, when blurring rendering is performed on a low-definition image, such as a low-quality photo taken by a mobile phone, the low-definition image usually needs to be restored into a high-definition image through an image restoration network, so that a better blurring rendering can be obtained. However, since the image restored by the image restoration network has certain errors, the errors are further enlarged in the subsequent blurring rendering process, and finally the blurring rendering effect of the image is reduced. SUMMARY

[0004] The purpose of the embodiments of the application is to provide an image processing method, device, electronic equipment and storage medium, which can improve the blurring rendering effect of an image.

[0005] In a first aspect, the embodiments of the application provide an image processing method, which comprises:

[0006] obtaining high-definition feature information and blurring feature information of a first image respectively;

[0007] fusing the high-definition feature information and the blurring feature information to obtain first fused feature information;

[0008] obtaining a blurred image based on the first fused feature information.

[0009] In a second aspect, the embodiments of the application provide an image processing device, which comprises:

[0010] an acquisition module configured to obtain high-definition feature information and blurring feature information of a first image respectively;

[0011] a fusion module configured to fuse the high-definition feature information and the blurring feature information to obtain first fused feature information;

[0012] the acquisition module is further configured to obtain a blurred image based on the first fused feature information.

[0013] In a third aspect, the embodiments of the application provide an electronic equipment, which comprises a processor and a memory. The memory stores programs or instructions that can be run on the processor. When the programs or instructions are executed by the processor, the steps of the image processing method according to the first aspect are implemented.

[0014] In a fourth aspect, an embodiment of the present application provides a readable storage medium, the readable storage medium storing a program or instructions, the program or instructions being executed by a processor to implement steps of the image processing method according to the first aspect.

[0015] In a fifth aspect, an embodiment of the present application provides a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, and the processor being configured to execute a program or instructions to implement the image processing method according to the first aspect.

[0016] In a sixth aspect, an embodiment of the present application provides a computer program product, the program product being stored in a storage medium, and the program product being executed by at least one processor to implement the image processing method according to the first aspect.

[0017] In the embodiment of the present application, the electronic device does not need to restore the first image into a high-definition image, but can obtain the first fusion feature information and the blurred image based on the fusion processing of the high-definition feature information and the blurred feature information after obtaining the high-definition feature information and the blurred feature information of the first image, so that the error introduced in the image blurring process is greatly reduced, and the blurring effect of the image is effectively improved. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 is a flowchart of an image processing method provided by an embodiment of the present application;

[0019] Figure 2 is a flowchart of an image processing method provided by an embodiment of the present application;

[0020] Figure 3 is a structural diagram of an image blurring model provided by an embodiment of the present application;

[0021] Figure 4 is a flowchart of an image processing method provided by an embodiment of the present application;

[0022] Figure 5 is a structural diagram of a depth prediction module provided by an embodiment of the present application;

[0023] Figure 6 is a flowchart of an image processing method provided by an embodiment of the present application;

[0024] Figure 7 is a flowchart of an image blurring model training method provided by an embodiment of the present application;

[0025] Figure 8 is a flowchart of a depth prediction module training method provided by an embodiment of the present application;

[0026] Figure 9 is a structural schematic diagram of an image processing device provided by an embodiment of the present application;

[0027] Figure 10 is a structural schematic diagram of an image processing device provided by an embodiment of the present application;

[0028] Figure 11 is a structural schematic diagram of an electronic device provided by an embodiment of the present application;

[0029] Figure 12 is a hardware structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0030] The technical solutions in the embodiments of the present application will be clearly described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some, but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art belong to the scope of protection of the present application.

[0031] The terms "first", "second", and the like in the specification of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are usually a category and do not limit the number of objects, for example, the first object can be one or more. In addition, "and / or" in the specification means at least one of the connected objects, and the character " / ", generally represents that the front and rear associated objects are in a "or" relationship.

[0032] The terms "at least one", "at least one of", and the like in the specification of the present application refer to any one of the objects, a combination of any two or more of the objects. For example, at least one of a, b, and c can mean "a", "b", "c", "a and b", "a and c", "b and c", and "a, b, and c", wherein a, b, and c can be single or multiple. Similarly, "at least two" refers to two or more, and its meaning is similar to "at least one".

[0033] The image processing method provided by the embodiments of the present application will be described in detail below in combination with the drawings and specific embodiments and application scenarios.

[0034] The image processing method provided by the embodiments of the present application can be applied to an image blurring rendering scene.

[0035] Image blur rendering refers to the aesthetic blur rendering of an out-of-focus image region in an image, which is widely used in photography and image processing to simulate the natural blur effect produced by a large aperture lens when taking a photo. Due to the influence of lens aperture and high magnification digital zoom, it is difficult for smart terminal devices such as mobile phones to directly achieve the high-definition blur capability of professional photography equipment when blurring low-definition images taken by the mobile phone, such as low-quality photos taken by the mobile phone.

[0036] In order to improve the high-definition rendering capability of terminal devices such as mobile phones, in the related art, the terminal device such as a mobile phone can first restore the taken low-definition image to a high-definition image using an image restoration network, and then perform blur rendering on the restored high-definition image. However, since the image restored by the image restoration network will have some errors, the errors will be further magnified in the subsequent blur rendering process, and finally the blur rendering effect of the image will be reduced.

[0037] Therefore, an image processing method is provided in the embodiments of the present application. The blur feature information of a first image is obtained. The high-definition feature information of the first image is obtained. The high-definition feature information and the blur feature information are fused to obtain first fused feature information. The blurred image is obtained based on the first fused feature information. In this way, the electronic device does not need to restore the first image to a high-definition image. After obtaining the high-definition feature information and the blur feature information of the first image, the first fused feature information can be obtained by fusing the high-definition feature information and the blur feature information, and the blurred image can be obtained. Therefore, the error introduced in the image blur process is greatly reduced, and the blur effect of the image is effectively improved.

[0038] The execution subject of the image processing method provided in the embodiments of the present application can be an image processing device. Illustratively, the image processing device can be an electronic device, or a functional component or functional entity in the electronic device. In the following, the image processing method provided in the embodiments of the present application will be illustratively described with the execution subject being an electronic device as an example.

[0039] Figure 1 is a flowchart of the image processing method provided in the embodiments of the present application. As shown in Figure 1 The image processing method provided in the embodiments of the present application can include the following steps 101 to 103.

[0040] Step 101, the electronic device obtains high-definition feature information and blur feature information of a first image respectively.

[0041] In some embodiments of the present application, the above-mentioned first image can be a low-definition image taken by a smart terminal device such as a mobile phone, for example, a low-quality photo taken by a mobile phone.

[0042] In some embodiments of the present application, the high-definition feature information and the blur feature information of the first image can be obtained by using a feature extraction network. Figure 1As shown in Figure 2 The "obtaining high-definition feature information of the first image" in step 101 can be implemented through steps 101a1 to 101a3 as follows.

[0043] In step 101a1, the electronic device inputs the first image into an image blurring model.

[0044] In some embodiments of the present application, as shown in Figure 3 The image blurring model 30 includes a first encoder 31, a high-definition feature extraction module 32, and a fusion module 33 connected in sequence.

[0045] In some embodiments of the present application, the basic structure of the image blurring model 30 can be a stable diffusion model of a U-Net network structure. For example, the basic structure of the image blurring model 30 can be a stable diffusion model 2.1, a stable diffusion model xl, etc., which can be determined according to actual needs, and the present embodiment does not make specific limitations here.

[0046] In step 101a2, the electronic device extracts a latent vector of the first image through the first encoder.

[0047] In some embodiments of the present application, the first encoder 31 can be a variational auto-encoder (VAE) for extracting the latent vector of the first image. The latent vector is a core concept in generative adversarial networks and other generative models such as VAE and stable diffusion model, which provides a diverse input basis for the generative model by sampling random noise from the latent space, thereby controlling the randomness of the generation process and the diversity of the output. The noise characteristics of the latent vector make the generator more robust to input perturbations, and by learning the distribution rules of the latent space, the model can generalize to generate samples outside the training data. For example, the stable diffusion model performs diffusion operations in the latent space, significantly reducing the computational cost of image generation. The concept of latent vector exists not only in generative adversarial networks, but also is a core component of VAE and diffusion models. In VAE, the latent space is encoded as a probability distribution.

[0048] In step 101a3, the electronic device extracts high-definition feature information from the latent vector through the high-definition feature extraction module.

[0049] In some embodiments of the present application, the structure of the high-definition feature extraction module 32 can be a down-sampling part of a stable diffusion model of a U-Net network structure.

[0050] In some embodiments of the present application, the electronic device can not add any noise to the latent vector generated by the first encoder 31 and directly input it into the high-definition feature extraction module 32.

[0051] In some embodiments of the present application, the high-definition feature extraction module is used to extract single-step high-definition feature information of the first image from the potential vector.

[0052] In some embodiments of the present application, as shown in Figure 3 The high-definition feature extraction module 32 includes N cascaded first down-sampling interpolation modules 321, N is an integer greater than 1; in combination with Figure 2 As shown in Figure 4 The step 101a3 can be implemented by the following steps A1 to A3:

[0053] Step A1, the electronic device extracts first feature information in the potential vector through the first down-sampling interpolation module in the N cascaded first down-sampling interpolation modules.

[0054] In some embodiments of the present application, the interpolation methods of the N cascaded first down-sampling interpolation modules 321 can be the same or different, which can be determined according to actual needs, and the present embodiment does not make specific limitations here.

[0055] It should be noted that the process of the electronic device extracting first feature information in the potential vector through the first down-sampling interpolation module in the N cascaded first down-sampling interpolation modules can refer to the following step A2, which extracts feature information from the (i-1)th feature information output by the (i-1)th down-sampling interpolation module through the ith down-sampling interpolation module in the N cascaded first down-sampling interpolation modules to obtain the ith feature information. To avoid repetition, this embodiment will not be repeated here.

[0056] Step A2, the electronic device extracts feature information from the (i-1)th feature information output by the (i-1)th down-sampling interpolation module through the ith down-sampling interpolation module in the N cascaded first down-sampling interpolation modules to obtain the ith feature information; i∈[2, N-1].

[0057] In some embodiments of the present application, each first down-sampling interpolation module 321 includes a first residual module 3211, a first self-attention module 3212, and a first cross-attention module 3213; the step A2 can be implemented by the following steps a1 to a3:

[0058] a1, the electronic device performs residual processing on the i first feature information through the first residual module in the (i-1)th first down-sampling interpolation module to obtain a first residual processing result.

[0059] In some embodiments of the present application, the first residual module 3211 is used to reduce the depth of the first feature information described above. The residual module is a breakthrough design in the field of deep learning, and its core value lies in solving the key problems of gradient disappearance and network degradation in the training of deep neural networks, providing a feasible solution for training extremely deep models. Through the jump connection and residual learning mechanism, it significantly improves the training efficiency and stability while ensuring the performance of the model, and becomes one of the basic modules of modern neural network architecture.

[0060] a2, the electronic device performs self-attention operation on the first residual processing result through the first self-attention module in the i-1th first down-sampling interpolation module, to obtain a first self-attention operation result.

[0061] In some embodiments of the present application, the first self-attention module 3212 described above can be a multi-head attention (Multi-Head Attention) module, which is an extended form of attention mechanism widely used in Transformer models. By running multiple independent attention mechanisms in parallel, it obtains the attention distribution of different subspaces of the first residual processing result, thereby more comprehensively capturing the potential multiple semantic associations in the first residual processing result.

[0062] a3, the electronic device performs cross-attention operation on the first self-attention operation result through the first cross-attention module in the i-1th first down-sampling interpolation module, to obtain the i-th first feature information.

[0063] In some embodiments of the present application, as shown in Figure 3 The image blurring model further includes a third decoder 34 connected to the fusion module 33, and the first cross-attention module 3213 is used for interaction between the first encoder 31 and the first decoder 34. In order to enable the decoder to more effectively utilize the information of the encoder, the cross-attention module is introduced. The cross-attention module is used to establish a correlation between the output of the first encoder 31 and the input of the third decoder 34 and calculate the attention weight.

[0064] Step A3, the electronic device performs feature extraction on the N-1th first feature information output by the N-1th first down-sampling interpolation module through the Nth first down-sampling interpolation module in the N cascaded first down-sampling interpolation modules, to obtain high-definition feature information.

[0065] It should be noted that the electronic device extracts the N-1th first feature information output by the N-1th first downsampling interpolation module through the Nth first downsampling interpolation module in the N cascaded first downsampling interpolation modules, and obtains the high-definition feature information. The process of extracting the high-definition feature information from the latent vector through the high-definition feature information extraction module can refer to the description of the above step A2. To avoid repetition, this embodiment will not be described here.

[0066] In some embodiments of the present application, the size of the output result of the N cascaded first downsampling interpolation modules can be reduced in turn. For example, N is 4, the size of the latent vector output by the first first downsampling interpolation module is 64*64, the size of the first feature information output by the second first downsampling interpolation module can be 32*32, the size of the first feature information output by the third first downsampling interpolation module can be 16*16, and the size of the high-definition feature information output by the fourth first downsampling interpolation module can be 8*8.

[0067] In this way, the electronic device extracts the first feature information in the latent vector through the first first downsampling interpolation module in the N cascaded first downsampling interpolation modules; extracts the i-1th first feature information output by the i-1th first downsampling interpolation module through the ith first downsampling interpolation module in the N cascaded first downsampling interpolation modules, and obtains the ith first feature information; and extracts the N-1th first feature information output by the N-1th first downsampling interpolation module through the Nth first downsampling interpolation module in the N cascaded first downsampling interpolation modules, and obtains the high-definition feature information. The high-definition feature information of the first image can be accurately extracted layer by layer through the N cascaded first downsampling interpolation modules.

[0068] In this way, the electronic device inputs the first image into the image blurring model; extracts the latent vector of the first image through the first encoder; and extracts the high-definition feature information from the latent vector through the high-definition feature extraction module, so that the high-definition feature information of the first image can be quickly and accurately extracted.

[0069] In some embodiments of the present application, as shown in Figure 3 The image blurring model 30 can further include a second encoder 35, a blurring feature extraction module 36 connected with the second encoder 35 and the feature fusion module 33. The "obtaining blurring feature information of the first image" in step 101 can be implemented by the following steps 101b1 to 101b4:

[0070] In step 101b1, the electronic device obtains the first disparity map of the first image.

[0071] In some embodiments of the present application, the above step 101b1 can be implemented through the following steps B1 and B2:

[0072] Step B1, the electronic device inputs the first image into a depth prediction module.

[0073] In some embodiments of the present application, as shown in Figure 5 The depth prediction module 51 can include a student module 51 and a teacher module 52 connected to the student module 51. The teacher module 52 can be a powerful pre-trained depth estimation model, such as any depth v2, deep pre-training transformer (DPT), deep Pro, etc., which has the ability to estimate the depth of an image. The student module 51 can be created using the same architecture as the teacher module, and can achieve the depth estimation performance of the teacher module 51.

[0074] Step B2, the electronic device obtains a first disparity map through the student module.

[0075] In some embodiments of the present application, the electronic device can input the first image into the student module 51 to obtain the first disparity map. The first disparity map represents the depth information of the first image, and includes the disparity values of each pixel point in the first image. The disparity map depicts the difference in pixel position of the same scene object at the same position in two cameras. Since the binocular camera is usually horizontally aligned, this difference is mainly in the horizontal direction. When a certain point X in the scene has coordinates (x, y) in the left camera, it will be mapped to (x+d, y) in the right camera imaging. d is the disparity value of the corresponding point in the disparity map, which is a measure of the relative position difference between cameras.

[0076] In this way, the electronic device inputs the first image into the depth prediction module, and obtains the first disparity map through the student module, which can improve the accuracy of the depth prediction result of the first image.

[0077] Step 101b2, the electronic device obtains a defocus map of the first image based on the first disparity map.

[0078] In some embodiments of the present application, the above step 102b2 can be implemented through the following steps C1 to C3:

[0079] Step C1, the electronic device identifies the first image to determine the focal position of the first image.

[0080] In some embodiments of the present application, the electronic device can identify the first image, determine the subject of the first image, then determine the subject area in which the subject is located in the first image, and then determine the center pixel point of the subject area as the focus position of the first image.

[0081] Step C2, the electronic device obtains the first disparity value of the focus position on the first disparity map.

[0082] In some embodiments of the present application, the electronic device can obtain the first disparity value of the pixel point at the focus position on the first disparity map.

[0083] For example, the pixel coordinates of the focus position of the first image are (x0, y0), and the electronic device can obtain the first disparity value of the pixel point with pixel coordinates (x0, y0) on the first disparity map.

[0084] Step C3, the electronic device obtains the defocus map based on the difference between the disparity values of each pixel point on the first disparity map and the first disparity value.

[0085] In some embodiments of the present application, the defocus map includes the blur radius of each pixel point in the first image.

[0086] In some embodiments of the present application, the electronic device can calculate the defocus map using the following formula:

[0087] r = K | D - D f | (1);

[0088] Wherein, D represents the disparity value of the first pixel point in the first image, D f represents the first disparity value, K represents the size of the dispersion circle, and r represents the blur radius of the first pixel point. The first pixel point is any pixel point in the disparity map.

[0089] In this way, the electronic device can quickly and accurately obtain the defocus map by identifying the first image, determining the focus position of the first image, obtaining the first disparity value of the focus position on the first disparity map, and obtaining the defocus map based on the difference between the disparity values of each pixel point on the first disparity map and the first disparity value.

[0090] Step 101b3, the electronic device obtains the feature vector of the defocus map through the second encoder.

[0091] In some embodiments of the present application, the second encoder 35 can be a variational autoencoder, and the second encoder is used to extract the feature vector of the defocus map.

[0092] Step 101b4, the electronic device extracts features from the feature vector of the defocus map through the blurring feature extraction module to obtain blurring feature information.

[0093] In some embodiments of the present application, as shown in Figure 3 the above-mentioned virtualization feature extraction module 36 includes N cascaded second down-sampling interpolation modules 361, and the above-mentioned virtualization feature information includes N virtualization feature information; and the step 101b4 can be implemented through the following steps D1 to D3:

[0094] Step D1, the electronic device performs feature extraction on the feature vector of the defocus image through the first second down-sampling interpolation module in the N cascaded second down-sampling interpolation modules, to obtain the first virtualization feature information.

[0095] Step D2, the electronic device performs feature extraction on the (i-1)th first virtualization feature information output by the (i-1)th first down-sampling interpolation module through the ith second down-sampling interpolation module in the N cascaded second down-sampling interpolation modules, to obtain the ith virtualization feature information; i∈[2, N-1];

[0096] Step D3, the electronic device performs feature extraction on the (N-1)th second feature information output by the (i-1)th second down-sampling interpolation module through the Nth second down-sampling interpolation module in the N cascaded second down-sampling interpolation modules, to obtain the Nth virtualization feature information.

[0097] It should be noted that the structure of the above-mentioned virtualization feature extraction module 36 is similar to that of the above-mentioned high-definition feature extraction module, so the process of the electronic device performing feature extraction on the feature vector of the defocus image through the above-mentioned virtualization feature extraction module to obtain virtualization feature information can refer to the above-mentioned A1, and this embodiment will not be described here again to avoid repetition.

[0098] In this way, the electronic device performs feature extraction on the feature vector of the defocus image through the virtualization feature extraction module to obtain virtualization feature information; performs feature extraction on the (i-1)th first virtualization feature information output by the (i-1)th first down-sampling interpolation module through the ith second down-sampling interpolation module in the N cascaded second down-sampling interpolation modules to obtain the ith virtualization feature information; and performs feature extraction on the (N-1)th second feature information output by the (i-1)th second down-sampling interpolation module through the Nth second down-sampling interpolation module in the N cascaded second down-sampling interpolation modules to obtain the Nth virtualization feature information, which can accurately extract the virtualization feature information of the first image layer by layer through the N cascaded second down-sampling interpolation modules.

[0099] Thus, the electronic device obtains the first parallax map of the first image, obtains the defocus map of the first image based on the first parallax map, obtains the feature vector of the defocus map through the second encoder, and extracts features of the feature vector of the defocus map through the virtualization feature extraction module to obtain the virtualization feature information of the first image, which can quickly and accurately obtain the virtualization feature information of the first image.

[0100] In step 102, the electronic device fuses the high-definition feature information and the virtualization feature information to obtain first fusion feature information.

[0101] In some embodiments of the present application, the above-mentioned step 102 can be implemented by the following step 102a:

[0102] In step 102a, the electronic device fuses the high-definition feature information and the virtualization feature information through the fusion module to obtain the first fusion feature information.

[0103] In some embodiments of the present application, as shown in Figure 3 The fusion module 33 includes N cascaded upsampling interpolation modules 331, and the above-mentioned step 102a can be implemented by the following steps 102a1 to 102a3:

[0104] In step 102a1, the electronic device performs feature fusion on the high-definition feature information and the Nth virtualization feature information through the first layer upsampling interpolation module in the N cascaded upsampling interpolation modules to obtain the first second fusion feature information.

[0105] In some embodiments of the present application, the interpolation methods of the N cascaded upsampling interpolation modules 331 can be the same or different, which can be determined according to actual needs, and the present embodiment does not make specific limitations here.

[0106] It should be noted that the process of obtaining the first second fusion feature information by performing feature fusion on the high-definition feature information and the Nth virtualization feature information through the first layer upsampling interpolation module in the N cascaded upsampling interpolation modules can refer to the description of the following step 102a2, that is, obtaining the i-th second fusion feature information by performing feature fusion on the (i-1)-th fusion feature information output by the (i-1)-th layer upsampling interpolation module and the (N-i)-th virtualization feature information through the i-th layer upsampling interpolation module in the N cascaded upsampling interpolation modules, and the present embodiment will not be repeated here to avoid repetition.

[0107] In step 102a2, the electronic device performs feature fusion on the (i-1)-th fusion feature information output by the (i-1)-th layer upsampling interpolation module and the (N-i)-th virtualization feature information through the i-th layer upsampling interpolation module in the N cascaded upsampling interpolation modules to obtain the i-th second fusion feature information; i∈[2, N-1].

[0108] In some embodiments of the present application, as shown in Figure 3 the second residual module 3311, the second self-attention module 3312, and the second cross-attention module 3313; the step 102a2 can be implemented through the following steps E1 to E3:

[0109] Step E1, the electronic device performs residual processing on the i-1th fusion feature information and the N-i th virtual feature information through the second residual module in the i-1th upsampling interpolation module to obtain a second residual processing result.

[0110] In some embodiments of the present application, the second residual module 3311 is used to reduce the depth of the i-1th fusion feature information and the N-i th virtual feature information.

[0111] Step E2, the electronic device performs self-attention operation on the second residual processing result through the second self-attention module in the i-1th upsampling interpolation module to obtain a second self-attention operation result.

[0112] In some embodiments of the present application, as shown in Figure 3 the second residual module 3311, the second self-attention module 3312, and the second cross-attention module 3313; the step 102a2 can be implemented through the following steps E1 to E3:

[0113] Step e1, the electronic device performs super-resolution operation on the second residual processing result through the low-rank adaptive super-resolution module to obtain a super-resolution operation result.

[0114] In some embodiments of the present application, the low-rank adaptive super-resolution module 33121 is obtained by training using low-definition image samples and pre-set high-definition image samples, and is used to improve the resolution of the second residual processing result.

[0115] Step e2, the electronic device performs virtual processing on the super-resolution operation result through the low-rank adaptive virtualization module to obtain a second self-attention operation result.

[0116] In some embodiments of the present application, the low-rank adaptive virtualization module 33122 is obtained by training using low-definition image samples and high-definition virtualization image samples, and is used for blurred rendering of the super-resolution operation result.

[0117] Thus, the electronic device performs super-resolution operation on the second residual processing result through the low-rank adaptive super-resolution module to obtain a super-resolution operation result, and performs self-attention operation on the super-resolution operation result through the low-rank adaptive virtualization module to obtain the second self-attention operation result, thereby accurately and quickly obtaining the second self-attention operation result.

[0118] In step E3, the electronic device performs cross-attention operation on the second self-attention operation result through the second cross-attention module in the i-1th layer upsampling interpolation module to obtain the i-th second fusion feature information.

[0119] It should be noted that the implementation process of performing cross-attention operation on the second self-attention operation result through the second cross-attention module in the i-1th layer upsampling interpolation module to obtain the i-th second fusion feature information in step E3 can refer to the related description of performing cross-attention operation on the first self-attention operation result through the first cross-attention module in the i-1th first downsampling interpolation module to obtain the i-th first feature information in step a3, and details are not repeated here.

[0120] Thus, the electronic device performs residual processing on the i-1th fusion feature information and the N-i-th virtualization feature information through the second residual module in the i-1th layer upsampling interpolation module to obtain a second residual processing result, performs self-attention operation on the second residual processing result through the second self-attention module in the i-1th layer upsampling interpolation module to obtain a second self-attention operation result, and performs cross-attention operation on the second self-attention operation result through the second cross-attention module in the i-1th layer upsampling interpolation module to obtain the i-th second fusion feature information, thereby accurately obtaining the i-th second fusion feature information.

[0121] In step 102a3, the electronic device performs feature extraction on the N-1th second fusion feature information and the 1st virtualization feature information output by the N-1th upsampling interpolation module through the Nth upsampling interpolation module in the N cascaded upsampling interpolation modules to obtain the first fusion feature information.

[0122] It should be noted that the process of performing feature extraction on the N-1th second fusion feature information and the 1st virtualization feature information output by the N-1th upsampling interpolation module through the Nth upsampling interpolation module in the N cascaded upsampling interpolation modules to obtain the first fusion feature information can refer to the related description of performing feature fusion on the i-1th fusion feature information and the N-i-th virtualization feature information output by the i-1th upsampling interpolation module through the i-th upsampling interpolation module in the N cascaded upsampling interpolation modules to obtain the i-th second fusion feature information in step 102a2, and details are not repeated here.

[0123] In some embodiments of the present application, the size of the output result of the N cascaded upsampling interpolation modules can increase in turn. For example, N is 4, the size of the latent vector output by the first upsampling interpolation module is 16*16, the size of the first feature information output by the second upsampling interpolation module can be 32*32, the size of the first feature information output by the third upsampling interpolation module can be 64*64, and the size of the high-definition feature information output by the fourth upsampling interpolation module can be 128*128.

[0124] In this way, the electronic device performs feature fusion on the high-definition feature information and the Nth blurred feature information through the first layer upsampling interpolation module in the N cascaded upsampling interpolation modules to obtain the first second fusion feature information; performs feature fusion on the (i-1)th fusion feature information output by the (i-1)th layer upsampling interpolation module and the N-i-th blurred feature information through the i-th layer upsampling interpolation module in the N cascaded upsampling interpolation modules to obtain the i-th second fusion feature information; and performs feature extraction on the (N-1)th second fusion feature information output by the (N-1)th layer upsampling interpolation module and the first blurred feature information through the Nth layer upsampling interpolation module in the N cascaded upsampling interpolation modules to obtain the first fusion feature information, which can be accurately extracted layer by layer through the N cascaded upsampling interpolation modules.

[0125] Step 103, the electronic device obtains a blurred image based on the first fusion feature information.

[0126] In some embodiments of the present application, the above step 103 can be implemented by the following step 103a:

[0127] Step 103a, the electronic device decodes the first fusion feature information through the third decoder to obtain a blurred image.

[0128] In some embodiments of the present application, the above first fusion feature information can be a latent space vector, and the above third decoder 34 is used to map the above first fusion feature information from the latent space back to the original image data space to obtain a blurred image.

[0129] In the image processing method provided in the embodiments of the present application, the electronic device obtains high-definition feature information of a first image; obtains blurred feature information of the first image; performs fusion processing on the high-definition feature information and the blurred feature information to obtain first fusion feature information; and obtains a blurred image based on the first fusion feature information. In this way, the electronic device does not need to restore the first image into a high-definition image, but can obtain the first fusion feature information by performing fusion processing on the high-definition feature information and the blurred feature information after obtaining the high-definition feature information and the blurred feature information of the first image, and obtain the blurred image, thereby greatly reducing the error introduced in the image blurring process and effectively improving the blurring effect of the image.

[0130] In some embodiments of the present application, before the step 101, the image processing method provided in the embodiments of the present application can further include the following steps 104a to 104e:

[0131] In step 104a, the electronic device performs image degradation processing on the preset high-definition image sample to obtain a low-definition image sample.

[0132] In some embodiments of the present application, the electronic device can use a general image degradation pipeline to perform image degradation processing on the preset high-definition image sample to obtain the low-definition image sample.

[0133] In some embodiments of the present application, the image degradation processing can include the following in sequence:

[0134] Blur processing, down-sampling processing, noise addition, and compression processing.

[0135] In some embodiments of the present application, the electronic device can use a random Gaussian kernel for blur processing and a bilinear interpolation for down-sampling processing.

[0136] In some embodiments of the present application, the noise added by the electronic device can be Gaussian noise, Poisson noise, etc., which can be determined according to actual needs, and the present embodiment does not make specific limitations here.

[0137] In step 104b, the electronic device obtains a high-definition blurred image sample corresponding to the preset high-definition image sample.

[0138] In some embodiments of the present application, the electronic device can render the preset high-definition image sample model to obtain the high-definition blurred image sample corresponding to the preset high-definition image sample.

[0139] In step 104c, the electronic device takes the low-definition image sample and the high-definition blurred image sample as a first training sample pair to train the second encoder, the blurred feature extraction module, and the low-rank adaptive blurring module.

[0140] In some embodiments of the present application, the second encoder, the virtualization feature extraction module, and the low-rank adaptive virtualization module are used for image virtualization, and thus can be used as a virtualization rendering module. The electronic device can set the learning rate to 5e-5, the number of training steps to 8000, and use the first training sample pair to train the virtualization rendering module to improve the performance of controllable virtualization rendering of the image virtualization model.

[0141] In step 104d, the electronic device uses the low-resolution image sample and the preset high-definition image sample as a second training sample pair to train the first encoder, the high-definition feature extraction module, and the low-rank adaptive super-resolution module.

[0142] In some embodiments of the present application, the first encoder, the high-definition feature extraction module, and the low-rank adaptive super-resolution module are used for image restoration, and thus can be used as an image restoration module. The electronic device can set the learning rate to 5e-5, the number of training steps to 8000, and use the second training sample pair to train the image restoration module to reduce the problem of performance decline caused by the conflict between the optimization objectives in the training processes of the image restoration module and the virtualization rendering module.

[0143] In some embodiments of the present application, the high-definition feature extraction module includes a low-rank adaptive module, and the electronic device can train the low-rank adaptive module using the second training sample pair to train the high-definition feature extraction module.

[0144] In step 104e, the electronic device alternately uses the first training sample pair and the second training sample pair to train the second encoder, the virtualization feature extraction module, and the low-rank adaptive virtualization module, and the first encoder, the high-definition feature extraction module, and the low-rank adaptive super-resolution module.

[0145] In some embodiments of the present application, the electronic device can continue to alternately train the virtualization rendering module and the image restoration module, that is, the electronic device can alternately use the first training sample pair and the second training sample pair to train the virtualization rendering module and the image restoration module.

[0146] In this way, the electronic device alternately uses the first training sample pair and the second training sample pair to train the second encoder, the virtualization feature extraction module, and the low-rank adaptive virtualization module, and the first encoder, the high-definition feature extraction module, and the low-rank adaptive super-resolution module. On the one hand, the image restoration and the image virtualization can be decoupled, and the performance of the image restoration and the virtualization rendering can be significantly improved. On the other hand, the problem of performance decline caused by the conflict between the optimization objectives and the imbalance of the training samples in the training processes of the image restoration and the image virtualization can be effectively inhibited, so that the image virtualization model can be better applied to edge devices.

[0147] In some embodiments of the present application, before step 101b2, the image processing method provided by the embodiments of the present application can further include steps 105a and 105b.

[0148] Step 105a, the electronic device performs image degradation processing on the first high-definition image to obtain a degraded first high-definition image.

[0149] It should be noted that the specific implementation process of step 105a can refer to the related description in step 104a described above, and to avoid repetition, this embodiment will not be described again this time.

[0150] Step 105b, the electronic device trains the student module by the teacher module, the first high-definition image and the degraded first high-definition image.

[0151] In some embodiments of the present application, step 105b can be implemented by steps 105b1-105b5 as follows:

[0152] Step 105b1, the electronic device inputs the first high-definition image into the teacher module and inputs the degraded first high-definition image into the student module.

[0153] In some embodiments of the present application, as shown in Figure 5 The teacher module 52 includes a third encoder 521 and a first decoder 522 connected with the third encoder 521, and the student module 51 includes a fourth encoder 511 and a second decoder 512 connected with the fourth encoder 511.

[0154] Step 105b2, the electronic device extracts the first high-definition image feature by the third encoder to obtain second feature information, and extracts the second feature information feature by the first decoder to obtain a second disparity map.

[0155] In some embodiments of the present application, the electronic device can extract the vector of the latent space from the first high-definition image by the third encoder 521 to obtain the second feature information.

[0156] In some embodiments of the present application, the electronic device can map the second feature information from the latent space back to the original image data space by the first decoder 522 to obtain the second disparity map.

[0157] Step 105b3, the electronic device extracts the degraded first high-definition image feature by the fourth encoder to obtain third feature information, and extracts the second feature information feature by the second decoder to obtain a third disparity map.

[0158] In some embodiments of the present application, the electronic device can extract the vector of the latent space from the first high-definition image after the degradation processing by the fourth encoder 511 described above, to obtain the third feature information.

[0159] In some embodiments of the present application, the electronic device can map the third feature information from the latent space back to the original image data space by the second decoder 512 described above, to obtain the third disparity map.

[0160] Step 105b4, the electronic device calculates a first difference value between the second feature information and the third feature information, and a second difference value between the second disparity map and the third disparity map.

[0161] In some embodiments of the present application, the electronic device can calculate the first difference value by using the following formula:

[0162] L1 = T1 - T2; (2)

[0163] Wherein, L1 represents the first difference value, T1 represents the second feature information, and T2 represents the third feature information.

[0164] In some embodiments of the present application, the electronic device can calculate the second difference value by using the following formula:

[0165] L2 = P1 - P2; (3)

[0166] Wherein, L2 represents the second difference value, P1 represents the second disparity map, and M2 represents the second disparity map.

[0167] Step 105b5, the electronic device takes the weighted value of the first difference value and the second difference value as the loss function of the student module, and trains the student module.

[0168] In some embodiments of the present application, the electronic device can calculate the weighted value by using the following formula:

[0169] L = a * L1 + b * L2; (4)

[0170] Wherein, L represents the weighted value, a represents the weight value of the first difference value, and b represents the weight value of the second difference value.

[0171] In some embodiments of the present application, the electronic device can take the weighted value as the loss function of the student module to adjust the parameters of the trained student module. After adjustment, the steps 105b1 to 105b5 are executed again, and the cycle is repeated until the weighted value is less than or equal to a preset threshold, and the training of the student module is determined to be completed.

[0172] Thus, the electronic device inputs the first high-definition image into the teacher module and inputs the first high-definition image after the degradation processing into the student module; extracts features of the first high-definition image through the third encoder to obtain second feature information, and extracts features of the second feature information through the first decoder to obtain a second disparity map; extracts features of the first high-definition image after the degradation processing through the fourth encoder to obtain third feature information, and extracts features of the second feature information through the second decoder to obtain a third disparity map; calculates a first difference value between the second feature information and the third feature information, and a second difference value between the second disparity map and the third disparity map; takes a weighted value of the first difference value and the second difference value as a loss function of the student module, and trains the student module, so as to improve the accuracy of the student module in extracting depth information of a low-definition image.

[0173] In the following, the image processing method provided by the embodiments of the present application will be described in further detail in combination with Figure 3 and Figure 5 FIG. 1 is a flowchart of an image processing method provided by an embodiment of the present application. Figure 6 As shown in FIG. 1, the image processing method includes the following steps 601 to 604.

[0174] In step 601, the electronic device pre-processes a low-quality image of an input image model.

[0175] For example, the electronic device can input the low-quality image of the low-resolution image into the first encoder 31, and directly input the generated latent vector into the high-definition feature extraction module 32 without adding any noise. At the same time, the low-resolution image is sent into the pre-trained depth prediction module 51 to obtain a disparity map for controllable blurring rendering.

[0176] In step 601, the electronic device pre-processes a low-quality image of an input image model.

[0177] For example, the electronic device can extract high-definition features from the low-quality image by using the high-definition feature extraction module 32.

[0178] In step 603, the electronic device performs controllable blurring rendering on the low-quality image.

[0179] For example, the electronic device can predict the depth information of the low-quality image through the depth prediction module 51, then calculate the defocus map pixel by pixel through formula (1), and then pass the blurring feature information extracted from the defocus map and the high-definition feature information extracted by the high-definition feature module 32 to the fusion module 33 through the blurring feature extraction module 36 to guide the generation of the blurring area.

[0180] For example, the blurring feature extraction module 36 can be a control network.

[0181] Exemplarily, the electronic device can use the focus clues obtained by the defocus map as guidance through the fusion module 33 to help the image blurring model separate the focused subject from the defocused area, while ensuring the subject is clear and the background presents a defocused blur effect.

[0182] Step 4, the electronic device outputs the generated high-definition blurred image.

[0183] Exemplarily, the electronic device can decode the output of the fusion module 33 in the image blurring model through the third decoder 34 to obtain the required high-definition blurred image.

[0184] Exemplarily, in order to improve the quality and controllability of the blurring rendering, the image processing framework proposed by the present application makes improvements on the basic diffusion model structure through multiple modules:

[0185] Module 1: controllable blurring rendering module.

[0186] This module is mainly composed of a depth prediction module, a second encoder, and a blurring feature extraction module of a control network structure. The depth prediction module is obtained by self-distillation of the existing depth prediction module, and the second encoder and the blurring feature extraction module of the control network structure directly copy the parameters of the high-definition feature extraction module during training to continue training. Since the input is a low-quality image, it is difficult to extract its depth information. The present application first predicts the depth information of the low-quality image through the trained depth prediction module, then calculates the defocus map pixel by pixel through formula (1), and then transmits the information of the defocus map to the blurring feature extraction module. The blurring feature information extracted from the defocus map by the blurring feature extraction module is input into the fusion module together with the high-definition feature information obtained by the single-step high-definition feature extraction module, which can generate a blurring target area guided by the defocus map while retaining the clarity of the subject.

[0187] Module 2: fusion module.

[0188] This module uses the focus clues obtained by the defocus map as guidance to help the model separate the focused subject from the defocused area, while ensuring the subject is clear and the background presents a defocused blur effect. Specifically, in the fusion module, the present application modifies the self-attention layer of the up-sampling area of the original diffusion network, adds a low-rank adaptive super-resolution module and a low-rank adaptive blurring module, and adjusts the attention map as follows:

[0189]

[0190] Where X represents the fusion feature information, Q, K and V are the query, key and value of the self-attention layer, and d is the scaling factor. is the focus attention mask of each pixel point in the defocus map.

[0191] The calculation can be performed by the following formula:

[0192]

[0193] Wherein, M is the mask at the focal point area in the defocus map.

[0194] Exemplarily, the focal point area can be a region of a preset size in the defocus map with the focal point position as the center.

[0195] Exemplarily, The relationship between different regions in the defocus map can be represented as 1 for the same region as the focal point region and 0 for other regions different from the focal point region. The size of the binary mask is adjusted to match the resolution required by the attention layer.

[0196] Exemplarily, the training step of the image blurring model is improved in the embodiments of the present application:

[0197] In view of the problem that the optimization objectives conflict with each other and the training samples are unbalanced in the image restoration and blurring rendering tasks, the original training process is improved and an alternating training strategy is proposed. The overall model training step is as shown in Figure 7 The specific steps are as follows:

[0198] Step 701, the electronic device initializes the model.

[0199] Exemplarily, the high-definition feature extraction module is initialized by using the parameters of existing open source models, such as Stable Diffusion 2.1, Stable Diffusion xl, etc., and then the low-rank adaptive module for image high-definition repair pre-trained by using paired high-definition and low-definition data is placed in the encoder and the high-definition feature extraction module, so as to reduce the training time of the model.

[0200] Exemplarily, the depth prediction module is obtained by pre-training, and the initialization parameters of the encoder and the control network copy the parameters of Stable Diffusion 2.1. The fusion module is located in the self-attention layer of each upsampling part of the diffusion network. The low-rank adaptive super-resolution module and the low-rank adaptive blurring module are placed in the original self-attention layer, and the weights of each are initialized.

[0201] Step 702, the electronic device trains the blurring rendering module.

[0202] Exemplarily, in the training process, the original parameters of the high-definition feature extraction module remain unchanged, and this step only trains the encoder and control network of the controllable blur rendering module, the low-rank adaptive blur module in the fusion module. Using the existing general image degradation pipeline, the original high-definition non-blur data is subjected to high-order degradation including blur (random Gaussian kernel), downsampling (bilinear interpolation), adding noise (Gaussian noise, Poisson noise), Joint Photographic Experts Group (JPEG) compression to obtain simulated low-definition data. The processed low-definition non-blur data and high-definition blur data are used as paired training data, the learning rate is set to 5e-5, and the training step number is 8000 steps, which is used to improve the performance of the controllable blur rendering model.

[0203] Step 703, the electronic device performs image restoration module training.

[0204] Exemplarily, in the training process, the parameters of the encoder and control network of the controllable blur rendering module and the low-rank adaptive blur module in the fusion module in step 802 remain unchanged, and this step only trains the low-rank adaptive module in the high-definition feature extraction module and the low-rank adaptive super-resolution module in the fusion module. Using low-quality and high-quality image pairs as training samples, the learning rate is set to 5e-5, and the training step number is 8000 steps, which reduces the problem of mutual conflict of optimization objectives in the training process of the two, resulting in a decline in effect.

[0205] Step 704, the electronic device continues to perform alternating training of blur rendering and image restoration.

[0206] Exemplarily, step 804 is the same as steps 702 and 703 in training parameters, and the training step numbers are 4000 steps and 4000 steps respectively, and the learning rate is set to 5e-6.

[0207] The existing depth estimation model has a significant performance on high-quality image data, but when applied to low-quality images, the accuracy of the depth estimation model deteriorates rapidly, thereby reducing the results of blur rendering. Therefore, the present application proposes a depth estimation module, which uses self-distillation to improve the depth estimation effect of the image. The whole training process is as shown in Figure 8 The specific steps are as follows:

[0208] Step 801, the electronic device sets the teacher module and the student module.

[0209] Exemplarily, the application first selects a powerful pre-trained depth estimation model as a teacher module, for example, any depth v2, DPT, depth Pro, etc., which has the ability to estimate the depth of an image. Then, a student module is created using the same architecture as the teacher module, with the goal of having the student module achieve the depth estimation performance of the teacher module in the initialization stage.

[0210] Step 802, the electronic device performs input processing.

[0211] During the training process, high-definition images are input into the teacher module, while simulated low-quality images obtained using the above degradation pipeline are input into the student module. This design allows the student module to learn how to process low-quality inputs in complex scenarios while using the teacher model as a reference standard.

[0212] Step 803, the electronic device performs feature extraction and distillation.

[0213] The application proposes a feature self-distillation method, which extracts image features through the encoders of the two modules for supervision, allowing the student module to learn the key features of the teacher module, thereby making the feature representation more consistent.

[0214] Step 804, the electronic device performs depth estimation optimization.

[0215] Exemplarily, as the training progresses, the depth estimation results are optimized by continuously adjusting the module parameters and learning rate. The student module not only can handle high-definition images, but also gradually improves its ability to estimate accuracy in a simulated low-quality environment. Throughout the training cycle, the output of the student module is regularly supervised with the output of the teacher module to obtain more accurate depth estimation results.

[0216] The image processing method provided in the embodiments of the present application can utilize the inherent blurring prior information of the diffusion model, and the blurring rendering and image restoration are innovatively integrated into a unified framework to overcome the shortcoming of the traditional method that cannot directly and effectively process low-definition images. Meanwhile, the out-of-focus image is used to enhance the performance of blurring rendering. In order to suppress the problem of mutual conflict of optimization targets and unbalanced training samples in the image restoration and blurring rendering tasks, the two tasks are decoupled by optimizing the attention mechanism and using an alternating training strategy, which significantly improves the effects of image restoration and blurring rendering. In addition, a depth estimation module is introduced to better adapt to low-quality inputs. The embodiments of the present application can effectively suppress the performance decline caused by the mutual conflict of optimization targets and unbalanced training samples in the training process of the integrated image restoration and blurring rendering tasks, and are conducive to the deployment of the image blurring model on edge devices. On the other hand, the high-definition feature prior information captured by the model can be reused, and image restoration and blurring rendering can be realized at the same time, thereby improving the quality and efficiency of high-definition blurring rendering. Meanwhile, the method successfully alleviates the task conflict by optimizing the attention mechanism and using the alternating training strategy, and gradually improves the performance of image restoration and blurring rendering, which shows its huge practical potential in the digital zoom scene.

[0217] It should be noted that each of the above method embodiments, or each of the various possible implementation manners of each method embodiment, can be executed alone or in combination with any two or more of them, and the specific execution can be determined according to actual use requirements, and the embodiments of the present application do not limit this.

[0218] The image processing method provided in the embodiments of the present application can utilize the inherent blurring prior information of the diffusion model, and the blurring rendering and image restoration are innovatively integrated into a unified framework to overcome the shortcoming of the traditional method that cannot directly and effectively process low-definition images. Meanwhile, the out-of-focus image is used to enhance the performance of blurring rendering. In order to suppress the problem of mutual conflict of optimization targets and unbalanced training samples in the image restoration and blurring rendering tasks, the two tasks are decoupled by optimizing the attention mechanism and using an alternating training strategy, which significantly improves the effects of image restoration and blurring rendering. In addition, a depth estimation module is introduced to better adapt to low-quality inputs. The embodiments of the present application can effectively suppress the performance decline caused by the mutual conflict of optimization targets and unbalanced training samples in the training process of the integrated image restoration and blurring rendering tasks, and are conducive to the deployment of the image blurring model on edge devices. On the other hand, the high-definition feature prior information captured by the model can be reused, and image restoration and blurring rendering can be realized at the same time, thereby improving the quality and efficiency of high-definition blurring rendering. Meanwhile, the method successfully alleviates the task conflict by optimizing the attention mechanism and using the alternating training strategy, and gradually improves the performance of image restoration and blurring rendering, which shows its huge practical potential in the digital zoom scene.

[0219] Figure 9 FIG. 9 is a structural schematic diagram of an image processing device 900 provided in the embodiments of the present application. The image processing device 900 includes an acquisition module 901 and a fusion module 902.

[0220] The acquisition module 901 is configured to acquire high-definition feature information and blurring feature information of a first image, respectively.

[0221] The fusion module 902 is configured to perform fusion processing on the high-definition feature information and the blurring feature information to obtain first fusion feature information.

[0222] The acquisition module 901 is further configured to obtain a blurring image based on the first fusion feature information.

[0223] In some embodiments of the present application, the acquisition module 901 is specifically configured to:

[0224] inputting the first image into an image hallucination model; the image hallucination model comprises a first encoder, a high-definition feature extraction module and a fusion module connected in sequence;

[0225] extracting a latent vector of the first image through the first encoder;

[0226] extracting the high-definition feature information from the latent vector through the high-definition feature extraction module;

[0227] The fusion module 902 is specifically configured to:

[0228] fuse the high-definition feature information and the hallucination feature information through the fusion module to obtain the first fusion feature information.

[0229] In some embodiments of the present application, the high-definition feature extraction module comprises N first down-sampling interpolation modules connected in cascade, N being an integer greater than 1;

[0230] The acquisition module 901 is specifically configured to:

[0231] extracting first feature information in the latent vector through a first down-sampling interpolation module in the N first down-sampling interpolation modules connected in cascade;

[0232] extracting the i-th first feature information through the i-th first down-sampling interpolation module in the N first down-sampling interpolation modules connected in cascade, the i-1-th first down-sampling interpolation module outputting the i-1-th first feature information; i ∈ [2, N-1];

[0233] extracting the N-th first feature information through the N-th first down-sampling interpolation module in the N first down-sampling interpolation modules connected in cascade, the N-1-th first down-sampling interpolation module outputting the N-1-th first feature information.

[0234] In some embodiments of the present application, each first down-sampling interpolation module comprises a first residual module, a first self-attention module and a first cross-attention module;

[0235] The acquisition module 901 is specifically configured to:

[0236] residual processing the i-th first feature information through the first residual module in the i-1-th first down-sampling interpolation module to obtain a first residual processing result;

[0237] self-attention operation on the first residual processing result through the first self-attention module in the i-1-th first down-sampling interpolation module to obtain a first self-attention operation result;

[0238] The first cross-attention module in the i-1th first down-sampling interpolation module performs cross-attention operation on the first self-attention operation result to obtain the i th first feature information.

[0239] In some embodiments of the present application, the image blurring model further comprises a second encoder, a blurring feature extraction module connected with the second encoder and the feature fusion module;

[0240] The acquisition module 901 is specifically configured to:

[0241] acquire a first disparity map of the first image;

[0242] acquire a defocus map of the first image based on the first disparity map;

[0243] acquire a feature vector of the defocus map through the second encoder;

[0244] perform feature extraction on the feature vector of the defocus map through the blurring feature extraction module to obtain the blurring feature information.

[0245] In some embodiments of the present application, the blurring feature extraction module comprises N cascaded second down-sampling interpolation modules, and the blurring feature information comprises N blurring feature information;

[0246] The acquisition module 901 is specifically configured to:

[0247] perform feature extraction on the feature vector of the defocus map through the first second down-sampling interpolation module in the N cascaded second down-sampling interpolation modules to obtain the first blurring feature information;

[0248] perform feature extraction on the i-1th first blurring feature information output by the i-1th first down-sampling interpolation module through the i th second down-sampling interpolation module in the N cascaded second down-sampling interpolation modules to obtain the i th blurring feature information; i∈[2, N-1];

[0249] perform feature extraction on the N-1th second feature information output by the i-1th second down-sampling interpolation module through the N th second down-sampling interpolation module in the N cascaded second down-sampling interpolation modules to obtain the N th blurring feature information.

[0250] In some embodiments of the present application, the fusion module comprises N cascaded up-sampling interpolation modules;

[0251] The acquisition module 901 is specifically configured to:

[0252] The first layer of the N cascaded up-sampling interpolation modules is configured to perform feature fusion on the high-definition feature information and the Nth virtual feature information to obtain first second fusion feature information.

[0253] The i-th layer of the N cascaded up-sampling interpolation modules is configured to perform feature fusion on the (i-1)th fusion feature information output by the (i-1)th layer of the up-sampling interpolation modules and the (N-i)th virtual feature information to obtain the i-th second fusion feature information; i ∈ [2, N-1].

[0254] The Nth layer of the N cascaded up-sampling interpolation modules is configured to perform feature extraction on the (N-1)th second fusion feature information output by the (N-1)th layer of the up-sampling interpolation modules and the first virtual feature information to obtain the first fusion feature information.

[0255] In some embodiments of the present application, each of the up-sampling interpolation modules includes a second residual module, a second self-attention module, and a second cross-attention module.

[0256] The obtaining module 901 is specifically configured to:

[0257] The second residual module in the (i-1)th layer of the up-sampling interpolation modules is configured to perform residual processing on the (i-1)th fusion feature information and the (N-i)th virtual feature information to obtain a second residual processing result.

[0258] The second self-attention module in the (i-1)th layer of the up-sampling interpolation modules is configured to perform self-attention operation on the second residual processing result to obtain a second self-attention operation result.

[0259] The second cross-attention module in the (i-1)th layer of the up-sampling interpolation modules is configured to perform cross-attention operation on the second self-attention operation result to obtain the i-th second fusion feature information.

[0260] In some embodiments of the present application, the second self-attention module includes a low-rank adaptive super-resolution module and a low-rank adaptive virtualization module connected to the low-rank adaptive super-resolution module.

[0261] The obtaining module 901 is specifically configured to:

[0262] The low-rank adaptive super-resolution module is configured to perform super-resolution operation on the second residual processing result to obtain a super-resolution operation result.

[0263] The low-rank adaptive virtualization module is configured to perform virtualization processing on the super-resolution operation result to obtain the second self-attention operation result.

[0264] In some embodiments of the present application, in combination Figure 9 As shown in Figure 10 The device 900 further includes a processing module 903 configured to:

[0265] perform image degradation processing on a preset high-definition image sample to obtain a low-definition image sample before obtaining the high-definition feature information of the first image;

[0266] obtain a high-definition blurred image sample corresponding to the preset high-definition image sample;

[0267] use the low-definition image sample and the high-definition blurred image sample as a first training sample pair to train the second encoder, the blurred feature extraction module, and the low-rank adaptive blurring module;

[0268] use the low-definition image sample and the preset high-definition image sample as a second training sample pair to train the first encoder, the high-definition feature extraction module, and the low-rank adaptive super-resolution module;

[0269] alternately use the first training sample pair and the second training sample pair to train the second encoder, the blurred feature extraction module, and the low-rank adaptive blurring module, and the first encoder, the high-definition feature extraction module, and the low-rank adaptive super-resolution module.

[0270] In some embodiments of the present application, the obtaining module 901 is specifically configured to:

[0271] identify the first image to determine the focal point position of the first image;

[0272] obtain a first disparity value at the focal point position on the first disparity map;

[0273] obtain the defocus map based on the difference between the disparity value of each pixel point on the first disparity map and the first disparity value.

[0274] In some embodiments of the present application, the obtaining module 901 is specifically configured to:

[0275] input the first image into a depth prediction module, the depth prediction module including a student module;

[0276] obtain the first disparity map through the student module.

[0277] In some embodiments of the present application, the depth prediction module further includes a teacher module connected to the student module;

[0278] The device 900 further includes a processing module 903 configured to:

[0279] The student module is used for performing image degradation processing on the first high-definition image to obtain a degraded first high-definition image before the first parallax map is obtained.

[0280] The teacher module, the first high-definition image and the degraded first high-definition image are used for training the student module.

[0281] In some embodiments of the present application, the processing module 903 is specifically used for:

[0282] The first high-definition image is input into the teacher module, and the degraded first high-definition image is input into the student module; the teacher module comprises a third encoder and a first decoder connected with the third encoder, and the teacher module comprises a fourth encoder and a second decoder connected with the fourth encoder.

[0283] The first high-definition image is extracted by the third encoder to obtain second feature information, and the second feature information is extracted by the first decoder to obtain a second parallax map.

[0284] The degraded first high-definition image is extracted by the fourth encoder to obtain third feature information, and the second feature information is extracted by the second decoder to obtain a third parallax map.

[0285] A first difference between the second feature information and the third feature information and a second difference between the second parallax map and the third parallax map are calculated.

[0286] The student module is trained by using a weighted value of the first difference and the second difference as a loss function of the student module.

[0287] In some embodiments of the present application, the image blurring module further comprises a third decoder connected with the feature fusion module.

[0288] The acquisition module 901 is specifically used for:

[0289] The first fused feature information is decoded by the third decoder to obtain the blurred image.

[0290] The image processing apparatus provided in the embodiments of the present application obtains the blurring feature information of the first image, obtains the high-definition feature information of the first image, performs fusion processing on the high-definition feature information and the blurring feature information to obtain first fusion feature information, and obtains the blurred image based on the first fusion feature information. In this way, the image processing apparatus does not need to restore the first image into a high-definition image, but can obtain the first fusion feature information by performing fusion processing on the high-definition feature information and the blurring feature information after obtaining the high-definition feature information and the blurring feature information of the first image, and obtain the blurred image, thereby greatly reducing the error introduced in the image blurring process and effectively improving the blurring effect of the image.

[0291] The image processing apparatus in the embodiments of the present application can be an electronic device or a component in an electronic device, for example, an integrated circuit or a chip. The electronic device can be a terminal or other devices other than the terminal. For example, the electronic device can be a mobile phone, a tablet computer, a notebook computer, a palm computer, a vehicle-mounted electronic device, a mobile Internet device, an augmented reality / virtual reality device, a robot, a wearable device, an ultra-mobile personal computer, a netbook, or a personal digital assistant, and can also be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine, or a self-service machine, and the like, and the embodiments of the present application are not limited in this regard.

[0292] The image processing apparatus in the embodiments of the present application can be a device with an operating system. The operating system can be an Android operating system, an ios operating system, or other possible operating systems, and the embodiments of the present application are not limited in this regard.

[0293] The image processing apparatus provided in the embodiments of the present application can implement each process of each embodiment of the image processing method described above, and details are not repeated here to avoid repetition.

[0294] Optionally, as shown in Figure 11 The embodiments of the present application also provide an electronic device 1100, which includes a processor 1101 and a memory 1102, and the memory 1102 stores programs or instructions executable on the processor 1101. When the programs or instructions are executed by the processor 1101, each step of the above-mentioned image processing method embodiments is implemented, and the same technical effects can be achieved. Details are not repeated here to avoid repetition.

[0295] It should be noted that the electronic device in the embodiments of the present application includes the mobile electronic device and the non-mobile electronic device described above.

[0296] Figure 12 A hardware structure schematic diagram of an electronic device according to an embodiment of the present application.

[0297] The electronic device 1200 includes, but is not limited to, a radio frequency unit 1201, a network module 1202, an audio output unit 1203, an input unit 1204, a sensor 1205, a display unit 1206, a user input unit 1207, an interface unit 1208, a memory 1209, and a processor 1210, etc.

[0298] Those skilled in the art can understand that the electronic device 1200 can also include a power supply (such as a battery) for supplying power to each component, and the power supply can be logically connected to the processor 1210 through a power management system, so as to realize the functions of managing charging, discharging, and power consumption management through the power management system. Figure 12 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and the electronic device can include more or fewer components than the figure, or combine certain components, or different component arrangements, which are not described here.

[0299] The processor 1210 is configured to respectively acquire high-definition feature information and blurred feature information of a first image.

[0300] The high-definition feature information and the blurred feature information are fused to obtain first fused feature information.

[0301] Based on the first fused feature information, a blurred image is obtained.

[0302] In some embodiments of the present application, the processor is specifically configured to:

[0303] The first image is input into an image blurring model; the image blurring model includes a first encoder, a high-definition feature extraction module, and a fusion module connected in sequence.

[0304] The first encoder extracts a latent vector of the first image.

[0305] The high-definition feature extraction module extracts the high-definition feature information from the latent vector.

[0306] The fusion module fuses the high-definition feature information and the blurred feature information to obtain the first fused feature information.

[0307] In some embodiments of the present application, the high-definition feature extraction module includes N first down-sampling interpolation modules connected in cascade, and N is an integer greater than 1.

[0308] The processor 1210 is specifically configured to:

[0309] extract first feature information in the latent vector through a first one of the N cascaded first down-sampling interpolation modules;

[0310] extract i-1th first feature information output by the i-1th first down-sampling interpolation module through an ith one of the N cascaded first down-sampling interpolation modules, and obtain i-th first feature information; i∈[2, N-1];

[0311] extract N-1th first feature information output by the N-1th first down-sampling interpolation module through an Nth one of the N cascaded first down-sampling interpolation modules, and obtain the high-definition feature information.

[0312] In some embodiments of the present application, each of the first down-sampling interpolation modules comprises a first residual module, a first self-attention module, and a first cross-attention module.

[0313] The processor 1210 is specifically configured to:

[0314] perform residual processing on the i-th first feature information through a first residual module in the i-1th first down-sampling interpolation module, and obtain a first residual processing result;

[0315] perform self-attention operation on the first residual processing result through a first self-attention module in the i-1th first down-sampling interpolation module, and obtain a first self-attention operation result;

[0316] perform cross-attention operation on the first self-attention operation result through a first cross-attention module in the i-1th first down-sampling interpolation module, and obtain the i-th first feature information.

[0317] In some embodiments of the present application, the image blurring model further comprises a second encoder, a blurring feature extraction module connected with the second encoder and the feature fusion module;

[0318] The processor 1210 is specifically configured to:

[0319] obtain a first disparity map of the first image;

[0320] obtain a defocus map of the first image based on the first disparity map;

[0321] obtain a feature vector of the defocus map through the second encoder;

[0322] extract feature information of the defocus map through the blurring feature extraction module.

[0323] In some embodiments of the present application, the blurring feature extraction module comprises N cascaded second down-sampling interpolation modules, and the blurring feature information comprises N blurring feature information;

[0324] The processor 1210 is specifically configured to:

[0325] The first blurring feature information is obtained by performing feature extraction on the feature vector of the defocus map through a first second down-sampling interpolation module in the N cascaded second down-sampling interpolation modules;

[0326] The i-th blurring feature information is obtained by performing feature extraction on the (i-1)-th first blurring feature information output by the (i-1)-th first down-sampling interpolation module through an i-th second down-sampling interpolation module in the N cascaded second down-sampling interpolation modules; i ∈ [2, N-1];

[0327] The N-th blurring feature information is obtained by performing feature extraction on the (N-1)-th second feature information output by the (i-1)-th second down-sampling interpolation module through an N-th second down-sampling interpolation module in the N cascaded second down-sampling interpolation modules.

[0328] In some embodiments of the present application, the fusion module comprises N cascaded up-sampling interpolation modules;

[0329] The processor 1210 is specifically configured to:

[0330] The first second fusion feature information is obtained by performing feature fusion on the high-definition feature information and the N-th blurring feature information through a first up-sampling interpolation module in the N cascaded up-sampling interpolation modules;

[0331] The i-th second fusion feature information is obtained by performing feature fusion on the (i-1)-th fusion feature information output by the (i-1)-th up-sampling interpolation module and the (N-i)-th blurring feature information through an i-th up-sampling interpolation module in the N cascaded up-sampling interpolation modules; i ∈ [2, N-1];

[0332] The first fusion feature information is obtained by performing feature extraction on the (N-1)-th second fusion feature information output by the (N-1)-th up-sampling interpolation module and the first blurring feature information through an N-th up-sampling interpolation module in the N cascaded up-sampling interpolation modules.

[0333] In some embodiments of the present application, each up-sampling interpolation module comprises a second residual module, a second self-attention module, and a second cross-attention module;

[0334] The processor 1210 is specifically configured to:

[0335] performing residual processing on the (i-1)th fusion feature information and the (N-i)th virtual feature information through a second residual module in the (i-1)th upsampling interpolation module to obtain a second residual processing result;

[0336] performing self-attention operation on the second residual processing result through a second self-attention module in the (i-1)th upsampling interpolation module to obtain a second self-attention operation result;

[0337] performing cross-attention operation on the second self-attention operation result through a second cross-attention module in the (i-1)th upsampling interpolation module to obtain the ith second fusion feature information.

[0338] In some embodiments of the present application, the second self-attention module includes a low-rank adaptive super-resolution module and a low-rank adaptive virtualization module connected to the low-rank adaptive super-resolution module.

[0339] The processor 1210 is specifically configured to:

[0340] performing super-resolution operation on the second residual processing result through the low-rank adaptive super-resolution module to obtain a super-resolution operation result;

[0341] performing virtualization processing on the super-resolution operation result through the low-rank adaptive virtualization module to obtain the second self-attention operation result.

[0342] In some embodiments of the present application, the processor 1210 is further configured to:

[0343] performing image degradation processing on a preset high-definition image sample to obtain a low-definition image sample before obtaining the high-definition feature information of the first image;

[0344] obtaining a high-definition virtual image sample corresponding to the preset high-definition image sample;

[0345] training the second encoder, the virtual feature extraction module and the low-rank adaptive virtualization module by taking the low-definition image sample and the high-definition virtual image sample as a first training sample pair;

[0346] training the first encoder, the high-definition feature extraction module and the low-rank adaptive super-resolution module by taking the low-definition image sample and the preset high-definition image sample as a second training sample pair;

[0347] training the second encoder, the virtual feature extraction module and the low-rank adaptive virtualization module and the first encoder, the high-definition feature extraction module and the low-rank adaptive super-resolution module alternately using the first training sample pair and the second training sample pair.

[0348] In some embodiments of the present application, the processor 1210 is specifically configured to:

[0349] identify the first image to determine a focal point position of the first image;

[0350] obtain a first disparity value at the focal point position on the first disparity map;

[0351] obtain the defocus map based on the difference between the disparity value of each pixel point on the first disparity map and the first disparity value.

[0352] In some embodiments of the present application, the processor 1210 is specifically configured to:

[0353] input the first image into a depth prediction module, wherein the depth prediction module comprises a student module;

[0354] obtain the first disparity map through the student module.

[0355] In some embodiments of the present application, the depth prediction module further comprises a teacher module connected to the student module;

[0356] The processor 1210 is further configured to:

[0357] perform image degradation processing on the first high-definition image to obtain a degraded first high-definition image before obtaining the first disparity map through the student module;

[0358] train the student module through the teacher module, the first high-definition image, and the degraded first high-definition image.

[0359] In some embodiments of the present application, the processor 1210 is specifically configured to:

[0360] input the first high-definition image into the teacher module and input the degraded first high-definition image into the student module; the teacher module comprises a third encoder and a first decoder connected to the third encoder, and the teacher module comprises a fourth encoder and a second decoder connected to the fourth encoder;

[0361] extract features of the first high-definition image through the third encoder to obtain second feature information, and extract features of the second feature information through the first decoder to obtain a second disparity map;

[0362] extract features of the degraded first high-definition image through the fourth encoder to obtain third feature information, and extract features of the second feature information through the second decoder to obtain a third disparity map;

[0363] calculate a first difference value between the second feature information and the third feature information, and a second difference value between the second disparity map and the third disparity map;

[0364] train the student module by taking the weighted value of the first difference value and the second difference value as a loss function of the student module.

[0365] In some embodiments of the present application, the image blurring module further comprises a third decoder connected to the feature fusion module.

[0366] The processor 1210 is specifically configured to:

[0367] The third decoder decodes the first fused feature information to obtain the blurred image.

[0368] In the electronic device provided by the embodiments of the present application, the electronic device obtains blurred feature information of a first image; obtains high-definition feature information of the first image; fuses the high-definition feature information and the blurred feature information to obtain first fused feature information; and obtains a blurred image based on the first fused feature information. In this way, the image processing device does not need to restore the first image into a high-definition image. After obtaining the high-definition feature information and the blurred feature information of the first image, the image processing device can fuse the high-definition feature information and the blurred feature information to obtain the first fused feature information and obtain the blurred image. Therefore, the error introduced in the image blurring process is greatly reduced, and the blurring effect of the image is effectively improved.

[0369] It should be understood that in the embodiments of the present application, the input unit 1204 can include a graphics processing unit (GPU) 12041 and a microphone 12042. The graphics processing unit 12041 processes image data of a still picture or a video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 1206 can include a display panel 12061, which can be configured in the form of a liquid crystal display, an organic light-emitting diode, etc. The user input unit 1207 includes at least one of a touch panel 12071 and other input devices 12072. The touch panel 12071 is also called a touch screen. The touch panel 12071 can include a touch detection device and a touch controller. The other input devices 12072 can include, but are not limited to, a physical keyboard, function keys (such as volume control keys, on-off keys, etc.), trackballs, mice, joysticks, and the like, which will not be described here.

[0370] The memory 1209 can be used to store software programs and various data. The memory 1209 can mainly include a first storage area storing programs or instructions and a second storage area storing data, wherein the first storage area can store an operating system, application programs or instructions required by at least one function (such as a sound playing function, an image playing function, etc.), and the like. In addition, the memory 1209 can include a volatile memory or a non-volatile memory, or the memory 1209 can include both volatile and non-volatile memories. The non-volatile memory can be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically EPROM (EEPROM), or a flash memory. The volatile memory can be a Random Access Memory (RAM), a Static RAM (SRAM), a Dynamic RAM (DRAM), a Synchronous DRAM (SDRAM), a Double Data Rate SDRAM (DDR SDRAM), an Enhanced SDRAM (ESDRAM), a Synch link DRAM (SLDRAM), and a Direct Rambus RAM (DRRAM). The memory 1209 in the embodiments of the present application includes but is not limited to these and any other suitable types of memories.

[0371] The processor 1210 can include one or more processing units; optionally, the processor 1210 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and an application program, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 1210.

[0372] The embodiments of the present application also provide a readable storage medium, the readable storage medium stores programs or instructions, the programs or instructions are executed by a processor to realize various processes of the above-mentioned image processing method embodiments, and the same technical effects can be achieved. To avoid repetition, details are not described here.

[0373] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes a computer readable storage medium, such as a computer readable only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0374] The embodiment of the present application further provides a chip, which comprises a processor and a communication interface, the communication interface is coupled with the processor, the processor is used for running programs or instructions to realize the processes of the above image processing method embodiments and achieve the same technical effects. To avoid repetition, details are not described herein.

[0375] It should be understood that the chip mentioned in the embodiment of the present application can also be referred to as a system level chip, a system chip, a chip system or a system on chip, etc.

[0376] The embodiment of the present application provides a computer program product, which is stored in a storage medium, and is executed by at least one processor to realize the processes of the above image processing method embodiments and achieve the same technical effects. To avoid repetition, details are not described herein.

[0377] It should be noted that in this document, the term "comprising" or "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the method and device in the embodiment of the present application is not limited to the order of performing the functions as shown or discussed, but can also include performing the functions in a substantially simultaneous manner or in the opposite order, for example, the described method can be performed in an order different from that described, and various steps can also be added, omitted or combined. In addition, the features described with reference to some examples can be combined in other examples.

[0378] Through the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned example methods can be realized by means of software and a necessary general hardware platform, and of course, can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a computer software product in essence or in the form of a part that contributes to the prior art, which is stored in a storage medium (such as a ROM / RAM, a magnetic disk, or an optical disk) and includes a plurality of instructions for causing a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application.

[0379] The embodiments of the present application are described above in combination with the drawings, but the present application is not limited to the above-mentioned specific embodiments, and the above-mentioned specific embodiments are only illustrative and not restrictive. Those skilled in the art can make many forms under the inspiration of the present application without departing from the scope of the present application and the scope protected by the claims.

Claims

1. An image processing method, characterized in that: The method comprises: respectively acquiring high-definition feature information and blur feature information of the first image; Fusing the high-definition feature information and the blurred feature information to obtain first fused feature information; A blurred image is obtained based on the first fused feature information.

2. The method according to claim 1, characterized in that The obtaining of high-definition feature information of the first image includes: Inputting the first image into an image blurring model; the image blurring model comprises a first encoder, a high-definition feature extraction module, and a fusion module connected in sequence; extracting a latent vector of the first image by the first encoder; extracting the high-definition feature information from the latent vector by the high-definition feature extraction module; The fusing the high-definition feature information and the blur feature information to obtain first fused feature information includes: The high-definition feature information and the blurred feature information are fused by the fusion module to obtain the first fused feature information.

3. The method according to claim 2, characterized in that The high-definition feature extraction module includes N cascaded first downsampling interpolation modules, where N is an integer greater than 1; The extracting the high-definition feature information from the latent vector by the high-definition feature extraction module includes: Extracting first feature information from the latent vector by using a first first downsampling interpolation module among the N cascaded first downsampling interpolation modules; Performing feature extraction on the i-1th first feature information output by the i-1th first downsampling interpolation module through the i-th first downsampling interpolation module among the N cascaded first downsampling interpolation modules to obtain the i-th first feature information; i∈[2, N-1]; The high-definition feature information is obtained by performing feature extraction on the N-1th first feature information output by the N-1th first down-sampling interpolation module through the Nth first down-sampling interpolation module in the N cascaded first down-sampling interpolation modules.

4. The method according to claim 3, characterized in that Each of the first downsampling interpolation modules includes a first residual module, a first self-attention module, and a first cross-attention module; The method further comprises: performing feature extraction on the (i-1)th first feature information output by the (i-1)th downsampling interpolation module through the (i)th downsampling interpolation module in the N cascaded first downsampling interpolation modules to obtain the (i)th first feature information. performing residual processing on the i first feature information by using the first residual module in the (i-1)th first downsampling interpolation module to obtain a first residual processing result; performing a self-attention operation on the first residual processing result by the first self-attention module in the (i-1)th first downsampling interpolation module, obtaining a first self-attention operation result; The first cross-attention operation is performed on the first self-attention operation result by the first cross-attention module in the i-1th first downsampling interpolation module to obtain the i-th first feature information.

5. The method according to claim 2, characterized in that The image blur model further includes a second encoder, and a blur feature extraction module connected to the second encoder and the feature fusion module; The acquiring of blur feature information of the first image includes: Acquire a first disparity map of the first image; acquiring a defocus map of the first image based on the first disparity map; Obtaining a feature vector of the defocus image by the second encoder; The blur feature extraction module extracts features from the feature vector of the defocus image to obtain the blur feature information.

6. The method according to claim 5, characterized in that The blur feature extraction module includes N cascaded second downsampling interpolation modules, and the blur feature information includes N blur feature information; The extracting a feature vector of the defocus image by the blur feature extraction module to obtain the blur feature information includes: Performing feature extraction on the feature vector of the defocus image by using the first second downsampling interpolation module in the N cascaded second downsampling interpolation modules to obtain first blur feature information; performing feature extraction on the (i-1)th first defocusing feature information output by the (i-1)th first defocusing feature information output by the (i-1)th first defocusing feature information, by the (i)th second defocusing feature information among the N cascaded second defocusing feature information modules, to obtain the (i)th defocusing feature information; i∈[2,N-1]; The Nth second down-sampling interpolation module in the N cascaded second down-sampling interpolation modules performs feature extraction on the N-1th second feature information output by the (i-1)th second down-sampling interpolation module to obtain the Nth blurring feature information.

7. The method according to claim 6, characterized in that The fusion module includes N cascaded upsampling interpolation modules; The fusing the high-definition feature information and the blur feature information by the fusion module to obtain the first fused feature information includes: Performing feature fusion on the high-definition feature information and the Nth blur feature information through the first upsampling interpolation module in the N cascaded upsampling interpolation modules to obtain first second fused feature information; Performing feature fusion on the i-1th fused feature information and the Ni-th blurring feature information output by the i-1th upsampling interpolation module in the N cascaded upsampling interpolation modules to obtain the i-th second fused feature information; i∈[2, N-1]; The N-th upsampling interpolation module in the N cascaded upsampling interpolation modules performs feature extraction on the N-1th second fused feature information and the first blurring feature information output by the N-1th upsampling interpolation module to obtain the first fused feature information.

8. The method according to claim 7, characterized in that Each of the upsampling interpolation modules includes a second residual module, a second self-attention module, and a second cross-attention module; The method further comprises: performing feature fusion on the i-1th fused feature information and the Ni-th blurring feature information output by the i-1th first downsampling interpolation module in the N cascaded upsampling interpolation modules through the i-th upsampling interpolation module in the layer, to obtain the i-th second fused feature information, including: Performing residual processing on the i-1th fused feature information and the Nith blurred feature information through a second residual module in the i-1th layer upsampling interpolation module to obtain a second residual processing result; performing a self-attention operation on the second residual processing result through the second self-attention module in the i-1th layer upsampling interpolation module, obtaining a second self-attention operation result; The second self-attention operation result is subjected to a cross-attention operation by the second cross-attention module in the i-1th layer upsampling interpolation module to obtain the i-th second fusion feature information.

9. The method according to claim 8, characterized in that The second self-attention module includes a low-rank adaptive super-resolution module and a low-rank adaptive blurring module connected to the low-rank adaptive super-resolution module; The performing a self-attention operation on the second residual processing result by the second self-attention module in the i-1th layer upsampling interpolation module to obtain a second self-attention operation result includes: Performing a super-resolution operation on the second residual processing result by the low-rank adaptive super-resolution module to obtain a super-resolution operation result; The super-resolution operation result is blurred by the low-rank adaptive blurring module to obtain the second self-attention operation result.

10. The method according to claim 9, characterized in that Before obtaining the high-definition feature information of the first image, the method further includes: Performing image degradation processing on the preset high-definition image samples to obtain low-definition image samples; Obtaining a high-definition blurred image sample corresponding to the preset high-definition image sample; Using the low-definition image sample and the high-definition blurred image sample as a first training sample pair, training the second encoder, the blur feature extraction module and the low-rank adaptive blur module; Using the low-definition image sample and the preset high-definition image sample as a second training sample pair, training the first encoder, the high-definition feature extraction module and the low-rank adaptive super-resolution module; The first training sample pairs and the second training sample pairs are alternately used to train the second encoder, the blur feature extraction module and the low-rank adaptive blur module, as well as the first encoder, the high-definition feature extraction module and the low-rank adaptive super-resolution module.

11. The method according to claim 5, characterized in that The acquiring a defocus map of the first image based on the disparity map includes: recognizing the first image and determining a focus position of the first image; Obtaining a first disparity value at the focus position on the first disparity map; The defocus map is obtained based on a difference between the disparity value of each pixel on the first disparity map and the first disparity value.

12. The method according to claim 5, characterized in that The acquiring a first disparity map of the first image includes: Inputting the first image into a depth prediction module, wherein the depth prediction module includes a student module; The first disparity map is obtained through the student module.

13. The method according to claim 12, characterized in that The depth preset module also includes a teacher module connected to the student module; Before obtaining the first disparity map through the student module, the method further includes: performing image degradation processing on the first high-definition image to obtain the first high-definition image after degradation processing; The student module is trained using the teacher module, the first high-definition image, and the first high-definition image after degradation processing.

14. The method according to claim 13, characterized in that The step of training the student module by using the teacher module, the first high-definition image, and the first high-definition image after degradation processing includes: Inputting the first high-definition image into the teacher module, and inputting the first high-definition image after degradation processing into the student module; the teacher module includes a third encoder and a first decoder connected to the third encoder, and the teacher module includes a fourth encoder and a second decoder connected to the fourth encoder; Extracting features of the first high-definition image using the third encoder to obtain second feature information, and extracting features of the second feature information using the first decoder to obtain a second disparity map; extracting features of the first high-definition image after degradation processing by the fourth encoder to obtain third feature information, and extracting features of the second feature information by the second decoder to obtain a third disparity map; calculating a first difference between the second feature information and the third feature information, and a second difference between the second disparity map and the third disparity map; The weighted value of the first difference and the second difference is used as the loss function of the student module to train the student module.

15. An image processing device, characterized in that: The device comprises: An acquisition module, configured to respectively acquire high-definition feature information and blur feature information of the first image; a fusion module, configured to fuse the high-definition feature information and the blurred feature information to obtain first fused feature information; The acquisition module is further configured to obtain a blurred image based on the first fusion feature information.

16. An electronic device, characterized in that: The apparatus comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the image processing method according to any one of claims 1 to 14 are implemented.

Images