Electronic equipment, execution method thereof and computer readable storage medium

Abstract

The embodiment of the invention provides electronic equipment, an execution method thereof and a computer readable storage medium, and relates to the field of artificial intelligence. The method comprises the following steps: acquiring position information of an object in an input image; predicting trailing blurring state information of the input image based on the position information and the input image; and processing the input image based on the predicted trailing blur state information to obtain an output image. Optionally, the method performed by the electronic device may be performed using an artificial intelligence model.

Description

Technical Field

[0001] This disclosure relates to the fields of image processing and artificial intelligence, and more particularly to a method performed by an electronic device, an electronic device, and a computer-readable storage medium. Background Technology

[0002] With the rapid development of various smart devices (such as smartphones and cameras), users have increasingly higher demands for image quality. Coma is an imaging error in optical systems caused by design flaws in lenses or other components. Light rays deviating from the optical axis cannot converge to a single point on the ideal imaging plane, forming a comet-like spot with a trailing tail. In actual shooting, due to lens hardware and optical imaging limitations, image quality degrades from the center to the edges, with the four corners furthest from the image center exhibiting the worst quality, displaying directional blurring, i.e., the coma phenomenon. As users' demands for image quality increase, so does their requirement for consistent global image quality. For example, in group photos, people in the corners should be clear; when photographing menus, text at the edges should be sharp. However, current imaging often exhibits inconsistent global quality, with blurring at the corners, affecting user experience and image understanding, failing to meet user needs. Summary of the Invention

[0003] According to a first aspect of the present disclosure, a method performed by an electronic device is provided, comprising: acquiring position information of an object in an input image; predicting trailing blur state information of the input image based on the position information and the input image; and performing processing on the input image based on the predicted trailing blur state information to obtain an output image.

[0004] Optionally, predicting the trailing blur state information of the input image based on the location information and the input image includes: performing feature extraction on the input image to obtain image features of the input image; obtaining the trailing blur features of the input image based on the image features; and obtaining the trailing blur state information based on the location information and the trailing blur features.

[0005] Optionally, obtaining the trailing blur feature based on the image features includes: dividing the image features into image features corresponding to the outer region of the input image and image features corresponding to the central region of the input image; dividing the image features corresponding to the outer region into image blocks of a first scale, and dividing the image features corresponding to the central region into image blocks of a second scale, wherein the first scale is smaller than the second scale; performing self-attention operations on the image blocks of the first scale and the image blocks of the second scale respectively to obtain a first feature and a second feature; performing cross-attention operations on the first feature and the second feature to obtain a third feature; and obtaining the trailing blur feature based on the third feature.

[0006] Optionally, obtaining the position information of the object in the input image includes: obtaining the position information of the object in the input image in the camera coordinate system based on camera parameters and the depth map corresponding to the input image.

[0007] Optionally, obtaining the trailing blur state information based on the location information and the trailing blur feature includes: performing a self-attention operation on the location information and the trailing blur feature to obtain a fourth feature; and obtaining the trailing blur state information based on the fourth feature.

[0008] Optionally, obtaining the trailing blur state information based on the fourth feature includes: obtaining polar coordinate encoding information corresponding to the input image; and performing upsampling on the fourth feature based on the polar coordinate encoding information to obtain the trailing blur state information.

[0009] Optionally, processing the input image based on the predicted trailing blur state information to obtain an output image includes: performing feature encoding, simulated aperture adjustment, and feature decoding operations sequentially on the input image based on the trailing blur state information to obtain the output image.

[0010] Optionally, based on the trailing blur state information, the input image is sequentially subjected to feature encoding, simulated aperture adjustment, and feature decoding operations to obtain the output image, including: performing feature encoding on the input image based on the trailing blur state information to obtain encoded features; performing simulated aperture adjustment on the encoded features based on the trailing blur state information and a preset aperture mapping pool to obtain aperture-adjusted features; and performing feature decoding on the aperture-adjusted features based on the trailing blur state information to obtain the output image.

[0011] Optionally, the trailing blur state information includes at least one of the following: direction information, degree information, and probability information of the trailing blur in at least one region.

[0012] Optionally, at least one of the feature encoding operation, the simulated aperture adjustment operation, and the feature decoding operation includes a first convolution operation, wherein performing the feature encoding operation, the simulated aperture adjustment operation, and the feature decoding operation sequentially on the input image based on the trailing blur state information to obtain the output image includes: adjusting the convolution kernel used in the first convolution operation based on the trailing blur state information.

[0013] Optionally, the step of performing feature encoding on the input image based on the trailing blur state information to obtain encoded features includes: performing a second convolution operation on the input image, and performing a first convolution operation on the features obtained through the second convolution operation based on the trailing blur state information to obtain encoded features; or, performing a first convolution operation on the input image to obtain encoded features.

[0014] Optionally, performing a simulated aperture adjustment operation on the encoded features based on the trailing blur state information and a preset aperture mapping pool to obtain the aperture-adjusted features includes: obtaining multiple aperture features corresponding to the multiple candidate aperture parameters based on the aperture mapping parameters corresponding to the multiple candidate aperture parameters in the preset aperture mapping pool; predicting aperture feature fusion weights based on the trailing blur state information; fusing the multiple aperture features based on the predicted aperture feature fusion weights to obtain a first fused feature; fusing the encoded features and the first fused feature to obtain a second fused feature; and obtaining the aperture-adjusted features based on the second fused feature.

[0015] Optionally, obtaining the aperture-adjusted features based on the second fusion feature includes: performing a first convolution operation on the second fusion feature based on the trailing blur state information, and performing a second convolution operation on the features obtained through the first convolution operation to obtain the aperture-adjusted features; or, performing a first convolution operation on the second fusion feature based on the trailing blur state information to obtain the aperture-adjusted features.

[0016] Optionally, the step of performing feature decoding on the aperture-adjusted features based on the trailing blur state information to obtain the output image includes: performing a second convolution operation on the aperture-adjusted features, performing a first convolution operation on the features obtained through the second convolution operation based on the trailing blur state information, and obtaining the output image based on the features obtained through the first convolution operation; or, performing a first convolution operation on the aperture-adjusted features based on the trailing blur state information, and obtaining the output image based on the features obtained through the first convolution operation.

[0017] Optionally, the trailing blur state information includes the direction information, degree information, and probability information of the trailing blur in at least one region. The first convolution operation includes: adjusting the value of the convolution kernel based on the degree information of the trailing blur, and adjusting the shape of the convolution kernel based on the direction information of the trailing blur; performing a convolution operation on the feature to be subjected to the first convolution operation using the adjusted convolution kernel to obtain the first convolution feature; and fusing the feature to be subjected to the first convolution operation and the first convolution feature based on the probability information of the trailing blur to obtain the output feature of the first convolution operation.

[0018] Optionally, adjusting the value of the convolution kernel based on the degree of trailing blur includes: obtaining convolution fusion weight information based on the degree of trailing blur; and obtaining the adjusted value of the convolution kernel based on the convolution fusion weight information and the convolution kernel to be adjusted.

[0019] Optionally, adjusting the shape of the convolution kernel based on the direction information of the trailing blur includes: estimating the initial convolution direction offset of the convolution kernel based on the features of the first convolution operation to be performed, determining the final convolution direction offset based on the estimated initial convolution direction offset and the direction information of the trailing blur, and adjusting the shape of the convolution kernel based on the final convolution direction offset.

[0020] Optionally, the aperture mapping parameters corresponding to multiple candidate aperture parameters in the preset aperture mapping pool are obtained in the following way: acquiring an image captured at a predetermined aperture and each candidate aperture parameter; predicting the aperture mapping parameters corresponding to each candidate aperture parameter using a pre-trained prediction model based on each candidate aperture parameter; modulating the image features of the image captured at the predetermined aperture based on the aperture mapping parameters corresponding to each candidate aperture parameter, and obtaining the target image corresponding to each candidate aperture parameter based on the modulated image features.

[0021] According to a second aspect of the present disclosure, an electronic device is provided, comprising: a memory; and a processor coupled to the memory and configured to perform the above-described method.

[0022] According to a third aspect of the present disclosure, a computer-readable storage medium is provided that stores instructions which, when executed by at least one processor, cause the at least one processor to perform the method described above.

[0023] According to a fourth aspect of the present disclosure, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the above-described method.

[0024] According to the technical solution provided by the embodiments of this disclosure, the position information of objects in an input image is obtained. Based on the position information and the input image, the trailing blur state information of the input image is predicted. The input image is then processed based on the predicted trailing blur state information to obtain an output image. Since the trailing blur in the image is related not only to the input image itself but also to the position information of objects in the input image, more accurate trailing blur state information can be predicted based on the position information and the input image. Processing the input image based on such predicted trailing blur state information can effectively improve the trailing blur in the input image. This allows for the improvement of the trailing blur in the input image by considering the characteristics of corner trailing blur without any hardware changes, resulting in a higher quality output image.

[0025] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0026] The accompanying drawings, which are incorporated in and form part of this specification, illustrate exemplary embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0027] Figure 1 This is a schematic diagram of the trailing blur formation.

[0028] Figure 2 This is a flowchart illustrating a method performed by an electronic device according to an embodiment of the present disclosure.

[0029] Figure 3 This is a schematic diagram illustrating the operation of obtaining the position information of an object in an input image in the camera coordinate system according to an embodiment of the present disclosure.

[0030] Figure 4 This is a schematic diagram illustrating the operation of the Coma prediction module according to an embodiment of the present disclosure.

[0031] Figure 5 This diagram illustrates the operation of a two-level attention module included in a Coma prediction module according to an embodiment of the present disclosure.

[0032] Figure 6 This is a schematic diagram illustrating the operation of the Coma attention module included in the Coma prediction module according to an embodiment of the present disclosure.

[0033] Figure 7 This is a schematic diagram illustrating the operation of the upsampling module included in the Coma prediction module according to an embodiment of the present disclosure.

[0034] Figure 8This is an example architecture diagram illustrating a method according to an embodiment of the present disclosure.

[0035] Figure 9 This is a schematic diagram illustrating an example of a method according to an embodiment of the present disclosure.

[0036] Figure 10 This is a schematic diagram illustrating a first convolution operation according to an embodiment of the present disclosure.

[0037] Figure 11 This is a schematic diagram illustrating the detailed operation of a first convolution operation according to an embodiment of the present disclosure.

[0038] Figure 12 This is a schematic diagram illustrating an example of kernel value adjustment according to an embodiment of the present disclosure.

[0039] Figure 13 This is a schematic diagram illustrating an example of convolution direction prediction according to an embodiment of the present disclosure.

[0040] Figure 14 This is a schematic diagram illustrating an example of fine-tuning the convolution direction according to an embodiment of the present disclosure.

[0041] Figure 15 This is a schematic diagram illustrating a comparison between the first convolution operation according to an embodiment of the present disclosure and conventional deformable convolution.

[0042] Figure 16 This is a schematic diagram illustrating a method including simulated aperture adjustment operation according to an embodiment of the present disclosure.

[0043] Figure 17 This is a schematic diagram illustrating aperture mapping according to an embodiment of the present disclosure.

[0044] Figure 18 A block diagram showing the establishment of an aperture mapping pool according to an embodiment of the present disclosure is shown.

[0045] Figure 19 The detailed process of establishing the aperture mapping pool according to an embodiment of the present disclosure is shown.

[0046] Figure 20 This is a schematic diagram illustrating a simulated aperture adjustment operation according to an embodiment of the present disclosure.

[0047] Figure 21 The detailed process of a simulated aperture adjustment operation according to an embodiment of the present disclosure is shown.

[0048] Figure 22 The diagram illustrates the application of the method according to embodiments of the present disclosure to a camera shooting scenario or an album image editing scenario.

[0049] Figure 23An example is shown of the method according to an embodiment of this disclosure being applied to a camera shooting scene.

[0050] Figure 24 This is a processing block diagram illustrating an example of applying a method according to an embodiment of the present disclosure to a camera shooting scene.

[0051] Figure 25 This is an example illustrating the application of the method according to embodiments of the present disclosure to a photo album image editing scenario.

[0052] Figure 26 This is an example illustrating the application of the method according to embodiments of this disclosure to a video recording scenario.

[0053] Figure 27 This is a schematic diagram illustrating an example of applying the method according to an embodiment of the present disclosure to a video recording scenario.

[0054] Figure 28 This is a block diagram illustrating an electronic device according to an embodiment of the present disclosure.

[0055] Figure 29 This is a schematic diagram illustrating the structure of an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0056] The following description, with reference to the accompanying drawings, is provided to aid in a thorough understanding of the various embodiments of this disclosure as defined by the claims and their equivalents. This description includes various specific details to aid understanding but should be considered exemplary only. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the various embodiments described herein without departing from the scope and spirit of this disclosure. Furthermore, for clarity and brevity, descriptions of well-known functions and structures may be omitted.

[0057] The terms and wording used in the following description and claims are not limited to their dictionary meanings, but are merely used by the inventors to enable a clear and consistent understanding of this disclosure. Therefore, it will be apparent to those skilled in the art that the following description of various embodiments of this disclosure is for illustrative purposes only and not for limiting the purpose of this disclosure as defined in the appended claims and their equivalents.

[0058] It should be understood that the singular forms of “a,” “an,” and “the” can also include plural references unless the context clearly indicates otherwise. Thus, for example, the reference to “component surface” includes referring to one or more such surfaces. When we say that an element is “connected” or “coupled” to another element, the element can be directly connected or coupled to the other element, or it can mean that the element and the other element are connected through an intermediate element. Furthermore, the use of “connected” or “coupled” herein can include wireless connections or wireless couplings.

[0059] The terms “comprising” or “may include” refer to the presence of a corresponding disclosed function, operation, or component that may be used in the various embodiments of this disclosure, rather than limiting the presence of one or more additional functions, operations, or features. Furthermore, the terms “comprising” or “having” may be interpreted as indicating certain characteristics, numbers, steps, operations, constituent elements, components, or combinations thereof, but should not be construed as excluding the possibility of the presence of one or more other characteristics, numbers, steps, operations, constituent elements, components, or combinations thereof.

[0060] The term "or" as used in the various embodiments of this disclosure includes any of the listed terms and all combinations thereof. For example, "A or B" may include A, may include B, or may include both A and B. When describing multiple (two or more) items, if the relationship between the multiple items is not explicitly defined, the multiple items may refer to one, more, or all of the multiple items. For example, the description "parameter A includes A1, A2, A3" can be implemented as parameter A includes A1 or A2 or A3, or it can be implemented as parameter A includes at least two of the three items A1, A2, and A3.

[0061] Unless otherwise defined, all terms used in this disclosure (including technical or scientific terms) have the same meaning as understood by one of those skilled in the art to which this disclosure pertains. Common terms as defined in dictionaries are to be interpreted as having a meaning consistent with the context in the relevant technical field and should not be interpreted ideally or overly formally unless expressly defined in this disclosure.

[0062] At least some of the functions of the device or electronic device provided in this disclosure embodiment can be implemented by an AI model, such as implementing at least one module of a plurality of modules of the device or electronic device by an AI model. AI-related functions can be executed by non-volatile memory, volatile memory, and a processor.

[0063] The processor may include one or more processors. In this case, the one or more processors may be general-purpose processors, such as central processing unit (CPU), application processor (AP), etc., or pure graphics processing unit, such as graphics processing unit (GPU), vision processing unit (VPU), and / or AI-specific processors, such as neural processing unit (NPU).

[0064] The one or more processors control the processing of input data based on predefined operating rules or artificial intelligence (AI) models stored in non-volatile and volatile memory. These predefined operating rules or AI models are provided through training or learning.

[0065] Here, "providing through learning" refers to obtaining predefined operating rules or an AI model with desired characteristics by applying a learning algorithm to multiple learning datasets. This learning can be performed within the device or electronic device itself, in which the AI ​​is executed according to the embodiment, and / or can be implemented via a separate server / system.

[0066] AI models can contain multiple neural network layers. Each layer has multiple weight values, and each layer performs neural network computations by calculating the input data of that layer (such as the computation results of the previous layer and / or the input data of the AI ​​model) and the multiple weight values ​​of the current layer. Examples of neural networks include, but are not limited to, convolutional neural networks (CNNs), deep neural networks (DNNs), recurrent neural networks (RNNs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), generative adversarial networks (GANs), and deep Q-networks.

[0067] A learning algorithm is a method of training a predetermined target device (e.g., a robot) using multiple learning data sets to enable, allow, or control the target device to make determinations or predictions. Examples of such learning algorithms include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0068] The methods provided in this disclosure may relate to one or more fields in the technical fields of speech, language, image, video, or data intelligence.

[0069] Optionally, in the context of speech or language, in the method performed by an electronic device according to this disclosure, a speech signal as an analog signal may be received via a speech input device (e.g., a microphone), and the speech portion may be converted into computer-readable text using an Automatic Speech Recognition (ASR) model. The user's utterance intent can be obtained by interpreting the converted text using a Natural Language Understanding (NLU) model. The ASR model or NLU model may be an artificial intelligence model. The artificial intelligence model may be processed by a dedicated artificial intelligence processor designed in a hardware architecture specified for processing the artificial intelligence model. Language understanding is a technique for recognizing and applying / processing human language / text, including, for example, natural language processing, machine translation, dialogue systems, question answering, or speech recognition / synthesis.

[0070] Optionally, when dealing with the field of images or videos, in the method performed by an electronic device according to this disclosure, output data can be obtained by using image data as input data for an artificial intelligence model. The methods of this disclosure can relate to the field of visual understanding in artificial intelligence technology, which is a technology for recognizing and processing things like human vision, and includes, for example, object recognition, object tracking, image retrieval, human recognition, scene recognition, 3D reconstruction / localization, or image enhancement.

[0071] Optionally, in the field of data intelligence processing, in the method performed by an electronic device according to this disclosure, during the reasoning or prediction phase, an artificial intelligence model can be used to perform prediction by using real-time input data. The processor of the electronic device can perform preprocessing operations on the data to transform it into a form suitable for use as input to the artificial intelligence model. Reasoning and prediction are techniques for making logical inferences and predictions by determining information, including, for example, knowledge-based reasoning, optimization prediction, preference-based planning, or recommendation.

[0072] In this application, the artificial intelligence model can be obtained through training. Here, "obtained through training" means obtaining a predefined operational rule or artificial intelligence model configured to perform desired features (or objectives) by training a basic artificial intelligence model with multiple training data using a training algorithm. The artificial intelligence model may include multiple neural network layers. Each of the multiple neural network layers includes multiple weight values, and neural network computation is performed by calculating the results of the previous layer and the multiple weight values.

[0073] The following description of several optional embodiments illustrates the technical solutions of this disclosure and the technical effects produced by these solutions. It should be noted that the following embodiments can be referenced, learned from, or combined with each other. Identical terms, similar features, and similar implementation steps in different embodiments will not be repeated.

[0074] As described in the background section, current imaging images often exhibit inconsistent global quality, with blurring at the four corners and edges, impacting user experience and image understanding, and failing to meet user needs. Corner blurring is an imaging deviation occurring in the edge regions of the image, influenced by both the inherent properties of optical imaging hardware and the relative position of the object being photographed. This causes the image quality to gradually decrease from the center outwards. A typical manifestation is the presence of trailing artifacts in specific directions at the corners, such as… Figure 1 As shown.

[0075] To address this issue, corner blur can be removed by improving hardware quality, for example, by designing and adjusting lens combinations to improve the quality of image edge areas. However, the inventors of this application have found that while this hardware improvement method can solve the corner blur problem to some extent, it is not only very costly but also leads to other more serious image quality problems, such as altering the shape of the subject. Furthermore, current software algorithms for removing corner blur only use ordinary image deblurring techniques without considering the characteristics of corner blur itself. However, corner blur is different from other types of blur; therefore, ordinary image deblurring techniques cannot completely solve the problem of corner blur.

[0076] To address the aforementioned issues, this disclosure proposes a method to effectively improve the four-corner trailing blur in an input image by taking into account the characteristics of four-corner trailing blur without any hardware changes.

[0077] The inventors of this application have discovered that corner blur (hereinafter also referred to as "blur") differs from ordinary blur, and the state of the blur is related to the position of the captured object. Therefore, this disclosure proposes to predict the blur state information using the position information of the object in the input image, and then guide the processing of the input image based on the blur state information, thereby achieving the removal of the blur.

[0078] Figure 2 This illustrates a method performed by an electronic device according to an embodiment of the present disclosure.

[0079] Reference Figure 2 In step S210, the position information of the object in the input image is obtained. According to an embodiment, the position information of the object in the input image in the camera coordinate system can be obtained based on camera parameters and the depth map corresponding to the input image, but the method of obtaining the position information is not limited to this. This disclosure does not impose any limitations on how the camera parameters and the depth map corresponding to the input image are obtained.

[0080] Figure 3 This is a schematic diagram illustrating the operation of obtaining the position information of an object in an input image in the camera coordinate system according to an embodiment of the present disclosure. Figure 3 As shown, P is a point on an object (e.g., a tree), and the position of P in the camera coordinate system is defined as P = (x, y, z). By transforming from pixel coordinates to image coordinates and then to camera coordinates, the position of P in the camera coordinate system can be finally determined. Figure 3 In this system, the origin of the pixel coordinate system is the top-left vertex of the pixel plane, the origin of the image coordinate system O′ is the center of the image plane, the origin of the camera coordinate system O is the optical center of the camera plane, the focal length f is the distance between the projection center (the origin O of the camera coordinate system) and the image plane, the optical axis is a line perpendicular to the image plane and passing through the projection center, and the origin O′ of the image coordinate system is defined as the intersection of the optical axis and the image plane. According to an embodiment, for example, z in the camera coordinate system can be obtained from the depth map corresponding to the input image. Assuming the projection point of P in the pixel coordinate system is P′(u,v), the position (x′,y′) of P in the image coordinate system can be solved using the following equation:

[0081]

[0082] Furthermore, the position of P in the camera coordinate system, P = (x, y, z), can then be solved using the following equation:

[0083]

[0084] Among them, c x and c y The displacement s is the origin. u and s v Let f1 be the pixel width and height, and f2 be the focal length offsets in the image coordinate system. In the equation above, Based on camera parameters, the position of object P in the camera coordinate system and its position in the pixel coordinate system can be determined. This relationship allows calculation of the object's position in the camera coordinate system. The object's position in the camera coordinate system is crucial for predicting the trailing blur state.

[0085] In step S220, based on the location information and the input image, the trailing blur state information of the input image is predicted. Since the degree of image trailing is related not only to the input image itself but also to the location information of objects in the input image, in the embodiments according to this disclosure, predicting the trailing blur state information of the input image based on the location information and the input image allows for more accurate prediction of the trailing blur state information.

[0086] In the following text, the operation for predicting the trailing blur state information of the input image is also referred to as "Coma prediction" or "Coma estimation", which can be performed by the Coma prediction module according to embodiments of this disclosure. Figure 4 This is a schematic diagram illustrating the operation of a Coma prediction module according to an embodiment of the present disclosure. Referring now to… Figure 4 The above operations performed by the Coma prediction module are described.

[0087] For example, step S220 may include: first, performing feature extraction on the input image to obtain image features of the input image; second, obtaining the trailing blur features of the input image based on the image features; and finally, obtaining the trailing blur state information based on the position information and the trailing blur features.

[0088] According to an embodiment, the trailing blur state information may include at least one of the following: direction information, degree information, and probability information of the trailing blur in at least one region, but is not limited thereto.

[0089] like Figure 4 As shown, the position F of an object in the input image in the camera coordinate system can be calculated based on the depth image and camera parameters. position The input image undergoes feature extraction to obtain image features F1. For example, a feature extraction model consisting of multiple convolutional layers can be used to perform feature extraction on the input image to obtain the image features F1 of the output image. Subsequently, the image features F1 pass through two-stage attention modules to obtain the trailing blur feature. Figure 4 is represented as F2 in

[0090] According to the embodiment, obtaining the trailing blur feature based on the image feature may include: dividing the image feature into an image feature corresponding to the peripheral region of the input image and an image feature corresponding to the central region of the input image; dividing the image feature corresponding to the peripheral region into image blocks of a first scale, and dividing the image feature corresponding to the central region into image blocks of a second scale, where the first scale is smaller than the second scale; performing self-attention operations on the image blocks of the first scale and the image blocks of the second scale respectively to obtain a first feature and a second feature; performing cross-attention operations on the first feature and the second feature to obtain a third feature; and obtaining the trailing blur feature based on the third feature.

[0091] For example, the operation of obtaining the trailing blur feature may be performed by Figure 4 the two-level attention module shown in Figure 5 FIG. shows an operation schematic diagram of the two-level attention module included in the Coma prediction module according to an embodiment of the present disclosure. As Figure 5 shown, the image feature F1 of the output image is divided into an image feature corresponding to the peripheral region of the input image and an image feature corresponding to the central region of the input image. The image feature corresponding to the peripheral region is divided into image blocks of a first scale s1, and the image feature corresponding to the central region is divided into image blocks of a second scale s2, where s1 < s2 (i.e., two levels of division are performed). This is because the peripheral region is more likely to generate corner blur than the central region, so the peripheral region can be divided into smaller-scale image blocks. The two different-scale image blocks are respectively subjected to self-attention (inner attention) calculation. For example, the two different-scale image blocks are respectively subjected to linear mapping, and the Transformer encoder is used to perform feature encoding on the results of the linear mapping. The feature obtained by performing the self-attention operation on the image blocks of the first scale s1 is the first feature, and the feature obtained by performing the self-attention operation on the image blocks of the second scale s2 is the second feature. Since the image blocks of the same level have a similar degree of trailing blur in different directions, the correlation between the image blocks can be recognized through the internal attention operation.

[0092] Subsequently, as Figure 5As shown, cross attention is performed on two levels of features (i.e., the first feature and the second feature) to ensure feature interaction between image patches at different levels. Typically, adjacent image patches have similar directions of trailing blur, but different intensities. To enhance the correlation between image patches at two levels, cross attention can be used to facilitate information flow between different image patches. Finally, a linear mapping is performed on the third feature obtained through the cross attention operation, and the shape of the linearly mapped feature is restored to be consistent with the scale of image feature F1 of the output image, obtaining feature F2, which indicates the trailing blur feature. According to embodiments of this disclosure, different processing strategies are adopted for the severely blurred peripheral region and the slightly blurred central region through a two-level attention module. Specifically, for the peripheral region level, since this level of region has a high degree of blur and blur probability, small-scale, high-density image patches are used to calculate its internal correlation. For the central region level, since this level of region has a low degree of blur and blur probability, large-scale, low-density image patches are used, which can help improve computational efficiency.

[0093] Return to reference Figure 4 ,like Figure 4 As shown, after obtaining the trailing blur feature F2, the positional information F can be used to further refine the model. position The trailing blur state information is obtained by combining the trailing blur feature F2. For example, a self-attention operation can be performed on the position information and the trailing blur feature to obtain a fourth feature, and then the trailing blur state information can be obtained based on the fourth feature. The above operation can be performed by... Figure 4 The Coma attention module shown is executed. Figure 6 This is a schematic diagram illustrating the operation of the Coma attention module included in the Coma prediction module according to an embodiment of the present disclosure. Figure 6 As shown, the input to this module includes a depth image, camera parameters, and a trailing blur feature F2 from the output of a two-stage attention module. The depth image and camera parameters are processed by a convolutional module to obtain positional information F. position Subsequently, F position The fourth feature F3 is obtained by performing self-attention calculation on F2, where the self-attention calculation methods for K, Q, and V are shown in the following equation:

[0094]

[0095] Where d is the normalization parameter and Softmax(*) is the activation function.

[0096] Return to reference Figure 4After obtaining the fourth feature F3, trailing blur state information can be obtained based on the fourth feature F3. According to an embodiment, obtaining trailing blur state information based on the fourth feature may include: obtaining polar coordinate encoding information corresponding to the input image; performing upsampling on the fourth feature based on the polar coordinate encoding information to obtain the trailing blur state information. The above operations can be performed by the upsampling module according to an embodiment of this disclosure. Figure 7 This is a schematic diagram illustrating the operation of the upsampling module included in the Coma prediction module according to an embodiment of the present disclosure. Figure 4 As shown, the input to the upsampling module can include polar coordinate encoded information and the fourth feature F3 output by the Coma attention module. For example, the polar coordinate encoded information can be obtained by transforming the pixel coordinate encoded information of the input image using the transformation relationship between the pixel coordinate system and the polar coordinate system. The upsampling module can include ordinary convolution calculation and positional attention calculation, such as... Figure 4 As shown in the medium-dark gray area, the positional attention calculation is as follows: Figure 4 The light gray area is shown. For any point P within a plane (width and height W and H respectively), its polar coordinates include the distance ρ from the center of the plane and the deflection angle θ. The greater the distance ρ, the greater the degree of trailing blur. Simultaneously, the direction of the deflection is related to the direction of the trailing blur. Therefore, by further incorporating the polar coordinate encoding information of the input image into the upsampling prediction (i.e., using positional attention based on polar coordinate encoding information to guide the upsampling prediction), the positional relationship of the trailing blur can be further enhanced, thereby obtaining more accurate trailing blur state information. For example, as... Figure 7 As shown, 1*1 convolution and 3*3 convolution can be performed sequentially on the polar coordinate encoded information. The convolution result, after passing through the activation function, can be multiplied with the result of ordinary convolution calculation on the fourth feature F3. The multiplication result is then added to F3 to obtain the final trailing blur state information.

[0097] The above has been combined Figures 4 to 7 Details of the predicted trailing fuzzy state information according to embodiments of the present disclosure are described.

[0098] Return to reference Figure 2After predicting the trailing blur state information of the input image, in step S230, processing can be performed on the input image based on the predicted trailing blur state information to obtain the output image. According to the embodiment, since the trailing blur is related to the position information of objects in the input image, more accurate trailing blur state information can be predicted based on the position information of objects in the input image and the input image itself. Processing the input image under the guidance of this more accurate trailing blur state information can effectively improve the trailing blur in the input image. This allows for the improvement of the trailing blur in the input image by considering the characteristics of corner trailing blur without any hardware changes, resulting in a higher quality output image.

[0099] Step S230 will now be described in further detail with reference to the accompanying drawings.

[0100] According to an embodiment, step S230 may include: based on the trailing blur state information, sequentially performing feature encoding, simulated aperture adjustment, and feature decoding operations on the input image to obtain an output image. That is, the feature encoding, simulated aperture adjustment (also known as "aperture simulation adaptive"), and feature decoding operations can be performed under the guidance of the trailing blur state information.

[0101] Figure 8 This is an example architecture diagram illustrating a method according to an embodiment of this disclosure. (e.g.) Figure 8 As shown, for example, as described above, the Coma prediction module according to embodiments of this disclosure can obtain the position information of an object in the input image in the camera coordinate system based on the depth map corresponding to the input image and camera parameters, and predict the trailing blur state information based on the position information and the input image. Optionally, the trailing blur state information can be further predicted by combining the polar coordinate encoding information of the input image. After predicting the trailing blur state information, at least one of the following operations can be guided based on the trailing blur state information: encoding operation, simulated aperture adjustment operation, and feature decoding operation. For example, as Figure 8 As shown, the trailing blur state information can be used to guide the encoder's encoding operation, the aperture simulation adaptive operation, and the decoder's decoding operation, thereby obtaining a high-quality output image.

[0102] According to an embodiment, based on the trailing blur state information, the input image is sequentially subjected to feature encoding, simulated aperture adjustment, and feature decoding operations to obtain an output image. This may include: performing feature encoding on the input image based on the trailing blur state information to obtain encoded features; performing simulated aperture adjustment on the encoded features based on the trailing blur state information and a preset aperture mapping pool to obtain aperture-adjusted features; and performing feature decoding on the aperture-adjusted features based on the trailing blur state information to obtain the output image.

[0103] According to the embodiments, at least one operation in the encoding operation can be guided based on the trailing blur state information, at least one operation in the simulated aperture adjustment operation can be guided based on the trailing blur state information, and at least one operation in the feature decoding operation can be guided based on the trailing blur state information. Figure 9 This is a schematic diagram illustrating an example of a method according to an embodiment of the present disclosure. For example, such as... Figure 9 As shown, at least one of the feature encoding operation, the simulated aperture adjustment operation, and the feature decoding operation may include a first convolution operation ( Figure 9 The operation indicated by the yellow area in the diagram, wherein, based on the trailing blur state information, the input image is sequentially subjected to feature encoding, simulated aperture adjustment, and feature decoding operations to obtain the output image, includes: adjusting the convolution kernel used in the first convolution operation based on the trailing blur state information. The first convolution operation can also be called a "dynamic convolution operation." The trailing blur state is different in different regions of the input image; for example, the degree and direction of the trailing blur are different in different regions. Therefore, traditional fixed-size convolution kernels are not suitable for removing trailing blur. If the input image is processed using a traditional fixed-size convolution kernel, it may result in the removal of the trailing blur while areas without trailing blur are also altered. Therefore, this disclosure further proposes a light-attention first convolution operation. In this light-attention first convolution operation, the convolution kernel is adjusted based on the trailing blur state information, for example, by adjusting the shape and value of the convolution kernel, thereby achieving dynamic adaptive convolution based on the characteristics of the trailing blur in different regions. This is beneficial for maintaining the areas without trailing blur unchanged while removing the trailing blur.

[0104] Figure 10 This is a schematic diagram illustrating a first convolution operation according to an embodiment of the present disclosure. Referring below... Figure 10 A first convolution operation according to an embodiment of the present disclosure is described. According to the embodiment, the trailing blur state information may include direction information (which can be represented by L[direction]), degree information (which can be represented by L[degree]), and probability information (which can be represented by L[weight]) of the trailing blur in at least one region. In this case, for example, the first convolution operation according to an embodiment of the present disclosure may include: adjusting the value of the convolution kernel based on the degree information of the trailing blur, and adjusting the shape of the convolution kernel based on the direction information of the trailing blur; performing a convolution operation on the feature to be subjected to the first convolution operation using the adjusted convolution kernel to obtain a first convolution feature; and fusing the feature to be subjected to the first convolution operation and the first convolution feature based on the probability information of the trailing blur to obtain the output feature of the first convolution operation.

[0105] For example, such as Figure 10 As shown, the value of the convolution kernel can be adjusted based on the degree of trailing blur (i.e., performing "convolution kernel value adjustment"). Figure 11 This is a schematic diagram illustrating the detailed operation of a first convolution operation according to an embodiment of the present disclosure.

[0106] According to an embodiment, adjusting the value of the convolution kernel based on the degree of trailing blur can include: obtaining convolution fusion weight information based on the degree of trailing blur; and obtaining the adjusted value of the convolution kernel based on the convolution fusion weight information and the convolution kernel to be adjusted. Figure 11 The green box in the middle shows the specific operations for adjusting the convolution kernel value. For example... Figure 11 As shown in the green box, the input for adjusting the convolution kernel values ​​is the program information L[degree] of the trailing blur obtained through Coma prediction. L[degree] is processed through multiple fully connected networks to calculate the output convolutional fusion weight information. For example, L[degree] sequentially passes through an average pooling layer, FC, ReLU, and Softmax to obtain a set of convolutional fusion weights w1 to w2. k After obtaining the convolution fusion weight information, the adjusted value of the convolution kernel can be obtained based on the convolution fusion weight information and the convolution kernel to be adjusted. Figure 12 This is a schematic diagram illustrating an example of convolution kernel value adjustment according to an embodiment of the present disclosure. According to the embodiment, the value of the convolution kernel may include the kernel's weights and biases. For example... Figure 12 As shown, the degree of blurring L is used by a fully connected network to predict a set of fusion weights w1 to w2. k Then, the weights and biases of the adjusted convolutional kernel can be calculated using the following formula.

[0107]

[0108] Among them, W i B i These are the weights and biases of the convolutional kernels to be adjusted. The weights and biases of the adjusted convolutional kernels are given by k, where k is the number of convolutional kernels and x is the input feature.

[0109] The first convolution operation according to the embodiment includes adjusting the value of the convolution kernel based on the degree of trailing blur, and also adjusting the shape of the convolution kernel based on the direction information of the trailing blur. According to the embodiment, adjusting the shape of the convolution kernel based on the direction information of the trailing blur includes: estimating an initial convolution direction offset of the convolution kernel based on the features to be subjected to the first convolution operation; determining a final convolution direction offset based on the estimated initial convolution direction offset and the direction information of the trailing blur; and adjusting the shape of the convolution kernel based on the final convolution direction offset. (Return to Reference) Figure 11 It can be based on the features of the convolution operation to be performed (also known as the input features f of the first convolution operation). in Estimate the initial convolution direction offset of the convolution kernel. This operation can be simply referred to as "convolution direction prediction," such as... Figure 11 As shown in the blue box. Figure 13 This is a schematic diagram illustrating an example of convolution direction prediction according to an embodiment of the present disclosure. Figure 13 As shown, based on the input features f of the first convolution operation in The initial convolution direction offset of the convolution kernel can be estimated, which may include a horizontal offset of offset_x and a vertical offset of offset_y. Subsequently, the convolution kernel to be adjusted (i.e., the original convolution kernel) can be displaced according to the estimated initial convolution direction offset, thereby changing the shape of the convolution kernel and obtaining a pre-deformed convolution kernel that is initially adapted to the input features.

[0110] Return to reference Figure 10 After obtaining the initial convolution direction offset through convolution direction prediction, further fine-tuning of the convolution direction can be performed based on the initial convolution direction offset and the direction information of the trailing blur. The fine-tuning of the convolution direction is as follows: Figure 11 As shown in the red box. Figure 11 As shown in the red box, the input for convolution direction fine-tuning includes the initial convolution direction offset. The directional information L[direction] of the trailing blur is obtained from the Coma prediction. The initial convolution direction can be shifted based on the directional information L[direction] of the trailing blur. Fine-tuning is performed to obtain the final convolution direction offset Δp. n . Figure 14 This is a schematic diagram illustrating an example of fine-tuning the convolution direction according to an embodiment of the present disclosure. Figure 14 As shown, the direction information L[direction] of the trailing blur can first be convolved with two branches to output scale vectors α and β, where α and β represent scale scaling and bias, respectively. Figure 14 As shown, the initial convolution direction offset, including the horizontal offset_x and vertical offset_y, is scaled and biased to obtain the fine-tuned convolution direction offset (i.e., the final convolution direction offset Δp). n For example, the final convolution direction offset Δp can be calculated according to the following equation. n :

[0111]

[0112] Subsequently, the shape of the convolution kernel can be adjusted based on the final convolution direction offset. Figure 14 In the diagram, the red convolution kernel represents the fine-tuned kernel shape, the yellow arrow indicates the offset direction at the corresponding position of the fine-tuned kernel, and the black arrow indicates the direction of the offset. Figure 13The initial convolution direction offset is estimated as shown in the figure. As can be seen from the figure, the shape of the fine-tuned convolution kernel matches the input features better.

[0113] Return operation Figure 10 and Figure 11 After adjusting the value of the convolution kernel based on the degree of trailing blur and adjusting the shape of the convolution kernel based on the direction of the trailing blur, the adjusted convolution kernel can be used to perform a convolution operation on the features to be subjected to the first convolution operation to obtain the first convolution feature. For example... Figure 10 and Figure 11 As shown, the adjusted convolution kernel is applied to the features (i.e., the input features f) to be subjected to the first convolution operation. in On the first convolution feature G, the trailing blur is obtained. Typically, trailing blur mainly occurs in the peripheral regions of the entire image, while it is less noticeable in the central region. Furthermore, within the same color region, trailing blur is more pronounced in high-frequency regions with complex textures, while it is less noticeable in low-frequency regions. Therefore, it is necessary to balance the central and peripheral regions, as well as the high-frequency and low-frequency regions. For this purpose, as... Figure 10 and Figure 11 As shown, the input feature f is based on the probability information L[weight] of the trailing blur. in The first convolution feature G is fused with the first convolution feature G to obtain the output feature f of the first convolution operation. out By applying probabilistic information based on trailing blur to the input feature f in By fusing with the first convolutional feature G, the blurry areas can be resolved while keeping the blur-free areas clear and unchanged.

[0114] For example, the input feature f can be expressed by the following formula. in The output feature f is obtained by fusing the first convolutional feature G with the first convolutional feature G. out :

[0115] f out =L[weight]*f in +(1-L[weight])*G

[0116] The above has been referred to Figures 10 to 14 The first convolution operation according to an embodiment of the present disclosure is described. Figure 15 This is a schematic diagram illustrating a comparison between the first convolution operation according to an embodiment of the present disclosure and conventional deformable convolution. For example... Figure 15 As shown, the input is a point with a value of 1.0, but it has a directional trailing blur. Traditional deformable convolution (as referenced above) Figure 13The described convolution direction prediction only adjusts the shape of the convolution kernel based on image features, but due to its lack of trailing blur state information, it cannot effectively remove trailing blur; for example, trailing blur of 0.5 cannot be effectively removed. However, the first convolution operation according to embodiments of this disclosure can adjust not only the shape of the convolution kernel based on trailing blur state information, but also the value of the convolution kernel based on the trailing blur state information, thus adaptively reducing trailing blur. For example, as... Figure 15 As shown, the trailing blur of a point with a resolution of 1.0 is reduced to 0.1, achieving better results than traditional deformable convolution.

[0117] As mentioned above Figure 8 and Figure 9 The feature encoding operation, the simulated aperture adjustment operation, and the feature decoding operation may include at least one first convolution operation. The first convolution operation included in the feature encoding operation, the simulated aperture adjustment operation, and the feature decoding operation can all be performed in the manner described above. Therefore, in the following description of the feature encoding operation, the simulated aperture adjustment operation, and the feature decoding operation, the first convolution operation involved in their processes will not be described again.

[0118] The encoding operation, the analog aperture adjustment operation, and the feature decoding operation will be introduced below.

[0119] According to embodiments, coded features can be obtained by performing feature encoding on the input image based on the trailing blur state information. For example, performing feature encoding on the input image based on the trailing blur state information to obtain coded features may include: performing a second convolution operation on the input image, and performing a first convolution operation on the features obtained through the second convolution operation based on the trailing blur state information to obtain coded features; or, performing a first convolution operation on the input image to obtain coded features. In the second convolution operation (also known as a "static convolution operation" or "conventional convolution operation"), the convolution kernel is not adjusted, and because the convolution kernel is not adjusted, it consumes fewer computational resources than the first convolution operation. Although the first convolution operation consumes more computational resources than the second convolution operation, as described above, because the convolution kernel is adjusted, using the first convolution operation can achieve better trailing blur removal results. If a better effect of removing trailing blur is required, the encoded features can be obtained entirely based on the first convolution operation. However, if a balance between computing resources and the effect of removing trailing blur is considered, the second convolution operation and the first convolution operation can be used together, with the second convolution operation performed first and then the first convolution operation performed.

[0120] like Figure 8 As shown, the encoder's input consists of the input image and the trailing blur state information obtained from Coma prediction. For example, as... Figure 9As shown, a second convolution operation can be performed on the input image at different scales (by...). Figure 9 The gray rectangle in the image indicates the first convolution operation based on the trailing blur state information. Figure 9 (The yellow rectangle in the image indicates the encoded features).

[0121] Subsequently, based on the trailing blur state information and the preset aperture mapping pool, a simulated aperture adjustment operation can be performed on the encoded features (by...). Figure 9 The orange rectangle in the image indicates the aperture adjustment feature. This "simulated aperture adjustment operation" is also known as "aperture simulation adaptive."

[0122] Research has found that another cause of trailing blur is the influence of the optical hardware system. However, as mentioned above, removing trailing blur by improving the hardware is not only costly but also leads to other more serious image quality problems. To address this, this disclosure proposes simulating aperture adjustment in the hidden space (also known as the feature space) to obtain features with locally optimal aperture effects and no corner blur, and then obtaining a high-quality output image based on such aperture-adjusted features.

[0123] Below, we will refer to Figures 16 to 20 The simulated aperture adjustment operation is described.

[0124] First, such as Figure 16 As shown, the input to the simulated aperture adjustment operation is the encoded feature output by the encoder. Figure 16 The middle is represented as F out The system uses tail blur state information from Coma prediction and a preset aperture mapping pool. By using the aperture mapping pool and tail blur state information to guide the simulation of different aperture effects for encoded features, features with optimal local aperture features and no corner blur can be obtained (i.e., features after aperture adjustment, in...). Figure 16 The middle is represented as F a ).

[0125] Since a preset aperture mapping pool is used in the simulated aperture adjustment operation, the establishment of the aperture mapping pool will be introduced first for ease of understanding. According to the embodiment, as... Figure 16 As shown, the preset aperture mapping pool is based on different aperture f. k Obtained by performing aperture mapping estimation (also known as "modulation parameter prediction"), the aperture mapping pool can include aperture mapping parameters corresponding to multiple candidate aperture parameters, for example, those corresponding to aperture f. k The corresponding aperture mapping parameter K K and K Q For example, the aperture mapping parameter K K and K Q It can be a convolution kernel, but it is not limited to that.

[0126] According to an embodiment, the aperture mapping parameters corresponding to multiple candidate aperture parameters are obtained in the following manner: acquiring an image captured at a predetermined aperture and each candidate aperture parameter; predicting the aperture mapping parameters corresponding to each candidate aperture parameter using a pre-trained prediction model based on each candidate aperture parameter; modulating the image features of the image captured at the predetermined aperture based on the aperture mapping parameters corresponding to each candidate aperture parameter, and obtaining the target image corresponding to each candidate aperture parameter based on the modulated image features.

[0127] like Figure 17 As shown, in an image signal processing system, different apertures can yield different RGB images through image signal processing. It is possible to establish a mapping from the RGB0 of an image captured at a predetermined aperture f0 to the RGB0 of a candidate aperture f0. k The image below RGB k The mapping. For example... Figure 17 As shown, the smaller / narrower the aperture size, the less light reaches the image sensor, the deeper the field of view, and the less blurred the background. Conversely, the larger / wider the aperture size, the more light reaches the image sensor, the shallower the field of view, and the more blurred the background.

[0128] Figure 18 A block diagram illustrating the establishment of an aperture mapping pool according to an embodiment of the present disclosure is shown. Figure 18 As shown, with the input being the candidate aperture f k In this case, the candidate aperture f can be predicted using a modulation parameter prediction model. k The corresponding aperture mapping parameters. The RGB0 of the image captured at a given predetermined aperture f0 and the candidate aperture f... k RGB images captured below k And with the parameters of the encoder (E) and decoder (D) fixed, the modulation parameter prediction model is based on f. k The predicted value of f k The corresponding aperture mapping parameters modulate the image features output by the encoder, enabling the decoder to obtain the RGB image by decoding the modulated image features. k .

[0129] Figure 19 The detailed process of establishing an aperture mapping pool according to embodiments of this disclosure is illustrated. To estimate the mapping between different apertures, a common VAE (Variational Autoencoder) model with pre-trained weights can be employed. The mapping between different apertures can be represented in various ways; for example, aperture mapping parameters can be represented using convolutional kernels. Figure 18 As shown, with a fixed encoder and decoder for the VAE, it is possible to base it on f kMLP is used as the modulation parameter prediction model to predict the aperture mapping parameter K in the hidden space. K and K Q Despite Figure 18 and Figure 19 The example shows the modulation parameter prediction model as an MLP, but the modulation parameter prediction model is not limited to MLP and can be various types of prediction models. Furthermore, the mapping for each aperture is independent of each other.

[0130] like Figure 18 and Figure 19 As shown, the aperture mapping pool can be created by following these steps:

[0131] Step 1: Select a candidate aperture parameter f k Candidate aperture parameters f are predicted using a modulation parameter prediction model. k The corresponding aperture mapping parameters. The modulation parameter prediction model can be set as needed; for example, it can be a fully connected network. Additionally, the mapping parameters output by the modulation parameter prediction model can also be set as needed; for example, they can be convolutional kernel weights.

[0132] Step 2: Using the VAE model, the input image is an image (RGB0) captured at a predetermined aperture of f0, and the output image is the candidate aperture parameter f. k RGB image of the target captured below k The encoder (E) and decoder (D) have fixed parameters, while the parameters of the modulation parameter prediction model are continuously adjusted to ensure that the mapping parameter K output by the modulation parameter prediction model is obtained. K and K Q Apply the following equation to the input feature F f The output feature F obtained by modulating the input features through latent space feature transformation can be decoded to obtain the candidate aperture parameter f. k The target image captured below.

[0133]

[0134] K = f(F) f ,K K )

[0135] Q = f(F) f ,K Q )

[0136] V = F f

[0137] Among them, K K ,K Q The parameters are the output parameters of the modulation parameter prediction model, where d is the normalization parameter, f(*) is the convolution operation, and F... fLet F be the input feature, F be the output feature, and Softmax(*) be the activation function. For example, if the input feature F... f If there are N channels, then K K ,K Q It can be [3*3*N*N] in size. For example... Figure 19 As shown, This is first implemented using the matrix multiplication function MatMul. Then, after passing through Softmax, we obtain... Then implement it through MatMul The output feature F is then obtained.

[0138] Step 3: Save the aperture mapping parameters corresponding to the candidate aperture parameters and add them to the aperture mapping pool. If all candidate aperture mapping parameters have been calculated, proceed to Step 4; otherwise, proceed to Step 1 and begin calculating the next aperture parameter.

[0139] Step 4: End and save the aperture mapping pool.

[0140] With the aperture mapping pool pre-established, such as Figure 9 and Figure 16 As shown, a simulated aperture adjustment operation can be performed on the encoded features based on the trailing blur state information and a preset aperture mapping pool to obtain the aperture-adjusted features. See below for reference. Figure 20 and Figure 21 The simulated aperture adjustment operation is described.

[0141] According to an embodiment, performing a simulated aperture adjustment operation on the encoded features based on trailing blur state information and a preset aperture mapping pool to obtain the aperture-adjusted features can include: obtaining multiple aperture features corresponding to multiple candidate aperture parameters based on aperture mapping parameters corresponding to multiple candidate aperture parameters in the preset aperture mapping pool; predicting aperture feature fusion weights based on the trailing blur state information; fusing the multiple aperture features based on the predicted aperture feature fusion weights to obtain a first fused feature; fusing the encoded features and the first fused feature to obtain a second fused feature; and obtaining the aperture-adjusted features based on the second fused feature. Through the above operations, aperture changes can be simulated to remove corner blur, enabling adaptive aperture simulation calculation in the latent space.

[0142] like Figure 20 As shown, assuming that the multiple candidate aperture parameters in the preset aperture mapping pool are f1 to f2, ... k This can be achieved by adjusting f1 to f k The corresponding aperture mapping parameters are respectively subjected to aperture feature mapping to obtain their respective aperture features, for example, such as Figure 21 As shown, it can be achieved by adjusting f1 to f kEach corresponding aperture mapping parameter undergoes a self-attention operation to obtain its respective aperture feature F. a1 to f ak The number of aperture features is the same as the number of candidate aperture parameters. The attention operation here can be performed in the same way as step 2 above. After obtaining the aperture features corresponding to each of the multiple candidate aperture parameters, dynamic fusion of aperture features can be performed based on the trailing blur state information output by the Coma prediction module. For example, as... Figure 21 As shown, based on the trailing blur state information, the aperture feature fusion weights Q[k, p, p] are predicted through convolutional computation (e.g., zero-convolutional computation), where k is the number of candidate aperture parameters in the aperture mapping pool, and p is the spatial scale of the fusion. Subsequently, different aperture features F... a1 F a2 F a k is fused according to the fusion weight Q to obtain the fused feature F. k (The first fusion feature mentioned above). By fusing aperture features, it is possible to achieve continuous aperture effects using discrete aperture mapping pools.

[0143] Next, the input feature f in (That is, the encoded features F from the encoder output) out The first fused feature F obtained by fusion with the aperture feature is... k The fusion is performed to obtain the output feature f. out That is, the second fusion feature mentioned above (in Figure 21 The middle is represented as F b For example, the encoded feature F can be first... out With the first fusion feature F k Feature concatenation is performed, and then channel-adaptive fusion is achieved using global average pooling and 1*1 convolution to obtain the second fused feature F. b Finally, based on the second fusion feature F b Obtain the characteristic F after aperture adjustment a .

[0144] According to an embodiment, obtaining the aperture-adjusted feature based on the second fusion feature includes: performing a first convolution operation on the second fusion feature based on the trailing blur state information, and performing a second convolution operation on the feature obtained through the first convolution operation to obtain the aperture-adjusted feature; or, performing a first convolution operation on the second fusion feature based on the trailing blur state information to obtain the aperture-adjusted feature. (See above for reference.) Figure 8 and Figure 9 As mentioned in the description, optionally, the simulated aperture adjustment operation may also include the first convolution operation mentioned above. In this case, when obtaining the second fused feature F...b Subsequently, the second fused feature F can be analyzed based on the trailing fuzzy state information. b Perform the first convolution operation to obtain the aperture-adjusted feature F. a Alternatively, as... Figure 21 As shown, in obtaining the second fusion feature F b Subsequently, the second fused feature F can be analyzed based on the trailing fuzzy state information. b Perform the first convolution operation, and then perform a second convolution operation on the features obtained through the first convolution operation to obtain the aperture-adjusted features F. a The details of the first convolution operation have already been described above, and will not be repeated here.

[0145] The above describes the simulated aperture adjustment operation. Please refer to the previous section. Figure 8 and Figure 9 After obtaining the aperture-adjusted features through simulated aperture adjustment, a decoder can be used to decode these features to obtain a high-quality output image. For example... Figure 8 and Figure 9 As shown, the output image can be obtained by performing feature decoding under the guidance of the trailing blur state information obtained by Coma prediction. That is, the output image is obtained by performing feature decoding operation on the aperture-adjusted features based on the trailing blur state information.

[0146] According to an embodiment, obtaining an output image by performing feature decoding on aperture-adjusted features based on trailing blur state information may include: performing a second convolution operation on the aperture-adjusted features, performing a first convolution operation on the features obtained through the second convolution operation based on the trailing blur state information, and obtaining the output image based on the features obtained through the first convolution operation; or, performing a first convolution operation on the aperture-adjusted features based on the trailing blur state information, and obtaining the output image based on the features obtained through the first convolution operation. For example, such as Figure 9 As shown in the schematic diagram of the decoder, a second convolution operation (marked by the gray rectangle in the decoder box) can be performed on the aperture-adapted features obtained through aperture simulation (i.e., simulated aperture adjustment) at different scales, followed by a first convolution operation (marked by the yellow rectangle in the decoder box). Finally, a high-quality output image is obtained by performing a further convolution operation on the features obtained from the first convolution operation. The details of the first convolution operation have already been described above and will not be repeated here.

[0147] The method performed by an electronic device according to embodiments of the present disclosure has been described above. The method according to embodiments of the present disclosure can improve the trailing blur in an input image by taking into account the characteristics of trailing blur without any hardware changes, thereby obtaining a higher quality output image.

[0148] The method according to the embodiments of this disclosure can be applied to a variety of scenarios. An example scenario to which the method according to the embodiments of this disclosure can be applied is briefly described below.

[0149] For example, the method according to embodiments of this disclosure can be applied to camera shooting scenarios and album image editing scenarios.

[0150] like Figure 22 As shown, when a user takes a photo with their mobile phone or selects a photo from their phone's album for editing, it can first be determined whether to perform corner blurring. If no (N), the process ends; if yes (Y), the corner blurring is removed using the method according to this embodiment, and an updated photo is output. Figure 22 As shown, trailing blur state information can be obtained based on Coma prediction. Specifically, this involves first extracting features from the captured photo or a photo selected from the album, then sequentially passing these features through a two-stage attention module, a Coma attention module, and upsampling prediction to obtain the trailing blur state information. The method for obtaining this trailing blur state information has already been described above and will not be repeated here. After obtaining the trailing blur state information, it can be used to guide the operation of the encoder, aperture simulation adaptation, and decoder, ultimately outputting and updating the photo.

[0151] Figure 23 Examples of applying the method according to embodiments of this disclosure to a camera shooting scenario are shown. Figure 23 As shown, after acquiring the captured image through the camera, it can first determine whether post-processing is required. If no (N), the processing ends. Otherwise, it can further determine whether to perform corner blurring. If N, the processing ends. Otherwise, the corner blurring is removed using the method according to the embodiments of this disclosure.

[0152] Figure 24 This is a processing block diagram illustrating an example of a method according to embodiments of the present disclosure being applied to a camera shooting scene. For example, such as... Figure 24 As shown, after acquiring an image through an image sensor, de-mosaic processing can be performed first, followed by color conversion, and then software image signal processing (ISP) can be performed. Subsequently, trailing blur removal can be performed using the method according to this disclosure. Finally, the image after trailing blur removal is compressed, and the compressed image is stored in a photo album. After performing trailing blur removal using the method according to this disclosure, the trailing blur in the image is effectively improved.

[0153] Figure 25 This illustrates an example of applying the method according to embodiments of this disclosure to a photo album image editing scenario. For example... Figure 25 As shown, users can select images to be processed from the album and choose to perform full-image processing or partial processing as desired. Whether performing full-image processing or partial processing, the method according to embodiments of this disclosure can be used to remove the trailing blur. The specific removal process has been described above and will not be repeated here. Optionally, the method according to embodiments of this disclosure can be applied to all graphics included in the album, or the trailing blur can be removed only from the image selected by the user.

[0154] Figure 26 This illustrates an example of applying the method according to embodiments of this disclosure to a video recording scenario. For example... Figure 26 As shown, this solution can be extended to address real-time processing issues during video recording. For example, it can be extended to determine whether the frame is a keyframe or an initial frame. If it is a keyframe or an initial frame, the Coma prediction parameters (i.e., the trailing blur state information) can be updated. If it is not a keyframe, the trailing blur state information predicted in the previous frame can be used for calculations in other modules to improve computational efficiency.

[0155] Figure 27 This is a schematic diagram illustrating an example of applying the method according to embodiments of the present disclosure to a video recording scenario. For example, when shooting video, people entering and leaving the frame is a common shooting technique. When a person is located at the edge of the frame, poor image quality is obtained due to trailing blur, which seriously affects the viewing experience of the video. The method of embodiments of the present disclosure can effectively remove the trailing blur that occurs when a person is located at the edge of the frame. For example, for each frame, after performing Coma prediction to obtain its trailing blur state information, the convolution kernel can be adjusted based on the trailing blur state information, and convolution operation can be performed using the adjusted convolution kernel. Optionally, as... Figure 27 As shown, to improve video processing speed, when the changes in the image of frame t are not significant, Coma prediction can be performed on frame 0 to obtain trailing blur state information. After obtaining the first convolution kernel based on the trailing blur state information, the trailing blur state information obtained based on frame 0 and the first convolution kernel can be applied to frames 1 through 2. t This processing allows for faster acquisition of higher quality videos.

[0156] The above has described some application scenarios of the method according to the embodiments of the present disclosure. However, the method according to the embodiments of the present disclosure is not limited to the above example scenarios, but can be applied to any scenario that requires trailing blur removal.

[0157] The electronic device according to embodiments of the present disclosure will now be briefly described. Figure 28 This is a block diagram illustrating an electronic device according to an embodiment of the present disclosure. (Refer to...) Figure 28 The electronic device 2800 may include a memory 2801 and a processor 2802, wherein the processor 2802 is coupled to the memory 2801 and configured to perform the methods described above.

[0158] According to embodiments of this disclosure, a computer program product is also provided, the computer program product including a computer program / instructions that, when executed by a processor, implement the methods described above.

[0159] This disclosure also provides an electronic device including at least one processor, and optionally, at least one transceiver coupled to the at least one processor and / or at least one memory, wherein the at least one processor is configured to perform the steps of the method provided in any optional embodiment of this disclosure.

[0160] Figure 29 The diagram shows a structural schematic of an electronic device to which an embodiment of the present invention applies, such as... Figure 29 As shown, Figure 29 The illustrated electronic device 4000 includes a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, for example, via a bus 4002. Optionally, the electronic device 4000 may further include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, each of the processor 4001, memory 4003, and transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of this disclosure. Optionally, the electronic device may be a first network node, a second network node, or a third network node.

[0161] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with this disclosure. Processor 4001 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0162] Bus 4002 may include a pathway for transmitting information between the aforementioned components. Bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 4002 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 29 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0163] The memory 4003 may be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium capable of carrying or storing computer programs and capable of being read by a computer, without limitation herein.

[0164] The memory 4003 is used to store computer programs or executable instructions that execute the embodiments of this disclosure, and is controlled by the processor 4001 to execute them. The processor 4001 is used to execute the computer programs or executable instructions stored in the memory 4003 to implement the steps shown in the foregoing method embodiments.

[0165] This disclosure provides a computer-readable storage medium storing a computer program or instructions that, when executed by at least one processor, can perform or implement the steps and corresponding content of the aforementioned method embodiments.

[0166] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, can implement the steps and corresponding content of the aforementioned method embodiments.

[0167] The terms “first,” “second,” “third,” “fourth,” “1,” “2,” etc. (if present) in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in a sequence other than that shown in the figures or text.

[0168] It should be understood that although arrows indicate various operation steps in the flowcharts of the embodiments of this disclosure, the order in which these steps are implemented is not limited to the order indicated by the arrows. Unless explicitly stated herein, in some implementation scenarios of the embodiments of this disclosure, the implementation steps in each flowchart can be executed in other orders as required. Furthermore, some or all of the steps in each flowchart may include multiple sub-steps or multiple stages based on the actual implementation scenario. Some or all of these sub-steps or stages can be executed at the same time, and each sub-step or stage can also be executed at different times. In scenarios where execution times differ, the execution order of these sub-steps or stages can be flexibly configured as required, and the embodiments of this disclosure do not limit this.

[0169] The above text and accompanying drawings are provided as examples only to help the reader understand this disclosure. They are not intended and should not be construed as limiting the scope of this disclosure in any way. Although certain embodiments and examples have been provided, it will be apparent to those skilled in the art, based on the content disclosed herein, that changes can be made to the illustrated embodiments and examples, and other similar implementations based on the technical concept of this disclosure can be adopted without departing from the scope of this disclosure, and these modifications and modifications are also within the protection scope of the embodiments of this disclosure.

Claims

1. A method performed by an electronic device, comprising: Obtain the location information of objects in the input image; Based on the location information and the input image, predict the trailing blur state information of the input image; The input image is processed based on the predicted trailing blur state information to obtain the output image.

2. The method according to claim 1, wherein, Based on the location information and the input image, predict the trailing blur state information of the input image, including: Feature extraction is performed on the input image to obtain the image features of the input image; Based on the image features, the trailing blur feature of the input image is obtained; The trailing blur state information is obtained based on the location information and the trailing blur feature.

3. The method according to claim 2, wherein, Obtaining trailing blur features based on the image features includes: The image features are divided into image features corresponding to the outer regions of the input image and image features corresponding to the central regions of the input image; The image features corresponding to the outer region are divided into image blocks of size 1, and the image features corresponding to the central region are divided into image blocks of size 2, wherein the first scale is smaller than the second scale; Perform self-attention operations on the first-scale image patch and the second-scale image patch respectively to obtain the first feature and the second feature; Perform a cross-attention operation on the first and second features to obtain the third feature; The trailing blur feature is obtained based on the third feature.

4. The method according to claim 1, wherein, The step of obtaining the position information of the object in the input image includes: obtaining the position information of the object in the input image in the camera coordinate system based on the camera parameters and the depth map corresponding to the input image.

5. The method according to claim 3, wherein, Obtaining the trailing blur state information based on the location information and the trailing blur feature includes: performing a self-attention operation on the location information and the trailing blur feature to obtain a fourth feature; and obtaining the trailing blur state information based on the fourth feature.

6. The method according to claim 5, wherein, Obtaining the trailing blur state information based on the fourth feature includes: Obtain the polar coordinate encoding information corresponding to the input image; Upsampling is performed on the fourth feature based on the polar coordinate encoding information to obtain the trailing blur state information.

7. The method according to claim 1, wherein, Based on the predicted trailing blur state information, the input image is processed to obtain an output image, including: Based on the trailing blur state information, feature encoding, simulated aperture adjustment, and feature decoding operations are sequentially performed on the input image to obtain the output image.

8. The method according to claim 7, wherein, Based on the trailing blur state information, the input image is sequentially subjected to feature encoding, simulated aperture adjustment, and feature decoding operations to obtain the output image, including: Based on the trailing blur state information, feature encoding operation is performed on the input image to obtain encoded features; Based on the trailing blur state information and the preset aperture mapping pool, a simulated aperture adjustment operation is performed on the encoded features to obtain the aperture-adjusted features. The output image is obtained by performing feature decoding on the features after aperture adjustment based on the trailing blur state information.

9. The method according to any one of claims 1 to 8, wherein, The trailing blur state information includes at least one of the following: direction information, degree information, and probability information of the trailing blur in at least one region.

10. The method according to claim 7, wherein, At least one of the feature encoding operation, the simulated aperture adjustment operation, and the feature decoding operation includes a first convolution operation. Specifically, the step of performing feature encoding, simulated aperture adjustment, and feature decoding operations sequentially on the input image based on the trailing blur state information to obtain the output image includes: adjusting the convolution kernel used in the first convolution operation based on the trailing blur state information.

11. The method according to claim 8, wherein, The step of performing feature encoding on the input image based on the trailing blur state information to obtain encoded features includes: Perform a second convolution operation on the input image, and perform a first convolution operation on the features obtained through the second convolution operation based on the trailing blur state information to obtain encoded features; or, Perform a first convolution operation on the input image to obtain encoded features.

12. The method according to claim 8, wherein, Based on the trailing blur state information and a preset aperture mapping pool, a simulated aperture adjustment operation is performed on the encoded features to obtain the aperture-adjusted features, including: Based on the aperture mapping parameters corresponding to multiple candidate aperture parameters in the preset aperture mapping pool, multiple aperture features corresponding to the multiple candidate aperture parameters are obtained; Predict aperture feature fusion weights based on the aforementioned trailing blur state information; The multiple aperture features are fused based on the predicted aperture feature fusion weights to obtain the first fused feature; The encoded feature and the first fusion feature are fused together to obtain the second fusion feature; The aperture-adjusted features are obtained based on the second fusion feature.

13. The method according to claim 12, wherein, The aperture-adjusted features are obtained based on the second fusion feature, including: Based on the trailing blur state information, a first convolution operation is performed on the second fused feature, and a second convolution operation is performed on the feature obtained through the first convolution operation to obtain the aperture-adjusted feature; or, Based on the trailing blur state information, the first convolution operation is performed on the second fused feature to obtain the aperture-adjusted feature.

14. The method according to claim 8, wherein, The step of performing feature decoding on the aperture-adjusted features based on the trailing blur state information to obtain the output image includes: Perform a second convolution operation on the features after aperture adjustment, perform a first convolution operation on the features obtained through the second convolution operation based on the trailing blur state information, and obtain the output image based on the features obtained through the first convolution operation; or, Based on the trailing blur state information, a first convolution operation is performed on the aperture-adjusted features, and the output image is obtained based on the features obtained through the first convolution operation.

15. The method according to any one of claims 10, 11, 13, and 14, wherein, The trailing blur state information includes direction information, degree information, and probability information of the trailing blur in at least one region, and the first convolution operation includes: The value of the convolution kernel is adjusted based on the degree of trailing blur, and the shape of the convolution kernel is adjusted based on the direction of the trailing blur. The first convolutional feature is obtained by performing a convolution operation on the features to be subjected to the first convolution operation using the adjusted convolution kernel; Based on the probabilistic information of trailing blur, the features to be performed on the first convolution operation and the first convolution feature are fused to obtain the output features of the first convolution operation.

16. The method according to claim 15, wherein, Adjusting the convolution kernel value based on the degree of trailing blur, including: Convolutional fusion weight information is obtained based on the degree of trailing blur. The adjusted value of the convolution kernel is obtained based on the convolution fusion weight information and the convolution kernel to be adjusted.

17. The method according to claim 15, wherein, Adjusting the shape of the convolution kernel based on the direction information of the trailing blur includes: estimating the initial convolution direction offset of the convolution kernel based on the features of the first convolution operation to be performed; determining the final convolution direction offset based on the estimated initial convolution direction offset and the direction information of the trailing blur; and adjusting the shape of the convolution kernel based on the final convolution direction offset.

18. The method according to claim 12, wherein, The aperture mapping parameters corresponding to multiple candidate aperture parameters in the preset aperture mapping pool are obtained through the following method: Acquire images captured at a predetermined aperture and parameters for each candidate aperture; Based on each candidate aperture parameter, a pre-trained prediction model is used to predict the aperture mapping parameters corresponding to each candidate aperture parameter; The image features of the image captured at the predetermined aperture are modulated based on the aperture mapping parameters corresponding to each candidate aperture parameter, and the target image corresponding to each candidate aperture parameter is obtained based on the modulated image features.

19. An electronic device comprising: Memory; A processor, coupled to the memory and configured to perform the method as claimed in any one of claims 1 to 18.

20. A computer-readable storage medium storing instructions that, when executed by at least one processor, cause the at least one processor to perform the method as claimed in any one of claims 1 to 18.

21. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the method as claimed in any one of claims 1 to 18.

Images