An image processing method, apparatus and electronic device

Abstract

This application provides an image processing method, apparatus, and electronic device. The method includes: acquiring feature information of a reference frame image and a target frame image; embedding the feature information of the reference frame image into a reference matrix Query; embedding the feature information of the target frame image into a first target matrix Key and a second target matrix Value; for an image position p in the reference matrix Query, performing a similarity measurement between the reference matrix Query and the first target matrix Key to obtain an alignment position in the first target matrix Key corresponding to the image position p; and performing alignment processing based on the alignment position and the second target matrix Value. This application can reduce the computational load of the alignment module in the image processing method.

Description

Technical Field

[0001] This application relates to the field of image processing, and more particularly to an image processing method, apparatus, and electronic device. Background Technology

[0002] In night scenes or low-light environments, single-frame images captured by imaging devices often exhibit high noise levels. Image fusion, which involves capturing multiple frames consecutively, effectively reduces noise while preserving rich image details, making it a mainstream low-light imaging solution. However, capturing multiple frames in night scenes or low light conditions can lead to significant camera shake due to the long shooting time, resulting in motion differences between the acquired frames. Therefore, an alignment module needs to be designed into the multi-frame fusion noise reduction method to achieve inter-frame alignment and ensure effective fusion.

[0003] Existing alignment modules use a PCD structure, which includes multiple convolutions, resulting in excessive computation and making it difficult to implement and apply the algorithm in practice. Summary of the Invention

[0004] This application provides an image processing method, apparatus, and electronic device. Based on an attention mechanism alignment structure, the alignment method reduces the number of convolutional layers and effectively reduces the computational load of the alignment module by matching local region information.

[0005] In a first aspect, embodiments of this application provide an image processing method, the method comprising: acquiring feature information of a reference frame image and a target frame image; embedding the feature information of the reference frame image into a reference matrix Query; embedding the feature information of the target frame image into a first target matrix Key and a second target matrix Value; for an image position p in the reference matrix Query, performing a similarity measurement between the reference matrix Query and the first target matrix Key to obtain an alignment position in the first target matrix Key corresponding to the image position p; and performing alignment processing based on the alignment position and the second target matrix Value.

[0006] In some possible implementations, the step of obtaining the alignment position of the image position p in the first target matrix Key by performing a similarity measurement between the reference matrix Query and the first target matrix Key, based on the image position p in the reference matrix Query, further includes: obtaining a first local region K of size s×s centered on the image position p in the first target matrix Key, corresponding to the image position p in the reference matrix Query. p (q); Obtain a second local region V of size s×s centered at the image position p on the second target matrix Value. p(q), where q is the position within the local region; for the image position p in the reference matrix Query, in the first local region K of the first target matrix Key. p The similarity measure is performed on (q) to obtain the s×s two-dimensional similarity matrix Sim. p (q); the two-dimensional similarity matrix Sim p The maximum value of (q) is the alignment position in the first target matrix Key corresponding to the image position p.

[0007] In some possible implementations, the alignment process based on the alignment position and the second target matrix Value further includes: adjusting the two-dimensional similarity matrix Sim. p (q) Perform nonmaximum suppression processing and generate a two-dimensional Gaussian kernel weight; based on the two-dimensional Gaussian kernel weight, apply the two-dimensional similarity matrix Sim p (q) Perform weighted Softmax normalization to obtain aggregate weights; apply the aggregate weights to the second local region V on the second target matrix Value. p (q) performs aggregation to obtain the aligned output.

[0008] In some possible implementations, the two-dimensional similarity matrix Sim p (q) Performing nonmaximum suppression processing further includes: through the first local region K p The maximum value position q is obtained by weighting the adaptive weights of each position q within (q). c ; with the maximum value at position q c A two-dimensional Gaussian kernel weight is generated around the center to perform nonmaximum suppression.

[0009] In some possible implementations, the second local region V on the second target matrix Value is based on the aggregation weight. p (q) Perform aggregation and output alignment results, including: transforming the aggregation weights to 1 row, s 2 The matrix of columns will represent the second local region V. p (q) is converted to s 2 A matrix of rows and columns C; for the aggregated weights and the second local region V on the second target matrix Value. p (q) Perform matrix multiplication, transform the dimension of the product, and obtain the aligned image features of the target frame image at each image position p.

[0010] In some possible implementations, the method further includes:

[0011] The process involves acquiring N frames of raw image data as input images; selecting the middle frame of the N frames of input images as a reference frame image; and selecting each frame of the other N-1 frames of images, excluding the reference frame image, as the target frame image.

[0012] In some possible implementations, the method further includes: fusing the aligned images of the reference frame image and the N-1 frames of the target frame image; and decoding the fused image to determine the output image.

[0013] Secondly, embodiments of this application provide an image processing apparatus, including:

[0014] The system includes a feature acquisition module for acquiring feature information of a reference frame image and a target frame image; a first feature embedding module for embedding the feature information of the reference frame image into a reference matrix Query; a second feature embedding module for embedding the feature information of the target frame image into a first target matrix Key and a second target matrix Value; a similarity measurement module for performing a similarity measurement between the reference matrix Query and the first target matrix Key for an image position p in the reference matrix Query, thereby obtaining the alignment position of the image position p in the first target matrix Key; and a position alignment module for performing alignment processing based on the alignment position and the second target matrix Value.

[0015] Among the possible implementations are:

[0016] The input module is used to acquire N frames of raw image data as input images; the selection module is used to select the middle frame of the N frames of input images as the reference frame image; and to select each frame of the other N-1 frames of images besides the reference frame image as the target frame image.

[0017] Among the possible implementations are:

[0018] The fusion module is used to fuse the reference frame image and the aligned image of the target frame image in N-1 frames; the output module is used to decode the fused image to determine the output image.

[0019] Thirdly, embodiments of this application provide an electronic device, including: a processor; a memory; and one or more computer programs, wherein the computer programs are stored in the memory, and when the computer programs are executed by the processor, the first device performs the method described in any of the first aspects.

[0020] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the method described in any of the first aspects.

[0021] Compared with the prior art, the beneficial effects of the embodiments of this application are as follows: The embodiments of this application propose an alignment approach based on an attention mechanism, which provides local region matching information to replace the work of multi-layer convolution in the prior art with less computation, effectively reducing the computational load of the alignment module. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of an existing PCD alignment module;

[0023] Figure 2 This is a schematic flowchart of the image data processing method provided in the embodiments of this application;

[0024] Figure 3 This is a flowchart illustrating step S4 of an image data processing method provided in another embodiment of this application;

[0025] Figure 4 This is a schematic diagram of a local region similarity measurement provided in an embodiment of this application;

[0026] Figure 5 This is a flowchart illustrating step S5 of the image data processing method provided in another embodiment of this application;

[0027] Figure 6 This is a schematic diagram illustrating the alignment effect of the image alignment method according to an embodiment of this application;

[0028] Figure 7 This is a schematic flowchart of an image data processing method provided in another embodiment of this application;

[0029] Figure 8 This is a flowchart of a multi-frame image fusion and noise reduction algorithm based on a local attention mechanism alignment structure provided in an embodiment of this application;

[0030] Figure 9 This is the specific design flow of the alignment module 202 based on the local attention mechanism provided in the embodiments of this application;

[0031] Figure 10 This is a schematic flowchart of an image data processing method provided in another embodiment of this application;

[0032] Figure 11 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0033] The terminology used in the implementation section of this application is for the purpose of explaining specific embodiments of this application only, and is not intended to limit this application.

[0034] Low-light imaging (or night scene imaging) of electronic devices (such as mobile phones) has always been a very important computational photography task. Compared with imaging tasks under normal lighting, low-light imaging has problems such as high noise, poor scene texture restoration, and more obvious motion ghosting.

[0035] To address the issue of poor low-light imaging quality on electronic devices, one approach is to extend the exposure time. However, portable electronic devices such as mobile phones typically lack professional stabilization equipment, and long exposure shooting while handheld inevitably introduces severe motion blur, further degrading the image quality.

[0036] Additionally, algorithmic approaches can be considered, such as multi-frame synthesis for noise reduction. Multi-frame noise reduction is a crucial and relatively mature low-light imaging solution. It involves continuously capturing multiple frames of normal or short-exposure data and then synthesizing them to achieve noise reduction and texture restoration. However, when capturing multi-frame images in night scenes or low-light environments, the long shooting time can lead to significant camera shake, resulting in motion between the captured frames. Because the motion varies between frames, motion ghosting artifacts are easily introduced. Therefore, a corresponding alignment module needs to be designed in the multi-frame fusion method to quickly and effectively achieve inter-frame alignment while ensuring the preservation of details in the fused result.

[0037] Existing multi-frame fusion algorithms all use the PCD structure as an alignment module.

[0038] Existing PCD alignment modules such as Figure 1 As shown, the PCD structure is a three-layer pyramid cascaded structure, containing a total of 8 convolutional layers and 3 deformable convolutional layers. The deformable convolutional layer, except for the last one, also includes a layer that generates sampling offset positions and modulation weights. Within a single scale of the PCD structure, the sampling offset for the deformable convolution is obtained through the convolutional layers. The logic of deformable convolution is to first sample the features using the sampling offset before performing convolution, thus distinguishing it from the traditional convolutional layer's operation of sampling convolution within a fixed square receptive field. Here, the sampling offset can be understood as the magnitude of relative motion between frames; therefore, deformable convolution can achieve feature alignment. However, for multi-frame algorithms, the aforementioned PCD module involves a large amount of convolutional computation, resulting in excessive computational cost and significant difficulties in practical applications or algorithm implementation.

[0039] Based on this, this application proposes an alignment approach based on an attention mechanism. By providing local region matching information, it replaces the work of multiple convolutions in the PCD module with less computation, effectively reducing the computational load of the alignment module. The alignment method and module of this application embodiment are described in detail below.

[0040] Figure 2 This is a flowchart illustrating an image data processing method provided in an embodiment of this application. This method can be executed by an electronic device; specifically, it can be executed by a processor in the electronic device. Further, it can be executed by an image capturing application in the electronic device, such as a camera application, or by the image capturing application calling relevant services provided by the operating system of the electronic device. Figure 2 As shown, the method may include:

[0041] Step S1: Obtain feature information of the reference frame image and the target frame image;

[0042] Specifically, this application proposes a multi-frame image fusion and noise reduction algorithm based on a local attention mechanism alignment structure. The multi-frame image consists of N frames, where N is a positive integer. The middle frame of the N input images is selected as the reference frame for alignment, and the other N-1 frames are the target frames. For each target frame, the alignment process is performed using the method of this application, and then the N frames are fused. This can greatly reduce the computational load of the multi-frame fusion algorithm.

[0043] For the input N frames of images, labeled I n (n = 1, ..., N), firstly, feature extraction is performed using an encoder to convert it into feature X. n :

[0044] X n =F(I n (1)

[0045] Where F(·) represents the feature transformation function of the encoder, which consists of multiple convolutional layers. For a selected reference frame, denoted as X... r For each target frame, it is marked as X n The feature information of the reference frame image and the target frame image is obtained through the feature transformation function described above.

[0046] Step S2: Embed the feature information of the reference frame image into the reference matrix Query;

[0047] Specifically, for the selected reference frame X r , its feature X r Embedded reference matrix Query, hereinafter abbreviated as matrix Q:

[0048] Q = Conv(X) r(2)

[0049] Here, Conv(·) represents a convolutional layer. This convolutional layer can maintain the original feature scale or downsample the features. Using different convolutional layers allows alignment to be performed at different scales, providing greater flexibility to adapt to different application scenarios.

[0050] Step S3: Embed the feature information of the target frame image into the first target matrix Key and the second target matrix Value;

[0051] Specifically, for each target frame X n , its feature X n Embed the first target matrix Key and the second target matrix Value, hereinafter referred to as matrices K and V:

[0052] K = Conv(X) n (3)

[0053] V = Conv(X) n (4)

[0054] Here, Conv(·) represents a convolutional layer. This convolutional layer can maintain the original feature scale or downsample the features. Using different convolutional layers allows alignment to be performed at different scales, providing greater flexibility for various application scenarios. The feature information of the target frame image is embedded into two matrix models K and V. The first target matrix K is used to measure the similarity with the reference frame image, or in other words, to find the alignment position in the target frame image. The second target matrix V is used to output the alignment result based on the alignment position, i.e., the feature information at the alignment position.

[0055] Step S4: For the image position p in the reference matrix Query, perform a similarity measurement between the reference matrix Query and the first target matrix Key to obtain the alignment position of the image position p in the first target matrix Key;

[0056] Specifically, in one feasible approach, for an image position p in a reference matrix Q, the most similar point in a first target matrix Key is found as the alignment position through similarity calculation, thus obtaining the alignment position of the image position p in the first target matrix Key.

[0057] In another feasible approach, based on the relatively small range of inter-frame image motion, for each image position p in the reference matrix Q, a local window centered on p is taken in the first target matrix Key, and the most similar point is found within the local window through similarity calculation as the alignment position, that is, the alignment position of the image position p in the first target matrix Key is obtained, which can further reduce the amount of computation.

[0058] Step S5: Perform alignment processing based on the alignment position and the second target matrix Value.

[0059] Specifically, for each image position p in the reference matrix Q, another local window centered on p is taken in the second target matrix Value. As described above, after obtaining the alignment position in the first target matrix Key corresponding to the image position p, the alignment result corresponding to the alignment position is taken in another local window of the second target matrix Value according to the alignment position.

[0060] In another specific implementation of the embodiments of this application, see [link to relevant documentation]. Figure 3 Step S4 further includes:

[0061] Step S41: Corresponding to the image position p in the reference matrix Query, obtain the first local region K of size s×s centered at the image position p on the first target matrix Key. p (q);

[0062] Specifically, such as Figure 4 The image shown is a reference diagram for measuring the similarity of local regions, such as... Figure 4 As shown, the reference matrix Q and the first target matrix K are H×W×C matrix blocks. For each position p in the reference matrix Q, it is denoted as Q. p Specifically, it can be represented as a 1×1×C matrix block. The first target matrix K correspondingly takes an s×s local region centered at p, which can be represented as an s×s×C matrix block, denoted as K. p (q), where C can be the number of channels in the feature map, and q is the position within a local region of the matrix block.

[0063] Step S42: Obtain a second local region V of size s×s centered at the image position p on the second target matrix Value. p (q), where q is the position within the local region;

[0064] The second target matrix V is an H×W×C matrix block. For each position p in the reference matrix Q, it can be specifically represented as a 1×1×C matrix block. Correspondingly, an s×s local region centered at p is taken in the second target matrix V, which can be specifically represented as an s×s×C matrix block, denoted as V. p (q), where C can be the number of channels in the feature map, and q is the position within a local region of the matrix block.

[0065] Step S43: For the image position p in the reference matrix Query, in the first target matrix Key, the s×s first local region K centered at the image position p.p The similarity measure is performed on (q) to obtain the s×s two-dimensional similarity matrix Sim. p (q);

[0066] Specifically, for the reference matrix Q p and the first local region K of the first target matrix p (q) Perform similarity measurement to obtain an s×s two-dimensional similarity matrix diagram, as shown below. Figure 4 As shown, similarity is calculated using matrix multiplication, and the formula is as follows:

[0067] Sim p (q)=Matmul(K p (q),Q p (5)

[0068] Here, Matmul(·) denotes matrix multiplication. To accommodate matrix multiplication, K... p (q) needs to be transformed into s through dimensionality transformation. 2 A matrix of rows and columns Q p Convert the matrix to a C-row, 1-column matrix, and finally perform a dimensionality transformation on the product to obtain an s×s two-dimensional similarity graph Sim. p (q).

[0069] Step S44, the two-dimensional similarity matrix Sim p The maximum value of (q) is the alignment position in the first target matrix Key corresponding to the image position p.

[0070] In one feasible implementation, the point most similar to image position p is found within an s×s local region window through similarity calculation; that is, a two-dimensional similarity matrix Sim is used. p The maximum value of (q) is the alignment position in the first target matrix Key corresponding to the image position p.

[0071] In another specific implementation of the embodiments of this application, see [link to relevant documentation]. Figure 5 Step S5 further includes:

[0072] Step S51, for the two-dimensional similarity matrix Sim p (q) Perform nonmaximum suppression processing and generate two-dimensional Gaussian kernel weights;

[0073] Since when the two-dimensional similarity matrix Sim p When multiple values ​​of similarity exist in a matrix, meaning there are multiple positions with the highest similarity, subsequent aggregation processing can cause blurring of feature details due to weighting. Therefore, it is necessary to adjust the two-dimensional similarity matrix Sim... pNon-maximum suppression is performed, which involves identifying the unique position with the highest similarity and applying non-maximum suppression to other noisy positions. In addition, when the input data contains significant noise, the two-dimensional similarity matrix Sim... p The accuracy can also be reduced due to noise interference, therefore it is also necessary to adjust the two-dimensional similarity matrix Sim. p Non-maximum suppression is performed to improve the two-dimensional similarity matrix Sim. p The accuracy.

[0074] Specifically, the maximum value position q is obtained by weighting the adaptive weights of each position q within the s×s local region window. c That is, the center position q for nonmaximum suppression is obtained through a weighted method. c :

[0075] q c =∑ q w p (q)·q (6)

[0076] Among them, w p (q) represents the adaptive weights of each point.

[0077] With the maximum value position q c Two-dimensional Gaussian kernel weights are generated around the center to perform nonmaximum suppression:

[0078]

[0079] Where σ is the standard deviation of the Gaussian function.

[0080] Step S52, based on the two-dimensional Gaussian kernel weights, apply the two-dimensional similarity matrix Sim p (q) Perform weighted Softmax normalization to obtain the aggregate weights;

[0081] As described above, with the maximum value at position q c After generating a two-dimensional Gaussian kernel weight centered at the center and performing non-maximum suppression processing, the two-dimensional similarity matrix Sim is then analyzed based on the two-dimensional Gaussian kernel weight. p The aggregate weights m are obtained by performing weighted Softmax normalization. p (q):

[0082]

[0083] Where α is the temperature coefficient, and q′ is each position within the local region window involved in the normalization operation.

[0084] As described above, the embodiments of this application design a two-dimensional Gaussian kernel generation method, which obtains the center position of the Gaussian kernel through a weighted method, and generates an s×s two-dimensional Gaussian weight g at that position. p This ensures the robustness of the alignment module on noisy data. By designing a robust nonmaximum suppression strategy, the stability of the alignment function can be guaranteed under high noise conditions.

[0085] Step S53, according to the aggregation weight, the second local region V on the second target matrix Value is... p (q) performs aggregation to obtain the aligned output.

[0086] Specifically, based on the above aggregation weight m p (q) For the second local region V p (q) Perform aggregation to obtain the aligned output:

[0087]

[0088] Matmul(·) represents matrix multiplication.

[0089] Specifically, by using the aggregation weight m p (q) Perform a dimensional transformation to 1 row, s 2 The matrix of columns will represent the second local region V. p (q) is converted to s 2 A matrix of rows and columns C; for the aggregate weight m p (q) and the second local region V on the second target matrix Value p (q) Perform matrix multiplication, then transform the calculated matrix product to obtain the aligned image features of the target frame image at each image position p. That is, to adapt to matrix multiplication, m p It needs to be transformed into 1 row by dimensional transformation. 2 A column matrix, V p Convert to s 2 A matrix of rows and columns C, where C is the number of channels in the feature map. Then, a dimensionality transformation is performed on the product to obtain... 1×1×C feature information at point p.

[0090] The image processing method in this embodiment is based on the Query, Key, Value model in the attention mechanism to design an alignment module. It matches inter-frame feature information of multiple frames, achieving effective alignment of inter-frame motion. Furthermore, by designing a robust non-maximum suppression strategy, it ensures the stability of the alignment function even under high-noise conditions. Figure 6The diagram shows the alignment effect of the image alignment method according to an embodiment of this application. Specifically, the upper and lower sets of figures show the movement of the hand and human body positions in the image before alignment relative to the reference frame image, while the image after alignment shows the effect of the hand and human body positions being aligned relative to the reference frame image. It can be seen that the image processing method of this application can achieve effective alignment of inter-frame motion.

[0091] Furthermore, referring to the steps and flow of the above image processing methods, the alignment module in the embodiment of this application only requires the design of one convolutional layer. Other steps, such as matrix multiplication and two-dimensional Gaussian kernel weight calculation, are performed within a local region, resulting in low computational cost. Therefore, the overall computational cost of the entire module is significantly lower than other technical solutions when performing multiple inter-frame alignments, enabling fast and effective inter-frame alignment. Specifically, the theoretical computational cost comparison between the alignment module of this application and the PCD alignment module is shown in the table below:

[0092] Table 1 Comparison of computational complexity between the alignment module and the PCD alignment module in this application.

[0093]

[0094] As can be seen from the comparison of computational load, for the same feature size image alignment processing, the computational load of the method in this application embodiment is 1.17T, which is much smaller than the computational load of 11.7T of the PCD alignment module. This can greatly improve the efficiency of the alignment module, quickly and effectively realize the inter-frame alignment function, and thus improve the effect of multi-frame fusion noise reduction.

[0095] In another specific implementation of this application's embodiments, the image alignment method described above is applied to a multi-frame image fusion and noise reduction algorithm, see [link to relevant documentation]. Figure 7 Before step S1: acquiring feature information of the reference frame image and the target frame image, the image processing method further includes:

[0096] Step S101: Obtain N frames of raw image data as input images;

[0097] In one feasible implementation, this application proposes a multi-frame image fusion and noise reduction algorithm based on a local attention mechanism alignment structure, the flowchart of which is shown below. Figure 8As shown, the neural network model 200 mainly comprises an encoder 201, an alignment module 202 based on a local attention mechanism, a fusion module 203, and a decoder 204. Multiple frames of input images are input into the neural network 200. First, the encoder 201 performs feature extraction. The extracted features are then input into the alignment module 202 for image alignment. The aligned images are then input into the fusion module 203 for concatenation and fusion. Finally, the decoder 204 decodes and outputs the final image. Specifically, the encoder 201, fusion module 203, and decoder 204 can have various implementation schemes, which are not intended to limit this application.

[0098] Step S102: Select the middle frame of the N input images as the reference frame image;

[0099] Specifically, the multi-frame image consists of N frames. The middle frame of the N input images is selected as the reference frame for alignment, and the other N-1 frames are the target frames. For each target frame, the image alignment method of this application embodiment is used for alignment processing, and then the N frames are fused together. This can greatly reduce the amount of computation and improve the efficiency of image fusion.

[0100] Alternatively, the middle frame of the N input images can be selected as the reference frame image, or any frame of the N input images can be selected as the reference frame image. Since the middle frame is used as the reference frame, its motion error is relatively balanced with other frames, and the accuracy of alignment processing will be relatively higher.

[0101] Step S103: Select each frame of the other N-1 frames besides the reference frame image as the target frame image;

[0102] Further, after selecting the reference frame image and the target frame image, alignment processing is performed. The reference frame image and each target frame image are input into the alignment module 202 based on the local attention mechanism for alignment processing to obtain the aligned image of the target frame image. The specific design process of the alignment module 202 based on the local attention mechanism is as follows: Figure 9 As shown, referring to the alignment method described above, it mainly includes eight steps: feature extraction, feature embedding, local region window expansion, matrix multiplication, non-maximum suppression, softmax normalization, matrix multiplication, and outputting aligned features. Based on this, this application's embodiment focuses on proposing an effective and lightweight alignment module. The alignment module is designed based on the Query, Key, and Value model in the attention mechanism, which can effectively reduce the computational load of the alignment module by providing local region matching information.

[0103] Further, see Figure 10After step S5: aligning the image based on the alignment position and the second target matrix Value, the image processing method further includes:

[0104] Step S401: Merge the reference frame image and the aligned images of the target frame image in N-1 frames;

[0105] As described above, image fusion is performed on each target frame image after alignment processing. Specifically, this is achieved by fusing the feature information of each aligned frame image. The resulting image is obtained by fusion:

[0106]

[0107] Where Conv(·) represents 3x3 convolution, and Concat(·) represents concatenation of feature information.

[0108] Step S402: Decode the fused image to determine the output image.

[0109] As described above, the fused image X f Decode the image to obtain the final output RGB image Y:

[0110] Y = F(X) f )

[0111] Where F(·) represents the feature transformation function of the decoder, which consists of multiple convolutional layers.

[0112] In another possible implementation, corresponding to the above image processing method, this application embodiment also provides an image processing apparatus, which includes:

[0113] The input module is used to acquire N frames of raw image data as input images;

[0114] The selection module is used to select an intermediate frame as a reference frame image from N input images; and to select each frame of the other N-1 frames as a target frame image.

[0115] An alignment module based on a local attention mechanism is used to perform alignment processing on each target frame image to obtain an aligned image of the target frame image;

[0116] A fusion module is used to fuse the reference frame image and the aligned image of each of the target frame images;

[0117] The output module is used to decode the fused image to determine the output image.

[0118] Specifically, the alignment module based on the local attention mechanism further includes:

[0119] The feature acquisition module is used to acquire feature information of the reference frame image and the target frame image;

[0120] The first feature embedding module is used to embed the feature information of the reference frame image into the reference matrix Query;

[0121] The second feature embedding module is used to embed the feature information of the target frame image into the first target matrix Key and the second target matrix Value.

[0122] The similarity measurement module is used to perform a similarity measurement between the reference matrix Query and the first target matrix Key for the image position p in the reference matrix Query, so as to obtain the alignment position of the image position p in the first target matrix Key.

[0123] The position alignment module is used to output the alignment result based on the alignment position and the second target matrix Value.

[0124] In summary, this application proposes an effective and lightweight alignment module based on an attention mechanism. By providing local region matching information, the computational load of the alignment module is effectively reduced. Furthermore, an image processing apparatus is proposed that performs multi-frame fusion and noise reduction processing on multiple frames of images based on the aforementioned alignment module, providing a practical and feasible low-light imaging solution.

[0125] This application also provides an electronic device for executing the image processing method described in the above embodiments, with reference to... Figure 11 This document illustrates a schematic diagram of an electronic device according to an embodiment of this application. The specific embodiments of this application do not limit the specific implementation of the electronic device.

[0126] like Figure 11 As shown, the electronic device 110 may include: a processor 1102, a communications interface 1104, a memory 1106, and a communications bus 1108.

[0127] in:

[0128] The processor 1102, communication interface 1104, and memory 1106 communicate with each other via communication bus 1108.

[0129] Communication interface 1104 is used to communicate with other electronic devices or servers.

[0130] The processor 1102 is used to execute program 1110, specifically to perform the relevant steps in the above data processing method embodiment.

[0131] Specifically, program 1110 may include program code that includes computer operation instructions.

[0132] The processor 1102 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application. The smart device may include one or more processors of the same type, such as one or more CPUs; or it may include processors of different types, such as one or more CPUs and one or more ASICs.

[0133] Memory 1106 is used to store program 1110. Memory 1106 includes SRAM memory.

[0134] Specifically, program 1110 can be used to cause processor 1102 to execute the steps described in any of the methods in the above embodiments. The specific implementation of each step in program 1110 can be found in the corresponding descriptions in the above method embodiments, and will not be repeated here. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the devices and modules described above can be referred to the corresponding process descriptions in the foregoing method embodiments, and will not be repeated here.

[0135] Based on the methods described in the above embodiments, this application provides a computer storage medium storing a computer program thereon, which, when executed by a processor, implements the methods described in the above embodiments.

[0136] Based on the methods described in the above embodiments, this application provides a computer program product that, when executed by a processor, implements the methods described in the above embodiments.

[0137] It should be noted that, depending on the implementation needs, the various components / steps described in the embodiments of this application can be broken down into more components / steps, or two or more components / steps or parts of the operation of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of this application.

[0138] The methods described in the embodiments of this application can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code downloaded over a network that is originally stored in a remote recording medium or a non-transitory machine-readable medium and will be stored in a local recording medium. Thus, the methods described herein can be processed by software stored on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an ASIC or FPGA). It is understood that the computer, processor, microprocessor controller, or programmable hardware includes storage components (e.g., RAM, ROM, flash memory, etc.) capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the navigation methods described herein are implemented. Furthermore, when a general-purpose computer accesses the code used to implement the navigation methods shown herein, the execution of the code transforms the general-purpose computer into a dedicated computer for executing the navigation methods shown herein.

[0139] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this application.

[0140] This application describes embodiments, but is not intended to limit them. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the embodiments. Therefore, all equivalent technical solutions also fall within the scope of the embodiments. The patent protection scope of the embodiments should be defined by the claims.

Claims

1. An image processing method, characterized in that, The method includes: Obtain feature information from the reference frame image and the target frame image; Embed the feature information of the reference frame image into the reference matrix Query; Embed the feature information of the target frame image into the first target matrix Key and the second target matrix Value; For the image position p in the reference matrix Query, a similarity measurement is performed between the reference matrix Query and the first target matrix Key to obtain the alignment position of the image position p in the first target matrix Key; Alignment is performed based on the alignment position and the second target matrix Value; The step of obtaining the alignment position of image position p in the first target matrix Key by performing a similarity measurement between the reference matrix Query and the first target matrix Key, for image position p, includes: For each image position p in the reference matrix Query, a local window centered on image position p is taken in the first target matrix Key, and the most similar point is found within the local window through similarity calculation as the alignment position, thus obtaining the alignment position of image position p in the first target matrix Key.

2. The method according to claim 1, characterized in that, in, For each image position p in the reference matrix Query, a local window centered on image position p is taken in the first target matrix Key, and the most similar point is found within the local window through similarity calculation as the alignment position, thus obtaining the alignment position of image position p in the first target matrix Key, including: Corresponding to the image position p in the reference matrix Query, obtain a first local region K of size s×s centered at the image position p on the first target matrix Key. p (q); Obtain a second local region V of size s×s centered at the image position p on the second target matrix Value. p (q), where q is the position within the local region; For the image position p in the reference matrix Query, in the first local region K of the first target matrix Key p The similarity measure is performed on (q) to obtain the s×s two-dimensional similarity matrix Sim. p (q); The two-dimensional similarity matrix Sim p The maximum value of (q) is the alignment position in the first target matrix Key corresponding to the image position p.

3. The method according to claim 2, characterized in that, in, The alignment process based on the alignment position and the second target matrix Value further includes: For the two-dimensional similarity matrix Sim p (q) Perform nonmaximum suppression processing and generate two-dimensional Gaussian kernel weights; Based on the aforementioned two-dimensional Gaussian kernel weights, the two-dimensional similarity matrix Sim... p (q) Perform weighted Softmax normalization to obtain the aggregate weights; Based on the aggregation weight, the second local region V on the second target matrix Value is... p (q) performs aggregation to obtain the aligned output.

4. The method according to claim 3, characterized in that, The two-dimensional similarity matrix Sim p (q) Non-maximum suppression processing also includes: Through the first local region K p The maximum value position q is obtained by weighting the adaptive weights of each position q within (q). c ; With the maximum value position q c A two-dimensional Gaussian kernel weight is generated around the center to perform nonmaximum suppression.

5. The method according to claim 4, characterized in that, The second local region V on the second target matrix Value is based on the aggregation weight. p (q) Perform aggregation and output alignment results, including: The aggregation weights are transformed to have 1 row and s. 2 The matrix of columns will represent the second local region V. p (q) is converted to s 2 A matrix with 1 row and C columns; For the aggregated weights and the second local region V on the second target matrix Value p (q) Perform matrix multiplication; The product is transformed to obtain the aligned image features of the target frame image at each image position p.

6. The method according to claim 1, characterized in that, Before obtaining the feature information of the reference frame image and the target frame image as described in the step, the method further includes: Obtain N frames of raw image data as input images; Select the middle frame of the N input images as the reference frame image; and Each of the N-1 other frames besides the reference frame image is selected as the target frame image.

7. The method according to claim 6, characterized in that, After performing alignment processing based on the alignment position and the second target matrix Value as described in step [1], the process further includes: The reference frame image and the aligned image of the target frame in the N-1 frames are fused together; The output image is determined by decoding the fused image.

8. An image processing apparatus, characterized in that, include: The feature acquisition module is used to acquire feature information of the reference frame image and the target frame image; The first feature embedding module is used to embed the feature information of the reference frame image into the reference matrix Query; The second feature embedding module is used to embed the feature information of the target frame image into the first target matrix Key and the second target matrix Value. The similarity measurement module is used to perform a similarity measurement between the reference matrix Query and the first target matrix Key for the image position p in the reference matrix Query, so as to obtain the alignment position of the image position p in the first target matrix Key. A position alignment module is used to perform alignment processing based on the alignment position and the second target matrix Value; The step of obtaining the alignment position of image position p in the first target matrix Key by performing a similarity measurement between the reference matrix Query and the first target matrix Key, for image position p, includes: For each image position p in the reference matrix Query, a local window centered on image position p is taken in the first target matrix Key, and the most similar point is found within the local window through similarity calculation as the alignment position, thus obtaining the alignment position of image position p in the first target matrix Key.

9. The image processing apparatus according to claim 8, characterized in that, Also includes: The input module is used to acquire N frames of raw image data as input images; The selection module is used to select the middle frame of the N-frame input images as the reference frame image; And select each of the other N-1 frames besides the reference frame image as the target frame image.

10. The image processing apparatus according to claim 9, characterized in that, Further includes: A fusion module is used to fuse the reference frame image and the aligned images of the N-1 target frame images; The output module is used to decode the fused image to determine the output image.

11. An electronic device, characterized in that, include: processor; Memory; And one or more computer programs, wherein the computer programs are stored in the memory, and when executed by the processor, the computer programs cause the electronic device to perform the method of any one of claims 1 to 7.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when run on a computer, causes the computer to perform the method described in any one of claims 1 to 7.

Images