Processing method, model, device, electronic equipment and storage medium of original image

Abstract

The embodiment of the present disclosure discloses a processing method, model, device, electronic equipment and storage medium of an original image, wherein the method comprises: determining the first input of the encoder network based on the original image to be processed; determining the first self-attention result based on the first input, the window-based self-attention network; determining the second self-attention result based on the first input, the local multiplicative self-attention network; determining the second input of the decoder network based on the first self-attention result and the second self-attention result; determining the third self-attention result based on the second input, the window-based self-attention network; determining the fourth self-attention result based on the second input, the local multiplicative self-attention network; and determining the target original image after restoration based on the third self-attention result and the fourth self-attention result. The present disclosure can effectively enhance the color information by realizing the self-attention of local features through the local multiplicative self-attention network, thereby improving the restoration effect.

Description

Technical Field

[0001] This disclosure relates to image processing technology, and in particular to a method, model, apparatus, electronic device, and storage medium for processing raw images. Background Technology

[0002] Raw images acquired by image sensors are of low quality before undergoing Image Signal Processing (ISP), requiring restoration to enhance their quality. Common techniques for this restoration include using the Transformer model (also known as a self-attention transform network). However, the Transformer, based on global self-attention, suffers from significant computational overhead when applied to image processing. To address this, window-based Transformer models are used to reduce computational cost. However, these window-based models rely on window-based self-attention, which leads to a dependence on learning from noisy pixels, resulting in poor image restoration quality. Summary of the Invention

[0003] To address the aforementioned technical problems such as poor original image restoration results, this disclosure is proposed. Embodiments of this disclosure provide a method, model, apparatus, electronic device, and storage medium for processing original images.

[0004] According to one aspect of the present disclosure, a method for processing an original image is provided, comprising: acquiring an original image to be processed; determining a first input to an encoder network based on the original image to be processed, wherein the encoder network is an encoder network in a pre-trained locally multiplicative encoder-decoder network model, and the locally multiplicative encoder-decoder network model further includes a decoder network; determining a first self-attention result based on the first input and a window-based self-attention network in the encoder network; determining a second self-attention result based on the first input and the locally multiplicative self-attention network in the encoder network; determining a second input to the decoder network based on the first self-attention result and the second self-attention result; determining a third self-attention result based on the second input and the window-based self-attention network in the decoder network; determining a fourth self-attention result based on the second input and the locally multiplicative self-attention network in the decoder network; and determining a restored target original image corresponding to the original image to be processed based on the third self-attention result and the fourth self-attention result.

[0005] According to another aspect of the present disclosure, an apparatus for processing an original image is provided, comprising: a first acquisition module for acquiring an original image to be processed; a first processing module for determining a first input to an encoder network based on the original image to be processed, wherein the encoder network is an encoder network in a pre-trained locally multiplicative encoder-decoder network model, and the locally multiplicative encoder-decoder network model further includes a decoder network; a second processing module for determining a first self-attention result based on the first input and a window-based self-attention network in the encoder network; a third processing module for determining a second self-attention result based on the first input and a locally multiplicative self-attention network in the encoder network; a fourth processing module for determining a second input to the decoder network based on the first self-attention result and the second self-attention result; a fifth processing module for determining a third self-attention result based on the second input and a window-based self-attention network in the decoder network; a sixth processing module for determining a fourth self-attention result based on the second input and a locally multiplicative self-attention network in the decoder network; and a seventh processing module for determining a restored target original image corresponding to the original image to be processed based on the third self-attention result and the fourth self-attention result.

[0006] According to another aspect of the present disclosure, a processing model for an original image is provided, comprising: a locally multiplicative encoder-decoder network, the locally multiplicative encoder-decoder network including an encoder network and a decoder network; the encoder network including a window-based self-attention network and a locally multiplicative self-attention network, the decoder network including a window-based self-attention network and a locally multiplicative self-attention network, wherein the window-based self-attention network in the encoder network is used to obtain a first self-attention result based on a first input determined from the original image to be processed, and the locally multiplicative self-attention network in the encoder network is used to obtain a second self-attention result based on the first input; the encoder network is further used to determine a second input to the decoder network based on the first self-attention result and the second self-attention result; the window-based self-attention network in the decoder network is used to obtain a third self-attention result based on the second input, the locally multiplicative self-attention network in the decoder network is used to obtain a fourth self-attention result based on the second input, and the decoder network is further used to obtain a decoding result based on the third self-attention result and the fourth self-attention result, the decoding result being used to obtain a restored target original image.

[0007] According to another aspect of the present disclosure, a computer-readable storage medium is provided, the storage medium storing a computer program for performing the original image processing method described in any of the above embodiments of the present disclosure.

[0008] According to another aspect of the present disclosure, an electronic device is provided, the electronic device comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory and execute the instructions to implement the original image processing method described in any of the above embodiments of the present disclosure.

[0009] Based on the original image processing method, model, apparatus, electronic device, and storage medium provided in the above embodiments of this disclosure, during the processing of the original image, attention operations are achieved by combining the locally multiplicative self-attention network in the encoder and decoder of the locally multiplicative encoder-decoder network model with a window-based self-attention network. The locally multiplicative self-attention network can focus on pixel-level information, thus achieving self-attention of local features. Combined with the attention results of the window-based self-attention network, color information can be effectively enhanced, avoiding the learning dependence of the window-based self-attention network on noisy pixels, thereby focusing more on beneficial pixels and effectively improving the restoration effect of the original image.

[0010] The technical solutions of this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0011] The above and other objects, features, and advantages of this disclosure will become more apparent from the more detailed description of the embodiments thereof in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the disclosure and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0012] Figure 1 This is an exemplary application scenario of the original image processing method provided in this disclosure;

[0013] Figure 2 This is a schematic flowchart of a method for processing an original image provided in an exemplary embodiment of this disclosure;

[0014] Figure 3 This is a schematic diagram of an original RGGB format image provided in an exemplary embodiment of this disclosure;

[0015] Figure 4 This is a schematic diagram of a 6x6 format raw image output by an X-Trans CMOS according to an exemplary embodiment of the present disclosure;

[0016] Figure 5 This is a schematic diagram of the overall network architecture of a locally multiplicative encoder-decoder network model provided in an exemplary embodiment of this disclosure;

[0017] Figure 6 This is a schematic flowchart of a method for processing an original image provided in another exemplary embodiment of this disclosure;

[0018] Figure 7 This is a schematic diagram of the structure of an encoder network provided in an exemplary embodiment of this disclosure;

[0019] Figure 8 This is a flowchart illustrating step 205 provided in an exemplary embodiment of this disclosure;

[0020] Figure 9 This is a schematic diagram of the network architecture of a locally multiplicative encoder-decoder network model provided in another exemplary embodiment of this disclosure;

[0021] Figure 10 This is a flowchart illustrating step 2052 provided in an exemplary embodiment of this disclosure;

[0022] Figure 11 This is a schematic flowchart of a method for processing an original image provided in yet another exemplary embodiment of this disclosure;

[0023] Figure 12 This is a schematic diagram of the structure of a locally multiplicative window-based self-attention transformation network provided in an exemplary embodiment of this disclosure;

[0024] Figure 13 This is a schematic diagram of the structure of a decoder network provided in an exemplary embodiment of this disclosure;

[0025] Figure 14 This is a flowchart illustrating step 208 provided in an exemplary embodiment of this disclosure;

[0026] Figure 15 This is a flowchart illustrating step 2021 provided in an exemplary embodiment of this disclosure;

[0027] Figure 16 This is a schematic diagram of the packing principle provided in an exemplary embodiment of this disclosure;

[0028] Figure 17 This is a schematic diagram of the structure of an input projection network provided in an exemplary embodiment of this disclosure;

[0029] Figure 18 This is a schematic diagram of the network architecture of a locally multiplicative encoder-decoder network model provided in yet another exemplary embodiment of this disclosure;

[0030] Figure 19 This is a schematic diagram comparing the computational complexity of W-MSA and L-MSA provided in an exemplary embodiment of this disclosure;

[0031] Figure 20 This is a schematic diagram of the structure of an apparatus for processing raw images provided in an exemplary embodiment of the present disclosure;

[0032] Figure 21 A schematic diagram of the structure of an apparatus for processing raw images provided in another exemplary embodiment of this disclosure;

[0033] Figure 22 This disclosure provides a schematic diagram of the structure of a feature extraction unit 5021 according to an exemplary embodiment;

[0034] Figure 23 This disclosure presents a schematic diagram of the structure of an application embodiment of an electronic device. Detailed Implementation

[0035] Hereinafter, exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. It is obvious that the described embodiments are merely some embodiments of the present disclosure, and not all embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described herein.

[0036] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0037] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of this disclosure are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them.

[0038] It should also be understood that in the embodiments disclosed herein, "a plurality of" may refer to two or more, and "at least one" may refer to one, two or more.

[0039] It should also be understood that any component, data or structure mentioned in the embodiments of this disclosure can generally be understood as one or more unless expressly defined or given to the contrary in the context.

[0040] Furthermore, the term "and / or" in this disclosure is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this disclosure generally indicates that the preceding and following related objects have an "or" relationship.

[0041] It should also be understood that the description of the various embodiments in this disclosure emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.

[0042] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0043] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0044] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0045] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0046] The embodiments disclosed herein can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate together with a wide range of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, and servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems, etc.

[0047] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are executed by remote processing devices linked through communication networks. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.

[0048] This disclosure outlines

[0049] In developing this disclosure, the inventors discovered that the raw images acquired by image sensors are of low quality before undergoing ISP (Image Signal Processing), requiring restoration to enhance their quality. In related technologies, the Transformer (also known as a self-attention transform network) model is typically used for raw image restoration. However, the Transformer is a global self-attention-based transform network model. Applying the Transformer model to the image domain results in a large computational load due to the computation of global self-attention. To address this issue, a window-based Transformer model has been applied to restore raw images to reduce computational load. However, the window-based Transformer model, through window-based self-attention, suffers from a learning dependency on noisy pixels, leading to poor raw image restoration results.

[0050] Exemplary Overview

[0051] Figure 1 This is an exemplary application scenario of the original image processing method provided in this disclosure.

[0052] In autonomous driving scenarios, raw images captured by image sensors (such as cameras) on the vehicle can serve as raw images to be processed. Raw images are the raw data output by the image sensors, such as RAW (raw, unprocessed) data. RAW data is the raw data from CMOS, CCD, and other image sensors that convert captured light source signals into digital signals. Specific examples include 2x2 formats such as RGGB, BGGR, GBRG, and GRBG, and 6x6 format data output by X-Trans CMOS, etc. Taking RGGB format as an example, RGGB represents the output format of RAW data, that is, the output order of different color channels in RAW data. R represents RED, G represents GREEN, and B represents BLUE. The raw image processing method disclosed herein is used to restore the raw image to be processed, obtaining the restored target raw image. Image signal processing is then performed on the target raw image to obtain an image in RGB (a superposition of the red, green, and blue color channels) format. Specifically, using the original image processing method disclosed herein, after obtaining the original image to be processed, the original image can be used as the input to a pre-trained locally multiplicative encoder-decoder network model. The locally multiplicative encoder-decoder network model can include an input projection network, an encoder network, and a decoder network, and may also include other related networks such as an output projection network, which can be configured according to actual needs. Based on the original image to be processed, the first input to the encoder network can be determined through the input projection network. The encoder network includes a window-based self-attention network and a locally multiplicative self-attention network, and may also include downsampling layers, feedforward neural networks, and other related networks, which can be configured according to actual needs. For example, the encoder network can include one or more encoders, each encoder including at least one locally multiplicative window-based self-attention transformation network and a downsampling layer. Each locally multiplicative window-based self-attention transformation network includes a locally multiplicative self-attention network and a window-based self-attention network, and may also include residual connections, layer normalization networks, feedforward neural networks, etc.Taking the first encoder as an example, its first locally multiplicative window-based self-attention transformation network performs window-based attention operations on the first input to obtain a first self-attention result. The locally multiplicative self-attention network performs locally multiplicative self-attention operations on the first input to obtain a second self-attention result. Then, the first and second self-attention results are combined to determine the second input of the decoder network. For example, the first and second self-attention results are multiplied to obtain a first target self-attention result, which is then added to the first input via the residual connection in the locally multiplicative window-based self-attention transformation network. Finally, the output of the locally multiplicative window-based self-attention transformation network is obtained through a layer normalization network, a feedforward neural network, etc. This output serves as the next locally multiplicative window-based self-attention transformation. The network input is processed in a similar manner. The output of the last locally multiplicative window-based self-attention transform network is downsampled and fed into the next encoder, and so on. After the last encoder has finished processing, the output of the encoder network is obtained. Based on the output of the encoder network, the second input of the decoder network can be determined. The decoder network can include at least one decoder. Each decoder can include an upsampling layer and at least one locally multiplicative window-based self-attention transform network. Each locally multiplicative window-based self-attention transform network includes a locally multiplicative self-attention network and a window-based self-attention network. The working principle of the locally multiplicative window-based self-attention transform network in the decoder is similar to that of the encoder, and will not be elaborated here. The difference between the decoder and the encoder is that the decoder needs to generate a high-resolution feature map from the low-resolution feature map through the upsampling layer. The decoder network outputs the decoding result, and the decoding result is passed through the output projection network to obtain the restored original target image. Since both the encoder and decoder include locally multiplicative self-attention networks and window-based self-attention networks, the locally multiplicative self-attention network can focus on pixel-level information, thus achieving self-attention of local features. Combined with the attention results of the window-based self-attention network, it can effectively enhance color information, avoid the learning dependence of the window-based self-attention network on noisy pixels, and thus focus more on beneficial pixels, effectively improving the restoration effect of the original image.

[0053] The original image processing method disclosed herein is not limited to autonomous driving scenarios, but can also be applied to any other scenarios that require image acquisition, such as applications involving mobile phones, monitors, educational equipment, and medical devices. The specific method can be set according to actual needs.

[0054] Exemplary methods

[0055] Figure 2This is a schematic flowchart illustrating a method for processing raw images according to an exemplary embodiment of this disclosure. This embodiment can be applied to electronic devices, specifically such as in-vehicle computing platforms, mobile phones, or servers. Figure 2 As shown, it includes the following steps:

[0056] Step 201: Obtain the original image to be processed.

[0057] The raw image to be processed can be raw data obtained from any image sensor, such as RAW (raw, unprocessed) data. RAW data is the raw data from the light source signals captured by image sensors such as CMOS and CCD, which are converted into digital signals. Specific examples include 2x2 format data such as RGGB, BGGR, GBRG, and GRBG, and 6x6 format data output from X-Trans CMOS, etc. Taking RGGB format as an example... Figure 3 This is a schematic diagram of a raw image in RGGB format provided in an exemplary embodiment of this disclosure. RGGB format represents the output format of RAW data, that is, the output order of different color channels in the RAW data, where R represents RED, G represents GREEN, and B represents BLUE. The raw image in this format consists of a 2*2 array of multiple RGGB arrays (see the dashed box in the figure). Figure 4 This is a schematic diagram of a 6*6 format original image output by X-TransCMOS provided in an exemplary embodiment of the present disclosure. It can be seen that the difference between it and the above-mentioned 2*2 original image is that the color channels are highly disordered and it is composed of multiple 6*6 arrays.

[0058] Step 202: Based on the original image to be processed, determine the first input of the encoder network. The encoder network is the encoder network in the pre-trained locally multiplicative encoder-decoder network model, which also includes a decoder network.

[0059] The first input can be image features of the original image to be processed extracted based on the input projection network. The locally multiplicative encoder-decoder network model is pre-trained and used for the restoration of the original image. The overall network architecture of the locally multiplicative encoder-decoder network model can be set according to actual needs, such as using a U-shaped architecture.

[0060] For example, Figure 5This is a schematic diagram of the overall network architecture of a locally multiplicative encoder-decoder network model provided in an exemplary embodiment of this disclosure. The network architecture includes an input projection network, an encoder network, a bottleneck network, a decoder network, and an output projection network. The encoder network and decoder network are bridged via skip connections. The input projection network can be any implementable network used for feature extraction from the original image; for example, it can be a convolutional neural network, specifically configured according to actual needs. The encoder network can include one or more encoders, each of which can include a window-based self-attention network and a locally multiplicative self-attention network. Each encoder can also include a layer normalization network, a feedforward network, residual connections, etc., specifically configured according to actual needs. The bottleneck network, located between the encoder network and the decoder network, can be composed of multiple locally multiplicative window-based self-attention transformation networks, each including both a locally multiplicative self-attention network and a window-based self-attention network. The decoder network can include one or more decoders, each of which can include a window-based self-attention network and a locally multiplicative self-attention network. Each decoder also includes an upsampling layer and other related networks. The output projection network extracts single-channel data from the decoding results of the decoder network to obtain the restored original image of the target. The output projection network can be a convolutional neural network, and the specific configuration can be determined according to actual needs.

[0061] Step 203: Determine the first self-attention result based on the first input and the window-based self-attention network in the encoder network.

[0062] Among them, the window-based self-attention network is used to implement self-attention operations within each window of the first input (or the layer normalization result of the first input) to reduce the amount of computation.

[0063] For example, the layer normalization result of the first input is divided into a first number of windows, and the computation of attention is restricted to each window.

[0064] Step 204: Determine the second self-attention result based on the first input and the locally multiplicative self-attention network in the encoder network.

[0065] Among them, the locally multiplicative self-attention network is used to focus on pixel-level information, thus enabling self-attention of local features, which can effectively enhance color information.

[0066] For example, the first number of windows, representing the normalized result of the first input layer, are further divided into a second number of sub-windows. Self-attention is performed within each sub-window to achieve self-attention of local features. The size of the sub-window can be determined based on the original image's data format. For example, for an original image in 2x2 format such as RGGB, the sub-window size can be set to 2x2. Each pixel position within the sub-window represents a single color feature. Through the self-attention of the sub-windows, demosaicing is achieved using neighboring pixels, compensating for the lack of other color information in a pixel. This effectively enhances color information, and adjacent pixels of the same color can provide crucial contextual information for noisy or defective pixels, thereby reducing noise, blurring, and other phenomena, further improving image quality.

[0067] It should be noted that steps 203 and 204 are not in any particular order.

[0068] Step 205: Based on the first self-attention result and the second self-attention result, determine the second input to the decoder network.

[0069] The second input can be the encoded result obtained by the encoder network. For a U-shaped architecture model, the second input can also be a feature map obtained by processing a bottleneck network after the encoder network. The specific settings can be configured according to actual needs. The decoder network can include one or more decoders to restore the low-resolution feature map generated by the encoder network to a feature map with the same resolution as the original image, and then obtain the restored target original image through output projection. The fusion method of the first self-attention result and the second self-attention result can be configured according to actual needs, such as fusion by multiplication. Since the encoder network may include multiple encoders, each encoder includes not only a window-based self-attention network and a locally multiplicative self-attention network, but also residual connections, layer normalization networks, feedforward networks, downsampling layers, and other related networks. Therefore, after the first and second self-attention results are fused, they still need to pass through other related networks and other encoders. If the network architecture is a U-shaped architecture, it also needs to pass through a bottleneck network after the encoder network to obtain the second input of the decoder network. The specific process can be determined according to the specific network structure of the locally multiplicative encoder-decoder network model, and will not be elaborated here.

[0070] Step 206: Determine the third self-attention result based on the window-based self-attention network in the second input and decoder network.

[0071] The window-based self-attention network in the decoder network operates on the same or similar principle as the window-based self-attention network in the encoder network, and will not be elaborated further here.

[0072] Step 207: Determine the fourth self-attention result based on the locally multiplicative self-attention network in the second input and decoder network.

[0073] The locally multiplicative self-attention network in the decoder network operates on the same or similar principle as the locally multiplicative self-attention network in the encoder network, and will not be elaborated further here.

[0074] It should be noted that steps 206 and 207 are not in any particular order.

[0075] Step 208: Based on the third self-attention result and the fourth self-attention result, determine the restored target original image corresponding to the original image to be processed.

[0076] Similar to the encoder network, each decoder in the decoder network includes not only a window-based self-attention network and a locally multiplicative self-attention network, but also related networks such as upsampling layers, residual connections, layer normalization networks, and feedforward networks. Therefore, after obtaining the third and fourth self-attention results of a decoder, the third and fourth self-attention results are fused together and then passed through other related networks and subsequent decoders of that decoder to obtain the output result of the decoder network. Based on the output result of the decoder network, the restored target original image corresponding to the original image to be processed is obtained through the output projection network, thereby realizing the restoration of the original image to be processed and improving the quality of the original image to be processed.

[0077] The original image processing method provided in this embodiment achieves attention operations by combining a locally multiplicative self-attention network in the encoder and decoder of a locally multiplicative encoder-decoder network model with a window-based self-attention network. The locally multiplicative self-attention network can focus on pixel-level information, thus achieving self-attention of local features. Combined with the attention results of the window-based self-attention network, it can effectively enhance color information, avoid the learning dependence of the window-based self-attention network on noisy pixels, and thus focus more on beneficial pixels, effectively improving the restoration effect of the original image.

[0078] Figure 6 This is a flowchart illustrating a method for processing an original image provided in another exemplary embodiment of this disclosure.

[0079] In an optional example, the encoder network includes at least one encoder, each encoder including at least one locally multiplicative window-based self-attention transform network and a downsampling layer; each locally multiplicative window-based self-attention transform network includes a locally multiplicative self-attention network and a window-based self-attention network; step 203, based on the first input and the window-based self-attention network in the encoder network, determines the first self-attention result, which may specifically include the following steps:

[0080] Step 2031: Based on the first input and the first locally multiplicative window-based self-attention transformation network of the first encoder in the encoder network, determine the first self-attention result.

[0081] Step 204, based on the first input and the locally multiplicative self-attention network in the encoder network, determines the second self-attention result, including:

[0082] Step 2041: Based on the first input and the first locally multiplicative window-based self-attention transformation network of the first encoder in the encoder network, determine the second self-attention result.

[0083] The encoder network includes one or more encoders connected sequentially. Each encoder includes at least one locally multiplicative window-based self-attention transformation network (Lm-Win Transformer Block) and a downsampling layer. The locally multiplicative window-based self-attention transformation networks are connected sequentially, and each network includes a locally multiplicative self-attention network (L-MSA) and a window-based self-attention network (W-MSA). For example, Figure 7This is a schematic diagram of the encoder network structure provided in an exemplary embodiment of this disclosure. In this example, the encoder network includes m (m is a positive integer) encoders. Each encoder includes two locally multiplicative window-based self-attention transformation networks (Lm-Win Transformer Blocks) and a downsampling layer. Each locally multiplicative window-based self-attention transformation network (Lm-Win Transformer Block) includes a locally multiplicative self-attention network (L-MSA) and a window-based self-attention network (W-MSA), as well as a fusion network of the first and second self-attention results and other related networks. The other related networks can be set according to actual needs, such as including a layer normalization network (LayerNorm) and residual connections. In locally enhanced feed-forward (LeFF) and other related networks, the first input enters the encoder network and is first encoded by the first encoder. Specifically, in the first encoder, it is first processed by a first locally multiplicative window-based self-attention transformation network, and then processed by subsequent locally multiplicative window-based self-attention transformation networks. Finally, it passes through a downsampling layer to obtain the encoding result of the first encoder (encoder 1). This encoding result is used as the input of the second encoder, and so on, until the last encoder (encoder m) completes the encoding, and the final encoding result of the encoder network is obtained.

[0084] This disclosure combines a locally multiplicative self-attention network and a window-based self-attention network during the encoding process to achieve self-attention of local features of the feature map of the original image to be processed while reducing the computational load, thereby enhancing color information and improving image quality.

[0085] In one optional example, Figure 8 This is a flowchart illustrating step 205 provided in an exemplary embodiment of this disclosure. In this example, step 205, determining the second input to the decoder network based on the first self-attention result and the second self-attention result, includes:

[0086] Step 2051: Determine the first target self-attention result based on the first self-attention result and the second self-attention result.

[0087] The first self-attention result is self-attention based on non-overlapping windows, while the second self-attention result is self-attention that is more localized relative to the window, reflecting the weight of each color channel, such as the weights of red, green, blue and green in their respective positions in RGGB, which can be called short-range weights. Multiplying the second self-attention result with the first self-attention result yields the first target self-attention, which can effectively enhance color information.

[0088] Step 2052: Based on the first target self-attention result, determine the first transformation result of the first locally multiplicative window-based self-attention transformation network in the first encoder.

[0089] Since each locally multiplicative window-based self-attention transformation network includes not only window-based self-attention networks and locally multiplicative self-attention networks, but also other related networks such as residual connections, layer normalization networks, feedforward networks, and downsampling layers, after obtaining the first target self-attention result, it is necessary to go through other related networks to obtain the first transformation result of the first locally multiplicative window-based self-attention transformation network.

[0090] Step 2053: Based on the first transformation result, the other locally multiplicative window-based self-attention transformation networks and downsampling layers in the encoder network (excluding the first locally multiplicative window-based self-attention transformation network), and the other encoders (excluding the first encoder), obtain the encoding result of the encoder network.

[0091] Since each encoder includes at least one locally multiplicative window-based self-attention transform network and a downsampling layer, after processing by the first locally multiplicative window-based self-attention transform network, the resulting first transform needs to be processed by other locally multiplicative window-based self-attention transform networks and downsampling layers to obtain the first encoded result of the first encoder. Since the encoder network includes at least one encoder, after obtaining the first encoded result of the first encoder, it needs to be encoded by other encoders to obtain the encoded result of the encoder network. The operating principle of each encoder is the same; please refer to the aforementioned first encoder for details, which will not be elaborated further.

[0092] Step 2054: Determine the second input based on the encoding result.

[0093] In this case, the specific network architecture of the locally multiplicative encoder-decoder network model can be set according to actual needs. For example, for a U-shaped architecture, there is a bottleneck network between the encoder network and the decoder network. In this case, the encoding result of the encoder network needs to be passed through the bottleneck network to obtain the second input of the decoder network. For other network architectures, the second input can be determined based on the encoding result of the encoder network according to the relationship between the encoder network and the decoder network. The specific settings can be set according to actual needs, and will not be elaborated here.

[0094] In an optional example, the attention computation of a window-based self-attention network can also introduce relative position encoding to improve model performance.

[0095] In an optional example, the locally multiplicative encoder-decoder network model further includes at least one locally multiplicative window-based self-attention transform network; step 2054, determining the second input based on the encoding result, includes: processing the encoding result based on at least one locally multiplicative window-based self-attention transform network to determine the second input.

[0096] In this model, at least one locally multiplicative window-based self-attention transformation network forms the bottleneck network of the locally multiplicative encoder-decoder network model. It is located between the encoder network and the decoder network in the U-shaped architecture and is used to capture long-distance dependencies of the low-resolution feature map output by the encoder network with a larger receptive field, thereby obtaining the second input of the decoder network.

[0097] For example, Figure 9 This is a schematic diagram of the network architecture of a locally multiplicative encoder-decoder network model provided in another exemplary embodiment of this disclosure. In this example, the output of the encoder network is connected to the input of the bottleneck network, and the output of the bottleneck network is connected to the decoder network. The bottleneck network includes at least one locally multiplicative window-based self-attention transformation network. Similar to the locally multiplicative window-based self-attention transformation network in the encoder network, each locally multiplicative window-based self-attention transformation network in the bottleneck network includes a locally multiplicative self-attention network and a window-based self-attention network, as well as other related networks. The specific working principles are described above and will not be repeated here.

[0098] This disclosure uses at least one locally multiplicative window-based self-attention transformation network as the bottleneck network of the U-shaped architecture. Benefiting from the hierarchical structure of the U-shaped architecture, when the feature map size is reduced to the window size, the locally multiplicative window-based self-attention transformation network at this stage has a large receptive field and can capture dependencies over longer distances, or even global dependencies.

[0099] In one optional example, Figure 10This is a flowchart illustrating step 2052 provided in an exemplary embodiment of this disclosure. In this example, each locally multiplicative window-based self-attention transformation network further includes a first residual connection, a first-layer normalization network, a locally enhanced feedforward network, and a second residual connection; step 2052, based on the first target self-attention result, determines the first transformation result of the first locally multiplicative window-based self-attention transformation network in the first encoder, including:

[0100] Step 20521: Determine the first connection result based on the first target self-attention result, the first input, and the first residual connection.

[0101] Specifically, the first target self-attention result and the first input are added together through the first residual connection to obtain the first connection result.

[0102] Step 20522: Based on the first connection result and the first layer normalized network, determine the first layer normalization result.

[0103] The first-layer normalization network performs layer normalization on the first connection result to obtain the first-layer normalization result.

[0104] Step 20523: Based on the first layer normalization result and the local enhanced feedforward network, determine the first enhancement result.

[0105] Among them, the Local Enhancement Feedforward Network (LeFF) improves the ability to capture local context information by adding deep convolutions to the feedforward network, thereby further improving the model performance.

[0106] Step 20524: Based on the first enhancement result, the first connection result, and the second residual connection, determine the first transformation result.

[0107] The first enhancement result and the first connection result are added together through the second residual connection to obtain the first transformation result.

[0108] This disclosure effectively enhances the model's ability to capture contextual information by locally enhancing the feedforward network, thereby further improving model performance.

[0109] In one optional example, Figure 11 This is a schematic flowchart of a method for processing an original image provided in another exemplary embodiment of the present disclosure. In this example, each locally multiplicative window-based self-attention transform network further includes a second normalization network; after determining the first input of the encoder network based on the original image to be processed in step 202, the method of the present disclosure further includes:

[0110] Step 203a: Based on the first input and the second-layer normalization network, obtain the second-layer normalization result corresponding to the first input.

[0111] The second-layer normalization network is used to normalize the first input, obtaining the normalized result. The purpose of layer normalization is to avoid gradient explosion and vanishing during backpropagation in the neural network, effectively improving model performance and reducing the error rate.

[0112] Step 2031, based on the first input, the first locally multiplicative window-based self-attention transform network in the first encoder of the encoder network, determines the first self-attention result, including:

[0113] Step 20311: Based on the second-layer normalization result and the first locally multiplicative window-based self-attention transformation network in the first encoder network, determine the first self-attention result.

[0114] The specific working principle of the encoder network is described above. The difference is that this step first performs layer normalization on the first input, and then performs window-based self-attention.

[0115] Step 2041, based on the first input and the first locally multiplicative window-based self-attention transform network in the first encoder of the encoder network, determines the second self-attention result, including:

[0116] Step 20411: Based on the second-layer normalization result and the first locally multiplicative window-based self-attention transformation network in the first encoder network, determine the second self-attention result.

[0117] Specifically, the first input is first normalized, and then locally multiplicative self-attention is performed.

[0118] For example, Figure 12 This is a schematic diagram of the structure of a locally multiplicative window-based self-attention transformation network provided in an exemplary embodiment of this disclosure. Taking the first locally multiplicative window-based self-attention transformation network of the first encoder as an example, the locally multiplicative window-based self-attention transformation network includes: a second-layer normalization network (LayerNorm), L-MSA, W-MSA, and a fusion network. First residual connection First layer normalization network (LayerNorm), locally enhanced feedforward network (LeFF), and second residual connection The first input is normalized by the second layer normalization network (LayerNorm). The normalized result of the second layer is used as the input of the locally multiplicative self-attention network (L-MSA) and the window-based self-attention network (W-MSA), respectively. The first self-attention result is obtained by the window-based self-attention network (W-MSA), and the second self-attention result is obtained by the locally multiplicative self-attention network (L-MSA). This indicates element-wise multiplication, specifically multiplying the corresponding elements of the first self-attention result and the second self-attention result to obtain the first target self-attention result. The first target self-attention result is then connected through the first residual. The first input is added to the first connection result to obtain the first connection result. The first connection result is then normalized through the first layer normalization network to obtain the first layer normalization result. The first layer normalization result is then passed through the Local Enhancement Feedforward Network (LeFF) to output the first enhancement result. The first enhancement result is then passed through the second residual connection. Adding the first connection result to the aforementioned result yields the first transformation result of the locally multiplicative window-based self-attention transformation network. The local enhancement feedforward network includes a first fully connected layer (FC), a token-to-feature map transformation layer (Tokens2Img), a depthwise convolutional layer (Depthwise Cov), a feature map-to-token transformation layer (Img2Tokens), a second fully connected layer, and a third residual connection. The normalized result of the first layer is passed through the fully connected layer to obtain the first result. Tokens2Img transforms the first result into a feature map, which is then subjected to depthwise convolution. The resulting convolution is converted back into tokens by Img2Tokens, passed through the second fully connected layer to obtain the second result. The second result is added to the normalized result of the first layer through the third residual connection to obtain the first enhancement result. The specific principles are not elaborated further.

[0119] For example, the encoding result output by the (l-1)th encoder is represented as X. l-1 The processing procedure of the first locally multiplicative window-based self-attention transform network of the l-th encoder is represented as follows:

[0120]

[0121]

[0122] Among them, X l This represents the output of the first locally multiplicative window-based self-attention transform network of the l-th encoder. W-MSA() represents the self-attention operation of the window-based self-attention network, LN() represents the layer normalization operation, L-MSA() represents the self-attention operation of the locally multiplicative self-attention network, and LeFF() represents the operation of the locally enhanced feedforward network. This is an intermediate value, corresponding to the first connection result of the first encoder mentioned above.

[0123] This disclosure effectively avoids gradient explosion and gradient vanishing during backpropagation by using layer normalization, thereby improving model performance and reducing the error rate.

[0124] In an optional example, step 20311, which determines the first self-attention result based on the second-layer normalization result and the first locally multiplicative window-based self-attention transformation network of the first encoder in the encoder network, includes: determining the self-attention result corresponding to a first number of windows of the second-layer normalization result based on the window-based self-attention network, wherein the first number of windows is obtained by dividing the second-layer normalization result; and taking the self-attention result corresponding to each window as the first self-attention result.

[0125] The first quantity can be set according to actual needs. The window for the first quantity is a non-overlapping window. For example, if the size of the normalized result of the second layer is C×H×W, where C, H, and W are the number of channels, height, and width, respectively, and the window size is M×M, then the first quantity... Perform self-attention operations within each window to obtain the self-attention results for each window. The self-attention results for each window are used as the first self-attention result.

[0126] For example, the result of the second-level normalization is represented as:

[0127]

[0128] Where N represents the number of windows, X 1 ,X 2 ,…,X N This represents the N windows that are divided.

[0129] For the i-th (i = 1, 2, ..., N) window, the attention operation is represented as follows:

[0130]

[0131] Where k represents the k-th attention head. This represents the attention result corresponding to the k-th attention head of the i-th window. These represent the projection matrices of the query vector (Q), key vector (K), and value vector (V) of the k-th head, respectively. `Attention()` represents the attention operation, as follows:

[0132]

[0133] Where Softmax() represents the normalization function, T represents the transpose, B represents the relative position deviation, and d k =C i / K h C i K represents the number of channels in the i-th window. h This indicates the number of heads in the window.

[0134] In an optional example, step 20411, which determines the second self-attention result based on the second-layer normalization result and the locally multiplicative self-attention network of the first locally multiplicative window-based self-attention transformation network in the first encoder of the encoder network, includes: determining the self-attention results corresponding to a second number of sub-windows based on the second-layer normalization result, wherein the second number of sub-windows is obtained by dividing the first number of windows; and taking the self-attention results corresponding to each sub-window as the second self-attention result.

[0135] The second quantity can be set according to actual needs, specifically determined by the size of the sub-windows. The size of the sub-windows can be determined based on the data format of the original image to be processed. For example, if the sub-window size is set to 2×2, then the number of sub-windows divided by each window is [number to be filled in]. Then the second quantity Perform self-attention operations within each sub-window to obtain the self-attention results corresponding to each sub-window, and use the self-attention results corresponding to each sub-window as the second self-attention result.

[0136] For example, the sub-window representation of the second-level normalization result is as follows:

[0137]

[0138] Where L represents the number of child windows, This represents the L sub-windows, where l represents the l-th encoder. From window X i Sub-windows are divided into sections.

[0139] For the j-th (j = 1, 2, ..., L) sub-window, the attention operation is represented as follows:

[0140]

[0141] in, These represent the projection matrices of the query vector, key vector, and value vector of the k-th head, respectively. `Attention()` represents the attention operation; see the preceding content for details.

[0142] In an optional example, step 2052, determining the first target self-attention result based on the first self-attention result and the second self-attention result, includes: multiplying the self-attention result corresponding to each sub-window belonging to the same window in the second self-attention result with the self-attention result corresponding to the window to which each sub-window belongs in the first self-attention result to obtain the fifth self-attention result corresponding to that window; and determining the first target self-attention result based on the fifth self-attention result corresponding to each window.

[0143] For example, the self-attention results corresponding to each sub-window of the k-th head of the i-th (i = 1, 2, ..., N) window are represented as follows:

[0144]

[0145] The self-attention result after multiplication of the k-th head of the i-th (i = 1, 2, ..., N) window. It is expressed as follows:

[0146]

[0147] The result of self-attention after multiplying K heads This forms the fifth self-attention result for the i-th (i = 1, 2, ..., N) window.

[0148] The self-attention result after multiplying the k-th head of N windows. It is expressed as follows:

[0149]

[0150] The self-attention results of all heads are concatenated along the channel dimension to obtain the first target self-attention result. In practical applications, the heads can be multiplied first and then concatenated, or concatenated first and then multiplied, depending on the actual needs.

[0151] This disclosure utilizes a locally multiplicative self-attention network to perform self-attention within a sub-window. This local multiplicative self-attention allows all feature maps to be rearranged according to a CFA (Color Filter Array)-based structure, making them more focused on effective pixel-level information and avoiding learning dependence on noisy pixels. Multiplying with the window's self-attention can effectively enhance color information, enabling de-mosaicing using neighboring pixels, compensating for the lack of other color information in a pixel, and providing key contextual information for noisy or defective pixels through adjacent pixels of the same color. This allows for better handling of pixels with severe noise and improves the restoration effect.

[0152] In an optional example, the decoder network includes at least one decoder, each decoder including an upsampling layer and at least one locally multiplicative window-based self-attention transformation network, each locally multiplicative window-based self-attention transformation network including a locally multiplicative self-attention network and a window-based self-attention network; step 206, based on the second input and the window-based self-attention network in the decoder network, determines the third self-attention result, including:

[0153] Step 2061: Determine the first sampling result based on the upsampling layer of the first decoder in the second input and decoder network.

[0154] The upsampling layer is used to improve the resolution of the feature map. The upsampling layer can use any feasible upsampling method, such as deconvolution (also known as transposed convolution), unpooling, bilinear interpolation, etc. The specific method can be set according to actual needs.

[0155] Step 2062: Based on the first sampling result and the first locally multiplicative window-based self-attention transformation network of the first decoder, determine the third self-attention result.

[0156] Among them, the locally multiplicative window-based self-attention transformation network and its network structure and working principle are similar to those of the encoder, and will not be described in detail here.

[0157] Step 207, based on the locally multiplicative self-attention network in the second input and decoder network, determines the fourth self-attention result, including:

[0158] Step 2071: Based on the first sampling result and the first locally multiplicative window-based self-attention transformation network of the first decoder, determine the fourth self-attention result.

[0159] The structure and working principle of locally multiplicative window-based self-attention transformation networks and their locally multiplicative self-attention networks are described above and will not be repeated here.

[0160] For example, Figure 13This is a schematic diagram of the decoder network structure provided in an exemplary embodiment of this disclosure. In this example, the decoder network includes m decoders, the same number as the encoders. Each decoder includes an upsampling layer and two sequentially connected locally multiplicative window-based self-attention transform networks (Lm-Win Transformer Blocks). Each locally multiplicative window-based self-attention transform network (Lm-WinTransformer Block) includes a locally multiplicative self-attention network (L-MSA) and a window-based self-attention network (W-MSA). It may also include other related networks such as layer normalization networks (LayerNorm), residual connections, and locally enhanced feedforward networks (LeFF). For specific structures, please refer to the foregoing content, which will not be repeated here.

[0161] In one optional example, Figure 14 This is a flowchart illustrating step 208 provided in an exemplary embodiment of this disclosure. Based on the third self-attention result and the fourth self-attention result, the restored target original image corresponding to the original image to be processed is determined, including:

[0162] Step 2081: Based on the third and fourth self-attention results, determine the second transformation result of the first locally multiplicative window-based self-attention transformation network of the first decoder.

[0163] Since the decoder network includes multiple decoders, and each decoder includes at least one locally multiplicative window-based self-attention transformation network, the third self-attention result and the fourth self-attention result are the outputs of two self-attention networks in the first locally multiplicative window-based self-attention transformation network of the first decoder. The first locally multiplicative window-based self-attention transformation network also includes other related networks. Therefore, the third self-attention result and the fourth self-attention result need to be fused and then passed through other related networks to obtain the second transformation result of the first locally multiplicative window-based self-attention transformation network of the first decoder.

[0164] Step 2082: Based on the second transformation result, the other locally multiplicative window-based self-attention transformation networks in the decoder network (excluding the first locally multiplicative window-based self-attention transformation network of the first decoder), and the other decoders (excluding the first decoder), obtain the decoding result of the decoder network.

[0165] After obtaining the second transformation result of the first locally multiplicative window-based self-attention transformation network of the first decoder, it is necessary to pass through other locally multiplicative window-based self-attention transformation networks in the first decoder, as well as other decoders following the first decoder in the decoder network, to obtain the decoding result of the decoder network.

[0166] Step 2083: Based on the decoding results, determine the original target image.

[0167] After the decoder network completes the decoding, the decoding result needs to be projected onto the original image format through the output projection network to obtain the restored target original image.

[0168] This disclosure achieves image feature reconstruction through a decoder network, thereby reconstructing the restored target original image corresponding to the original image to be processed. In each decoder of the decoder network, a locally multiplicative self-attention network (L-MSA) and a window-based self-attention network (W-MSA) are also used to effectively enhance color information and further improve image quality while reducing the amount of computation. For specific effects, please refer to the aforementioned encoder network, which will not be repeated here.

[0169] In an optional example, the locally multiplicative encoder-decoder network model further includes an output projection network and a third residual connection; step 2083, determining the target original image based on the decoding result, includes: projecting the decoding result onto the output projection network to obtain a first output result; and determining the target original image based on the first output result and the original image to be processed through the third residual connection.

[0170] The output projection network can employ any feasible method, such as linear projection, and can be configured according to actual needs. The first output obtained from the projection is added to the original image to be processed through a third residual connection to obtain the target original image. This third residual connection accelerates the learning speed of the entire network during the model training phase.

[0171] In an optional example, the locally multiplicative encoder-decoder network model also includes an input projection network;

[0172] Step 202, based on the original image to be processed, determines the first input to the encoder network, including:

[0173] Step 2021: Based on the input projection network, perform feature extraction on the original image to be processed to obtain the first image features.

[0174] The input projection network can be any feasible method, such as a convolutional neural network, to extract the low-level features of the original image to be processed.

[0175] Step 2022: Use the first image features as the first input.

[0176] In one optional example, Figure 15 This is a flowchart illustrating step 2021 provided in an exemplary embodiment of this disclosure. In this example, step 2021, based on the input projection network, performs feature extraction on the original image to be processed to obtain first image features, including:

[0177] Step 20211: Pack the original image to be processed to obtain the color branch information of the target channel number corresponding to the original image to be processed.

[0178] The packing process involves grouping pixels of the same color from the original image to be processed into clusters and concatenating them along the channel dimension. For example... Figure 16 This is a schematic diagram of the packing principle provided by an exemplary embodiment of this disclosure. In this example, the RGGB format is still used as an example, where R1-R4 represent the red channels. The original 4×4 image is packed into a 2×2 image with 4 channels to obtain the color branch information of 4 channels.

[0179] Step 20212: Based on the color branch information, determine the first convolution result of the color branch.

[0180] The color branch can be convolved using a first convolutional neural network to obtain the first convolution result for the color branch. The first convolutional neural network can be configured according to actual needs, such as a 3×3 convolution kernel, a stride of 1, and padding of 1; the specific configuration is not limited.

[0181] Step 20213: Based on the first convolution result, determine the color branch features.

[0182] Specifically, a second convolutional neural network can be used to convolve the result of the first convolution, followed by an activation function to extract color branch features. The second convolutional neural network can be configured according to actual needs, such as a 1×1 convolution kernel, a stride of 1, and padding of 0; the specific configuration is not limited. The activation function can be any feasible activation function, such as the sigmoid activation function, and can be configured according to actual requirements.

[0183] Step 20214: Based on the original image to be processed, determine the second convolution result of the spatial branch.

[0184] Specifically, a third convolutional neural network can be used to perform convolution operations on the original image to obtain the second convolution result of the spatial branch. The third convolutional neural network can be configured according to actual needs, such as a 3×3 convolution kernel, a stride of 2, and padding of 1, without any specific limitations.

[0185] Step 20215: Based on the second convolution result, determine the spatial branch features.

[0186] Specifically, the spatial branch features can be obtained by performing a convolution operation on the result of the second convolution using a fourth convolutional neural network, followed by an activation function. The fourth convolutional neural network can be configured according to actual needs, such as a 1×1 convolution kernel, a stride of 1, and padding of 0; the specific configuration is not limited. The activation function can be any feasible activation function, such as the sigmoid activation function, and can be configured according to actual needs.

[0187] Steps 20211-20213 and steps 20214-20215 are not in any particular order.

[0188] Step 20216: Determine the first fusion feature based on the spatial branching feature and the first convolution result.

[0189] Among them, the spatial branch feature and the first convolution result can be multiplied to determine the first fusion feature.

[0190] Step 20217: Determine the second fusion feature based on the color branch feature and the second convolution result.

[0191] Among them, the color branch feature and the second convolution result can be multiplied to determine the second fusion feature.

[0192] Step 20218: The first fusion feature and the second fusion feature are spliced ​​together to obtain the first image feature.

[0193] The first fusion feature and the second fusion feature can be spliced ​​together using a concat network to obtain the first image feature.

[0194] For example, Figure 17This is a schematic diagram of the input projection network provided in an exemplary embodiment of this disclosure. This input projection network can be called a Bi-directional Fusion Projection module (BFP). Here, Color route represents the color branch, Spatial route represents the spatial branch, Packing represents the packing operation, and Conv k3,s1,p1 represents a convolution operation with a 3×3 kernel, stride = s, and padding of 1. Conv k1,s1,p0 represents a convolution operation with a 1×1 kernel, stride of 1, and padding of 0; Sigmoid is the activation function; Conv k3,s2,p1 represents a 3×3 kernel, stride of 2, and padding of 1; and Concat represents the concatenation operation.

[0195] For example, the specific operation of inputting the projection network can be represented as follows:

[0196]

[0197]

[0198] F BFP =Concat(F spatial ,F color )

[0199] Where I represents the original image to be processed, F color F represents the first fusion feature. spatial F represents the second fusion feature. BFP Let DS(I) represent the first image feature, DS(I) represent the Convk3,s2,p1 convolution operation of the spatial branch, specifically representing a 2x downsampling, and the result of the operation is the second convolution result. P(I) represents the packing operation on I, and the result is the color branch information. 3×3 (P(I)) represents the convolution operation on the color branch information, and the result of the operation is the first convolution result. Conv 1×1 (Conv 3×3 (P(I))) indicates that the first convolution result is convolved again, and Sigmoid(Conv) is used. 1×1 (Conv 3×3 (P(I)))) represents the activation operation, and the result of the operation is the color branch feature. Indicates cross-multiplication. This indicates that the color branch feature is multiplied by the corresponding element of the second convolution result to obtain the second fused feature, Sigmoid(Conv). 1×1The result of the (DS(I)) operation is a spatial branch feature, and Concat() represents a concatenation operation, specifically concatenating along the channel dimension.

[0200] This disclosure utilizes two different attributes of the original image to be processed through color branching and spatial branching. The spatial branch considers pixels in the same color filter, while the color branch maintains the spatial structure of the original image. Compared with the shortcomings of the prior art, which only reduces the image resolution by packing and breaks the compact spatial order within adjacent pixels, this disclosure can make full use of color information and spatial information through cross-fusion, thereby avoiding the shortcomings of the prior art and further improving the restoration effect of the original image.

[0201] In one optional example, Figure 18 This is a schematic diagram of the network architecture of a locally multiplicative encoder-decoder network model provided in another exemplary embodiment of this disclosure. In this example, information is passed between the encoder network and the decoder network through skip connections. Specifically, the output of the last locally multiplicative window-based self-attention transform network of each encoder is concatenated with the upsampled output of the corresponding decoder and used as the input of the locally multiplicative window-based self-attention transform network of the decoder. By using skip connections to achieve information passing between the encoder and decoder, modulation between the input and output can be realized, which can effectively improve model performance and convergence. The specific working principles of other parts are described above and will not be repeated here.

[0202] In one optional example, Figure 19 This is a schematic diagram comparing the computational complexity of W-MSA and L-MSA according to an exemplary embodiment of this disclosure. The window size of W-MSA is M×M, while the sub-window size of L-MSA is 2×2. The feature maps of the windows are rearranged according to the relative positions of the sub-windows. Short-rang weights represent short-distance weights, and self-attention represents self-attention. It can be seen that the self-attention complexity of the k-th head in the W-MSA window is O((MM)). 2 *d k ) = O(M 4 d k The computational complexity of L-MSA is O((2×2)). 2 ×d k ×(M / 2)×(M / 2))=O(M 2 ×d k ×4), it can be seen that the computational complexity of L-MSA is less than that of W-MSA.

[0203] In one optional example, the training process of the locally multiplicative encoder-decoder network model is as follows: A certain logarithmic number of low-quality and high-quality original training images are acquired as the training set. The low-quality original training images serve as the input to the model network, while the high-quality original training images are compared with the predicted images output by the network. The network loss is calculated using a preset loss function and used to adjust the network parameters until the network loss meets the preset conditions, thus obtaining the trained locally multiplicative encoder-decoder network model. The preset loss function can be set according to actual needs, such as the L1 loss function, the Charbonnier loss function, etc., without specific limitations. The L1 loss function is a loss function based on minimum absolute deviation. It calculates the network loss by taking the absolute value of the difference between the target value (in this disclosure, the pixel value of the high-quality original training image) and the pixel value of the predicted high-quality original image output by the model, and summing the absolute differences of all pixels. The Charbonnier loss function is an improvement on the L1 loss function. Compared to the L1 loss function, the Charbonnier loss function is more stable and can improve the robustness of network training convergence.

[0204] This disclosure utilizes a locally multiplicative self-attention network combined with a window-based self-attention network, a locally enhanced feedforward network, and a bidirectional fusion projection network to achieve efficient restoration of low-quality original images, reducing noise and blurring, and providing high-quality original images for downstream tasks. The method disclosed herein can be used as a preprocessing step in an ISP or as input for tasks such as semantic segmentation and object detection of the original image, depending on actual needs.

[0205] The embodiments or optional examples disclosed above can be implemented individually or in any combination without conflict. The specific implementation can be set according to actual needs, and this disclosure does not limit it.

[0206] Any of the original image processing methods provided in this disclosure can be executed by any suitable device with data processing capabilities, including but not limited to: terminal devices and servers. Alternatively, any of the original image processing methods provided in this disclosure can be executed by a processor, such as by a processor executing any of the original image processing methods mentioned in this disclosure by calling corresponding instructions stored in memory. Further details will not be elaborated below.

[0207] In an exemplary embodiment, this disclosure also provides a processing model for an original image, including a locally multiplicative encoder-decoder network, which includes an encoder network and a decoder network. The encoder network includes a window-based self-attention network and a locally multiplicative self-attention network, and the decoder network includes a window-based self-attention network and a locally multiplicative self-attention network. The window-based self-attention network in the encoder network is used to obtain a first self-attention result based on a first input determined from the original image to be processed. The locally multiplicative self-attention network in the encoder network is used to obtain a second self-attention result based on the first input. The encoder network is also used to determine a second input to the decoder network based on the first and second self-attention results. The window-based self-attention network in the decoder network is used to obtain a third self-attention result based on the second input. The locally multiplicative self-attention network in the decoder network is used to obtain a fourth self-attention result based on the second input. The decoder network is also used to obtain a decoding result based on the third and fourth self-attention results. The decoding result is used to obtain the restored target original image.

[0208] The specific structure of the locally multiplicative encoding / decoding network is described in the aforementioned embodiments and will not be repeated here.

[0209] Exemplary device

[0210] Figure 20 This is a schematic diagram of the structure of an apparatus for processing raw images provided in an exemplary embodiment of this disclosure. The apparatus of this embodiment can be used to implement corresponding method embodiments of this disclosure, such as... Figure 20 The device shown includes: a first acquisition module 501, a first processing module 502, a second processing module 503, a third processing module 504, a fourth processing module 505, a fifth processing module 506, a sixth processing module 507, and a seventh processing module 508.

[0211] The first acquisition module 501 is used to acquire the original image to be processed; the first processing module 502 is used to determine the first input of the encoder network based on the original image to be processed acquired by the first acquisition module 501, wherein the encoder network is the encoder network in a pre-trained locally multiplicative encoder-decoder network model, and the locally multiplicative encoder-decoder network model also includes a decoder network; the second processing module 503 is used to determine the first self-attention result based on the first input determined by the first processing module 502 and the window-based self-attention network in the encoder network; the third processing module 504 is used to determine the first self-attention result based on the first input determined by the first processing module 502 and the locally multiplicative self-attention network in the encoder network. The first processing module 505 determines the second self-attention result; the second processing module 506 determines the second input of the decoder network based on the first and second self-attention results; the third processing module 507 determines the fourth self-attention result based on the second input determined by the fourth processing module 505 and the window-based self-attention network in the decoder network; the fourth processing module 508 determines the restored target original image corresponding to the original image to be processed based on the third and fourth self-attention results.

[0212] Figure 21 This is a schematic diagram of the structure of a raw image processing apparatus provided in another exemplary embodiment of this disclosure.

[0213] In an optional example, the encoder network includes at least one encoder, each encoder including at least one locally multiplicative window-based self-attention transform network and a downsampling layer; each locally multiplicative window-based self-attention transform network includes a locally multiplicative self-attention network and a window-based self-attention network; the second processing module 503 includes: a first processing unit 5031, for determining a first self-attention result based on the first input and the window-based self-attention network of the first locally multiplicative window-based self-attention transform network of the first encoder in the encoder network; the third processing module 504 includes: a second processing unit 5041, for determining a second self-attention result based on the first input and the window-based self-attention network of the first locally multiplicative window-based self-attention transform network of the first encoder in the encoder network.

[0214] In an optional example, the fourth processing module 505 includes: a third processing unit 5051, a fourth processing unit 5052, a fifth processing unit 5053, and a sixth processing unit 5054.

[0215] The third processing unit 5051 is used to determine the first target self-attention result based on the first self-attention result and the second self-attention result; the fourth processing unit 5052 is used to determine the first transformation result of the first locally multiplicative window-based self-attention transformation network in the first encoder based on the first target self-attention result; the fifth processing unit 5053 is used to obtain the encoding result of the encoder network based on the first transformation result, other locally multiplicative window-based self-attention transformation networks and downsampling layers in the encoder network other than the first locally multiplicative window-based self-attention transformation network, and other encoders other than the first encoder; the sixth processing unit 5054 is used to determine the second input based on the encoding result.

[0216] In an optional example, the locally multiplicative encoding / decoding network model further includes at least one locally multiplicative window-based self-attention transformation network; the sixth processing unit 5054 is specifically used to: process the encoding result based on at least one locally multiplicative window-based self-attention transformation network to determine the second input.

[0217] In an optional example, each locally multiplicative window-based self-attention transformation network further includes a first residual connection, a first-layer normalization network, a locally enhanced feedforward network, and a second residual connection.

[0218] The fourth processing unit 5052 is specifically used for: determining a first connection result based on the first target self-attention result, the first input, and the first residual connection; determining a first layer normalization result based on the first connection result and the first layer normalization network; determining a first enhancement result based on the first layer normalization result and the local enhancement feedforward network; and determining a first transformation result based on the first enhancement result, the first connection result, and the second residual connection.

[0219] In an optional example, each locally multiplicative window-based self-attention transformation network further includes a second-layer normalization network; the apparatus of this disclosure further includes: an eighth processing module 509, used to obtain a second-layer normalization result corresponding to the first input based on the first input and the second-layer normalization network; correspondingly, the first processing unit 5031 is specifically used to determine a first self-attention result based on the second-layer normalization result and the window-based self-attention network of the first locally multiplicative window-based self-attention transformation network in the first encoder of the encoder network; the second processing unit 5041 is specifically used to determine a second self-attention result based on the second-layer normalization result and the locally multiplicative self-attention network of the first locally multiplicative window-based self-attention transformation network in the first encoder of the encoder network.

[0220] In an optional example, the first processing unit 5031 is specifically used to: determine the self-attention results corresponding to a first number of windows of the second-layer normalization result based on a window-based self-attention network, wherein the first number of windows is obtained by dividing the second-layer normalization result; and take the self-attention results corresponding to each window as the first self-attention result.

[0221] The second processing unit 5041 is specifically used to: determine the self-attention results corresponding to a second number of sub-windows of the second layer normalization result based on the locally multiplicative self-attention network, wherein the second number of sub-windows is obtained by dividing the first number of windows; and take the self-attention results corresponding to each sub-window as the second self-attention results.

[0222] The third processing unit 5051 is specifically used to: multiply the self-attention results corresponding to each sub-window belonging to the same window in the second self-attention result with the self-attention results corresponding to the windows to which each sub-window belongs in the first self-attention result to obtain the fifth self-attention result corresponding to the window; and determine the first target self-attention result based on the fifth self-attention results corresponding to each window.

[0223] In an optional example, the decoder network includes at least one decoder, each decoder including an upsampling layer and at least one locally multiplicative window-based self-attention transformation network, each locally multiplicative window-based self-attention transformation network including a locally multiplicative self-attention network and a window-based self-attention network; the fifth processing module 506 includes: a first determining unit 5061 and a second determining unit 5062.

[0224] The first determining unit 5061 is used to determine a first sampling result based on the second input and the upsampling layer of the first decoder in the decoder network; the second determining unit 5062 is used to determine a third self-attention result based on the first sampling result and the window-based self-attention network of the first locally multiplicative window-based self-attention transformation network of the first decoder.

[0225] The sixth processing module 507 includes: a third determining unit 5071, used to determine a fourth self-attention result based on the first sampling result and the first locally multiplicative window-based self-attention transformation network of the first decoder.

[0226] In an optional example, the seventh processing module 508 includes: a fourth determining unit 5081, a fifth determining unit 5082, and a sixth determining unit 5083.

[0227] The fourth determining unit 5081 is used to determine the second transformation result of the first locally multiplicative window-based self-attention transformation network of the first decoder based on the third self-attention result and the fourth self-attention result; the fifth determining unit 5082 is used to obtain the decoding result of the decoder network based on the second transformation result, other locally multiplicative window-based self-attention transformation networks in the decoder network other than the first locally multiplicative window-based self-attention transformation network of the first decoder, and other decoders other than the first decoder; the sixth determining unit 5083 is used to determine the target original image based on the decoding result.

[0228] In an optional example, the locally multiplicative codec network model also includes an output projection network and a third residual connection;

[0229] The sixth determining unit 5083 is specifically used for: performing output projection on the decoding result based on the output projection network to obtain the first output result; and determining the target original image based on the first output result and the original image to be processed through the third residual connection.

[0230] In an optional example, the locally multiplicative encoder-decoder network model also includes an input projection network;

[0231] The first processing module 502 includes a feature extraction unit 5021, which is used to extract features from the original image to be processed based on the input projection network, obtain first image features, and use the first image features as the first input.

[0232] In one optional example, Figure 22 This is a schematic diagram of the structure of a feature extraction unit 5021 provided in an exemplary embodiment of the present disclosure. The feature extraction unit 5021 includes: a packing subunit 50211, a first convolution subunit 50212, a first determining subunit 50213, a second convolution subunit 50214, a second determining subunit 50215, a first fusion subunit 50216, a second fusion subunit 50217, and a splicing subunit 50218.

[0233] Packaging subunit 50211 is used to package the original image to be processed to obtain color branch information of the target number of channels corresponding to the original image to be processed; First convolution subunit 50212 is used to determine the first convolution result of the color branch based on the color branch information; First determination subunit 50213 is used to determine the color branch feature based on the first convolution result; Second convolution subunit 50214 is used to determine the second convolution result of the spatial branch based on the original image to be processed; Second determination subunit 50215 is used to determine the spatial branch feature based on the second convolution result; First fusion subunit 50216 is used to determine the first fusion feature based on the spatial branch feature and the first convolution result; Second fusion subunit 50217 is used to determine the second fusion feature based on the color branch feature and the second convolution result; Stitching subunit 50218 is used to stitch the first fusion feature and the second fusion feature to obtain the first image feature.

[0234] Exemplary electronic devices

[0235] This disclosure also provides an electronic device, including: a memory for storing computer programs;

[0236] A processor is configured to execute a computer program stored in the memory, wherein when the computer program is executed, it implements the original image processing method described in any of the above embodiments of the present disclosure.

[0237] Figure 23 This is a schematic diagram of an application embodiment of the electronic device disclosed herein. In this embodiment, the electronic device 10 includes one or more processors 11 and a memory 12.

[0238] The processor 11 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 10 to perform desired functions.

[0239] The memory 12 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 11 may execute the program instructions to implement the methods of the various embodiments of this disclosure described above and / or other desired functions. Various contents such as input signals, signal components, and noise components may also be stored in the computer-readable storage medium.

[0240] In one example, the electronic device 10 may also include an input device 13 and an output device 14, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0241] For example, the input device 13 may be the microphone or microphone array described above, used to capture the input signal of the sound source.

[0242] In addition, the input device 13 may also include, for example, a keyboard, a mouse, etc.

[0243] The output device 14 can output various information to the outside, including determined distance information, direction information, etc. The output device 14 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0244] Of course, for the sake of simplicity, Figure 23 Only some of the components of the electronic device 10 relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 10 may include any other suitable components depending on the specific application.

[0245] Exemplary computer program products and computer-readable storage media

[0246] In addition to the methods and apparatus described above, embodiments of this disclosure may also be computer program products comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps of the methods according to various embodiments of this disclosure as described in the "Exemplary Methods" section above.

[0247] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0248] Furthermore, embodiments of this disclosure may also be computer-readable storage media having computer program instructions stored thereon, which, when executed by a processor, cause the processor to perform the steps in the methods according to various embodiments of this disclosure described in the "Exemplary Methods" section above.

[0249] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0250] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0251] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0252] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0253] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.

[0254] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.

[0255] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0256] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A method for processing an original image, comprising: Obtain the original image to be processed; Based on the original image to be processed, the first input of the encoder network is determined. The encoder network is the encoder network in a pre-trained locally multiplicative encoder-decoder network model, and the locally multiplicative encoder-decoder network model also includes a decoder network. Based on the first input and the window-based self-attention network in the encoder network, a first self-attention result is determined; Based on the first input and the locally multiplicative self-attention network in the encoder network, a second self-attention result is determined. Based on the first self-attention result and the second self-attention result, the second input of the decoder network is determined; Based on the second input and the window-based self-attention network in the decoder network, a third self-attention result is determined; Based on the second input and the locally multiplicative self-attention network in the decoder network, a fourth self-attention result is determined; Based on the third self-attention result and the fourth self-attention result, the restored target original image corresponding to the original image to be processed is determined.

2. The method of claim 1, wherein, The encoder network includes at least one encoder, each encoder including at least one locally multiplicative window-based self-attention transform network and a downsampling layer; each locally multiplicative window-based self-attention transform network includes the locally multiplicative self-attention network and the window-based self-attention network. The determination of the first self-attention result based on the first input and the window-based self-attention network in the encoder network includes: Based on the first input and the first window-based self-attention network of the first locally multiplicative self-attention transformation network in the first encoder network, the first self-attention result is determined. The step of determining the second self-attention result based on the first input and the locally multiplicative self-attention network in the encoder network includes: The second self-attention result is determined based on the first input and the locally multiplicative self-attention network of the first locally multiplicative window-based self-attention transformation network in the first encoder network.

3. The method of claim 2, wherein, Determining the second input of the decoder network based on the first self-attention result and the second self-attention result includes: Based on the first self-attention result and the second self-attention result, a first target self-attention result is determined; Based on the first target self-attention result, determine the first transformation result of the first locally multiplicative window-based self-attention transformation network in the first encoder; Based on the first transformation result, the other locally multiplicative window-based self-attention transformation networks in the encoder network (excluding the first locally multiplicative window-based self-attention transformation network), the downsampling layer, and the other encoders (excluding the first encoder), the encoding result of the encoder network is obtained. Based on the encoding result, the second input is determined.

4. The method of claim 3, wherein, The locally multiplicative encoding / decoding network model further includes at least one locally multiplicative window-based self-attention transform network; determining the second input based on the encoding result includes: The encoding result is processed based on the at least one locally multiplicative window-based self-attention transformation network to determine the second input.

5. The method of claim 3, wherein, Each of the locally multiplicative window-based self-attention transformation networks further includes a first residual connection, a first-layer normalization network, a local enhancement feedforward network, and a second residual connection; The step of determining the first transformation result of the first locally multiplicative window-based self-attention transformation network in the first encoder based on the first target self-attention result includes: Based on the first target self-attention result, the first input, and the first residual connection, the first connection result is determined; Based on the first connection result and the first layer normalized network, the first layer normalization result is determined; Based on the first layer normalization result and the local enhancement feedforward network, the first enhancement result is determined; Based on the first enhancement result, the first connection result, and the second residual connection, the first transformation result is determined.

6. The method of claim 3, wherein, Each of the locally multiplicative window-based self-attention transformation networks also includes a second normalization network; After determining the first input to the encoder network based on the original image to be processed, the process further includes: Based on the first input and the second-layer normalization network, the second-layer normalization result corresponding to the first input is obtained; The window-based self-attention network, based on the first input and the first locally multiplicative window-based self-attention transform network in the first encoder of the encoder network, determines the first self-attention result, including: Based on the second-layer normalization result and the first window-based self-attention network of the first locally multiplicative window-based self-attention transformation network in the encoder network, the first self-attention result is determined. The determination of the second self-attention result by the locally multiplicative self-attention network based on the first input and the first locally multiplicative window-based self-attention transform network in the first encoder network includes: The second self-attention result is determined based on the second-layer normalization result and the first locally multiplicative window-based self-attention transformation network in the encoder network.

7. The method of claim 6, wherein, The determination of the first self-attention result based on the second-layer normalization result and the first locally multiplicative window-based self-attention transformation network in the first encoder of the encoder network includes: Based on the window-based self-attention network, a first number of windows corresponding to the second layer normalization result are determined, wherein the first number of windows are obtained by dividing the second layer normalization result; The self-attention result corresponding to each window is taken as the first self-attention result; The determination of the second self-attention result based on the second-layer normalization result and the first locally multiplicative window-based self-attention transformation network in the encoder network includes: Based on the locally multiplicative self-attention network, the self-attention results corresponding to a second number of sub-windows of the second layer normalization result are determined. The second number of sub-windows are obtained by dividing the first number of windows. The self-attention result corresponding to each of the sub-windows is taken as the second self-attention result; The step of determining the first target self-attention result based on the first self-attention result and the second self-attention result includes: Multiply the self-attention results corresponding to each sub-window belonging to the same window in the second self-attention result by the self-attention results corresponding to the windows to which each sub-window belongs in the first self-attention result to obtain the fifth self-attention result corresponding to that window; Based on the fifth self-attention result corresponding to each window, the first target self-attention result is determined.

8. The method of claim 1, wherein, The decoder network includes at least one decoder, each decoder including an upsampling layer and at least one locally multiplicative window-based self-attention transformation network, each locally multiplicative window-based self-attention transformation network including the locally multiplicative self-attention network and the window-based self-attention network; The determination of the third self-attention result based on the second input and the window-based self-attention network in the decoder network includes: Based on the second input and the upsampling layer of the first decoder in the decoder network, a first sampling result is determined; Based on the first sampling result and the first window-based self-attention network of the first locally multiplicative self-attention transformation network of the first decoder, the third self-attention result is determined. The determination of the fourth self-attention result based on the second input and the locally multiplicative self-attention network in the decoder network includes: Based on the first sampling result and the first locally multiplicative window-based self-attention transformation network of the first decoder, the fourth self-attention result is determined.

9. The method according to claim 8, wherein, The step of determining the restored target original image corresponding to the original image to be processed based on the third self-attention result and the fourth self-attention result includes: Based on the third self-attention result and the fourth self-attention result, the second transformation result of the first locally multiplicative window-based self-attention transformation network of the first decoder is determined; Based on the second transformation result, the other locally multiplicative window-based self-attention transformation networks in the decoder network (excluding the first locally multiplicative window-based self-attention transformation network of the first decoder), and the other decoders (excluding the first decoder), the decoding result of the decoder network is obtained. Based on the decoding results, the original target image is determined.

10. The method of claim 9, wherein, The locally multiplicative encoding / decoding network model also includes an output projection network and a third residual connection; Determining the target original image based on the decoding result includes: Based on the output projection network, the decoding result is projected to obtain a first output result; Based on the first output result and the original image to be processed, the target original image is determined through the third residual connection.

11. The method of any one of claims 1-10, wherein, The locally multiplicative encoding / decoding network model also includes an input projection network; Determining the first input to the encoder network based on the original image to be processed includes: Based on the input projection network, feature extraction is performed on the original image to be processed to obtain the first image features; Use the first image features as the first input.

12. The method of claim 11, wherein, The step of extracting features from the original image to be processed based on the input projection network to obtain first image features includes: The original image to be processed is packaged to obtain the color branch information of the target number of channels corresponding to the original image to be processed. Based on the color branch information, determine the first convolution result of the color branch; Based on the first convolution result, the color branch features are determined; Based on the original image to be processed, determine the second convolution result of the spatial branch; Based on the second convolution result, spatial branch features are determined; Based on the spatial branching features and the first convolution result, the first fusion feature is determined; Based on the color branch features and the second convolution result, a second fusion feature is determined; The first fusion feature and the second fusion feature are spliced ​​together to obtain the first image feature.

13. A processing apparatus for an original image, comprising: The first acquisition module is used to acquire the original image to be processed; The first processing module is used to determine the first input of the encoder network based on the original image to be processed. The encoder network is the encoder network in a pre-trained locally multiplicative encoder-decoder network model, and the locally multiplicative encoder-decoder network model further includes a decoder network. The second processing module is used to determine the first self-attention result based on the first input and the window-based self-attention network in the encoder network; The third processing module is used to determine the second self-attention result based on the first input and the locally multiplicative self-attention network in the encoder network; The fourth processing module is used to determine the second input of the decoder network based on the first self-attention result and the second self-attention result; The fifth processing module is used to determine the third self-attention result based on the second input and the window-based self-attention network in the decoder network; The sixth processing module is used to determine the fourth self-attention result based on the second input and the locally multiplicative self-attention network in the decoder network; The seventh processing module is used to determine the restored target original image corresponding to the original image to be processed based on the third self-attention result and the fourth self-attention result.

14. A computer-readable storage medium storing a computer program for performing the processing method of the original image according to any one of claims 1-12.

15. An electronic device, the electronic device comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the original image processing method according to any one of claims 1-12.

Images