Image processing method and device, electronic equipment and storage medium

Abstract

This disclosure provides an image processing method, apparatus, electronic device, and storage medium. The method includes: determining a first tensor corresponding to an image to be processed; determining, based on the first tensor, a first key tensor for row aggregation, a second key tensor for column aggregation, a first value tensor for row aggregation, and a second value tensor for column aggregation corresponding to the first tensor; determining a query tensor based on the second tensor; performing an attention operation on the first key tensor and the query tensor to obtain a first attention weight; performing an attention operation on the second key tensor and the query tensor to obtain a second attention weight; determining a target output tensor based on the first attention weight, the first value tensor, the second attention weight, and the second value tensor; and determining an image processing result based on the target output tensor. This disclosure achieves dimensionality reduction of the key and value tensors, effectively reducing the computational complexity of attention operations and thus improving image processing efficiency.

Description

Technical Field

[0001] This disclosure relates to computer vision technology, and in particular to an image processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] Recently, Transformer neural network models have been increasingly used for cross-space (or cross-view) feature association learning tasks. This task is mainly achieved through cross-space attention, which refers to the attention operation between different data modalities (e.g., image features in the image coordinate system and BEV features in the bird's-eye view (BEV) coordinate system, image features and text features, etc.) and between different data in the same data modality. The core of implementing cross-space attention is the multi-head attention mechanism (MHA). Since image features are two-dimensional feature maps, each pixel in the feature map is used as a token during the attention operation, resulting in a very large number of input tokens. For example, the number of tokens for a single feature map may reach tens of thousands. In practical applications, there are usually multiple feature maps, leading to excessive computational cost of cross-space attention operations and low image processing efficiency. Summary of the Invention

[0003] To address the aforementioned technical problems such as low image processing efficiency, this disclosure is proposed. Embodiments of this disclosure provide an image processing method, apparatus, electronic device, and storage medium.

[0004] According to one aspect of the present disclosure, an image processing method is provided, comprising: determining a first tensor corresponding to an image to be processed; determining, based on the first tensor, a first key tensor for row aggregation, a second key tensor for column aggregation, a first value tensor for row aggregation, and a second value tensor for column aggregation corresponding to the first tensor; determining a query tensor based on the second tensor, wherein the second tensor is a tensor to be subjected to cross-attention operation with the first tensor; performing an attention operation based on the first key tensor and the query tensor to obtain a first attention weight; performing an attention operation based on the second key tensor and the query tensor to obtain a second attention weight; determining a target output tensor based on the first attention weight, the first value tensor, the second attention weight, and the second value tensor; and determining an image processing result corresponding to the image to be processed based on the target output tensor.

[0005] According to another aspect of the present disclosure, an image processing apparatus is provided, comprising: a first determining module, configured to determine a first tensor corresponding to an image to be processed; a first processing module, configured to determine, based on the first tensor, a first key tensor for row aggregation, a second key tensor for column aggregation, a first value tensor for row aggregation, and a second value tensor for column aggregation corresponding to the first tensor; a second processing module, configured to determine a query tensor based on the second tensor, wherein the second tensor is a tensor to be subjected to cross-attention operation with the first tensor; a third processing module, configured to perform an attention operation based on the first key tensor and the query tensor to obtain a first attention weight; a fourth processing module, configured to perform an attention operation based on the second key tensor and the query tensor to obtain a second attention weight; a fifth processing module, configured to determine a target output tensor based on the first attention weight, the first value tensor, the second attention weight, and the second value tensor; and a sixth processing module, configured to determine an image processing result corresponding to the image to be processed based on the target output tensor.

[0006] According to another aspect of the present disclosure, a computer-readable storage medium is provided, the storage medium storing a computer program for performing the image processing method described in any of the above embodiments of the present disclosure.

[0007] According to another aspect of the present disclosure, an electronic device is provided, the electronic device comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory and execute the instructions to implement the image processing method described in any of the above embodiments of the present disclosure.

[0008] Based on the image processing method, apparatus, electronic device, and storage medium provided in the above embodiments of this disclosure, by determining the first key tensor for row aggregation, the second key tensor for column aggregation, the first value tensor for row aggregation, and the second value tensor for column aggregation corresponding to the first tensor corresponding to the image to be processed, dimensionality reduction of the key tensor and value tensor is achieved for cross-attention operation, which can effectively reduce computational complexity and thus improve image processing efficiency.

[0009] The technical solutions of this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0010] The above and other objects, features, and advantages of this disclosure will become more apparent from the more detailed description of the embodiments thereof in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the disclosure and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0011] Figure 1 This is an exemplary application scenario of the image processing method provided in this disclosure;

[0012] Figure 2 This is a schematic flowchart of an image processing method provided in an exemplary embodiment of this disclosure;

[0013] Figure 3 This is a schematic diagram of a first key tensor, a first value tensor, a second key tensor, and a second value tensor provided in an exemplary embodiment of this disclosure;

[0014] Figure 4 This is a flowchart illustrating step 202 provided in an exemplary embodiment of this disclosure;

[0015] Figure 5 This is a schematic diagram illustrating the aggregation principle of a first tensor provided in an exemplary embodiment of this disclosure;

[0016] Figure 6 This is a flowchart illustrating step 202 provided in another exemplary embodiment of this disclosure;

[0017] Figure 7 This is a schematic diagram illustrating the mapping principle of the first tensor provided in an exemplary embodiment of this disclosure;

[0018] Figure 8 This is a schematic flowchart of an image processing method provided in another exemplary embodiment of this disclosure;

[0019] Figure 9 This is a schematic diagram of the weight value of a single token provided in an exemplary embodiment of this disclosure;

[0020] Figure 10 This is a schematic diagram of a first output tensor and a second output tensor provided in an exemplary embodiment of this disclosure;

[0021] Figure 11 This is a schematic diagram of the structure of a DETR network provided in an exemplary embodiment of this disclosure;

[0022] Figure 12 This is a schematic diagram of the structure of an image processing apparatus provided in an exemplary embodiment of the present disclosure;

[0023] Figure 13This is a schematic diagram of the structure of the first processing module 502 provided in an exemplary embodiment of this disclosure;

[0024] Figure 14 This is a schematic diagram of the structure of the first processing module 502 provided in another exemplary embodiment of this disclosure;

[0025] Figure 15 This is a schematic diagram of the structure of an image processing apparatus provided in another exemplary embodiment of the present disclosure;

[0026] Figure 16 This is a schematic diagram of the structure of an application embodiment of the electronic device disclosed herein. Detailed Implementation

[0027] Hereinafter, exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present disclosure, and not all embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described herein.

[0028] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0029] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of this disclosure are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them.

[0030] It should also be understood that in the embodiments disclosed herein, "a plurality of" may refer to two or more, and "at least one" may refer to one, two or more.

[0031] It should also be understood that any component, data or structure mentioned in the embodiments of this disclosure can generally be understood as one or more unless expressly defined or given to the contrary in the context.

[0032] Furthermore, the term "and / or" in this disclosure is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this disclosure generally indicates that the preceding and following related objects have an "or" relationship.

[0033] It should also be understood that the description of the various embodiments in this disclosure emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.

[0034] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0035] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0036] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0037] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0038] The embodiments disclosed herein can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate together with a wide range of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, and servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems, etc.

[0039] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are executed by remote processing devices linked through communication networks. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.

[0040] This disclosure outlines

[0041] In developing this disclosure, the inventors discovered that recently, Transformer neural network models have been increasingly used for cross-space (or cross-view) feature association learning tasks. This task is primarily achieved through cross-space attention, which refers to attention operations between different data modalities (e.g., image features in the image coordinate system and BEV features in the bird's-eye view (BEV) coordinate system, or image features and text features, etc.) and between different data within the same data modality. The core of cross-space attention is the multi-head attention mechanism (MHA). Since image features are two-dimensional feature maps, each pixel in the feature map is used as a token during attention operations, resulting in a very large number of input tokens. For example, the number of tokens for a single feature map can reach tens of thousands. In practical applications, there are usually multiple feature maps, leading to excessive computational complexity in cross-space attention operations and low image processing efficiency.

[0042] Exemplary Overview

[0043] Figure 1 This is an exemplary application scenario of the image processing method provided in this disclosure.

[0044] In the field of autonomous driving, scenarios involving BEV (Bird's Eye View) semantic segmentation typically require cross-attention between the image features of the image to be processed and the BEV embedding features. (See [link to relevant documentation]). Figure 1The image to be processed (i.e., the input image) is processed by a feature extraction network to obtain its corresponding first image features. The first image features, after being encoded with spatial location, are used as the first tensor of the cross-space attention. The BEV embedding, after initialization, is used as the second tensor of the cross-space attention. The first tensor and the second tensor perform a cross-space attention operation. That is, the first tensor is used to determine the key tensor and value tensor of the cross-space attention operation, and the second tensor is used to determine the query tensor of the cross-space attention operation. Attention weights are obtained based on the query tensor and the key tensor. The attention weights are applied to the value tensor to obtain the target output tensor of the attention operation. The application of attention weights to the value tensor means that the pixel values ​​of each channel of the value tensor are weighted and summed based on the attention weights to obtain the target output tensor. Specifically, for each token in the query tensor (a C-dimensional vector formed by the pixel values ​​of the C channels at each pixel position), multiplying the token by the transpose of the key tensor yields the attention weight corresponding to that token. This attention weight is then used as the weight for the pixel values ​​at each position in each channel of the value tensor. The weighted sum of the pixel values ​​in each channel of the value tensor is then performed to obtain the output result corresponding to that token. The output results of all tokens form the target output tensor. The target output tensor is then processed by a BEV decoder and semantic segmentation prediction to obtain the BEV semantic segmentation result corresponding to the image to be processed. Using the image processing method disclosed herein, when performing cross-attention operations on the first and second tensors, the method can determine the first key tensor for row-oriented aggregation, the second key tensor for column-oriented aggregation, the first value tensor for row-oriented aggregation, and the second value tensor for column-oriented aggregation based on the first tensor. Based on the second tensor, the query tensor is determined. Attention operations are performed on the first key tensor and the query tensor to obtain the first attention weight. Attention operations are also performed on the second key tensor and the query tensor to obtain the second attention weight. Based on the first attention weight, the first value tensor, the second attention weight, and the second value tensor, the target output tensor is determined. Since the determined tensors are key and value tensors aggregated along the axis (row or column), dimensionality reduction of the key and value tensors is achieved for cross-attention operations, which can significantly reduce computational complexity and effectively improve image processing efficiency.

[0045] The image processing method disclosed herein is not limited to the aforementioned autonomous driving scenarios. In practical applications, it can be applied to any scenario requiring cross-attention operations, such as scenarios where a network architecture based on DETR (DEtectionTransformer) and its series (e.g., Deformable DETR (Deformable DEtectionTransformer)) is needed to complete a certain task. DETR is the visual version of Transformer and can be used for object detection and panoramic segmentation. The DETR decoder includes an encoder-decoder attention layer, which is a cross-attention layer. Its key tensor and value tensor are determined based on the feature map extracted by the encoder, and its query tensor is determined based on the object query. Therefore, the image processing method disclosed herein can be used to implement cross-attention to reduce computational complexity. The specific settings can be configured according to actual needs.

[0046] Exemplary methods

[0047] Figure 2 This is a schematic flowchart of an exemplary embodiment of the image processing method provided in this disclosure. This embodiment can be applied to electronic devices, specifically servers or terminals, such as... Figure 2 As shown, it includes the following steps:

[0048] Step 201: Determine the first tensor corresponding to the image to be processed.

[0049] The image to be processed can be any image that needs processing, such as an image requiring object detection, an image requiring semantic segmentation, etc., without specific limitations. The image to be processed may include one or more images. The first tensor corresponding to the image to be processed is a tensor obtained by performing certain processing (such as feature extraction) on the image to be processed, used for cross-attention operations, such as the aforementioned... Figure 1 The result of adding the first image feature extracted and the spatial location coding tensor is used as the first tensor. The specific determination of the first tensor can be set according to actual needs, which will not be elaborated here.

[0050] Step 202: Based on the first tensor, determine the first key tensor for row aggregation, the second key tensor for column aggregation, the first value tensor for row aggregation, and the second value tensor for column aggregation corresponding to the first tensor.

[0051] In row-oriented aggregation, the first key tensor is the key tensor after dimensionality reduction in the row direction. For example, the size of the first tensor is C×H1×W1, where C, H1, and W1 represent the number of channels, height, and width, respectively. The resulting first key tensor is C×H1×1, and the W1 dimensions of each row are aggregated into 1 dimension. The row dimension of the dimensionality-reduced first key tensor can be set according to actual needs. For example, the width W of the dimensionality-reduced first key tensor can be W1 / 2, W1 / 4, 2, 1, etc., without any specific limitation. Similarly, the second key tensor in column-oriented aggregation is the key tensor after dimensionality reduction in the column direction. For example, the size of the resulting second key tensor is C×1×W1, and the H1 dimensions of each column are aggregated into 1 dimension or a multi-dimensional tensor smaller than H1. Likewise, the first value tensor in row-oriented aggregation is the value tensor after dimensionality reduction in the row direction, and the second value tensor in column-oriented aggregation is the value tensor after dimensionality reduction in the column direction. It should be noted that the sizes of the first key tensor, second key tensor, first value tensor, and second value tensor must be the same. Since both the key tensor and the value tensor have been dimensionality reduced, the computational cost of attention operations can be reduced.

[0052] For example, Figure 3 This is a schematic diagram of a first key tensor, a first value tensor, a second key tensor, and a second value tensor provided in an exemplary embodiment of this disclosure. In this example, the first key tensor and the first value tensor are row-wise reduced to tensors with W=1, and the second key tensor and the second value tensor are column-wise reduced to tensors with H=1. In this case, the original two-dimensional attention operation can be transformed into two one-dimensional attention operations, which can greatly reduce the computational complexity of the attention operation and improve image processing efficiency.

[0053] Optionally, for the dimensionality-reduced first key tensor, second key tensor, first value tensor, and second value tensor, one can first perform row-wise and column-wise aggregation on the first tensor, and then perform mapping based on the row-wise aggregation result to obtain the row-wise aggregated first key tensor and first value tensor, and perform mapping based on the column-wise aggregation result to obtain the column-wise aggregated second key tensor and second value tensor; alternatively, one can first map the first tensor to obtain the key tensor and value tensor, and then perform row-wise and column-wise aggregation on the key tensor and value tensor respectively to obtain the first key tensor, first value tensor, second key tensor, and second value tensor. This embodiment does not limit the scope. The first tensor can be mapped to the key tensor and value tensor through a preset mapping rule, which will not be elaborated further.

[0054] Step 203: Based on the second tensor, determine the query tensor, which is the tensor to be cross-attention operated on with the first tensor.

[0055] The second tensor can be determined based on the actual application requirements of cross-attention, such as the BEVembedding mentioned above, and is not specifically limited. It can be mapped to a query tensor using certain mapping rules, which will not be elaborated further.

[0056] Steps 202 and 203 are not in any particular order.

[0057] Step 204: Perform attention operations based on the first key tensor and the query tensor to obtain the first attention weight.

[0058] The attention operation between the first key tensor and the query tensor refers to multiplying the C-dimensional vector at each position in the query tensor as a token with the transpose of the C-dimensional vector at each position in the first key tensor (inner product, i.e., multiplying C values ​​and summing them) to obtain the weight value corresponding to each position in the query tensor. The weight values ​​corresponding to each position in the query tensor form the first attention weights.

[0059] Step 205: Perform attention operations based on the second key tensor and the query tensor to obtain the second attention weights.

[0060] The attention operation principle of this step is the same as or similar to that of step 204, and will not be repeated here.

[0061] Steps 204 and 205 are not in any particular order.

[0062] Step 206: Determine the target output tensor based on the first attention weight, the first value tensor, the second attention weight, and the second value tensor.

[0063] Specifically, a first attention weight is applied to a first-value tensor to obtain a first-output tensor, and a second attention weight is applied to a second-value tensor to obtain a second-output tensor. The target output tensor is obtained by combining the first and second output tensors. The first attention weight includes attention weights corresponding to multiple tokens. Applying the first attention weight to the first-value tensor means weighting and summing the pixel values ​​of each channel of the first-value tensor based on the attention weight corresponding to each token in the first attention weight to obtain the first output tensor.

[0064] Step 207: Determine the image processing result corresponding to the image to be processed based on the target output tensor.

[0065] After obtaining the target output tensor of the cross-attention process, subsequent processing can be performed based on the target output tensor, such as decoding and output projection, to obtain the image processing result corresponding to the image to be processed. The specific subsequent processing can be set according to actual needs, and this embodiment does not limit it.

[0066] The image processing method provided in this embodiment determines the first key tensor for row aggregation, the second key tensor for column aggregation, the first value tensor for row aggregation, and the second value tensor for column aggregation based on the first tensor corresponding to the image to be processed. This achieves dimensionality reduction of the key tensor and value tensor, which is used for cross-attention operations. This can effectively reduce computational complexity and thus improve image processing efficiency.

[0067] Figure 4 This is a flowchart illustrating step 202 provided in an exemplary embodiment of this disclosure.

[0068] In one optional example, step 202 may specifically include the following steps:

[0069] Step 2021a: Based on the first tensor, determine the first aggregation tensor for row-oriented aggregation and the second aggregation tensor for column-oriented aggregation.

[0070] Row-wise aggregation and column-wise aggregation can be implemented using pooling operations. A first aggregated tensor is obtained by performing row-wise aggregation on the first tensor, and a second aggregated tensor is obtained by performing column-wise aggregation on the first tensor. Row-wise aggregation reduces the dimensionality of the width W1 of the first tensor, and column-wise aggregation reduces the dimensionality of the height H1 of the first tensor. The specific dimensionality after reduction can be set according to actual needs. For example, the width of the first tensor can be aggregated to one dimension to obtain the first aggregated tensor, and the height of the first tensor can be aggregated to one dimension to obtain the second aggregated tensor.

[0071] Step 2022a: Determine the first key tensor based on the first aggregation tensor and the first mapping rule.

[0072] The first mapping rule can be set according to actual needs, such as the first mapping matrix W corresponding to the first key tensor. K .

[0073] Step 2023a: Determine the second key tensor based on the second aggregation tensor and the second mapping rule.

[0074] The second mapping rule can be set according to actual needs, such as the second mapping matrix corresponding to the second key tensor.

[0075] In practical applications, the second mapping matrix and the first mapping matrix can be the same mapping matrix W. K .

[0076] Step 2024a: Determine the first value tensor based on the first aggregation tensor and the third mapping rule.

[0077] The third mapping rule can be the third mapping matrix corresponding to the first value tensor.

[0078] Step 2025a: Determine the second value tensor based on the second aggregation tensor and the fourth mapping rule.

[0079] The fourth mapping rule can be the fourth mapping matrix corresponding to the second-valued tensor. In practical applications, the fourth mapping matrix and the third mapping matrix can be the same mapping matrix W. V .

[0080] For example, Figure 5 This is a schematic diagram illustrating the aggregation principle of a first tensor provided in an exemplary embodiment of this disclosure. The first tensor is aggregated row-wise to obtain a first aggregated tensor, and the first aggregated tensor is then processed by W... K The mapping yields the first key tensor, and the first aggregation tensor is obtained through W. V The mapping yields a first-value tensor, which is then aggregated columnwise to obtain a second-aggregated tensor. This second-aggregated tensor is then processed by W. K The mapping yields the second key tensor, and the second aggregation tensor is obtained through W. V The mapping yields a second-valued tensor.

[0081] This disclosure achieves dimensionality reduction of the key tensor and value tensor by aggregating the first tensor in the row direction and column direction to obtain a first aggregated tensor in the row direction and a second aggregated tensor in the column direction. Then, based on the first aggregated tensor after aggregation, it maps to a first key tensor and a first value tensor, and based on the second aggregated tensor, it maps to a second key tensor and a second value tensor, thereby reducing the computational complexity of attention operations.

[0082] In an optional example, step 2021a, which determines the first aggregation tensor for row-oriented aggregation and the second aggregation tensor for column-oriented aggregation based on the first tensor, includes: performing a first preset pooling operation on the first tensor in the row direction to determine the first aggregation tensor; and performing a second preset pooling operation on the first tensor in the column direction to determine the second aggregation tensor.

[0083] The specific pooling methods for the first and second preset pooling operations can be set according to actual needs. For example, the aggregation of the first tensor can be achieved through a pooling layer in a neural network. The pooling operation can be average pooling, max pooling, etc. The purpose of the pooling operation is to reduce the dimensionality of the input features; the specific principle will not be elaborated further.

[0084] This disclosure achieves aggregation of the first tensor through pooling operations, thereby reducing the dimensionality of the key tensor and value tensor and decreasing the computational cost of attention operations.

[0085] Figure 6 This is a flowchart illustrating step 202 provided in another exemplary embodiment of this disclosure.

[0086] In an optional example, step 202, based on the first tensor, determines the first key tensor for row-oriented aggregation, the second key tensor for column-oriented aggregation, the first value tensor for row-oriented aggregation, and the second value tensor for column-oriented aggregation, including:

[0087] Step 2021b: Based on the first tensor, determine the third key tensor and the third value tensor.

[0088] The first tensor can be mapped to the third key tensor using the mapping matrix corresponding to the key tensor. Similarly, the first tensor can be mapped to the third value tensor based on the mapping matrix corresponding to the value tensor. The third key tensor and the third value tensor are tensors of the same size as the first tensor.

[0089] Step 2022b: Perform an aggregation operation on the third key tensor in the row direction to obtain the first key tensor.

[0090] The principle of row aggregation is similar to that of the aggregation of the first tensor mentioned above, and will not be repeated here.

[0091] Step 2023b: Perform an aggregation operation on the third bond tensor in the column direction to obtain the second bond tensor.

[0092] The principle of column-oriented aggregation operation is explained above and will not be repeated here.

[0093] Step 2024b: Perform an aggregation operation on the third-value tensor along the row direction to obtain the first-value tensor.

[0094] The principle of row aggregation operation is explained above and will not be repeated here.

[0095] Step 2025b: Perform an aggregation operation on the third-value tensor along the column to obtain the second-value tensor.

[0096] The principle of column-oriented aggregation operation is explained above and will not be repeated here.

[0097] For example, Figure 7 This is a schematic diagram illustrating the mapping principle of a first tensor provided in an exemplary embodiment of this disclosure. The first tensor is mapped via the mapping matrix W. K The mapping yields a third key tensor. This third key tensor is then aggregated row-wise to obtain the first key tensor. The third key tensor is then aggregated column-wise to obtain the second key tensor. The first tensor is then mapped by matrix W. V The mapping yields a third-valued tensor, which is then aggregated in a row direction to obtain a first-valued tensor. Finally, the third-valued tensor is aggregated in a column direction to obtain a second-valued tensor.

[0098] This disclosure achieves dimensionality reduction of key and value tensors by first mapping a first tensor to a third key tensor and a third value tensor of the same size, and then performing row-wise and column-wise aggregation on the third key tensor and the third value tensor to obtain a first key tensor, a first value tensor, a second key tensor, and a second value tensor. The aforementioned "same size" refers to the third key tensor and the third value tensor having the same width, height, and number of channels, for example, both being C×H1×W1.

[0099] Figure 8 This is a schematic flowchart of an image processing method provided in another exemplary embodiment of this disclosure.

[0100] In an optional example, step 204, which involves performing an attention operation based on the first key tensor and the query tensor to obtain the first attention weights, includes:

[0101] Step 2041: For any C-dimensional vector at the first position in the query tensor, multiply the C-dimensional vector by the transpose of the C-dimensional vector at each second position in the first key tensor to obtain the weight value corresponding to the first position, where C represents the number of channels in the query tensor and the first key tensor.

[0102] For example, the query tensor is represented as Q = Q [1:H2,1:W2,1:C] The first bond tensor is represented as K = K [1:H1,1:W,1:C] W is less than the width W1 of the first tensor. For each token in the query tensor, it is represented as Q. [i,j,1:C] =[q1,q2,…,q C Let i = 1, 2, ..., H2, j = 1, 2, ..., W2, and let K be the C-dimensional vector representing each second position in the first bond tensor. [l,k,1:C] =[k1,k2,…,k C Let l = 1, 2, ..., H1, k = 1, 2, ..., W. Then, the weight value corresponding to the i-th row and j-th column position in the query tensor is represented as follows:

[0103]

[0104] Among them, Z [1:H1,1:W] Let H1×W be the tensor, which contains H1×W weight values. For each second position in the first key tensor, the weight value at that second position is represented as q1k1+q2k2+…+q C k C .

[0105] For example, Figure 9This is a schematic diagram of the weight value of a single Token provided in an exemplary embodiment of this disclosure. For each Token, the weight value corresponding to the Token obtained by the attention operation with the first key tensor (C×H1×W) is of size H1×W. Taking W=1 as an example, the query tensor includes H2×W2 Tokens, so H2×W2 weight values ​​of size H1×W (called the first weight value) will be obtained.

[0106] Step 2042: Determine the first attention weight based on the weight values ​​corresponding to each first position.

[0107] The H2×W2 weight values ​​of size H1×W obtained above are used as the first attention weights.

[0108] This disclosure performs attention operations on the first key tensor and the query tensor based on row-oriented aggregation to obtain the first attention weight, which greatly reduces the amount of computation compared to the existing full attention operations on key tensors.

[0109] In an optional example, step 205, which involves performing an attention operation based on the second key tensor and the query tensor to obtain second attention weights, includes:

[0110] Step 2051: For any C-dimensional vector at the first position in the query tensor, multiply the C-dimensional vector by the transpose of the C-dimensional vector at each third position in the second key tensor to obtain the weight value corresponding to the first position, where C represents the number of channels in the query tensor and the second key tensor.

[0111] Step 2052: Determine the second attention weights based on the weight values ​​corresponding to each first position.

[0112] The specific operating principles of steps 2051-2052 are described in steps 2041-2042 above, and will not be repeated here.

[0113] In an optional example, step 206, which determines the target output tensor based on the first attention weights, the first value tensor, the second attention weights, and the second value tensor, includes:

[0114] Step 2061: Determine the first output tensor based on the first attention weight and the first value tensor.

[0115] Specifically, the first attention weight is applied to the first value tensor to obtain the first output tensor. In other words, the pixel values ​​of each channel in the first value tensor are weighted and summed based on the first attention weight to obtain the first output tensor.

[0116] Step 2062: Determine the second output tensor based on the second attention weight and the second value tensor.

[0117] The specific operation of this step is similar to that of step 2061, and will not be repeated here. The second output tensor obtained is the same size as the first output tensor.

[0118] For example, Figure 10 This is a schematic diagram of a first output tensor and a second output tensor provided in an exemplary embodiment of this disclosure. In this example, a single token is used as an example. Each token's corresponding first weight value is applied to the first value tensor to obtain a C×1×1 first output result. Specifically, for each channel in the first value tensor, the value at each position of the H1×W first weight value is used as the weight of the corresponding pixel value in the first value tensor. Based on this first weight value, the H1×W pixel values ​​of that channel in the first value tensor are weighted and summed to obtain a 1×1 output result for that channel. The 1×1 output results of C channels form a C×1×1 first output result. Querying the tensor for H2×W2 tokens yields H2×W2 C×1×1 first output results, which form a C×H2×W2 first output tensor. Similarly, the second weight value corresponding to each Token is applied to the second value tensor to obtain a second output result of C×1×1. Querying the H2×W2 Tokens corresponding to the tensor will yield H2×W2 second output results of C×1×1. These H2×W2 second output results of C×1×1 form a second output tensor of C×H2×W2.

[0119] Step 2063: Determine the target output tensor based on the first output tensor and the second output tensor.

[0120] The target output tensor can be obtained by fusing the first output tensor and the second output tensor. The specific fusion method can be set according to actual needs.

[0121] This disclosure further reduces computation by applying the attention weights obtained from the aggregated key tensors (first key tensor and second key tensor) to the aggregated value tensors (first value tensor and second value tensor).

[0122] In an optional example, step 2063, based on the first output tensor and the second output tensor, determines the target output tensor by adding the first output tensor to the second output tensor to obtain the target output tensor.

[0123] The addition of the first output tensor and the second output tensor means the addition of corresponding elements of the two tensors.

[0124] In an optional example, step 203, based on the second tensor, determines the query tensor, including:

[0125] Step 2031: Determine the query tensor based on the second tensor and the fifth mapping rule.

[0126] The fifth mapping rule can be set according to actual needs, such as to query the mapping matrix W corresponding to the tensor. Q The specifics will not be elaborated further.

[0127] In an optional example, step 207, based on the target output tensor, determines the image processing result corresponding to the image to be processed, including:

[0128] Step 2071: Add the target output tensor to the second tensor to obtain the third tensor.

[0129] The target output tensor and the second tensor can be added together through residual connection to obtain the third tensor.

[0130] Step 2072: Normalize the third tensor to obtain the fourth tensor.

[0131] The normalization method of the third tensor can be set according to actual needs, such as layer normalization (LN), batch normalization (BN), etc., without any specific limitation.

[0132] Step 2073: Based on the fourth tensor, determine the image processing result corresponding to the image to be processed.

[0133] The fourth tensor can also be decoded and output to determine the image processing result corresponding to the image to be processed. The specific settings can be configured according to actual needs.

[0134] For example, Figure 11This is a schematic diagram of the structure of a DETR network provided in an exemplary embodiment of this disclosure, wherein Image features represent image features, Multi-Head Self-Attention represents a multi-head self-attention network, Add&Norm represents addition and normalization, FFN (Feed Forward Networks) represents a feedforward network, Multi-HeadAttention represents a multi-head attention network, Object queries represent object query features, Class represents classification, Bounding Box represents a prediction box, Encoder represents an encoder, Decoder represents a decoder, N× represents N encoders, and M× represents M decoders. As can be seen, in the DETR network, each decoder includes a multi-head attention network. Its key and value tensors are determined based on image features obtained from the encoder combined with spatial location encoding, while its query tensor is determined based on initializeable object queries and the normalized result of its self-attention. For this DETR network, the image processing method of this disclosure can be used to perform axial (row and column) aggregation on the key and value tensors in the multi-head attention network, and then perform attention operations with the query tensor to effectively reduce the computational complexity of the attention operation and improve image processing efficiency. The target output tensor obtained by multi-head attention needs to be added to and normalized with its previous normalized result to obtain a fourth tensor. This fourth tensor is then processed by a subsequent feedforward network (FFN), Add & Norm, and the FFN of the two prediction heads to obtain the classification and prediction boxes, which serve as the image processing result corresponding to the image to be processed. The specific principle of DETR will not be elaborated here. Similarly, in any other application of neural networks with multi-head attention, the multi-head attention method of this disclosure can be used to reduce computational complexity.

[0135] In an optional example, step 201, determining the first tensor corresponding to the image to be processed, includes:

[0136] Step 2011: Extract features from the image to be processed to obtain the first image features corresponding to the image to be processed.

[0137] The feature extraction method can be set according to actual needs. For example, feature extraction can be based on a feature extraction network, which can be any implementable network, such as a convolutional neural network, an encoder network, etc.

[0138] Step 2012: Determine the first tensor based on the first image features and the spatial location encoding tensor corresponding to the first image features.

[0139] The first image feature can be added to the spatial location encoding tensor to obtain the first tensor.

[0140] In an optional example, the first key tensor and the first value tensor can be obtained by aggregating the row direction into a 1-dimensional (width W=1) structure, and the second key tensor and the second value tensor can be aggregating the column direction into a 1-dimensional (height H=1) structure. This can transform the original two-dimensional attention operation into two one-dimensional attention operations, which greatly reduces the computational complexity.

[0141] For example, the computational complexity O1 of the original two-dimensional attention operation is:

[0142] O1 = H2 × W2 × (H1 × W1)

[0143] The computational complexity of aggregating into rows and columns and calculating separately is:

[0144] O2 = H2 × W2 × (H1 + W1)

[0145] As can be seen, the computational complexity has changed from multiplication to addition. For images, the amount of computation is greatly reduced, usually by one to two orders of magnitude.

[0146] The image processing method disclosed herein significantly reduces the computational cost of MHA (Multi-Head Attention) in networks such as Transformer by aggregating the rows and columns of key vectors and value vectors. This also greatly reduces the memory consumption of the computation process, thereby significantly improving image processing efficiency.

[0147] The embodiments or optional examples disclosed herein can be implemented individually or in any combination without conflict. The specific implementation can be set according to actual needs, and this disclosure does not limit it.

[0148] Any of the image processing methods provided in this disclosure can be executed by any suitable device with data processing capabilities, including but not limited to: terminal devices and servers. Alternatively, any of the image processing methods provided in this disclosure can be executed by a processor, such as by a processor executing any of the image processing methods mentioned in this disclosure by calling corresponding instructions stored in memory. Further details will not be elaborated below.

[0149] Exemplary device

[0150] Figure 12 This is a schematic diagram of the structure of an image processing apparatus provided in an exemplary embodiment of the present disclosure. The apparatus of this embodiment can be used to implement the corresponding method embodiments of the present disclosure, such as… Figure 12The device shown includes: a first determining module 501, a first processing module 502, a second processing module 503, a third processing module 504, a fourth processing module 505, a fifth processing module 506, and a sixth processing module 507.

[0151] A first determining module 501 is used to determine a first tensor corresponding to the image to be processed; a first processing module 502 is used to determine, based on the first tensor determined by the first determining module 501, a first key tensor for row aggregation, a second key tensor for column aggregation, a first value tensor for row aggregation, and a second value tensor for column aggregation corresponding to the first tensor; a second processing module 503 is used to determine a query tensor based on the second tensor, wherein the second tensor is a tensor to be cross-attention operated with the first tensor; a third processing module 504 is used to perform an attention operation based on the first key tensor and the query tensor to obtain a first attention weight; a fourth processing module 505 is used to perform an attention operation based on the second key tensor and the query tensor to obtain a second attention weight; a fifth processing module 506 is used to determine a target output tensor based on the first attention weight, the first value tensor, the second attention weight, and the second value tensor; a sixth processing module 507 is used to determine the image processing result corresponding to the image to be processed based on the target output tensor determined by the fifth processing module 506.

[0152] In one optional example, Figure 13 This is a schematic diagram of the structure of a first processing module 502 provided in an exemplary embodiment of the present disclosure. In this example, the first processing module 502 includes: a first determining unit 5021a, a second determining unit 5022a, a third determining unit 5023a, a fourth determining unit 5024a, and a fifth determining unit 5025a.

[0153] The first determining unit 5021a is used to determine a first aggregation tensor for row-oriented aggregation and a second aggregation tensor for column-oriented aggregation based on the first tensor; the second determining unit 5022a is used to determine the first key tensor based on the first aggregation tensor and a first mapping rule; the third determining unit 5023a is used to determine the second key tensor based on the second aggregation tensor and a second mapping rule; the fourth determining unit 5024a is used to determine the first value tensor based on the first aggregation tensor and a third mapping rule; and the fifth determining unit 5025a is used to determine the second value tensor based on the second aggregation tensor and a fourth mapping rule.

[0154] In an optional example, the first determining unit 5021a is specifically used to: perform a first preset pooling operation on the first tensor in the row direction to determine the first aggregated tensor; and perform a second preset pooling operation on the first tensor in the column direction to determine the second aggregated tensor.

[0155] In one optional example, Figure 14 This is a schematic diagram of the structure of a first processing module 502 provided in another exemplary embodiment of the present disclosure. In this example, the first processing module 502 includes: a first processing unit 5021b, a second processing unit 5022b, a third processing unit 5023b, a fourth processing unit 5024b, and a fifth processing unit 5025b.

[0156] The first processing unit 5021b is used to determine a third key tensor and a third value tensor based on the first tensor; the second processing unit 5022b is used to perform an aggregation operation on the third key tensor in the row direction to obtain the first key tensor; the third processing unit 5023b is used to perform an aggregation operation on the third key tensor in the column direction to obtain the second key tensor; the fourth processing unit 5024b is used to perform an aggregation operation on the third value tensor in the row direction to obtain the first value tensor; and the fifth processing unit 5025b is used to perform an aggregation operation on the third value tensor in the column direction to obtain the second value tensor.

[0157] Figure 15 This is a schematic diagram of the structure of an image processing apparatus provided in another exemplary embodiment of the present disclosure.

[0158] In an optional example, the third processing module 504 includes a sixth processing unit 5041 and a sixth determining unit 5042.

[0159] The sixth processing unit 5041 is used to multiply the C-dimensional vector of any first position in the query tensor with the transpose of the C-dimensional vector of each second position in the first key tensor to obtain the weight value corresponding to the first position, where C represents the number of channels in the query tensor and the first key tensor; the sixth determining unit 5042 is used to determine the first attention weight based on the weight values ​​corresponding to each first position.

[0160] In an optional example, the fourth processing module 505 includes: a seventh processing unit 5051 and a seventh determining unit 5052.

[0161] The seventh processing unit 5051 is used to multiply the C-dimensional vector of any first position in the query tensor with the transpose of the C-dimensional vector of each third position in the second key tensor to obtain the weight value corresponding to the first position, where C represents the number of channels in the query tensor and the second key tensor; the seventh determining unit 5052 is used to determine the second attention weight based on the weight values ​​corresponding to each first position.

[0162] In an optional example, the fifth processing module 506 includes an eighth determining unit 5061, a ninth determining unit 5062, and a tenth determining unit 5063.

[0163] The eighth determining unit 5061 is used to determine a first output tensor based on the first attention weight and the first value tensor; the ninth determining unit 5062 is used to determine a second output tensor based on the second attention weight and the second value tensor; the tenth determining unit 5063 is used to determine the target output tensor based on the first output tensor and the second output tensor.

[0164] In an optional example, the tenth determining unit 5063 is specifically used to: add the first output tensor to the second output tensor to obtain the target output tensor.

[0165] In an optional example, the second processing module 503 includes a mapping unit 5031, used to determine the query tensor based on the second tensor and the fifth mapping rule.

[0166] In an optional example, the sixth processing module 507 includes an eighth processing unit 5071, a normalization unit 5072, and a ninth processing unit 5073.

[0167] The eighth processing unit 5071 is used to add the target output tensor to the second tensor to obtain a third tensor; the normalization unit 5072 is used to normalize the third tensor to obtain a fourth tensor; the ninth processing unit 5073 is used to determine the image processing result corresponding to the image to be processed based on the fourth tensor.

[0168] In an optional example, the first determining module 501 includes a feature extraction unit 5011 and a tenth processing unit 5012.

[0169] The feature extraction unit 5011 is used to extract features from the image to be processed to obtain the first image features corresponding to the image to be processed; the tenth processing unit 5012 is used to determine the first tensor based on the first image features and the spatial location encoding tensor corresponding to the first image features.

[0170] Exemplary electronic devices

[0171] This disclosure also provides an electronic device, including: a memory for storing computer programs;

[0172] A processor is configured to execute a computer program stored in the memory, wherein when the computer program is executed, it implements the image processing method described in any of the above embodiments of the present disclosure.

[0173] Figure 16 This is a schematic diagram of an application embodiment of the electronic device disclosed herein. In this embodiment, the electronic device 10 includes one or more processors 11 and a memory 12.

[0174] The processor 11 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 10 to perform desired functions.

[0175] The memory 12 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 11 may execute the program instructions to implement the methods of the various embodiments of this disclosure described above and / or other desired functions. Various contents such as input signals, signal components, and noise components may also be stored in the computer-readable storage medium.

[0176] In one example, the electronic device 10 may also include an input device 13 and an output device 14, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0177] For example, the input device 13 may be the microphone or microphone array described above, used to capture the input signal of the sound source.

[0178] In addition, the input device 13 may also include, for example, a keyboard, a mouse, etc.

[0179] The output device 14 can output various information to the outside, including determined distance information, direction information, etc. The output device 14 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0180] Of course, for the sake of simplicity, Figure 16 Only some of the components of the electronic device 10 relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 10 may include any other suitable components depending on the specific application.

[0181] Exemplary computer program products and computer-readable storage media

[0182] In addition to the methods and apparatus described above, embodiments of this disclosure may also be computer program products comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps of the methods according to various embodiments of this disclosure as described in the "Exemplary Methods" section above.

[0183] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0184] Furthermore, embodiments of this disclosure may also be computer-readable storage media having computer program instructions stored thereon, which, when executed by a processor, cause the processor to perform the steps in the methods according to various embodiments of this disclosure described in the "Exemplary Methods" section above.

[0185] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0186] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0187] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0188] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0189] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.

[0190] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.

[0191] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0192] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. An image processing method, comprising: Determine the first tensor corresponding to the image to be processed; Based on the first tensor, determine the first key tensor for row-oriented aggregation, the second key tensor for column-oriented aggregation, the first value tensor for row-oriented aggregation, and the second value tensor for column-oriented aggregation corresponding to the first tensor; wherein, the first key tensor is the key tensor after dimensionality reduction in the row direction, and the second key tensor is the key tensor after dimensionality reduction in the column direction; the first value tensor is the value tensor after dimensionality reduction in the row direction, and the second value tensor is the value tensor after dimensionality reduction in the column direction. Based on the second tensor, the query tensor is determined, where the second tensor is the tensor to be subjected to cross-attention operation with the first tensor; A first attention weight is obtained by performing an attention operation based on the first key tensor and the query tensor. A second attention weight is obtained by performing an attention operation based on the second key tensor and the query tensor. Based on the first attention weight, the first value tensor, the second attention weight, and the second value tensor, determine the target output tensor; Based on the target output tensor, determine the image processing result corresponding to the image to be processed; The step of determining the target output tensor based on the first attention weight, the first value tensor, the second attention weight, and the second value tensor includes: Based on the first attention weight and the first value tensor, determine the first output tensor; The second output tensor is determined based on the second attention weight and the second value tensor; The target output tensor is determined based on the first output tensor and the second output tensor.

2. The method according to claim 1, wherein, The step of determining, based on the first tensor, the first key tensor for row-oriented aggregation, the second key tensor for column-oriented aggregation, the first value tensor for row-oriented aggregation, and the second value tensor for column-oriented aggregation corresponding to the first tensor includes: Based on the first tensor, determine the first aggregation tensor for row-oriented aggregation and the second aggregation tensor for column-oriented aggregation; Based on the first aggregation tensor and the first mapping rule, the first key tensor is determined; Based on the second aggregation tensor and the second mapping rule, the second key tensor is determined; Based on the first aggregation tensor and the third mapping rule, the first value tensor is determined; Based on the second aggregation tensor and the fourth mapping rule, the second value tensor is determined.

3. The method according to claim 2, wherein, The step of determining the first aggregation tensor for row-oriented aggregation and the second aggregation tensor for column-oriented aggregation based on the first tensor includes: Perform a first preset pooling operation on the first tensor in the row direction to determine the first aggregate tensor; Perform a second preset pooling operation on the first tensor in the column direction to determine the second aggregated tensor.

4. The method according to claim 1, wherein, The step of determining, based on the first tensor, the first key tensor for row-oriented aggregation, the second key tensor for column-oriented aggregation, the first value tensor for row-oriented aggregation, and the second value tensor for column-oriented aggregation corresponding to the first tensor includes: Based on the first tensor, determine the third key tensor and the third value tensor; The third key tensor is aggregated along the row direction to obtain the first key tensor; The third key tensor is aggregated in the column direction to obtain the second key tensor; The third-value tensor is aggregated along the row direction to obtain the first-value tensor; The third-value tensor is aggregated along the column direction to obtain the second-value tensor.

5. The method according to claim 1, wherein, The step of performing attention operations based on the first key tensor and the query tensor to obtain the first attention weight includes: For any C-dimensional vector at a first position in the query tensor, multiply the C-dimensional vector by the transpose of the C-dimensional vector at each second position in the first key tensor to obtain the weight value corresponding to the first position, where C represents the number of channels in the query tensor and the first key tensor. The first attention weight is determined based on the weight value corresponding to each of the first positions.

6. The method according to claim 1, wherein, The step of performing attention operations based on the second key tensor and the query tensor to obtain the second attention weights includes: For any C-dimensional vector at a first position in the query tensor, multiply the C-dimensional vector by the transpose of the C-dimensional vector at each third position in the second key tensor to obtain the weight value corresponding to the first position, where C represents the number of channels in the query tensor and the second key tensor. The second attention weight is determined based on the weight value corresponding to each of the first positions.

7. The method according to claim 1, wherein, Determining the target output tensor based on the first output tensor and the second output tensor includes: The first output tensor is added to the second output tensor to obtain the target output tensor.

8. The method according to claim 1, wherein, The process of determining the query tensor based on the second tensor includes: The query tensor is determined based on the second tensor and the fifth mapping rule.

9. The method according to claim 1, wherein, Determining the image processing result corresponding to the image to be processed based on the target output tensor includes: Add the target output tensor to the second tensor to obtain the third tensor; Normalize the third tensor to obtain the fourth tensor; Based on the fourth tensor, the image processing result corresponding to the image to be processed is determined.

10. The method according to claim 1, wherein, Determining the first tensor corresponding to the image to be processed includes: Feature extraction is performed on the image to be processed to obtain the first image feature corresponding to the image to be processed; The first tensor is determined based on the first image features and the spatial location encoding tensor corresponding to the first image features.

11. An image processing apparatus, comprising: The first determining module is used to determine the first tensor corresponding to the image to be processed; The first processing module is configured to determine, based on the first tensor, a first key tensor for row-oriented aggregation, a second key tensor for column-oriented aggregation, a first value tensor for row-oriented aggregation, and a second value tensor for column-oriented aggregation corresponding to the first tensor; wherein, the first key tensor is a key tensor after dimensionality reduction in the row direction, the second key tensor is a key tensor after dimensionality reduction in the column direction; the first value tensor is a value tensor after dimensionality reduction in the row direction, and the second value tensor is a value tensor after dimensionality reduction in the column direction. The second processing module is used to determine the query tensor based on the second tensor, where the second tensor is the tensor to be subjected to cross-attention operation with the first tensor; The third processing module is used to perform attention operations based on the first key tensor and the query tensor to obtain the first attention weight; The fourth processing module is used to perform attention operations based on the second key tensor and the query tensor to obtain the second attention weight; The fifth processing module is used to determine the target output tensor based on the first attention weight, the first value tensor, the second attention weight, and the second value tensor. The sixth processing module is used to determine the image processing result corresponding to the image to be processed based on the target output tensor; The fifth processing module includes: The eighth determining unit is used to determine the first output tensor based on the first attention weight and the first value tensor; The ninth determining unit is used to determine the second output tensor based on the second attention weight and the second value tensor; The tenth determining unit is used to determine the target output tensor based on the first output tensor and the second output tensor.

12. A computer-readable storage medium storing a computer program for performing the image processing method according to any one of claims 1-10.

13. An electronic device, the electronic device comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the image processing method according to any one of claims 1-10.

Images