Image processing method and device, electronic equipment and storage medium

Abstract

Embodiments of the present disclosure disclose an image processing method and device, electronic equipment and a storage medium, wherein the method comprises: obtaining a query tensor, a key tensor and a value tensor corresponding to a to-be-processed image; performing attention operations on the query tensor and the key tensor respectively in rows and in columns to obtain row output weights corresponding to each row and column output weights corresponding to each column; obtaining a first output tensor based on the row output weights corresponding to each row, the column output weights corresponding to each column and the value tensor; and determining an image processing result corresponding to the to-be-processed image based on the first output tensor. Embodiments of the present disclosure realize forming tokens of pixels at each position of an image, without patching the image, thereby effectively preserving the detailed information of the image and effectively reducing the calculation amount of the attention operation, and solving the problem of easy loss of image details in the prior art based on the patch method.

Description

Technical Field

[0001] This disclosure relates to computer vision technology, and in particular to an image processing method and apparatus, electronic devices, and storage media. Background Technology

[0002] Recently, the Transformer has been introduced into the field of computer vision, forming a new visual neural network model, namely the Vision Transformer (ViT). Since the Transformer neural network model originated from NLP (Natural Language Processing) tasks, its advantage lies in using multi-head attention (MHA) to model long-distance (or temporal) relationships in parallel. The data object it processes is text, which is one-dimensional data with a limited vocabulary in each sentence and a small number of input tokens. However, in the field of computer vision, the data object it processes becomes an image, which is two-dimensional data. If each pixel is used as a token, the computational burden faced by MHA becomes intractable. To address this problem, existing technologies usually divide the image into patches, with each patch forming a token, thus making MHA computationally feasible. However, dividing the image into patches requires merging pixels from multiple locations to form a single token, which can easily lead to the loss of a lot of detailed information in the image. Summary of the Invention

[0003] To address the aforementioned technical problem of image detail loss due to the patch-based approach, this disclosure is proposed. Embodiments of this disclosure provide an image processing method and apparatus, an electronic device, and a storage medium.

[0004] According to one aspect of the present disclosure, an image processing method is provided, comprising: obtaining a query tensor, a key tensor, and a value tensor corresponding to an image to be processed; performing attention operations by row and by column based on the query tensor and the key tensor respectively to obtain row output weights corresponding to each row and column output weights corresponding to each column respectively; obtaining a first output tensor based on the row output weights corresponding to each row, the column output weights corresponding to each column, and the value tensor; and determining an image processing result corresponding to the image to be processed based on the first output tensor.

[0005] According to another aspect of the present disclosure, an image processing apparatus is provided, comprising: a first acquisition module, configured to acquire a query tensor, a key tensor, and a value tensor corresponding to an image to be processed; a first processing module, configured to perform attention operations on the query tensor and the key tensor, respectively, by row and by column, to obtain row output weights corresponding to each row and column output weights corresponding to each column; a second processing module, configured to obtain a first output tensor based on the row output weights corresponding to each row, the column output weights corresponding to each column, and the value tensor; and a third processing module, configured to determine an image processing result corresponding to the image to be processed based on the first output tensor.

[0006] According to another aspect of the present disclosure, a computer-readable storage medium is provided, the storage medium storing a computer program for performing the image processing method described in any of the above embodiments of the present disclosure.

[0007] According to another aspect of the present disclosure, a chip is provided, the chip being connected to or including a memory, for reading and executing a computer program stored in the memory to implement the image processing method described in any of the above embodiments of the present disclosure.

[0008] According to another aspect of the present disclosure, an electronic device is provided, the electronic device comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory and execute the instructions to implement the image processing method described in any of the above embodiments of the present disclosure.

[0009] Based on the image processing method, apparatus, electronic device, and storage medium provided in the above embodiments of this disclosure, by improving MHA, attention operations are performed on the query tensor and key tensor corresponding to the image to be processed, respectively, by row and by column, to obtain the row output weights corresponding to each row and the column output weights corresponding to each column. Then, based on the row output weights corresponding to each row and the column output weights corresponding to each column, the value tensor is applied to obtain the first output tensor of the MHA module. The image processing result corresponding to the image to be processed is then determined based on the first output tensor. This realizes the formation of a token for each pixel position in the image without patching the image, thereby effectively preserving the image's detailed information. Furthermore, by performing attention operations within rows and columns, the computational load of attention operations is effectively reduced. Thus, while achieving the computational feasibility of MHA, the effectiveness of image detailed information is guaranteed, solving the problem that the existing patch-based method easily leads to the loss of image details.

[0010] The technical solutions of this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0011] The above and other objects, features, and advantages of this disclosure will become more apparent from the more detailed description of the embodiments thereof in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the disclosure and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0012] Figure 1 This is a schematic diagram of the network structure of ViT, which is an exemplary application scenario of the image processing method provided in this disclosure;

[0013] Figure 2 This is a schematic flowchart of an image processing method provided in an exemplary embodiment of this disclosure;

[0014] Figure 3 This is a schematic flowchart of an image processing method provided in another exemplary embodiment of this disclosure;

[0015] Figure 4 This is a schematic diagram illustrating the principle of row attention operation provided in an exemplary embodiment of this disclosure;

[0016] Figure 5 This is a schematic diagram illustrating the principle of column attention operation provided in an exemplary embodiment of this disclosure;

[0017] Figure 6 This is a schematic flowchart of an image processing method provided in yet another exemplary embodiment of this disclosure;

[0018] Figure 7 This is a schematic diagram of a row output tensor provided in an exemplary embodiment of this disclosure;

[0019] Figure 8 This is a schematic diagram illustrating the principle of element-wise addition provided in an exemplary embodiment of this disclosure;

[0020] Figure 9 This is a flowchart illustrating step 201 provided in an exemplary embodiment of this disclosure;

[0021] Figure 10 This is a flowchart illustrating step 204 provided in an exemplary embodiment of this disclosure;

[0022] Figure 11 This is a schematic diagram of the structure of an image processing apparatus provided in an exemplary embodiment of the present disclosure;

[0023] Figure 12 This is a schematic diagram of the structure of the first processing module 502 provided in an exemplary embodiment of this disclosure;

[0024] Figure 13 This is a schematic diagram of the structure of the first processing module 502 provided in another exemplary embodiment of this disclosure;

[0025] Figure 14 This is a schematic diagram of the structure of the third processing module 503 provided in an exemplary embodiment of this disclosure;

[0026] Figure 15 This is a schematic diagram of the structure of the first acquisition module 501 provided in an exemplary embodiment of this disclosure;

[0027] Figure 16 This is a schematic diagram of the structure of the third processing module 504 provided in an exemplary embodiment of this disclosure;

[0028] Figure 17 This is a schematic diagram of the structure of an application embodiment of the electronic device disclosed herein. Detailed Implementation

[0029] Hereinafter, exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present disclosure, and not all embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described herein.

[0030] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0031] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of this disclosure are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them.

[0032] It should also be understood that in the embodiments disclosed herein, "a plurality of" may refer to two or more, and "at least one" may refer to one, two or more.

[0033] It should also be understood that any component, data or structure mentioned in the embodiments of this disclosure can generally be understood as one or more unless expressly defined or given to the contrary in the context.

[0034] Furthermore, the term "and / or" in this disclosure is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this disclosure generally indicates that the preceding and following related objects have an "or" relationship.

[0035] It should also be understood that the description of the various embodiments in this disclosure emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.

[0036] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0037] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0038] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0039] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0040] The embodiments disclosed herein can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate together with a wide range of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, and servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems, etc.

[0041] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are executed by remote processing devices linked through communication networks. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.

[0042] This disclosure outlines

[0043] In the process of realizing this disclosure, the inventors discovered that after introducing Transformer into the field of computer vision, the data object to be processed changes from text to images. Since images are two-dimensional data, if each pixel is used as a token, the computational load faced by MHA will become intractable. To address this problem, existing technologies usually divide the image into patches (i.e., image blocks), with each image block forming a token, thereby achieving computational feasibility for MHA. However, dividing the image into patches requires merging pixels from multiple locations to form a token, which easily leads to the loss of many detailed information of the image.

[0044] Exemplary Overview

[0045] Figure 1This is a schematic diagram of the network structure of ViT, which is an exemplary application scenario of the image processing method provided in this disclosure. This network structure can be applied to image classification scenarios. The input image is processed through an embedding operation to obtain a tensor A. Positional encodings are added to tensor A to obtain the input tensor B for the encoder. This network structure has N encoders, where N is a positive integer and can be set according to actual needs. The processing flow of each encoder is as follows (taking the first encoder as an example): The input tensor B undergoes a LayerNorm operation to obtain tensor C, which is then fed into the MHA (Multilayer Perceptron). The output of MHA is tensor D. Tensor D is added to the tensor B before LayerNorm to obtain tensor E. Then, another LayerNorm operation is performed to obtain tensor F. Finally, it passes through a multilayer perceptron (MLP) to obtain tensor I. Tensor I is added to the input tensor E before the second LayerNorm to obtain tensor J, which serves as the input tensor for the next encoder. This process is repeated, and so on, until a tensor L is output after passing through N encoders. Finally, it undergoes an Average operation... After pooling (AvgPooling) and softmax operations, the final category output is obtained. Using the method disclosed herein, by improving MHA (still taking the MHA of the first encoder as an example), the query tensor Q, key tensor K, and value tensor V corresponding to the image to be processed can be obtained based on tensor C. Based on the query tensor and key tensor, attention operations are performed row-wise and column-wise respectively to obtain the row output weights and column output weights corresponding to each row and each column. Based on the row output weights, column output weights, and the value tensor, the first output tensor is obtained, which is the tensor D output by MHA. This achieves the formation of tokens from pixels at each position in the image without patching the image, thus effectively preserving image detail information. Furthermore, performing attention operations within rows and columns effectively reduces the computational cost of attention operations. Therefore, while achieving the computational feasibility of MHA, the effectiveness of image detail information is guaranteed, solving the problem that existing patch-based methods easily lead to the loss of image details.

[0046] Optionally, the image processing method disclosed herein is not limited to the above-mentioned image classification application scenarios, but is applicable to any image processing scenario that requires MHA.

[0047] Exemplary methods

[0048] Figure 2This is a schematic flowchart of an exemplary embodiment of the image processing method provided in this disclosure. This embodiment can be applied to electronic devices, such as terminal devices, servers, etc., specifically, for example, on an in-vehicle computing platform. Figure 2 As shown, it includes the following steps:

[0049] Step 201: Obtain the query tensor, key tensor, and value tensor corresponding to the image to be processed.

[0050] The image to be processed can be any image that needs to be processed. The query tensor, key tensor, and value tensor corresponding to the image to be processed can be obtained by performing certain feature extraction on the image to be processed and then using certain mapping rules. The specific feature extraction and mapping rules can be set according to actual needs, and this embodiment does not limit them.

[0051] Step 202: Based on the query tensor and the key tensor, perform attention operations by row and by column respectively to obtain the row output weights corresponding to each row and the column output weights corresponding to each column.

[0052] Among them, performing attention operations by row refers to performing attention operations on the same rows of the query tensor and the key tensor within the row. Similarly, performing attention operations by column refers to performing attention operations on the same columns of the query tensor and the key tensor within the column.

[0053] For example, taking the first row of the query tensor and the first row corresponding to the key tensor as an example, we use the pixel at each position in the row as a token to perform attention operations. The principle is similar for other rows and each column.

[0054] Attention is performed on a row-by-row basis. Each row receives a corresponding row output weight, and each column receives a corresponding column output weight. The row output weight and column output weight include the row weight value and column weight value applied to the value tensor, respectively.

[0055] Step 203: Based on the row output weights corresponding to each row, the column output weights corresponding to each column, and the value tensor, obtain the first output tensor.

[0056] Specifically, the row output weights corresponding to each row are applied to the value tensor to obtain the row output tensor; and the column output weights corresponding to each column are applied to the value tensor to obtain the column output tensor. Then, the first output tensor is obtained based on the row output tensor and the column output tensor.

[0057] Step 204: Based on the first output tensor, determine the image processing result corresponding to the image to be processed.

[0058] After obtaining the first output tensor, subsequent processing can be performed based on the first output tensor to determine the final image processing result. The subsequent processing can be set according to actual needs, and this embodiment does not impose any limitations. For example, when steps 201-203 are used in a TiV encoder, the encoder may include other related processing operations besides the steps described above, such as LayerNorm operations, MLP, etc. Furthermore, TiV may include multiple encoders, and there may be pooling operations, Softmax operations, etc., after the encoder. Therefore, the corresponding subsequent processing can include at least one of these other related operations.

[0059] The image processing method provided in this embodiment obtains the corresponding row output weights and column output weights by querying the row and column attention operations of the key and value tensors, respectively. These weights are then applied to the value tensor to obtain the row output tensor and column output tensor. Based on the row and column output tensors, a first output tensor is determined for subsequent image processing to obtain the image processing result. This method forms a token for each pixel in the image without patching the image, thus effectively preserving the image's detailed information. Furthermore, the attention operations performed within rows and columns effectively reduce the computational cost of attention operations. Therefore, while achieving the computational feasibility of MHA, the method ensures the effectiveness of image detail information and solves the problem that existing patch-based methods are prone to image detail loss.

[0060] Figure 3 This is a schematic flowchart of an image processing method provided in another exemplary embodiment of this disclosure.

[0061] In an optional example, the dimensions of both the query tensor and the key tensor are H*W*C; H, W, and C represent the height, width, and number of channels of the tensor, respectively; the corresponding step 202 may include:

[0062] Step 2021: Based on the i-th row of the query tensor and the i-th row of the key tensor, perform an attention operation to obtain the row output weight corresponding to the i-th row; i = 1, 2, ..., H.

[0063] For example, Figure 4 This is a schematic diagram illustrating the principle of row attention operation provided in an exemplary embodiment of this disclosure. The query tensor and key tensor have a height H = 3, a width W = 4, and a number of channels C = 3. Taking the dark gray first row as an example, the row output weight matrix corresponding to the first row obtained by the attention operation is W*W. Within the row, the pixel at each position is used as a token for the attention operation. That is, the pixel value vector of C channels at each position is used as a token.

[0064] Step 2022: Based on the j-th column of the query tensor and the j-th column of the key tensor, perform an attention operation to obtain the column output weight corresponding to the j-th column; j = 1, 2, ..., W.

[0065] For example, Figure 5 This is a schematic diagram illustrating the principle of column attention operation provided in an exemplary embodiment of this disclosure. Taking the dark gray 4th column as an example, the column output weight matrix corresponding to the 4th column obtained by performing the attention operation is H*H. Within the column, each pixel is used as a token for the attention operation.

[0066] It should be noted that steps 2021 and 2022 are not in any particular order.

[0067] Figure 6 This is a schematic flowchart of an image processing method provided in another exemplary embodiment of the present disclosure.

[0068] In an optional example, step 2021 performs an attention operation based on the i-th row of the query tensor and the i-th row of the key tensor to obtain the row output weights corresponding to the i-th row, including:

[0069] Step 20211: Multiply the i-th row of the query tensor by the transpose of the i-th row of the key tensor to obtain the first row of results.

[0070] For example, taking H = W = C = 3 as an example, the i-th row of the query tensor can be represented as a W*C matrix Q. [i，1：W，1：C] :

[0071]

[0072] Among them, Q [i，1：W，1：C] This means querying the W*C matrix corresponding to the i-th row of tensor Q, where 1:W represents the index from 1 to W in the width direction, and 1:C represents the index from 1 to C in the channel direction, i = 1, 2, ..., H. This represents the pixel feature value of the k-th channel at the position of the j-th column in the i-th row.

[0073] The transpose of the i-th row of the key tensor can be represented as a C*W matrix.

[0074]

[0075] Where T represents the transpose, i = 1, 2, ..., H, This represents the pixel feature value of the k-th channel at the position of the j-th column in the i-th row.

[0076] The first row of results is represented as a W*W matrix Z. i :

[0077]

[0078] Where i = 1, 2, ..., H, Q represents [i，1：W，1：C] The sth line and The sum of the products of corresponding elements in the t-th column, i.e.:

[0079]

[0080]

[0081] …

[0082]

[0083] Step 20212: Based on the first row of results and the number of channels of the query tensor, obtain the second row of results.

[0084] Where the number of channels is represented by C, the result in the second row is represented as a W*W matrix L. i :

[0085]

[0086] Step 20213: Perform a softmax operation on the second row result to obtain the row output weight corresponding to the i-th row.

[0087] For example, the row output weights corresponding to the i-th row are represented as a W*W matrix M. i :

[0088] M i =softmax(L i )

[0089] Here, softmax() represents the softmax function, and the specific principle will not be elaborated here.

[0090] Optionally, softmax(L i ) can be L i Normalization is performed row-wise. For example, if the first row has 5 original eigenvalues ​​B1-B5, the nth (n = 1, 2, ..., 5) eigenvalue Cn obtained through normalization is based on these 5 original eigenvalues. For example:

[0091]

[0092] Here, exp() represents the exponential function.

[0093] This example is only used to illustrate the principle of inline attention operations. Its specific representation can be set according to actual needs and is not limited to the matrix representation mentioned above.

[0094] In one optional example, each row can be processed either serially or in parallel. To improve data processing efficiency, parallel processing is used.

[0095] In an optional example, step 2022 performs an attention operation based on the j-th column of the query tensor and the j-th column of the key tensor to obtain the column output weights corresponding to the j-th column, including:

[0096] Step 20221: Multiply the j-th column of the query tensor by the transpose of the j-th column of the key tensor to obtain the result of the first column.

[0097] In this context, the j-th column of the query tensor can be represented as an H*W matrix, the transpose of the j-th column of the key tensor can be represented as a W*H matrix, and the corresponding first column result obtained from the j-th column is represented as an H*H matrix.

[0098] The specific operating principle of this step is similar to that of step 20211 above, and will not be repeated here.

[0099] Step 20222: Based on the results in the first column and the number of channels of the query tensor, obtain the results in the second column.

[0100] The second column of the result is represented as an H*H matrix.

[0101] The specific operating principle of this step is similar to that of step 20212 above, and will not be repeated here.

[0102] Step 20223: Perform a softmax operation on the second column result to obtain the column output weight corresponding to the j-th column.

[0103] The column output weights corresponding to the j-th column are represented as an H*H matrix.

[0104] The specific operating principle of this step is similar to that of step 20213 above, and will not be repeated here.

[0105] In one optional example, the columns can be processed either serially or in parallel. To improve data processing efficiency, parallel processing is used.

[0106] In an optional example, step 203, based on the row output weights corresponding to each row, the column output weights corresponding to each column, and the value tensor, obtains the first output tensor, including:

[0107] Step 2031: Perform a weighted operation on the corresponding rows of the value tensor based on the row output weights of each row to obtain the row output results of each row. Based on the row output results of each row, obtain the row output tensor.

[0108] Here, the row output weights corresponding to the i-th row are represented by a W*W matrix M. i For example, the i-th row of a value tensor is represented as a W*C matrix V. [i，1：W，1：C] The output result corresponding to the i-th row is represented as a W*C matrix.

[0109]

[0110] Among them, M i V [i，1：W，1：C] The rules of matrix multiplication are followed, and the specifics will not be elaborated here.

[0111] The combined row output results are then represented as a tensor O of size H*W*C. R .

[0112] For example, Figure 7 This is a schematic diagram of a row output tensor provided in an exemplary embodiment of this disclosure.

[0113] In one optional example, each row can be processed either serially or in parallel. To improve data processing efficiency, parallel processing is used.

[0114] Step 2032: Perform weighted operations on the corresponding columns of the value tensor based on the column output weights of each column to obtain the column output results of each column, and obtain the column output tensor based on the column output results of each column.

[0115] Here, the column output weights corresponding to the j-th column are represented as an H*H matrix N. j For example, the j-th column of a value tensor is represented as an H*C matrix V. [1：H，j，1：C] The output result corresponding to the j-th column is represented as an H*C matrix.

[0116]

[0117] The obtained column output tensor is represented as a tensor O of size H*W*C. C .

[0118] The specific operating principle of this step is similar to that of step 2031 above, and will not be repeated here.

[0119] In one optional example, each row can be processed either serially or in parallel. To improve data processing efficiency, parallel processing is used.

[0120] Steps 2031 and 2032 are not in any particular order.

[0121] Step 2033: Add the row output tensor and column output tensor element by element to obtain the first output tensor.

[0122] Element-wise addition refers to adding the pixel feature values ​​at the same position in the row output tensor and the column output tensor. That is, the first output tensor O is represented as:

[0123] O = O R +O C

[0124] For example, Figure 8 This is a schematic diagram illustrating the principle of element-wise addition provided in an exemplary embodiment of this disclosure. This example only shows element-wise addition for one channel. For multiple channels, each channel is added in the manner described in this example, and the specific details will not be repeated.

[0125] In one optional example, Figure 9 This is a flowchart illustrating step 201 provided in an exemplary embodiment of this disclosure. In this example, step 201, obtaining the query tensor, key tensor, and value tensor corresponding to the image to be processed, includes:

[0126] Step 2011: Perform feature embedding operation on the image to be processed to obtain the corresponding first tensor.

[0127] Among them, the feature embedding operation refers to the embedding operation, which converts the image to be processed into a tensor in a low-dimensional space. The specific operation can adopt the methods commonly used in the relevant fields, which will not be elaborated here.

[0128] Step 2012: Add the first tensor to the corresponding spatial location encoding tensor to obtain the second tensor.

[0129] Among them, the spatial location encoding tensor is used to embed the position of the token sequence and guide attention alignment. The specific embedding of the spatial location encoding can adopt the methods commonly used in the relevant fields, which will not be elaborated here.

[0130] Step 2013: Normalize the second tensor to obtain the third tensor.

[0131] The normalization method can be set according to actual needs, such as using the layernorm normalization method.

[0132] Step 2014: Determine the query tensor based on the third tensor and the first mapping rule.

[0133] The first mapping rule is used to map the third tensor to the query tensor. The first mapping rule can be set according to actual needs, such as by querying the weight matrix W of the tensor Q. q Map the third tensor T to the query tensor Q, where Wq ∈R C*C (i.e., a C*C matrix), T∈R H*W*C H, W, and C represent the height, width, and number of channels of the tensor, respectively. Specifically, the third tensor T can be viewed as H*W vectors of C dimensions, using W... q Multiplying by each of the C-dimensional vectors in the third tensor T yields the query tensor Q, represented as follows:

[0134] Q = Q [1：H，1：W，1：C]

[0135] Q [i，j，1：C] =W q T [i，j，1：C]

[0136] Among them, Q [i，j，1：C] T represents the C-dimensional vector at the i-th row and j-th column position of Q. [i，j，1：C] That is, it represents the C-dimensional vector at the i-th row and j-th column position of T.

[0137] Step 2015: Determine the key tensor based on the third tensor and the second mapping rule.

[0138] The second mapping rule is similar to the first mapping rule, and can be implemented using the weight matrix W of the key tensor. k The third tensor T is mapped to the key tensor K. For details on the principle, please refer to step 2014 above, which will not be repeated here.

[0139] Step 2016: Determine the value tensor based on the third tensor and the third mapping rule.

[0140] The third mapping rule is similar to the first mapping rule, and can be implemented using the weight matrix W of the value tensor. v The third tensor T is mapped to the value tensor V. For the specific principle, please refer to step 2014 above, which will not be repeated here.

[0141] It should be noted that steps 2014 through 2016 are not in any particular order.

[0142] In one optional example, Figure 10 This is a flowchart illustrating step 204 of an exemplary embodiment of the present disclosure. In this example, step 204, which determines the image processing result corresponding to the image to be processed based on the first output tensor, includes:

[0143] Step 2041: Add the first output tensor to the second tensor to obtain the fourth tensor.

[0144] Step 2042: Normalize the fourth tensor to obtain the fifth tensor.

[0145] The specific normalization method can be set according to actual needs, such as using layernorm normalization.

[0146] Step 2043: Input the fifth tensor into the multilayer perceptron module to obtain the sixth tensor.

[0147] The multilayer perceptron module is the MLP in ViT, and its specific principles will not be elaborated here.

[0148] Step 2044: Add the sixth quantity to the fourth quantity to obtain the seventh quantity.

[0149] Step 2045: Based on the seventh tensor, determine the image processing result corresponding to the image to be processed.

[0150] Steps 2041-2045 above implement the process from MHA output to encoder output in one encoder in ViT. In practical applications, ViT can include multiple encoders, such as N encoders. The seventh tensor is then used as the input of the next encoder to continue encoding, and so on, until the N encoder processes are completed. The output tensor obtained is then used for subsequent processing to obtain the image processing result corresponding to the image to be processed. The subsequent processing can be set according to actual needs. Different subsequent processing can be different for different scenarios, such as image classification scenarios, image segmentation, etc. There is no specific limitation.

[0151] In an optional example, step 2045, which determines the image processing result corresponding to the image to be processed based on the seventh tensor, includes: performing an average pooling operation on the seventh tensor to obtain a pooling result; and performing a softmax operation on the pooling result to obtain the image processing result corresponding to the object to be processed.

[0152] When ViT has multiple encoders, the seventh tensor can be the tensor output by the last encoder.

[0153] In this example, average pooling is performed on the seventh tensor to obtain the pooling result. Then, a softmax operation is performed on the pooling result, yielding the image classification result corresponding to the object to be processed. In practical applications, other processing can be performed on the seventh tensor according to actual needs to achieve the desired purpose.

[0154] Using the image processing method disclosed herein, the computational complexity of MHA (measured by the total number of token interactions) is:

[0155] H×W 2 +W×H 2 =HW(H+W)

[0156] The computational complexity of existing MHA technologies is:

[0157] H×W×(H×W)=H 2 W 2

[0158] As can be seen, the computational complexity of MHA disclosed in this paper is reduced from H×W in the prior art to H+W, and multiplication is changed to addition. The overall computational load will be greatly reduced, for example, by 1-2 orders of magnitude. Therefore, memory consumption will also be greatly reduced, making it possible for ViT image processing to use the pixels at each position as tokens without patching the image. This fully preserves the original dimension of the input two-dimensional data, thereby effectively preserving the detailed information of the image and realizing attention learning for high-resolution input.

[0159] Any of the image processing methods provided in this disclosure can be executed by any suitable device with data processing capabilities, including but not limited to: terminal devices and servers. Alternatively, any of the image processing methods provided in this disclosure can be executed by a processor, such as by a processor executing any of the image processing methods mentioned in this disclosure by calling corresponding instructions stored in memory. Further details will not be elaborated below.

[0160] Exemplary device

[0161] Figure 11 This is a schematic diagram of the structure of an image processing apparatus provided in an exemplary embodiment of the present disclosure. The apparatus of this embodiment can be used to implement the corresponding method embodiments of the present disclosure, such as… Figure 11 The device shown includes: a first acquisition module 501, a first processing module 502, a second processing module 503, and a third processing module 504.

[0162] The first acquisition module 501 is used to acquire the query tensor, key tensor, and value tensor corresponding to the image to be processed.

[0163] The first processing module 502 is used to perform attention operations on rows and columns respectively based on the query tensor and key tensor obtained by the first acquisition module 501, so as to obtain the row output weights corresponding to each row and the column output weights corresponding to each column.

[0164] The second processing module 503 is used to obtain the first output tensor based on the row output weights corresponding to each row, the column output weights corresponding to each column, and the value tensors obtained by the first processing module 502.

[0165] The third processing module 504 is used to determine the image processing result corresponding to the image to be processed based on the first output tensor obtained by the second processing module 503.

[0166] In one optional example, Figure 12This is a schematic diagram of the structure of the first processing module 502 provided in an exemplary embodiment of this disclosure. In this example, the dimensions of both the query tensor and the key tensor are H*W*C; H, W, and C represent the height, width, and number of channels of the tensor, respectively; the first processing module 502 includes: a first processing unit 5021 and a second processing unit 5022. The first processing unit 5021 is used to perform an attention operation based on the i-th row of the query tensor and the i-th row of the key tensor obtained by the first acquisition module 501 to obtain the row output weight corresponding to the i-th row; i = 1, 2, ..., H; the second processing unit 5022 is used to perform an attention operation based on the j-th column of the query tensor and the j-th column of the key tensor obtained by the first acquisition module 501 to obtain the column output weight corresponding to the j-th column; j = 1, 2, ..., W.

[0167] Figure 13 This is a schematic diagram of the structure of the first processing module 502 provided in another exemplary embodiment of this disclosure.

[0168] In an optional example, the first processing unit 5021 may include a first processing subunit 50211, a second processing subunit 50212, and a third processing subunit 50213. The first processing subunit 50211 is used to multiply the i-th row of the query tensor by the transpose of the i-th row of the key tensor to obtain a first row result; the second processing subunit 50212 is used to obtain a second row result based on the first row result obtained by the first processing subunit 50211 and the number of channels of the query tensor; the third processing subunit 50213 is used to perform a softmax operation on the second row result to obtain the row output weight corresponding to the i-th row.

[0169] In an optional example, the second processing unit 5022 may include a fourth processing subunit 50221, a fifth processing subunit 50222, and a sixth processing subunit 50223. The fourth processing subunit 50221 is used to multiply the j-th column of the query tensor by the transpose of the j-th column of the key tensor to obtain a first column result; the fifth processing subunit 50222 is used to obtain a second column result based on the first column result obtained by the fourth processing subunit 50221 and the number of channels in the query tensor; the sixth processing subunit 50223 is used to perform a softmax operation on the second column result obtained by the fifth processing subunit 50222 to obtain the column output weight corresponding to the j-th column.

[0170] In one optional example, Figure 14This is a schematic diagram of the structure of the third processing module 503 provided in an exemplary embodiment of this disclosure. In this example, the third processing module 503 includes: a third processing unit 5031, a fourth processing unit 5032, and a fifth processing unit 5033. The third processing unit 5031 is used to perform a weighted operation on the corresponding rows of the value tensor based on the row output weights corresponding to each row, to obtain the row output results corresponding to each row, and to obtain a row output tensor based on the row output results corresponding to each row. The fourth processing unit 5032 is used to perform a weighted operation on the corresponding columns of the value tensor based on the column output weights corresponding to each column, to obtain the column output results corresponding to each column, and to obtain a column output tensor based on the column output results corresponding to each column. The fifth processing unit 5033 is used to add the row output tensor obtained by the third processing unit 5031 and the column output tensor obtained by the fourth processing unit 5032 element-wise to obtain a first output tensor.

[0171] In one optional example, Figure 15 This is a schematic diagram of the structure of the first acquisition module 501 provided in an exemplary embodiment of the present disclosure. In this example, the first acquisition module 501 includes: a feature embedding unit 5011, an encoding embedding unit 5012, a first normalization unit 5013, a first mapping unit 5014, a second mapping unit 5015, and a third mapping unit 5016. Feature embedding unit 5011 is used to perform feature embedding operation on the image to be processed to obtain the corresponding first tensor; encoding embedding unit 5012 is used to add the first tensor obtained by feature embedding unit 5011 to the corresponding spatial location encoding tensor to obtain a second tensor; first normalization unit 5013 is used to normalize the second tensor obtained by encoding embedding unit 5012 to obtain a third tensor; first mapping unit 5014 is used to determine the query tensor based on the third tensor obtained by first normalization unit 5013 and the first mapping rule; second mapping unit 5015 is used to determine the key tensor based on the third tensor obtained by first normalization unit 5013 and the second mapping rule; third mapping unit 5016 is used to determine the value tensor based on the third tensor obtained by first normalization unit 5013 and the third mapping rule.

[0172] In one optional example, Figure 16This is a schematic diagram of the structure of the third processing module 504 provided in an exemplary embodiment of this disclosure. In this example, the third processing module 504 includes: a sixth processing unit 5041, a second normalization unit 5042, a seventh processing unit 5043, an eighth processing unit 5044, and a ninth processing unit 5045. The sixth processing unit 5041 is used to add the first output tensor to the second tensor to obtain a fourth tensor; the second normalization unit 5042 is used to normalize the fourth tensor obtained by the sixth processing unit 5041 to obtain a fifth tensor; the seventh processing unit 5043 is used to input the fifth tensor obtained by the second normalization unit 5042 into the multilayer perceptron module to obtain a sixth tensor; the eighth processing unit 5044 is used to add the sixth tensor obtained by the seventh processing unit 5043 to the fourth tensor obtained by the sixth processing unit 5041 to obtain a seventh tensor; and the ninth processing unit 5045 is used to determine the image processing result corresponding to the image to be processed based on the seventh tensor obtained by the eighth processing unit 5044.

[0173] In an optional example, the ninth processing unit 5045 is specifically used to: perform an average pooling operation on the seventh tensor to obtain a pooling result; and perform a softmax operation on the pooling result to obtain the image processing result corresponding to the object to be processed.

[0174] Exemplary electronic devices

[0175] This disclosure also provides an electronic device, including: a memory for storing computer programs;

[0176] A processor is configured to execute a computer program stored in the memory, wherein when the computer program is executed, it implements the image processing method described in any of the above embodiments of the present disclosure.

[0177] Figure 17 This is a schematic diagram of an application embodiment of the electronic device disclosed herein. In this embodiment, the electronic device 10 includes one or more processors 11 and a memory 12.

[0178] The processor 11 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 10 to perform desired functions.

[0179] The memory 12 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 11 may execute the program instructions to implement the image processing methods of the various embodiments of this disclosure described above and / or other desired functions. Various contents such as input signals, signal components, and noise components may also be stored in the computer-readable storage medium.

[0180] In one example, the electronic device 10 may also include an input device 13 and an output device 14, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0181] For example, the input device 13 may be the microphone or microphone array described above, used to capture the input signal of the sound source.

[0182] In addition, the input device 13 may also include, for example, a keyboard, a mouse, etc.

[0183] The output device 14 can output various information to the outside, including determined distance information, direction information, etc. The output device 14 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0184] Of course, for the sake of simplicity, Figure 17 Only some of the components of the electronic device 10 relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 10 may include any other suitable components depending on the specific application.

[0185] Exemplary computer program products and computer-readable storage media

[0186] In addition to the methods and apparatus described above, embodiments of this disclosure may also be computer program products comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps in the image processing methods according to various embodiments of this disclosure as described in the "Exemplary Methods" section of this specification.

[0187] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0188] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions that, when executed by a processor, cause the processor to perform the steps in the image processing methods according to various embodiments of this disclosure as described in the "Exemplary Methods" section above.

[0189] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0190] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0191] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0192] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0193] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.

[0194] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.

[0195] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0196] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. An image processing method, comprising: Obtain the query tensor, key tensor, and value tensor corresponding to the image to be processed; Based on the query tensor and the key tensor, attention operations are performed by row and by column respectively to obtain the row output weights corresponding to each row and the column output weights corresponding to each column. Based on the row output weights corresponding to each row, the column output weights corresponding to each column, and the value tensor, a first output tensor is obtained; Based on the first output tensor, determine the image processing result corresponding to the image to be processed; Wherein, the dimensions of the query tensor and the key tensor are both H*W*C; H, W, and C represent the height, width, and number of channels of the tensor, respectively; The step of performing attention operations on rows and columns based on the query tensor and the key tensor respectively to obtain the row output weights corresponding to each row and the column output weights corresponding to each column includes: Based on the i-th row of the query tensor and the i-th row of the key tensor, an attention operation is performed to obtain the row output weight corresponding to the i-th row; i = 1, 2, ..., H; Based on the h-th column of the query tensor and the j-th column of the key tensor, an attention operation is performed to obtain the column output weight corresponding to the j-th column; j = 1, 2, ..., W.

2. The method according to claim 1, wherein, The attention operation, based on the i-th row of the query tensor and the i-th row of the key tensor, is performed to obtain the row output weight corresponding to the i-th row, including: Multiply the i-th row of the query tensor by the transpose of the i-th row of the key tensor to obtain the first row result; Based on the first row of results and the number of channels of the queried tensor, the second row of results is obtained; Perform a softmax operation on the second row result to obtain the row output weight corresponding to the i-th row; The attention operation, based on the j-th column of the query tensor and the j-th column of the key tensor, is performed to obtain the column output weight corresponding to the j-th column, including: Multiply the j-th column of the query tensor by the transpose of the j-th column of the key tensor to obtain the first column result; Based on the results in the first column and the number of channels of the queried tensor, the results in the second column are obtained; Perform a softmax operation on the results of the second column to obtain the column output weight corresponding to the j-th column.

3. The method according to claim 1, wherein, The step of obtaining the first output tensor based on the row output weights corresponding to each of the rows, the column output weights corresponding to each of the columns, and the value tensor includes: Based on the row output weights corresponding to each row, a weighted operation is performed on the corresponding rows of the value tensor to obtain the row output results corresponding to each row. Based on the row output results corresponding to each row, a row output tensor is obtained. The corresponding columns of the value tensor are weighted according to the column output weights corresponding to each column to obtain the column output results corresponding to each column. Based on the column output results corresponding to each column, the column output tensor is obtained. The first output tensor is obtained by adding the row output tensor and the column output tensor element by element.

4. The method according to any one of claims 1-3, wherein, The step of obtaining the query tensor, key tensor, and value tensor corresponding to the image to be processed includes: Perform feature embedding operation on the image to be processed to obtain the corresponding first tensor; Add the first tensor to the corresponding spatial location encoding tensor to obtain the second tensor; Normalize the second tensor to obtain the third tensor; The query tensor is determined based on the third tensor and the first mapping rule; The key tensor is determined based on the third tensor and the second mapping rule; The value tensor is determined based on the third tensor and the third mapping rule.

5. The method according to claim 4, wherein, The step of determining the image processing result corresponding to the image to be processed based on the first output tensor includes: Add the first output tensor to the second tensor to obtain the fourth tensor; Normalize the fourth tensor to obtain the fifth tensor; The fifth tensor is input into the multilayer perceptron module to obtain the sixth tensor; Add the sixth tensor to the fourth tensor to obtain the seventh tensor; Based on the seventh tensor, the image processing result corresponding to the image to be processed is determined.

6. The method according to claim 5, wherein, The step of determining the image processing result corresponding to the image to be processed based on the seventh tensor includes: Perform average pooling on the seventh tensor to obtain the pooling result; The pooling result is subjected to a softmax operation to obtain the image processing result corresponding to the image to be processed.

7. An image processing apparatus, comprising: The first acquisition module is used to acquire the query tensor, key tensor, and value tensor corresponding to the image to be processed; A first processing module is configured to perform attention operations on the query tensor and the key tensor, respectively, by row and by column, to obtain the row output weights corresponding to each row and the column output weights corresponding to each column; wherein the size of the query tensor and the key tensor is H*W*C; H, W, and C represent the height, width, and number of channels of the tensor, respectively. The first processing module includes a first processing unit and a second processing unit. The first processing unit is configured to perform attention operations on the i-th row of the query tensor and the i-th row of the key tensor to obtain the row output weights corresponding to the i-th row; i = 1, 2, ..., H. The second processing unit is configured to perform attention operations on the j-th column of the query tensor and the j-th column of the key tensor to obtain the column output weights corresponding to the j-th column; j = 1, 2, ..., W. The second processing module is used to obtain a first output tensor based on the row output weights corresponding to each row, the column output weights corresponding to each column, and the value tensor. The third processing module is used to determine the image processing result corresponding to the image to be processed based on the first output tensor.

8. A computer-readable storage medium storing a computer program for performing the image processing method according to any one of claims 1-6.

9. An electronic device, the electronic device comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the image processing method according to any one of claims 1-6.

Images