Image processing method and device, storage and computing integrated chip and electronic device

Abstract

The embodiments of the present application relate to an image processing method and device thereof, a storage-computing integrated chip and an electronic device. The image processing method comprises: dividing an image to be processed into blocks to obtain N initial image blocks, where N is an integer greater than 1; projecting each of the initial image blocks respectively to obtain M image feature vectors corresponding to each of the initial image blocks respectively, where M is an integer greater than 1; transmitting the M×N image feature vectors to a storage-computing array according to a preset rule, so that the storage-computing array performs matrix multiplication and addition operations on the received image feature vectors to obtain N fused feature vectors corresponding one to one to the corresponding initial image blocks; and reconstructing according to the N fused feature vectors to generate a target image.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image technology, and in particular, to an image processing method and device thereof, a storage and computing integrated chip, and an electronic device. Background Art

[0002] Super-resolution technology is an important means to improve image resolution in the post-processing stage of mobile phone photography. However, while traditional super-resolution algorithms improve clarity and sharpness, they can hardly avoid increasing noise or sacrificing image details in the process of noise reduction. Therefore, artificial intelligence super-resolution (AISR) can be used to improve the above problems, but the amount of data that AISR needs to process is large, which makes it difficult for AISR to process quickly enough to meet people's needs. Summary of the invention

[0003] The embodiments of the present application provide an image processing method and device thereof, a storage and computing integrated chip, and an electronic device, which can improve the computing speed of the image processing method.

[0004] An image processing method, comprising:

[0005] Divide the image to be processed into blocks to obtain N initial image blocks, where N is an integer greater than 1;

[0006] Projecting each of the initial image blocks respectively to obtain M image feature vectors corresponding to each of the initial image blocks respectively, where M is an integer greater than 1;

[0007] According to a preset rule, the M×N image feature vectors are transmitted to a storage and calculation array, so that the storage and calculation array performs a matrix multiplication and addition operation on the received image feature vectors to obtain N fused feature vectors corresponding to the corresponding initial image blocks one by one;

[0008] Reconstruction is performed based on the N fused feature vectors to generate a target image.

[0009] A storage-computing integrated chip, comprising:

[0010] A processing module, used for dividing the image to be processed into blocks to obtain N initial image blocks, projecting each of the initial image blocks respectively to obtain M image feature vectors corresponding to each of the initial image blocks respectively, and transmitting the M×N image feature vectors to a storage and calculation array according to a preset rule, where N and M are both integers greater than 1;

[0011] A storage and calculation array connected to the processing module, the storage and calculation array comprising a plurality of storage and calculation units, and the storage and calculation array is used to perform matrix multiplication and addition operations on the acquired image feature vectors;

[0012] The processing module is further configured to use the operation result as a fused feature vector corresponding to the initial image block, and to perform reconstruction based on a plurality of the fused feature vectors to generate a target image.

[0013] An image processing device, comprising:

[0014] A block division module, used for dividing the image to be processed into blocks to obtain N initial image blocks, where N is an integer greater than 1;

[0015] A projection module, used to project each of the initial image blocks respectively to obtain M image feature vectors corresponding to each of the initial image blocks respectively, where M is an integer greater than 1;

[0016] A vector transmission module, used for transmitting the M×N image feature vectors to a storage and calculation array according to a preset rule, so that the storage and calculation array performs matrix multiplication and addition operations on the received image feature vectors to obtain N fused feature vectors corresponding to the corresponding initial image blocks one by one;

[0017] A reconstruction module is used to reconstruct according to the N fused feature vectors to generate a target image.

[0018] An electronic device includes the storage and computing integrated chip as described above.

[0019] The above-mentioned image processing method and its device, storage and computing integrated chip and electronic device, the image processing method includes: dividing the image to be processed into blocks to obtain N initial image blocks, wherein N is an integer greater than 1; projecting each of the initial image blocks respectively to obtain M image feature vectors corresponding to each of the initial image blocks respectively, wherein M is an integer greater than 1; transmitting the M×N image feature vectors to the storage and computing array according to a preset rule, so that the storage and computing array performs matrix multiplication and addition operations on the received image feature vectors to obtain N fused feature vectors corresponding one to one to the corresponding initial image blocks; reconstructing according to the N fused feature vectors to generate a target image. In an embodiment of the present application, matrix multiplication and addition operations are implemented through the storage and computing array. After the processor generates data that needs to be subjected to matrix multiplication and addition operations, the data is directly stored in the storage and computing array. The processor can control the storage and computing array to perform subsequent operations by issuing commands, thereby greatly reducing the time for moving data, thereby providing an image processing method with a faster computing speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 One of the flowcharts of the image processing method according to an embodiment;

[0022] Figure 2 A schematic diagram of a block operation according to an embodiment;

[0023] Figure 3 This is a second flowchart of an image processing method according to an embodiment;

[0024] Figure 4 It is a module structure block diagram corresponding to step 304 to step 308;

[0025] Figure 5 A flowchart of a method for obtaining the attention weight matrix corresponding to the i-th initial image block in an embodiment;

[0026] Figure 6 A schematic diagram of the structure of a plurality of second storage and computing devices according to an embodiment;

[0027] Figure 7 A flowchart of a method for obtaining the fused feature vector corresponding to the i-th initial image block in an embodiment;

[0028] Figure 8 A schematic diagram of the structure of a plurality of first storage and computing devices according to an embodiment;

[0029] Fig. 9 A flowchart of an embodiment of reconstructing a target image according to a plurality of fused feature vectors;

[0030] Fig.10 A schematic diagram of a block operation according to an embodiment;

[0031] Fig.11 A schematic diagram of an efficient Transformer structure according to an embodiment;

[0032] Fig.12 This is a schematic diagram of the structure of a storage-computing integrated chip according to an embodiment;

[0033] Fig.13 A schematic diagram of the structure of a storage and computing array according to an embodiment;

[0034] Fig.14 FIG. 4 is a structural block diagram of an image processing device according to an embodiment. DETAILED DESCRIPTION

[0035] In order to facilitate understanding of the embodiments of the present application, the embodiments of the present application will be described more fully below with reference to the relevant drawings. The preferred embodiments of the embodiments of the present application are given in the drawings. However, the embodiments of the present application can be implemented in many different forms and are not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the embodiments of the present application more thorough and comprehensive.

[0036] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those generally understood by those skilled in the art of the technical field of the embodiments of the present application. The terms used herein in the specification of the embodiments of the present application are only for the purpose of describing specific embodiments and are not intended to limit the embodiments of the present application. The term "and / or" used herein includes any and all combinations of one or more related listed items.

[0037] In the description of the embodiments of the present application, it should be understood that the terms "upper", "lower", "vertical", "horizontal", "inside", "outside", etc. indicate orientations or positional relationships based on the methods or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the embodiments of the present application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the embodiments of the present application.

[0038] It is understood that the terms "first", "second", etc. used in the present application may be used herein to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish a first element from another element. For example, without departing from the scope of the present application, a first initial image block may be referred to as a second initial image block, and similarly, a second initial image block may be referred to as a first initial image block. Both the first initial image block and the second initial image block are initial image blocks, but they are not the same initial image block.

[0039] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined. In the description of the present application, the meaning of "several" is at least one, such as one, two, etc., unless otherwise clearly and specifically defined.

[0040] The embodiment of the present application provides an image processing method, specifically an image processing method of AISR. Among them, artificial intelligence (AI) refers to a system and machine that can imitate human intelligence to perform tasks and iteratively improve itself based on the collected information. That is, artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that the machine has the functions of perception, reasoning and decision-making. The image processing method of the embodiment of the present application is applied to a processor, and the processor can be but not limited to a device with data processing functions such as a CPU and a GPU. The above-mentioned processor is connected to a storage and calculation array to implement the steps of matrix multiplication and addition operations in the image processing method through the storage and calculation array. Among them, the storage and calculation array refers to an array structure that uses a memory for data processing. The storage and calculation array can be but not limited to a memristor array, a phase change memory (Phase Change Memory, PCM) array, a static random access memory (Static Random-Access Memory, SRAM) array, a dynamic random access memory (Dynamic Random Access Memory, DRAM) array, a flash memory (Flash) array, etc. Different storage devices have different physical properties and different ways of completing calculation logic, which are not specifically limited in this embodiment.

[0041] Figure 1 is a flowchart of an image processing method according to an embodiment, referring to Figure 1 In this embodiment, the image processing method includes steps 102 to 108.

[0042] Step 102: Divide the image to be processed into blocks to obtain N initial image blocks, where N is an integer greater than 1.

[0043] The image to be processed may be any image. For example, the image to be processed may be an image captured by a camera on an electronic device. Of course, in a possible implementation, the image to be processed may not be an image captured by a camera on a terminal, but an image or a screenshot acquired in real time from a network, which is not limited in this embodiment. Blocking can be achieved by the unfold function. Figure 2 This is a schematic diagram of a block operation according to an embodiment, referring to Figure 2 , the image to be processed can include multiple channels, such as Figure 2 In the embodiment, the image to be processed includes three channels. By dividing the image into blocks, a sliding local area block can be extracted from the image to be processed, thereby dividing the image into multiple small patches, i.e., initial image blocks. For example, if the image to be processed includes 192*192 pixels, 16 initial image blocks can be obtained according to the image to be processed by dividing the image into blocks, wherein each initial image block includes 48*48 pixels.

[0044] Optionally, in order to avoid too many initial image blocks obtained by block segmentation, each initial image block can include a larger number of pixels to reduce the amount of subsequent data processing. In order to make the transition of the target image generated after super-resolution processing more natural and the image clearer, each initial image block can include a smaller number of pixels to achieve more accurate processing. However, it can be understood that the embodiment of the present application performs super-resolution processing based on the similarity between different initial image blocks. Therefore, it should also be avoided that the number of pixels in each initial image block is too small to prevent errors in the matching and comparison results of different initial image blocks.

[0045] Furthermore, the Embedding corresponding to each initial image block can be obtained. The literal translation of Embedding in Chinese is embedding. In simple terms, Embedding refers to vector mapping, that is, using vectors to represent entities. In this embodiment, it can be understood that the corresponding initial image block is represented by a vector, and the Embedding carries the feature information of the corresponding initial image block. By obtaining the Embedding corresponding to each initial image block, it is convenient to perform calculations and analysis based on the initial image block in subsequent steps.

[0046] Step 104 : Project each of the initial image blocks respectively to obtain M image feature vectors corresponding to each of the initial image blocks respectively, where M is an integer greater than 1.

[0047] Among them, in order to obtain the global information of the image, each initial image block can be projected separately to obtain M image feature vectors of each initial image block. Through different projection methods, the features of the initial image block in different dimensions or details can be obtained, so as to achieve accurate analysis of the image to be processed. That is, M image feature vectors corresponding to the i-th initial image block can be obtained through M different projection methods. Among them, i is a positive integer less than or equal to the total number of initial image blocks. Optionally, the method of determining the image feature vector of the i-th initial image block can be linear projection or nonlinear projection, which is not limited in this embodiment.

[0048] Step 106, transmitting the M×N image feature vectors to a storage and computing array according to a preset rule, so that the storage and computing array performs matrix multiplication and addition operations on the received image feature vectors to obtain N fused feature vectors corresponding one-to-one to the corresponding initial image blocks.

[0049] Among them, the image to be processed is divided into N initial image blocks, each of which includes M image feature vectors. Therefore, M×N image feature vectors are generated through the above processing steps. In Single Image Super Resolution (SISR), similar initial image blocks in the image to be processed can be used as references to each other. The similarity between the i-th initial image block and each initial image block can be obtained through the attention mechanism, so that the texture details of the i-th image block can be restored based on other initial image blocks. The matrix multiplication and addition operation in this step can be used as a specific implementation method of the above mutual reference, so as to achieve accurate processing of image details.

[0050] Step 108: Reconstruct according to the N fused feature vectors to generate a target image.

[0051] Among them, reconstruction may include at least one of multiple processing methods such as splicing, fusion, upsampling, etc., which is not limited in this embodiment. In this embodiment, the fusion feature vector corresponding to the ith initial image block is a comprehensive calculation result including the attributes of the ith initial image block itself and the attributes of other image blocks, so that the detail features of the ith initial image block can be supplemented without losing the main features of the ith initial image block, thereby providing an image block with more details. Based on image blocks with more details, a target image with higher resolution can be obtained through reconstruction.

[0052] It is understandable that in the relevant technology, the Information Multi-distillation Network (IMDN) and the like include a large number of channel split and concat operations. Therefore, data needs to be frequently moved through the memory I / O interface, and the speed of data movement is often slower than the speed of calculation, so the above-mentioned large amount of data movement process greatly limits the processing speed of AISR. Specifically, after the processor generates data that requires matrix multiplication and addition operations and stores them in the memory, it is necessary to obtain the data that requires matrix multiplication and addition operations from the memory again and transmit them back to the processor before subsequent matrix multiplication and addition operations can be performed, which results in a longer time required for data movement.

[0053] In this embodiment, a new image processing method is proposed, which does not require a large number of channel separation and connection operations, thereby greatly reducing the amount of data that needs to be transmitted. Moreover, in this embodiment, matrix multiplication and addition operations are implemented through a storage and calculation array. After the processor generates data that needs to be matrix multiplication and addition operations, the data is directly stored in the storage and calculation array. The processor can control the storage and calculation array to perform subsequent operations by issuing commands, thereby greatly reducing the time for transporting data, and can also reduce the time for analog-to-digital conversion of data output by the storage and calculation array, thereby providing an image processing method with a faster operation speed.

[0054] Figure 3 This is a second flowchart of an image processing method according to an embodiment, referring to Figure 3 In this embodiment, the image processing method includes steps 302 to 310.

[0055] Specifically, step 302 of this embodiment is Figure 1 Step 102 of the embodiment is the same as step 310. Figure 1 Step 108 of the embodiment is the same and will not be described again here. Figure 1 Step 104 of the embodiment includes step 304 of the embodiment, Figure 1 Step 106 in includes steps 306 to 308 of this embodiment.

[0056] Step 304: Project each of the initial image blocks based on three different projection matrices to obtain a first eigenvector Q, a second eigenvector K, and a third eigenvector V corresponding to each of the initial image blocks. Specifically, Figure 4 For the module structure diagram corresponding to step 304 to step 308, refer to Figure 4 , Q, K, and V represent the three matrices obtained after linearly projecting the initial image block, representing query, key, and value, respectively. Specifically, Q is the first eigenvector Q obtained by linearly projecting the initial image block through the first projection matrix, K is the second eigenvector K obtained by linearly projecting the initial image block through the second projection matrix, and V is the third eigenvector V obtained by linearly projecting the initial image block through the third projection matrix. Among them, the first projection matrix, the second projection matrix, and the third projection matrix are different, and the specific values ​​of the first projection matrix, the second projection matrix, and the third projection matrix can be obtained through training. Moreover, each initial image block uses the same first projection matrix, the same second projection matrix, and the same third projection matrix, so that the similarity of different initial image blocks in the corresponding dimensions can be obtained.

[0057] Optionally, in some embodiments, each of the initial image blocks can be projected separately through a multi-head projection matrix to implement image processing of a multi-head attention mechanism. Specifically, taking two heads as an example, after projecting the same initialization image block, the corresponding two first eigenvectors Q (Q1, Q2), two second eigenvectors K (K1, K2) and third eigenvectors V (V1, V2) can be obtained, so as to obtain more details of each initial image block, that is, to perform more accurate image processing. It can be understood that the data processing method of each head is the same, which will not be repeated in this embodiment.

[0058] Step 306: transmit the N first eigenvectors Q and the N second eigenvectors K to the storage array for matrix multiplication and addition operations, so as to obtain the attention weight matrices corresponding to each of the initial image blocks.

[0059] Specifically, the first eigenvector Q and the second eigenvector K are both one-dimensional vectors, and therefore, the result of the multiplication and addition operation of the first eigenvector Q and the second eigenvector K is a numerical value, which reflects the degree of similarity between the two initial image blocks. The higher the degree of similarity between the two initial image blocks, the closer the result value after the multiplication and addition operation is to 1. After performing matrix multiplication and addition operations on the first eigenvector Q of the i-th initial image block and the second eigenvector K of each initial image block, corresponding N numerical values ​​can be obtained respectively, and each numerical value reflects the degree of similarity between the i-th initial image block and each initial image block.

[0060] Exemplarily, taking three initial image blocks, each eigenvector including four eigenvalues ​​as an example, the first eigenvector Q of the first initial image block is [q11, q12, q13, q14], the second eigenvector K of the first initial image block is [k11, k12, k13, k14], the second eigenvector K of the second initial image block is [k21, k22, k23, k24], and the second eigenvector K of the third initial image block is [k31, k32, k33, k34]. The matrix multiplication and addition result of the first eigenvector Q of the first initial image block and the second eigenvector K of the first initial image block is q11*k11+q12*k12+q13*k13+q14*k14, and the first eigenvector Q of the first initial image block is [q11, q12, q13, q14], the second eigenvector K of the first initial image block is [k11, k12, k13, k14], the second eigenvector K of the second initial image block is [k21, k22, k23, k24], and the second eigenvector K of the third initial image block is [k31, k32, k33, k34]. The matrix multiplication and addition result of the eigenvector Q and the second eigenvector K of the second initial image block is q11*k21+q12*k22+q13*k23+q14*k24, and the matrix multiplication and addition result of the first eigenvector Q of the first initial image block and the second eigenvector K of the third initial image block is q11*k31+q12*k32+q13*k33+q14*k34, so that the attention weight matrix corresponding to the first initial image block can be determined according to q11*k11+q12*k12+q13*k13+q14*k14, q11*k21+q12*k22+q13*k23+q14*k24 and q11*k31+q12*k32+q13*k33+q14*k34. That is, the attention weight matrix formed based on the above N numerical values ​​can obtain the relationship between similar image blocks in the image to be processed, that is, it reflects the global relationship between the i-th initial image block and each initial image block in the image to be processed, so that the super-resolution processing has more references. It should be noted that q11*k11+q12*k12+q13*k13+q14*k14, q11*k21+q12*k22+q13*k23+q14*k24 and q11*k31+q12*k32+q13*k33+q14*k34 can be directly used as multiple attention weight values ​​in the attention weight matrix, or other data processing can be performed on q11*k11+q12*k12+q13*k13+q14*k14, q11*k21+q12*k22+q13*k23+q14*k24 and q11*k31+q12*k32+q13*k33+q14*k34, and the processed data can be used as multiple attention weight values ​​in the attention weight matrix, which is not limited in this embodiment.

[0061] Step 308, respectively transmit the attention weight matrix and the third eigenvector V corresponding to each of the initial image blocks to the storage array for matrix multiplication and addition operations to obtain N fused eigenvectors corresponding one by one to the corresponding initial image blocks.

[0062] Specifically, after obtaining the similarity between different initial image blocks, the matrix multiplication and addition operation between the attention weight matrix and the third eigenvector V can be regarded as a weighted summation of the third eigenvector V to serve as a new fusion feature vector. Exemplarily, taking the above three initial image blocks, each eigenvector includes four eigenvalues ​​as an example, the third eigenvector V of the first initial image block is [v11, v12, v13, v14], the third eigenvector V of the second initial image block is [v21, v22, v23, v24], and the third eigenvector V of the third initial image block is [v31, v32, v33, v34]. Combined with the attention weight matrix corresponding to the first initial image block generated in the previous step, the fusion feature vector corresponding to the first initial image block can be determined. The eigenvector is [v11(q11*k11+q12*k12+q13*k13+q14*k14)+v21(q11*k21+q12*k22+q13*k23+q14*k24)+v31(q11*k31+q12*k32+q13*k33+q14*k34),v12(q11*k11+q12*k12+q13*k13+q14*k14)+v22(q11*k21+q12*k22+q13*k23+q14*k24) k24)+v32(q11*k31+q12*k32+q13*k33+q14*k34),v13(q11*k11+q12*k12+q13*k13+q14*k14)+v23(q11*k21+q 12*k22+q13*k23+q14*k24)+v33(q11*k31+q12*k32+q13*k33+q14*k34),v14(q11*k11+q12*k12+q13*k13+q14* k14)+v24(q11*k21+q12*k22+q13*k23+q14*k24)+v34(q11*k31+q12*k32+q13*k33+q14*k34)], it can be understood that the initial image block with greater similarity has a positive attention weight, so when calculating, weighting based on the attention weight can accurately compensate for the details in the current initial image block through the features in the similar initial image blocks, and reduce the influence of the non-similar initial image blocks on the current initial image block. The above steps 306 to 308 correspond to Figure 4 The process of dot product attention in .

[0063] In this embodiment, for each initial image block, the similarity between it and all the initial image blocks in the image to be processed is calculated, and then a fused feature vector is obtained based on the similarity and the image feature vector as the feature expression of the initial image block. Based on the above strategy, it is possible to effectively avoid using local information for judgment, thereby solving the problem that the image super-resolution processing result is easily affected by local noise, and improving the accuracy of image super-resolution processing.

[0064] Further, continue to refer to Figure 4 In one embodiment, a reduction (Reduction) and expansion (Expansion) operation set in pairs can be used to reduce the number of parameters. Specifically, before the initial image block is linearly projected, the number of channels can be reduced by a reduction operation, thereby reducing the amount of data that needs to be processed during the linear projection and dot product attention operations. For example, the initial image block with four channels can be reduced to three channels. Correspondingly, after completing the matrix multiplication operation and obtaining the fused feature vector O, an expansion operation can be performed to restore the original number of channels. For example, the image processing result with three channels can be expanded to four channels, thereby improving the accuracy of subsequent image processing operations. Among them, the number of channels to be reduced and expanded can be determined according to the computing power of the dot product attention.

[0065] In one embodiment, the first eigenvector Q, the second eigenvector K and the third eigenvector V can be split into s segments of equal length by a feature splitting module (Feature Split Module, FSM). For example, taking the first eigenvector Q as an example, if the first eigenvector Q is [q11, q12, q1, q14, q15, q16, q17, q18, q19], it can be split into 3 sub-eigenvectors, namely Q1 [q11, q12, q13], Q2 [q14, q15, q16] and Q3 [q17, q18, q19], and the splitting methods of the second eigenvector K and the third eigenvector V are similar. It is understandable that if the first eigenvector Q includes 9 eigenvalues, when the first eigenvector Q and the second eigenvector K which also includes 9 eigenvalues ​​perform a matrix multiplication operation, it is necessary to complete the multiplication and addition operations between 9 parameters and 9 parameters at one time. If the first eigenvector Q and the second eigenvector K are both split into only 3 eigenvalues, it is only necessary to complete the multiplication and addition operations between 3 parameters and 3 parameters at one time, thereby greatly reducing the amount of calculation for each operation. It is understandable that the length of each segment after the split can be determined according to the size of the storage array. The larger the size of the storage array, the more multiplication and addition operations of parameters can be supported at one time. It should be noted that the above segmentation method is only used for exemplary explanation, and is not used to limit the scope of protection of this application. The length of each segment after the split can be much greater than 3. Correspondingly, based on the above splitting operation, the generated fused feature vector also includes multiple segments, namely O1, O2 and O3, and it is necessary to process the multiple segments of data through a concat operation to generate the final fused feature vector O.

[0066] Figure 5 is a flowchart of a method for obtaining the attention weight matrix corresponding to the i-th initial image block in an embodiment, referring to Figure 5 In one embodiment, the method for obtaining the attention weight matrix corresponding to the i-th initial image block includes steps 502 to 504.

[0067] Step 502: Obtain the first dot product sum of the first eigenvector Q of the i-th initial image block and the second eigenvector K of each initial image block.

[0068] Wherein, 1≤n≤N, and n is an integer. Specifically, the storage and computing array includes at least m second storage and computing devices, the first eigenvector Q includes m first eigenvalues, the second eigenvector K includes m second eigenvalues, and m≥2, and m is an integer. The following steps can be used to obtain the multiple first dot product sums corresponding to the i-th initial image block, and the obtaining method includes steps 5022 to 5026. Figure 6is a schematic diagram of the structure of multiple second storage and computing devices in an embodiment, combined with reference Figure 5 and Figure 6 .

[0069] Step 5022: store the m first eigenvalues ​​of the i-th initial image block in m second storage and computing devices in a one-to-one correspondence.

[0070] Specifically, the storage and computing array may include at least one row, that is, the storage and computing array includes at least one word line, and each word line is connected to m second storage and computing devices, and multiple second storage and computing devices connected to the same word line are used to store the first feature vector Q of the same initial image block. Exemplarily, still taking the above three initial image blocks, each eigenvector includes four eigenvalues ​​as an example, the first eigenvector Q of the first initial image block is [q11, q12, q13, q14], and the above four eigenvalues ​​q11, q12, q13 and q14 are stored one-to-one in the four second storage computing devices connected to the first row, the first eigenvector Q of the second initial image block is [q21, q22, q23, q24], and the above four eigenvalues ​​q21, q22, q23 and q24 are stored one-to-one in the four second storage computing devices connected to the second row, and the first eigenvector Q of the third initial image block is [q31, q32, q33, q34], and the above four eigenvalues ​​q31, q32, q33 and q34 are stored one-to-one in the four second storage computing devices connected to the third row. Through the above array storage method, matrix multiplication and addition operations can be performed on multiple initial image blocks at one time, thereby improving the operation speed. It is understandable that when the size of the storage and computing array is small, fewer rows of second storage and computing devices may be provided accordingly, which is not limited in this embodiment.

[0071] Step 5024: Transmit the m second feature vectors K of each of the initial image blocks to the m second storage and computing devices in a one-to-one correspondence.

[0072] Specifically, multiple eigenvalues ​​in the second eigenvector K of the same initial image block are transmitted to multiple bit lines BL in a one-to-one correspondence. Exemplarily, still taking the aforementioned three initial image blocks, each eigenvector includes four eigenvalues ​​as an example, the second eigenvector K of the first initial image block is [k11, k12, k13, k14], the second eigenvector K of the second initial image block is [k21, k22, k23, k24], and the second eigenvector K of the third initial image block is [k31, k32, k33, k34]. The second eigenvectors K of different initial image blocks can be input into the storage and calculation array in time-sharing, for example, [k11, k12, k13, k14] can be input at the first moment, [k21, k22, k23, k24] can be input at the second moment, and [k31, k32, k33, k34] can be input at the third moment.

[0073] Step 5026: Obtain the first dot product sum corresponding to the i-th initial image block according to the signal output by the storage and calculation array.

[0074] Specifically, based on the above structure and data input method, the data line SL of the storage array can output the result of the matrix multiplication and addition operation. Exemplarily, the data line SL1 outputs the result of the matrix multiplication and addition operation of the first eigenvector Q of the first initial image block and the second eigenvector K of the first initial image block at the first moment as q11*k11+q12*k12+q13*k13+q14*k14, the data line SL1 outputs the result of the matrix multiplication and addition operation of the first eigenvector Q of the first initial image block and the second eigenvector K of the second initial image block at the second moment as q11*k21+q12*k22+q13*k23+q14*k24, and the data line SL1 outputs the result of the matrix multiplication and addition operation of the first eigenvector Q of the first initial image block and the second eigenvector K of the third initial image block at the third moment as q11*k31+q12*k32+q13*k33+q14*k34.

[0075] Step 504: normalize the N first dot product sums of the ith initial image block, and obtain the attention weight matrix of the ith initial image block according to the normalized first dot product sums.

[0076] Among them, normalization is to limit the data to a certain range after processing it through a certain algorithm. Data normalization is a basic work of data mining. Different evaluation indicators often have different dimensions and dimensional units, which will affect the results of data analysis. In order to eliminate the dimensional influence between indicators, it is necessary to normalize the data and solve the comparability problem between data indicators.

[0077] In this embodiment, the normalization operation can avoid the attention weight value from exceeding the range, causing errors in the subsequent calculation results, and avoiding the problem of slow convergence speed. Moreover, based on the specific storage and calculation array, the processor can directly store each eigenvalue in the first eigenvector Q to the second storage and calculation device after obtaining the first eigenvector Q, and then obtain the second eigenvector K of each initial image block in turn, and send it to the bit line of the storage and calculation array step by step in time-sharing, so that the storage and calculation array can perform corresponding operations, which can effectively reduce the amount of data that needs to be transmitted between the processor and the storage and calculation array, thereby avoiding data congestion on the data transmission path and increasing the speed of generating the attention weight matrix.

[0078] In one embodiment, continue to refer to Figure 4, the normalization of the N first dot product sums of the i-th initial image block includes the maximum value normalization (Scale) and the exponential function normalization (Softmax) performed in sequence. Specifically, the N first dot product sums of the i-th initial image block are subjected to maximum value normalization, and the N first dot product sums after the maximum value normalization are subjected to exponential function normalization. Among them, the maximum value normalization refers to the proportional enlargement or reduction of all data based on the maximum value and the minimum value of multiple data. The exponential function normalization refers to "compressing" a vector z containing an arbitrary real number into another real vector σ(z) so that the range of each element is between (0,1) and the sum of all elements is 1. This function is more than in the multi-classification problem. The exponential function normalization can make the distance of data with a large gap larger. Therefore, in this embodiment, through the maximum value normalization and the exponential function normalization, both a better data result range can be obtained and the key part of the data can be accurately presented.

[0079] In one embodiment, the storage and computing array includes at least N first storage and computing devices, the attention weight matrix includes N attention weight values, Figure 7 is a flowchart of a method for obtaining the fused feature vector corresponding to the i-th initial image block in an embodiment, referring to Figure 7 The method for obtaining the fused feature vector corresponding to the i-th initial image block includes steps 702 to 706. Figure 8 is a schematic diagram of the structure of multiple first storage and computing devices according to an embodiment, with reference to Figure 7 and Figure 8 .

[0080] Step 702: Store the N attention weight values ​​of the i-th initial image block in the N first storage and computing devices in a one-to-one correspondence.

[0081] Specifically, the storage and computing array may include at least one row, that is, the storage and computing array includes at least one word line, and each word line is connected to N first storage and computing devices, and multiple first storage and computing devices connected to the same word line are used to store the attention weight matrix of the same initial image block. Exemplarily, still taking the aforementioned three initial image blocks, each eigenvector includes four eigenvalues ​​as an example, the attention weight matrix of the first initial image block includes three weight values, namely q11*k11+q12*k12+q13*k13+q14*k14, q11*k21+q12*k22+q13*k23+q14*k24 and q11*k31+q12*k32+q13*k33+q14*k34, and the above three attention weight values ​​are stored one by one in the three first storage and computing devices connected to the first row. The attention weight matrix of the second initial image block includes three weight values, namely q21*k11+q22*k12+q23*k13+q24*k14, q21*k21+q22*k22+q23*k23+q24*k24 and q21*k31+q22*k32+q23*k33+q24*k34, and the above three attention weight values ​​are stored one by one in the three first storage and computing devices connected to the second row. Through the above array storage method, matrix multiplication and addition operations can be performed on multiple initial image blocks at a time, thereby improving the operation speed. It can be understood that when the size of the storage and computing array is small, fewer rows of first storage and computing devices can be set accordingly, and this embodiment is not limited.

[0082] Step 704: Transmit the third feature vectors V of the N initial image blocks to the N first storage and computing devices in a one-to-one correspondence.

[0083] Specifically, multiple eigenvalues ​​in the third eigenvector V of the same initial image block are transmitted to the same bit line BL ( Figure 8 Only the first eigenvalue in each third eigenvector V is shown in the figure). Multiple eigenvalues ​​in the third eigenvector V of the same initial image block can be input into the storage array in a time-sharing manner. For example, v11 can be input into the first bit line BL1 at the first moment, v12 can be input into the first bit line BL1 at the second moment, v13 can be input into the first bit line BL1 at the third moment, and v14 can be input into the first bit line BL1 at the fourth moment.

[0084] Step 706: Obtain the fused feature vector of the i-th initial image block according to the signal output by the storage and calculation array.

[0085] Specifically, based on the above structure and data input method, the result of the matrix multiplication and addition operation can be output by the data line SL of the storage and calculation array. Exemplarily, the data line SL1 outputs the matrix multiplication and addition operation result of the attention weight matrix of the first initial image block and the third eigenvector V of the first initial image block at the first moment as v11(q11*k11+q12*k12+q13*k13+q14*k14)+v21(q11*k21+q12*k22+q13*k23+q14*k24)+v31(q11*k31+q12*k32+q13*k33+q14*k3 4), the data line SL2 outputs the matrix multiplication and addition operation result of the attention weight matrix of the second initial image block and the third eigenvector V of the second initial image block at the first moment, which is v11(q21*k11+q22*k12+q23*k13+q24*k14)+v21(q21*k21+q22*k22+q23*k23+q24*k24)+v31(q21*k31+q22*k32+q23*k33+q24*k34).

[0086] Fig. 9 is a flowchart of an embodiment of reconstructing a target image based on a plurality of fused feature vectors, referring to Fig. 9 In this embodiment, the reconstruction is performed based on the multiple fused feature vectors to generate a target image, including steps 902 to 908.

[0087] Step 902: Generate corresponding fused image blocks according to the fused feature vectors. The fusion method can be set as needed, for example, it can be at least one of splicing or weighted addition.

[0088] Step 904: Process each of the fused image blocks by a multilayer perceptron to generate a perceptual image block. The spliced ​​result is input into a multilayer perceptron (MLP) model to mix two different heterogeneous features. Optionally, the perceptron model can be a three-layer perceptron.

[0089] Step 906: fold the plurality of perceived image blocks to obtain a perceived image. Fig.10 This is a schematic diagram of a block operation according to an embodiment, referring to Fig.10 , multiple patches can be folded into an overall perceived image through the fold function.

[0090] Step 908: upsample the perceived image to obtain the target image. Specifically, upsampling can be achieved by algorithms such as bilinear interpolation, transposed convolution, upsampling (unsampling), uppooling (unpooling), and sub-pixel convolution (Pixel Shuffle), which are not limited in this embodiment.

[0091] In this embodiment, through a series of operations of converting the vector into an image block and then converting the image block into an image, the image block can be restored to generate an image that can be output. Moreover, combined with the upsampling operation, the resolution of the image can be further improved to obtain a target image with a target resolution.

[0092] Fig.11 A schematic diagram of an efficient Transformer structure according to an embodiment of the present invention is shown in FIG. Fig.11 , before projecting each of the initial image blocks, it also includes: performing layer normalization on each of the initial image blocks; and / or before processing each of the fused image blocks by a multi-layer perceptron to generate a perceived image block, it also includes: performing layer normalization on each of the perceived image blocks. Wherein, layer normalization refers to normalizing different channels of the same sample. Specifically, the calculation process of layer normalization is shown in the following three formulas:

[0093]

[0094]

[0095]

[0096] Where H is the number of nodes in the Lth layer of Reduction or MLP, and a l is the input vector of this layer, is the input vector of this layer after normalization, μ and σ are the mean and variance of this layer respectively. Fig.11 In the illustrated embodiment, the efficient Transformer structure performs layer normalization twice to achieve the best layer normalization effect.

[0097] Further, continue to refer to Fig.11, the multi-head attention unit and the multi-layer perceptron unit are connected through residual networks (Residual Networks, ResNets), and the multi-layer perceptron unit is also connected through residual networks when outputting. It is understandable that as the number of layers increases, the effect on the training set will have a degradation problem (degradation problem). The reason is that as the network becomes deeper and deeper, training becomes more and more difficult, and network optimization becomes more and more difficult. Although theoretically speaking, the deeper the network, the better the effect should be, but in fact, due to the difficulty of training, a network that is too deep will have degradation problems, and the effect is not as good as a relatively shallow network. In this embodiment, the residual network can effectively suppress the above-mentioned degradation problem, thereby improving the image generation effect.

[0098] In one of the embodiments, the storage and computing array includes a plurality of memristors, and before dividing the image to be processed into blocks to obtain N initial image blocks, it also includes performing shallow feature extraction on the image to be processed through convolution operations based on the storage and computing array to obtain a shallow feature image. Among them, the receptive field of shallow feature extraction is smaller, so it can utilize more fine-grained feature information, and at this time, the overlapping area of ​​the receptive field corresponding to each pixel point is also smaller, thereby ensuring that the shallow feature extraction network can capture more details. Accordingly, dividing the image to be processed into blocks to obtain N initial image blocks includes dividing the shallow feature image into blocks to obtain N initial image blocks. In this embodiment, based on the deep features extracted from the shallow features and the efficient Transformer structure, more accurate super-resolution processing of the image can be achieved.

[0099] In one of the embodiments, before upsampling the perceived image, the method further includes performing channel fusion on the perceived image through convolution operation based on the storage array. In this embodiment, the number of channels can be reduced by channel fusion, and effective information can be extracted from the fused features. Moreover, by setting the channel fusion operation before upsampling, the processing pressure during the upsampling process can be effectively reduced. Furthermore, the shallow feature image and the perceived image after channel fusion can be summed to realize a residual network structure to suppress the aforementioned training degradation problem, thereby improving the image generation effect. Accordingly, the upsampling of the perceived image includes upsampling the image obtained by summation. Among them, the above-mentioned channel fusion and shallow feature extraction can both be realized through convolution operation, and the above-mentioned convolution operation can also be realized based on the storage array. In some embodiments, the above-mentioned channel fusion and / or shallow feature extraction can also be realized through the aforementioned efficient Transformer structure, which is not limited in this embodiment.

[0100] In one embodiment, the above upsampling operation can be implemented by sub-pixel convolution. The specific method of sub-pixel convolution (PixelShuffle) includes: receiving a low-resolution input image, first obtaining r by convolution 2 The feature map of each channel (the size of the feature map is consistent with the input low-resolution image), and then a high-resolution image is obtained by a periodic shuffling method, wherein r is an upscaling factor, that is, the image enlargement factor. The storage and computing array includes a plurality of memristors, and the upsampling of the perceived image includes upsampling the perceived image by sub-pixel convolution based on the storage and computing array.

[0101] It should be understood that, although the steps in the flowcharts of each flowchart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in each flowchart may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0102] The embodiment of the present application also provides a storage-computing integrated chip, Fig.12 FIG. 1 is a schematic diagram of a storage-computing integrated chip according to an embodiment of the present invention. Fig.12 The storage and computing integrated chip includes a processing module and a storage and computing array.

[0103] Among them, the processing module is used to divide the image to be processed into blocks to obtain N initial image blocks, project each of the initial image blocks respectively, so as to obtain M image feature vectors corresponding to each of the initial image blocks respectively, and transmit the M×N image feature vectors to the storage and calculation array according to the preset rules, wherein N and M are both integers greater than 1. The storage and calculation array is connected to the processing module, and the storage and calculation array includes a plurality of storage and calculation units, and the storage and calculation array is used to perform matrix multiplication and addition operations on the acquired image feature vectors. Among them, the processing module is also used to use the operation result as the fusion feature vector corresponding to the initial image block, and reconstruct according to the plurality of fusion feature vectors to generate a target image. In an embodiment, the matrix multiplication and addition operation is realized by the storage and calculation array, and after the processor generates the data that needs to be subjected to the matrix multiplication and addition operation, the data is directly stored in the storage and calculation array, and the processor can control the storage and calculation array to perform subsequent operations by issuing a command, thereby greatly reducing the time of transporting data, and can also reduce the time of analog-to-digital conversion of the data output by the storage and calculation array, thereby providing a storage and calculation integrated chip with a faster operation speed.

[0104] Fig.13 A schematic diagram of the structure of a storage and computing array according to an embodiment of the present invention is shown in FIG. Fig.13 In this embodiment, the storage and calculation array further includes a plurality of word lines WL, a plurality of bit lines BL and a plurality of data lines SL, and the storage and calculation unit includes a memristor and a switch transistor. The memristor is respectively connected to the bit line BL and the first electrode of the switch transistor, the control electrode of the switch transistor is connected to the word line WL, and the second electrode of the switch transistor is connected to the data line SL. The plurality of switch transistors in the same row are connected to the same word line WL and the same data line SL, and the plurality of memristors in the same column are connected to the same bit line BL.

[0105] The present application also provides an image processing device, Fig.14 is a structural block diagram of an image processing device according to an embodiment, Fig.14, the image processing device includes a blocking module 1402, a projection module 1404, a vector transmission module 1406 and a reconstruction module 1408. The blocking module 1402 is used to block the image to be processed to obtain N initial image blocks, where N is an integer greater than 1. The projection module 1404 is used to project each of the initial image blocks respectively to obtain M image feature vectors corresponding to each of the initial image blocks respectively, where M is an integer greater than 1. The vector transmission module 1406 is used to transmit the M×N image feature vectors to the storage array according to a preset rule, so that the storage array performs matrix multiplication and addition operations on the received image feature vectors to obtain N fused feature vectors corresponding to the corresponding initial image blocks. The reconstruction module 1408 is used to reconstruct according to the N fused feature vectors to generate a target image. In this embodiment, matrix multiplication and addition operations are implemented through a storage and calculation array. After the processor generates data that requires matrix multiplication and addition operations, the data is directly stored in the storage and calculation array. The processor can control the storage and calculation array to perform subsequent operations by issuing commands, thereby greatly reducing the time for moving data, thereby providing an image processing device with a faster computing speed.

[0106] The division of each module in the above-mentioned image processing device is only for illustration. In other embodiments, the image processing device can be divided into different modules as needed to complete all or part of the functions of the above-mentioned image processing device. For the specific definition of the image processing device, please refer to the definition of the image processing method above, which will not be repeated here. Each module in the above-mentioned image processing device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0107] The embodiment of the present application also provides an electronic device, including the storage and computing integrated chip as described above. The electronic device can be any terminal device such as a mobile phone, a tablet computer, a PDA (Personal Digital Assistant), a POS (Point of Sales), a car computer, a wearable device, etc. In this embodiment, based on the aforementioned storage and computing integrated chip, an electronic device with a faster image processing speed is provided.

[0108] The embodiment of the present application further provides a computer-readable storage medium, one or more non-volatile computer-readable storage media containing computer-executable instructions, which, when executed by one or more processors, enable the processors to perform the steps of the image processing method.

[0109] A computer program product comprising instructions, when executed on a computer, causes the computer to perform an image processing method.

[0110] Any reference to memory, storage, database or other medium used in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM), which is used as an external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlin second eigenvector K) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0111] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0112] The above-described embodiments only express several implementation methods of the embodiments of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the embodiments of the present application, and these all belong to the protection scope of the embodiments of the present application. Therefore, the protection scope of the patent of the embodiments of the present application shall be subject to the attached claims.

Claims

1. An image processing method, characterized in that: include: Divide the image to be processed into blocks to obtain N initial image blocks, where N is an integer greater than 1; Projecting each of the initial image blocks based on three different projection matrices respectively to obtain a first eigenvector, a second eigenvector and a third eigenvector corresponding to each of the initial image blocks respectively; Transmitting the N first eigenvectors and the N second eigenvectors to a storage array for matrix multiplication and addition operations, respectively, to obtain attention weight matrices corresponding to the initial image blocks; The attention weight matrix and the third eigenvector corresponding to each of the initial image blocks are respectively transmitted to the storage array for matrix multiplication and addition operations to obtain N fused eigenvectors corresponding to the corresponding initial image blocks one by one; Reconstruction is performed based on the N fused feature vectors to generate a target image.

2. The image processing method according to claim 1, characterized in that: The storage and computing array includes at least N first storage and computing devices, the attention weight matrix includes N attention weight values, and the method for obtaining the fusion feature vector corresponding to the i-th initial image block includes: The N attention weight values ​​of the i-th initial image block are stored one by one in the N first storage and computing devices; i is a positive integer less than or equal to the total number of initial image blocks; Transmitting the third feature vectors of the N initial image blocks to the N first storage and computing devices one by one; The fused feature vector of the i-th initial image block is obtained according to the signal output by the storage and calculation array.

3. The image processing method according to claim 1, characterized in that: The method for obtaining the attention weight matrix corresponding to the i-th initial image block includes: Obtaining the first dot product sum of the first feature vector of the i-th initial image block and the second feature vectors of each initial image block, wherein i is a positive integer less than or equal to the total number of initial image blocks; Normalize the N first dot product sums of the i-th initial image block, and obtain the attention weight matrix of the i-th initial image block according to the normalized first dot product sums.

4. The image processing method according to claim 3, characterized in that: The storage and computing array includes at least m second storage and computing devices, the first eigenvector includes m first eigenvalues, the second eigenvector includes m second eigenvalues, m≥2, and m is an integer, and a method for obtaining a plurality of first dot product sums corresponding to the i-th initial image block includes: The m first characteristic values ​​of the i-th initial image block are stored in the m second storage and computing devices in a one-to-one correspondence; Transmitting the m second feature vectors of each of the initial image blocks to the m second storage and computing devices one by one; Obtain a first dot product sum corresponding to the i-th initial image block according to the signal output by the storage and calculation array.

5. The image processing method according to claim 3, characterized in that: The normalizing the N first dot product sums of the i-th initial image block comprises: Performing maximum value normalization on the N first dot product sums of the i-th initial image block; The N first dot product sums after the maximum value normalization are normalized using an exponential function.

6. The image processing method according to claim 1, characterized in that: The reconstructing according to the N fused feature vectors to generate a target image comprises: Generate corresponding fused image blocks respectively according to each of the fused feature vectors; Processing each of the fused image blocks respectively by a multi-layer perceptron to generate a perceptual image block; Folding a plurality of the perceived image blocks to obtain a perceived image; The perceived image is upsampled to obtain the target image.

7. The image processing method according to claim 6, characterized in that: The storage and computing array includes a plurality of memristors, and before dividing the image to be processed into blocks to obtain N initial image blocks, the method further includes: Based on the storage and calculation array, a convolution operation is performed on the image to be processed to extract shallow features, so as to obtain a shallow feature image; The step of dividing the image to be processed into blocks to obtain N initial image blocks includes: The shallow feature image is divided into blocks to obtain N initial image blocks.

8. The image processing method according to claim 7, characterized in that: Before upsampling the perceived image, the method further includes: Performing channel fusion on the perceived image through convolution operation based on the storage and computing array; Summing the shallow feature image and the perceived image after channel fusion; The up-sampling of the perceived image comprises: The summed image is upsampled.

9. The image processing method according to claim 6, characterized in that: The storage and computing array includes a plurality of memristors, and upsampling the perceived image includes: Sub-pixel convolution is performed on the perceived image based on the storage and computation array.

10. The image processing method according to claim 6, characterized in that: Before projecting the initial image blocks respectively, the method further includes: performing layer normalization on the initial image blocks respectively; and / or Before the fused image blocks are processed by a multi-layer perceptron to generate the perceived image blocks, the method further includes: performing layer normalization on the perceived image blocks.

11. A storage and computing integrated chip, characterized in that: include: A processing module, used for dividing the image to be processed into blocks to obtain N initial image blocks; Projecting each of the initial image blocks based on three different projection matrices to obtain first eigenvectors, second eigenvectors and third eigenvectors corresponding to each of the initial image blocks; transmitting N of the first eigenvectors and N of the second eigenvectors to a storage array for matrix multiplication and addition operations to obtain attention weight matrices corresponding to each of the initial image blocks; transmitting the attention weight matrix and the third eigenvector corresponding to each of the initial image blocks to the storage array to obtain N fused eigenvectors corresponding to the corresponding initial image blocks, wherein N is an integer greater than 1; A storage and calculation array, connected to the processing module, comprising a plurality of storage and calculation units, and used for performing matrix multiplication and addition operations on the acquired attention weight matrix and the third eigenvector; The processing module is further configured to use the operation result as a fused feature vector corresponding to the initial image block, and to perform reconstruction based on a plurality of the fused feature vectors to generate a target image.

12. The storage-computing integrated chip according to claim 11, characterized in that: The storage and computing array further includes a plurality of word lines, a plurality of bit lines and a plurality of data lines, and the storage and computing unit includes a memristor and a switch transistor; wherein, The memristor is connected to the bit line and the first electrode of the switch transistor respectively, the control electrode of the switch transistor is connected to the word line, and the second electrode of the switch transistor is connected to the data line; The plurality of switch transistors in the same row are connected to the same word line and the same data line, and the plurality of memristors in the same column are connected to the same bit line.

13. An image processing device, characterized in that: include: A block division module, used for dividing the image to be processed into blocks to obtain N initial image blocks, where N is an integer greater than 1; A projection module, used for projecting each of the initial image blocks based on three different projection matrices, so as to respectively obtain a first eigenvector, a second eigenvector and a third eigenvector corresponding to each of the initial image blocks; A vector transmission module, used to transmit the N first eigenvectors and the N second eigenvectors to the storage array for matrix multiplication and addition operations, so as to obtain the attention weight matrix corresponding to each of the initial image blocks; and transmit the attention weight matrix and the third eigenvector corresponding to each of the initial image blocks to the storage array for matrix multiplication and addition operations, so as to obtain N fused eigenvectors corresponding to the corresponding initial image blocks one by one; A reconstruction module is used to reconstruct according to the N fused feature vectors to generate a target image.

14. An electronic device, characterized in that: It includes the storage-computing integrated chip as described in claim 11 or 12.

Images