In-memory processing-oriented structure fusion method, device and equipment

Abstract

The invention provides an in-memory processing-oriented structure fusion method, device and equipment. The method comprises the following steps: acquiring a to-be-converted low-resolution image; sequentially inputting the to-be-converted low-resolution images into a pre-constructed fusion lookup table for table lookup processing according to a preset receptive field size and a preset traversal sequence to obtain a converted high-resolution image; the fusion lookup table is an overall relation table which is in one-to-one correspondence with the high-resolution output images and is established for the low-resolution input images according to the size of a preset receptive field in a multi-branch fusion processing mode by adopting a pre-training multi-branch fusion processing network; the pre-trained multi-branch fusion processing network is a multi-branch convolutional neural network based on a space attention mechanism. According to the image processing method and device, the overall relation table is obtained by fusing the multi-branch data, redundant calculation paths are eliminated, deep multiplexing of storage resources is achieved at the same time, the hardware requirement in the deployment process is reduced, and the real-time performance of image processing is improved.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a structure fusion method, device and equipment for in-memory processing. Background Art

[0002] With the continuous improvement of mobile device performance and users' increasing demand for image quality, the demand for high-resolution images has become particularly urgent on edge devices such as smartphones and tablets. To meet this demand, a variety of methods have emerged, ranging from traditional interpolation algorithms to modern deep learning-based super-resolution techniques. Deep learning-based methods, in particular, can significantly improve the quality of low-resolution images, but these methods typically require extensive computing resources and storage space, which conflicts with the limited computing power and storage capacity of edge devices.

[0003] To address these challenges, a lookup table-based image super-resolution method has been proposed and is gaining traction. This method uses a pre-trained convolutional neural network model to store the mapping between low-resolution and high-resolution images in a lookup table. This method trades significant storage resources for time efficiency, effectively reducing the computational burden of the actual inference process. This method is particularly suitable for edge devices because it significantly reduces the computing power required for real-time processing, enabling high-quality image super-resolution even in resource-constrained environments.

[0004] However, although the lookup table-based approach solves some of the problems, it still has some limitations. First, although the storage overhead of the lookup table has been reduced from exponential to linear, when using multi-branch lookup tables for parallel processing, the storage resources it occupies are still huge, making it difficult to deploy on low-end embedded devices. Second, when the system relies on multiple lookup tables for calculations, frequent memory accesses will lead to increased latency. At the same time, the excessive number of lookup tables will also extend the calculation time, affecting the real-time performance of image processing. These problems limit the widespread application of existing methods on edge devices, especially in application scenarios that require low storage consumption and high real-time performance. Summary of the Invention

[0005] In order to solve the above problems existing in the prior art, the present invention provides a structure fusion method, device and equipment for in-memory processing.

[0006] The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0007] In a first aspect, the present invention provides a structure fusion method for in-memory processing, comprising:

[0008] Get the low-resolution image to be converted;

[0009] The low-resolution image to be converted is input into a pre-built fusion lookup table in sequence according to a preset receptive field size and a preset traversal order for lookup processing to obtain a converted high-resolution image; the fusion lookup table is an overall relationship table that is established for the low-resolution input image according to the preset receptive field size and the high-resolution output image under a multi-branch fusion processing method using a pre-trained multi-branch fusion processing network; the pre-trained multi-branch fusion processing network is a multi-branch convolutional neural network based on a spatial attention mechanism.

[0010] Optionally, the low-resolution image to be converted is sequentially input into a pre-built fusion lookup table according to a preset receptive field size and a preset traversal order for table lookup processing to obtain a converted high-resolution image, including:

[0011] Split the low-resolution image to be converted into multiple low-resolution image sub-blocks to be converted according to a preset traversal order and a preset receptive field size;

[0012] According to the pixel values ​​corresponding to the multiple low-resolution image sub-blocks to be converted, a table lookup process is performed in the fusion lookup table to obtain multiple converted high-resolution image sub-blocks;

[0013] The plurality of converted high-resolution image sub-blocks are spliced ​​together in a splicing order corresponding to a preset traversal order to obtain a converted high-resolution image.

[0014] Optionally, the preset traversal order includes: an order from top to bottom and from left to right, an order from top to bottom and from right to left, an order from bottom to top and from left to right, and an order from bottom to top and from right to left.

[0015] Optionally, the pre-trained multi-branch fusion processing network includes: a plurality of preset CNN networks and a plurality of spatial attention mechanism SA networks; the number of the preset CNN networks is equal to the number of the spatial attention mechanism SA networks;

[0016] Each preset CNN network is connected in parallel with the spatial attention mechanism SA network, and connected to each other through their respective output ends to form multiple independent branch networks;

[0017] Multiple branch networks are connected in parallel and interconnected through their respective output ends to form a pre-trained multi-branch fusion processing network.

[0018] Optionally, the process of constructing the fusion lookup table includes:

[0019] Acquire multiple low-resolution input images; the low-resolution input images are index blocks with a preset receptive field size obtained by splitting the relationship table construction samples; the relationship table construction samples are sample images consistent with the low-resolution image scene to be converted;

[0020] Input the low-resolution input image into multiple branch networks respectively to obtain multiple branch mapping results;

[0021] The multiple branch mapping results corresponding to each low-resolution input image are averaged to obtain multiple high-resolution output images;

[0022] A one-to-one corresponding index relationship table is constructed for the low-resolution input image and the high-resolution output image to obtain a fusion lookup table.

[0023] Optionally, the training process of the pre-trained multi-branch fusion processing network includes:

[0024] Obtaining a training sample image pair; wherein the training sample image pair is provided with a one-to-one correspondence between a low-resolution input sample image and a high-resolution output sample image;

[0025] Input the training sample images into the initial multi-branch fusion processing network for training;

[0026] The initial multi-branch fusion processing network that meets the preset stopping condition is used as the pre-trained multi-branch fusion processing network; the structure of the initial multi-branch fusion processing network is the same as that of the pre-trained multi-branch fusion processing network;

[0027] The preset stopping conditions include: the number of iterative training times is greater than the iteration threshold or the value of the loss function corresponding to the initial multi-branch fusion processing network is continuously less than the loss threshold.

[0028] Optionally, according to the pixel values ​​corresponding to the plurality of low-resolution image sub-blocks to be converted, a table lookup process is performed in a fusion lookup table to obtain a plurality of converted high-resolution image sub-blocks, including:

[0029] According to the pixel values ​​corresponding to the multiple low-resolution image sub-blocks to be converted, a traversal search method is used to perform table lookup processing in the fusion lookup table to obtain the matched multiple converted high-resolution image sub-blocks.

[0030] In a second aspect, the present invention provides a structure fusion device for in-memory processing, the structure fusion device for in-memory processing comprising: an acquisition unit and a table lookup unit;

[0031] The acquisition unit is used to: acquire the low-resolution image to be converted;

[0032] The table lookup unit is used to: input the low-resolution image to be converted into a pre-built fusion lookup table in sequence according to a preset receptive field size and a preset traversal order for table lookup processing to obtain a converted high-resolution image; the fusion lookup table is an overall relationship table that is established for the low-resolution input image according to the preset receptive field size and the high-resolution output image under a multi-branch fusion processing method using a pre-trained multi-branch fusion processing network; the pre-trained multi-branch fusion processing network is a multi-branch convolutional neural network based on a spatial attention mechanism.

[0033] In a third aspect, the present invention provides a structural fusion device for in-memory processing, comprising: a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor. When the structural fusion device for in-memory processing is running, the processor and the storage medium communicate through the bus, and the processor executes the machine-readable instructions to perform the steps of the structural fusion method for in-memory processing as described in any one of the first aspects above.

[0034] The present invention provides a structural fusion method, device, and apparatus for in-memory processing. The structural fusion method for in-memory processing includes: obtaining a low-resolution image to be converted; inputting the low-resolution image to be converted into a pre-constructed fusion lookup table in sequence according to a preset receptive field size and a preset traversal order for lookup processing to obtain a converted high-resolution image; the fusion lookup table is a pre-trained multi-branch fusion processing network, which establishes a one-to-one correspondence between the low-resolution input image and the high-resolution output image according to a preset receptive field size under a multi-branch fusion processing mode; the pre-trained multi-branch fusion processing network is a multi-branch convolutional neural network based on a spatial attention mechanism. In the present invention, a pre-trained multi-branch fusion processing network is constructed by adopting a multi-branch convolutional neural network based on a spatial attention mechanism, effectively improving the modeling capability of the network; secondly, the multi-branch data is fused to obtain an overall relationship table, eliminating redundant calculation paths while achieving deep reuse of storage resources. That is, combined with the hardware mapping rules of storage and computing integration, the originally scattered lookup tables are fused, significantly reducing resource consumption and shortening inference latency, reducing hardware requirements during deployment, and improving the real-time performance of image processing.

[0035] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A schematic flow chart of a structure fusion method for in-memory processing provided by an embodiment of the present invention;

[0037] Figure 2 The schematic diagram of the structure of the pre-trained multi-branch fusion processing network is shown as an example;

[0038] Figure 3 A schematic diagram of a structure fusion device for in-memory processing provided by an embodiment of the present invention;

[0039] Figure 4 A schematic structural diagram of a structure fusion device for in-memory processing provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0040] The present invention aims to overcome bottlenecks such as high hardware resource consumption, long inference time, and low storage resource utilization in parallel multi-branch lookup tables. To address these issues, the present invention proposes a "structural fusion method for in-memory processing," which introduces a spatial attention mechanism into the network and fuses parallel lookup tables through structural reparameterization. This method achieves deep reuse of hardware storage resources, reduces hardware resource consumption, and shortens inference time.

[0041] The present invention will be further described in detail below with reference to specific examples, but the embodiments of the present invention are not limited thereto.

[0042] In order to reduce the hardware requirements during the deployment process and improve the real-time performance of image processing, an embodiment of the present invention provides a structure fusion method for in-memory processing. Figure 1 The flowchart of a structure fusion method for in-memory processing provided by an embodiment of the present invention is as follows. Figure 1 Shown, including:

[0043] S101: Obtain a low-resolution image to be converted.

[0044] It should be noted that the low-resolution image to be converted can be any image that needs to be reconstructed with high resolution or super resolution.

[0045] S102 , inputting the low-resolution image to be converted into a pre-built fusion lookup table in sequence according to a preset receptive field size and a preset traversal order, performing table lookup processing, and obtaining a converted high-resolution image.

[0046] It should be noted that the fusion lookup table adopts a pre-trained multi-branch fusion processing network. Under the multi-branch fusion processing method, an overall relationship table is established for the low-resolution input image according to the preset receptive field size, which corresponds one to one with the high-resolution output image; the pre-trained multi-branch fusion processing network is a multi-branch convolutional neural network based on the spatial attention mechanism.

[0047] An embodiment of the present invention provides a structural fusion method for in-memory processing. By adopting a multi-branch convolutional neural network based on a spatial attention mechanism, a pre-trained multi-branch fusion processing network is constructed, which effectively improves the modeling ability of the network; secondly, the multi-branch data is fused to obtain an overall relationship table, eliminating redundant calculation paths while realizing deep reuse of storage resources. That is, combined with the hardware mapping rules of integrated storage and computing, the originally scattered lookup tables are fused, which significantly reduces resource consumption and shortens inference latency, reduces hardware requirements during deployment and improves the real-time performance of image processing.

[0048] Optionally, S102 may specifically include:

[0049] Split the low-resolution image to be converted into multiple low-resolution image sub-blocks to be converted according to a preset traversal order and a preset receptive field size;

[0050] According to the pixel values ​​corresponding to the multiple low-resolution image sub-blocks to be converted, a table lookup process is performed in the fusion lookup table to obtain multiple converted high-resolution image sub-blocks;

[0051] The plurality of converted high-resolution image sub-blocks are spliced ​​together in a splicing order corresponding to a preset traversal order to obtain a converted high-resolution image.

[0052] Optionally, the preset traversal order includes: an order from top to bottom and from left to right, an order from top to bottom and from right to left, an order from bottom to top and from left to right, and an order from bottom to top and from right to left.

[0053] Optionally, the pre-trained multi-branch fusion processing network includes: a plurality of preset CNN networks and a plurality of spatial attention mechanism SA networks; the number of the preset CNN networks is equal to the number of the spatial attention mechanism SA networks;

[0054] Each preset CNN network is connected in parallel with the spatial attention mechanism SA network, and connected to each other through their respective output ends to form multiple independent branch networks;

[0055] Multiple branch networks are connected in parallel and interconnected through their respective output ends to form a pre-trained multi-branch fusion processing network.

[0056] Figure 2 The schematic diagram of the structure of the pre-trained multi-branch fusion processing network is shown as an example. Figure 2As shown in the figure, the pre-trained multi-branch fusion processing network (SA network) uses multiple parallel CNN networks to extract image features, and the settings of the convolution kernel groups in each parallel branch are consistent. The spatial attention mechanism SA network is introduced in each parallel branch to enable the network to focus on the area of ​​interest. And the size and shape of the first layer convolution kernel of the CNN network in each branch are the same as those of the spatial attention SA network. After the input data (low-resolution input image) passes through the branch CNN network, the feature map F is obtained. After the input data passes through the SA network, the feature vector SAF with the spatial attention mechanism is obtained. The SAF with spatial weight is weighted to the corresponding position of the corresponding channel of the feature map F through the scale(F,SAF) operation to highlight the area of ​​interest and complete the recalibration of the feature map. Then the feature map FBranch of each branch is used. x The final network output (high-resolution output image) is obtained by adding and averaging.

[0057] FBranch x =scale(F x ,SAF x )=F x (c) ⊙SAF x (c) ;

[0058] Where x is an integer not less than 0, used to represent the x-th branch; F x (c) Represents the cth channel of the feature map F of the xth branch; SAF x (c) Represents the cth channel of the feature map SAF of the xth branch; the ⊙ operator represents the multiplication of the corresponding positions of the two feature maps; FBranch x Represents the feature map of the x-th branch output, scale(F x ,SAF x ) indicates F x With SAF x Perform spatial weighting processing.

[0059]

[0060] ModelOut represents the final network output; n represents the total number of branches.

[0061] Optionally, the process of constructing the fusion lookup table includes:

[0062] Acquire multiple low-resolution input images; the low-resolution input images are index blocks with a preset receptive field size obtained by splitting the relationship table construction samples; the relationship table construction samples are sample images consistent with the low-resolution image scene to be converted;

[0063] Input the low-resolution input image into multiple branch networks respectively to obtain multiple branch mapping results;

[0064] The multiple branch mapping results corresponding to each low-resolution input image are averaged to obtain multiple high-resolution output images;

[0065] A one-to-one corresponding index relationship table is constructed for the low-resolution input image and the high-resolution output image to obtain a fusion lookup table.

[0066] In this embodiment, establishing a lookup table is to enumerate all input data of the pre-trained multi-branch fusion processing network, use the input data as the lookup table index, and the corresponding output of the network as the index value of the lookup table, so as to obtain a lookup table containing the mapping relationship between the low-resolution image and the high-resolution image learned by the network.

[0067] Generate an index based on the input data and input it into the network model. Then, obtain the corresponding output through the pre-trained multi-branch fusion processing network, and thus obtain the lookup table corresponding to each structure, namely:

[0068]

[0069] SALUT x [index]=SA x ((data1,data2,…,data m ));

[0070] CLUT x [index]=CNN x ((data1,data2,…,data m ));

[0071] Among them, index represents the lookup table index generated according to the low-resolution input image; m represents the size of the receptive field, i represents the i-th receptive field, and data i Indicates the pixel value corresponding to the i-th receptive field, data i The value of SALUT is 0 to 255. x [index] represents the low-resolution input image of the input spatial attention network, CLUT x [index] represents the low-resolution input image of the preset CNN network, SA x Represents spatial attention network, CNN x Represents the preset CNN network, data1, data2,…, data m They represent the pixel values ​​corresponding to the 1, 2, …, m receptive fields of the input low-resolution image.

[0072] Furthermore, in this embodiment, the lookup table FuLUT of each parallel branch is first established through the above-mentioned pre-trained multi-branch fusion processing network. x , since each parallel branch lookup table FuLUT x They also have the same receptive field size and shape, so we can also use the lookup table structure to reparameterize the idea and establish the input index and each parallel branch FuLUT x The mapping relationship of the output summed and averaged results is used to map the FuLUT of each parallel branch. x The fused lookup table is used as the final fused lookup table.

[0073] Optionally, the training process of the pre-trained multi-branch fusion processing network includes:

[0074] Obtaining a training sample image pair; wherein the training sample image pair is provided with a one-to-one correspondence between a low-resolution input sample image and a high-resolution output sample image;

[0075] Input the training sample images into the initial multi-branch fusion processing network for training;

[0076] The initial multi-branch fusion processing network that meets the preset stopping condition is used as the pre-trained multi-branch fusion processing network; the structure of the initial multi-branch fusion processing network is the same as that of the pre-trained multi-branch fusion processing network;

[0077] The preset stopping conditions include: the number of iterative training times is greater than the iteration threshold or the value of the loss function corresponding to the initial multi-branch fusion processing network is continuously less than the loss threshold.

[0078] Optionally, according to the pixel values ​​corresponding to the plurality of low-resolution image sub-blocks to be converted, a table lookup process is performed in a fusion lookup table to obtain a plurality of converted high-resolution image sub-blocks, including:

[0079] According to the pixel values ​​corresponding to the multiple low-resolution image sub-blocks to be converted, a traversal search method is used to perform table lookup processing in the fusion lookup table to obtain the matched multiple converted high-resolution image sub-blocks.

[0080] To demonstrate the effectiveness of the in-memory processing-oriented structural fusion method provided by this invention, a simulation experiment was conducted. The experiment used a pipeline structure, n = 2 branches, uniformly sampled lookup tables, and a receptive field size of 4 for each branch. The memory resource usage and inference time before and after lookup table fusion were compared. Specific data is shown in Table 1.

[0081] Table 1 Comparison of resource usage and inference time before and after lookup table structure fusion (r=2)

[0082] Storage resource overhead Inference time of Set5 dataset Inference time of Set14 dataset Before structural fusion 1.3MB 1.68s 4.52s After structural fusion 0.327MB 0.57s 1.44s

[0083] Here, r represents the super-resolution magnification. The Set5 dataset is a small dataset containing 5 high-resolution images and their corresponding low-resolution versions. The Set14 dataset is slightly larger than the Set5 dataset and contains 14 high-resolution images and their corresponding low-resolution versions.

[0084] The experimental results show that the proposed method has significant advantages in terms of storage resource utilization and model inference time compared with the previous methods: compared with the lookup table structure before fusion, the storage resource usage after fusion is reduced by about The inference speed is increased by about By integrating lookup tables into their structure, this method not only effectively reduces the storage resource overhead of the lookup tables but also eliminates the redundant interpolation calculation paths used in previous methods, reducing memory access times and significantly shortening model inference latency. Experimental data demonstrates that this method combines the advantages of short inference time with low resource requirements, providing a viable hardware solution for the deployment of intelligent edge devices.

[0085] In summary, this invention successfully alleviates the storage resource usage and inference delay problems of the lookup table by integrating innovations in the lookup table structure and introducing a spatial attention mechanism in the training network. It reduces storage overhead while shortening the model inference time, providing new ideas for in-memory processing methods for deep learning.

[0086] The method provided in the embodiment of the present invention can be applied to electronic devices. Specifically, the electronic devices can be desktop computers, portable computers, smart mobile terminals, servers, etc., which are not limited in the embodiment of the present invention.

[0087] Based on the same inventive concept, an embodiment of the present invention further provides a structure fusion device for in-memory processing. Figure 3 A schematic diagram of a structure fusion device for in-memory processing provided by an embodiment of the present invention is shown in FIG. Figure 3 As shown, it includes: an acquisition unit 301 and a table lookup unit 302;

[0088] The acquisition unit 301 is used to: acquire a low-resolution image to be converted;

[0089] The table lookup unit 302 is used to: input the low-resolution image to be converted into a pre-constructed fusion lookup table in accordance with a preset receptive field size and a preset traversal order, and perform table lookup processing to obtain a converted high-resolution image; the fusion lookup table is an overall relationship table that is established for the low-resolution input image according to the preset receptive field size and the high-resolution output image under a multi-branch fusion processing mode using a pre-trained multi-branch fusion processing network; the pre-trained multi-branch fusion processing network is a multi-branch convolutional neural network based on a spatial attention mechanism.

[0090] Figure 4 A schematic diagram of the structure of a device for in-memory processing provided in an embodiment of the present invention includes a processor 410, a storage medium 420, and a bus 430. The storage medium 420 stores machine-readable instructions executable by the processor 410. When the device for in-memory processing operates, the processor 410 communicates with the storage medium 420 via the bus 430, and the processor 410 executes the machine-readable instructions to perform the steps of the above-described method embodiment. The specific implementation methods and technical effects are similar and will not be repeated here.

[0091] The storage medium may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage. Alternatively, the storage medium may be at least one storage device located away from the processor.

[0092] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components.

[0093] It should be noted that the terms "first," "second," and the like are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in sequences other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention.

[0094] In the description of this specification, the reference terms "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" mean that the specific features or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features or characteristics described can be combined in any suitable manner in any one or more embodiments or examples. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification.

[0095] Although the present invention is described herein in conjunction with various embodiments, in the process of implementing the claimed invention, those skilled in the art can understand and implement other variations of the above-mentioned disclosed embodiments by viewing the drawings and the disclosed content. In the description of the present invention, the word "comprising" does not exclude other components or steps, "one" or "an" does not exclude multiple situations, and the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined. In addition, certain measures are recorded in different embodiments, but this does not mean that these measures cannot be combined to produce good results.

[0096] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention cannot be considered to be limited to these descriptions. For those skilled in the art of the present invention, several simple deductions or substitutions can be made without departing from the concept of the present invention, and all of these should be considered to fall within the scope of protection of the present invention.

Claims

1. A structural fusion method for in-memory processing, characterized in that: include: Get the low-resolution image to be converted; Inputting the low-resolution image to be converted into a pre-built fusion lookup table in sequence according to a preset receptive field size and a preset traversal order for table lookup processing to obtain a converted high-resolution image; The fusion lookup table adopts a pre-trained multi-branch fusion processing network. Under the multi-branch fusion processing mode, an overall relationship table is established for the low-resolution input image according to the preset receptive field size and the high-resolution output image. The pre-trained multi-branch fusion processing network is a multi-branch convolutional neural network based on the spatial attention mechanism.

2. The structure fusion method for in-memory processing according to claim 1 is characterized in that: The low-resolution image to be converted is sequentially input into a pre-built fusion lookup table according to a preset receptive field size and a preset traversal order for table lookup processing to obtain a converted high-resolution image, including: Splitting the to-be-converted low-resolution image into a plurality of to-be-converted low-resolution image sub-blocks according to the preset traversal order and the preset receptive field size; According to the pixel values ​​corresponding to the plurality of low-resolution image sub-blocks to be converted, a table lookup process is performed in the fusion lookup table to obtain a plurality of converted high-resolution image sub-blocks; The plurality of converted high-resolution image sub-blocks are spliced ​​together in a splicing order corresponding to the preset traversal order to obtain the converted high-resolution image.

3. The structure fusion method for in-memory processing according to claim 2, characterized in that: The preset traversal order includes: an order from top to bottom and from left to right, an order from top to bottom and from right to left, an order from bottom to top and from left to right, and an order from bottom to top and from right to left.

4. The structure fusion method for in-memory processing according to claim 2, characterized in that: The pre-trained multi-branch fusion processing network includes: multiple preset CNN networks and multiple spatial attention mechanism SA networks; the number of the preset CNN networks is equal to the number of the spatial attention mechanism SA networks; Each of the preset CNN networks is connected in parallel to the spatial attention mechanism SA network, and is interconnected through their respective output ends to form multiple independent branch networks; A plurality of the branch networks are connected in parallel and are interconnected through their respective output ends to form the pre-trained multi-branch fusion processing network.

5. The structure fusion method for in-memory processing according to claim 4 is characterized in that: The construction process of the fusion lookup table includes: Acquire a plurality of low-resolution input images; the low-resolution input images are index blocks obtained by splitting the relationship table construction samples and having the same size as the preset receptive field; the relationship table construction samples are sample images that are consistent with the scene of the low-resolution image to be converted; Inputting the low-resolution input image into the plurality of branch networks respectively to obtain a plurality of branch mapping results; averaging the multiple branch mapping results corresponding to each of the low-resolution input images to obtain multiple high-resolution output images; A one-to-one corresponding index relationship table is constructed for the low-resolution input image and the high-resolution output image to obtain the fusion lookup table.

6. The structure fusion method for in-memory processing according to claim 4, characterized in that: The training process of the pre-trained multi-branch fusion processing network includes: Acquire a training sample image pair; wherein the training sample image pair is provided with a one-to-one correspondence between a low-resolution input sample image and a high-resolution output sample image; Inputting the training sample images into an initial multi-branch fusion processing network for training; The initial multi-branch fusion processing network that meets the preset stopping condition is used as the pre-trained multi-branch fusion processing network; the initial multi-branch fusion processing network has the same structure as the pre-trained multi-branch fusion processing network; The preset stopping condition includes: the number of iterative training times is greater than the iteration threshold or the value of the loss function corresponding to the initial multi-branch fusion processing network is continuously less than the loss threshold.

7. The structure fusion method for in-memory processing according to claim 2, characterized in that: The step of performing a lookup in the fusion lookup table according to pixel values ​​corresponding to the plurality of low-resolution image sub-blocks to be converted, and obtaining a plurality of corresponding converted high-resolution image sub-blocks, comprises: According to the pixel value sizes corresponding to the multiple low-resolution image sub-blocks to be converted, a table lookup process is performed in the fusion lookup table in a traversal search manner to obtain the multiple matched converted high-resolution image sub-blocks.

8. A structural fusion device for in-memory processing, characterized in that: The structure fusion device for in-memory processing includes: an acquisition unit and a table lookup unit; The acquisition unit is used to: acquire the low-resolution image to be converted; The table lookup unit is used to: input the low-resolution image to be converted into a pre-constructed fusion lookup table in sequence according to a preset receptive field size and a preset traversal order for table lookup processing to obtain a converted high-resolution image; the fusion lookup table is an overall relationship table that is established for the low-resolution input image according to the preset receptive field size and the high-resolution output image under a multi-branch fusion processing mode using a pre-trained multi-branch fusion processing network; the pre-trained multi-branch fusion processing network is a multi-branch convolutional neural network based on a spatial attention mechanism.

9. A structural fusion device for in-memory processing, characterized in that: include: A processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor. When the structure fusion device for in-memory processing is running, the processor and the storage medium communicate via the bus, and the processor executes the machine-readable instructions to perform the steps of the structure fusion method for in-memory processing as described in any one of claims 1 to 7.

Images