Retina bionic-driven super-resolution reconstruction model and implementation method thereof

Abstract

The invention discloses a retina bionics-driven super-resolution reconstruction model and an implementation method thereof. The reconstruction model is composed of a shallow extraction and lateral suppression module (LIM) and a deep feature extraction and reconstruction module (RM). The shallow layer extraction simulates the processing process of photoreceptor cells in a biological vision system on optical signals, and low-level feature information is extracted from a low-resolution image so as to realize primary processing of image signals; the side inhibition module LIM simulates a horizontal cell side inhibition mechanism to capture signals, and performs different intensity inhibition on features; the deep feature extraction adopts multi-layer feature fusion and a layered structure, imitates multiple functions of bipolar cells, gradually extracts and fuses feature information of different layers, and improves the naturalness of the whole image; the reconstruction module RM is composed of an up-sampling block and a convolution, simulates ganglion cells, and is responsible for generating a final high-resolution reconstruction image. According to the method, a satisfactory reconstruction result is obtained for an image in a complex scene, and meanwhile it is proved that a retina biological system is feasible to be migrated to an image super-resolution reconstruction technology.

Description

Technical Field

[0001] The invention belongs to the technical field of image reconstruction, and relates to a retinal bionics-driven super-resolution reconstruction model and an implementation method thereof. Background Art

[0002] Single Image Super-resolution Reconstruction (SISR) is a key research direction in the field of computer vision, which aims to reconstruct a high-resolution image with clear details from a low-resolution image to improve image quality and visual perception. Super-resolution reconstruction technology is widely used in medical imaging, public security, satellite remote sensing and other fields.

[0003] Researchers use the powerful feature extraction and nonlinear mapping capabilities of convolutional neural networks (CNNs) to reconstruct images clearly and realistically. The SRCNN model proposed by Dong et al. pioneered the use of deep learning in image super-resolution reconstruction. For details, see "Dong C, Loy CC, He K, and Tang X 2015 Image super-resolution using deep convolutional networks IEEE transactions on pattern analysis and machine intelligence. 38295-307". The VDSR model proposed by Kim et al. uses very deep convolutional networks and introduces residual learning structures to improve the quality of super-resolution reconstruction. For details, see "Kim J, Lee JK, and Lee KM 2016 Accurate image super-resolution using very deep convolutional networks Proceedings of the IEEE conference on computer vision and pattern recognition. pp 1646-1654". Subsequently, Tong et al. designed the SRDenseNet model, which used a residual structure to concatenate the features extracted from dense blocks, which not only alleviated the gradient vanishing problem but also strengthened the information flow. For details, see "Tong T, Li G, Liu X, and Gao Q 2017 Image super-resolution using dense skip connections Proceedings of the IEEE international conference on computer vision. pp 4799-4807". Zhang et al. first introduced the channel attention mechanism to weight the feature channels and proposed the RCAN model, which can more effectively extract and utilize the beneficial features in the image. For details, see "Zhang Y, Li K, Li K, Wang L, Zhong B, and Fu Y 2018 Image super-resolution using very deep residual channel attention networks Proceedings of the European conference on computer vision (ECCV). pp 286-301".The EMASRN network proposed by Zhu et al. uses progressive multi-scale feature extraction blocks and expectation-maximization attention blocks, while taking advantage of the design of feedback network structure to better balance performance and applicability. For details, see "Zhu X, Guo K, Ren S, Hu B, Hu M, and Fang H 2021 Lightweight image super-resolution with expectation-maximization attention mechanism IEEE Transactions on Circuits and Systems for Video Technology. 321273-1284". Meng et al. used convolutional layers of different scales to fully extract spatial information from different regions of the original image, and used channel stripping operations to compress the model. For details, see "Meng Z, Zhang J, Li X, and Zhang L 2022 Lightweight image super-resolution based on local interaction of multi-scale features and global fusion Mathematics. 101096". Chen et al. used various cross-scale information for multi-scale feature fusion, and used local-global spatial attention blocks to enhance basic features and suppress irrelevant information. For details, see "Chen J, Wang W, and Xing FTH2023Multi-feature fusion attention network for single image super-resolutionIET image processing.171389-1402". Han et al. improved network performance through multi-scale recursive attention feature fusion network blocks. For details, see "Han X, Wang L, Wang X, Zhang P, and Xu H 2023A multi-scalerecursive attention feature fusion network for image super-resolutionreconstruction algorithm Sensors.239458".Liu et al. proposed an improved asymmetric Laplacian pyramid network by making full use of various size features. For details, see "Liu X, Qiao S, Zhang T, Zhao C, and Yao X2024Single-image super-resolution based on an improved asymmetric Laplacianpyramid structure Digital Signal Processing.145104321". Cai et al. proposed a multi-branch feature multiplexing fusion network and designed a mixed multi-layer attention mechanism to achieve multi-use of features and multi-level fusion of features at different levels. For details, see "Cai Y, Gao G, Jia Z, and Lai H 2022Image reconstruction of multibranchfeature multiplexing fusion network with mixedmultilayer attention RemoteSensing.142029". Chen et al. designed a multi-level information compensation model to compensate for the information lost during channel compression and the information that is difficult to recover during the channel recovery process of Unet-like networks, thereby realizing multi-level feature extraction and application. For details, see "Chen Y, Xia R, Yang K, and Zou K 2024 MICU: Image super-resolution via multi-level information compensation and U-net Expert Systems with Applications. 245123111". The above deep learning models have achieved significant performance improvements in the field of image reconstruction, but these models still have certain limitations in generalization ability and model optimization when facing complex and changeable real-world problems. With the vigorous development of bionics, its application in deep learning provides new ideas for overcoming these limitations. With its unique perspective, bionics draws inspiration from biological systems, which not only optimizes the efficiency and adaptability of the model, but also greatly improves the model's ability to simulate and solve complex real-world problems.

[0004] With the help of Hawkeye algorithm, Zhang et al. combined obstacle avoidance and target tracking by visual segmentation and proposed a bionic dynamic path planning algorithm for micro UAV based on filtering and Hawkeye vision fusion. For details, see "Zhang Z, Wang S, Chen J, and Han Y 2023A bionic dynamic path planning algorithm of the micro UAVbased on the fusion of deep neural network optimization / filtering and hawk-eye vision IEEE Transactions on Systems, Man, and Cybernetics: Systems. 533728-3740". Palekar et al. reduced the feature dimension by selecting features based on the adaptive red colobus bionic model and used the hunger game bionic model, which was conducive to the simplification of the loss function and the updating of the weight parameters. For details, see "Palekar V 2024Adaptive optimized residual convolutional image annotation model with bionic feature selection model Computer Standards and Interfaces. 87103780". Lin et al. designed a pre-enhancement network by simulating the information transmission mechanism of the lateral geniculate nucleus of the retina, thereby enhancing the ability of the encoding network to extract details and local features. For details, see "Lin C, Zhang Z, and Hu Y 2022 Bio-inspired feature enhancement network for edge detection Applied Intelligence. 5211027-11042". Wang et al. studied two receptive field models of retinal neurons, six dual-mode cell imaging mechanisms of rattlesnakes and their mathematical models and PCNN models, and achieved effective fusion of infrared and visible light images. For details, see "Wang Q, Yan X, Xie W, and Wang Y 2024 Image Fusion Method Based on Snake Visual Imaging Mechanism and PCNN Sensors. 243077".The above literature shows that combining biological systems with deep learning helps explain the decision-making process of the model, enables the network to perform effective data enhancement, better process local details and global contextual information, achieve feature extraction and learning strategy improvement to adapt to complex scenarios, and can continuously evolve and optimize.

[0005] The inventors have found that the existing single-image super-resolution reconstruction method based on deep learning has poor reconstruction effect and has the following main defects: (1) the network runs slowly; (2) the edge detail information of the reconstructed image is not obvious; (3) the contrast of the reconstructed image is poor. Summary of the invention

[0006] In order to solve the above problems, the present invention provides a retinal bionics-driven super-resolution reconstruction model, which imitates the structure and function design modules of retinal neuron cells, provides new ideas for network construction, effectively improves the network's performance in retaining image details and enhancing perceptual quality, and improves the network's ability to process complex image information.

[0007] The second object of the present invention is to provide a retinal bionics driven super-resolution reconstruction model and its implementation method.

[0008] A third object of the present invention is to provide an electronic device.

[0009] A fourth object of the present invention is to provide a computer storage medium.

[0010] The technical solution adopted by the present invention is a retinal bionics-driven super-resolution reconstruction model and its implementation method, including shallow layer extraction, lateral inhibition module (LIM), deep layer feature extraction and reconstruction module (RM);

[0011] The shallow extraction simulates the process of photoreceptor cells in the biological visual system processing light signals, extracts low-level feature information from low-resolution images, and realizes preliminary processing of image signals.

[0012] The lateral inhibition module LIM simulates the lateral inhibition mechanism of horizontal cells to capture signals and suppress features with different intensities. The core function of the lateral inhibition module is to simulate the lateral inhibition mechanism of horizontal cells. It uses 3×3 convolution to represent the dendrites of horizontal cells, captures and integrates local signals from neighboring photoreceptor cells, and then uses the Sigmoid function to convert this information into an inhibition coefficient to simulate the inhibition strength of horizontal cells on surrounding cells. Finally, the inhibition coefficient is multiplied element by element with the original input to achieve the lateral inhibition effect on the local area.

[0013] The deep feature extraction adopts multi-layer feature fusion and hierarchical structure, imitating the multiple functions of bipolar cells, gradually extracting and fusing feature information at different levels to improve the naturalness of the overall image;

[0014] The reconstruction module RM, simulating ganglion cells, is responsible for generating the final high-resolution reconstructed image.

[0015] The hierarchical structure of the deep feature extraction is composed of a hierarchical fusion network HFN, a 3×3 convolution and a residual structure, which realizes the integration and enhancement of features and improves the naturalness of the overall image;

[0016] The hierarchical fusion network HFN includes two parts: signal enhancement and information fusion;

[0017] The signal enhancement part uses 6 enhanced hierarchical feature fusion blocks (EHFFBs), the output of the previous layer is propagated to the next layer layer by layer to form a dense connection, and high-frequency features at different levels are gradually extracted;

[0018] The information fusion part uses concat connection and residual connection to connect the high-frequency features extracted by the enhanced hierarchical feature fusion block EHFFB to enhance the feature extraction and fusion capabilities;

[0019] The multi-layer feature fusion of the deep feature extraction is responsible for extracting spatial context extension features, performing multi-level fusion of features and retaining enhanced structural information through the enhanced hierarchical feature fusion block EHFFB, and optimizing the contribution of feature channels through the channel attention mechanism, thereby effectively improving the quality and details of the reconstructed image;

[0020] The enhanced hierarchical feature fusion block EHFFB is composed of five dilated convolutions with a convolution kernel size of 3×3 and dilation rates of 1, 1, 1, 2, and 2, respectively, to capture spatial features at different levels, and the number of channels of each convolution kernel remains unchanged when outputting;

[0021] The five convolution kernels in the EHFFB are fused with the same dilation rate through Concat, and the 1×1 convolution and ECA are used to achieve effective fusion and enhancement of features, thereby improving the representation ability of the model;

[0022] The reconstruction module RM simulates ganglion cells and is responsible for generating the final high-resolution reconstructed image;

[0023] The reconstruction module RM consists of different numbers of upsampling blocks and one 3×3 convolution to achieve image reconstruction effects with different magnifications;

[0024] The upsampling block consists of one 3×3 convolution and one sub-pixel convolution, simulating the highly dense connections between ganglion cells and pyramidal cells in the fovea region to respond to different visual stimuli.

[0025] A retinal bionics-driven super-resolution reconstruction model and its implementation method are reconstructed according to the following steps:

[0026] S1, extract low-level feature information from low-resolution images to achieve preliminary processing of image signals;

[0027] S2, lateral inhibition module LIM receives shallow layer extraction information, integrates the information of local area cells, expands the receptive field, and enhances the contour contrast of the visual image;

[0028] The side inhibition module performs the inhibition feedback according to the following formula:

[0029] X 0 =Sigmoid(C 3×3 (X′))*X′

[0030] Among them, X′ represents the output of shallow feature extraction, C 3×3 represents a convolution kernel of size 3×3, * represents element-by-element multiplication, X 0 Represents the output result of the lateral inhibition module;

[0031] S3, deep extraction extracts multiple information through enhanced hierarchical feature fusion block EHFFB for enhanced processing, so as to extract features at different levels and fuse and filter them. The hierarchical fusion network HFN simultaneously captures the global structural information and local details of the image and integrates them in itself, providing high-quality feature input for subsequent image reconstruction;

[0032] S4, the reconstruction module uses the low-level features extracted from the shallow layer and the high-level features extracted from the deep layer to perform super-resolution reconstruction on the low-resolution image according to the integrated feature map;

[0033] The reconstruction module is reconstructed according to the following formula:

[0034] Y=C 3×3 (f PixelShuffle (C 3×3 (X out )))

[0035] Among them, f PixelShuffle is a sub-pixel convolution upsampling operation, and Y represents the reconstructed output result.

[0036] Furthermore, in the deep feature extraction of S3, the enhanced hierarchical feature fusion block EHFFB performs feature information enhancement at different levels according to the following formula:

[0037]

[0038] F 2=ReLU(C 3×3 (F 1 ))

[0039] F 3 =ReLU(C 3×3 (F 2 +X i-1 ))

[0040] F 4 =C 1×1 ({F 2 ,F 3})

[0041]

[0042] F 6 =C 1×1 ({F 1 ,F 5})

[0043] F 7 =ReLU(C 3×3 (F 6 ))

[0044] F 8 =C 1×1 ({F 2 ,F 3 ,F 7})

[0045] X i =ECA(F 8 )+X i-1

[0046] Among them, F k represents the output of the kth (k = {1, 2, 3, 4, 5, 6, 7, 8}) convolution kernel, C 3×3 represents the convolution kernel of size 3×3, C 2 3×3 indicates that the convolution kernel size is 3×3, the dilation rate is 2, {·} indicates the concatenation operation, and X i-1 Is the input of the i-th (i=1, ..., n) EHFFB module. i It is the output of the i-th (i=1, ..., n) EHFFB module.

[0047] Furthermore, in the deep feature extraction of S3, the hierarchical fusion network HFN performs multi-level information fusion according to the following formula:

[0048] X 1 =f EHFFB (X 0 )

[0049] X n =f EHFFB (X n-1 )

[0050] X n+1 =f EHFFB (C 1×1 {X 1 ,…,X n})

[0051] X out =C 3×3 (X n+1 )+X′

[0052] Among them, X 0 represents the output of the lateral inhibition module, f EHFFB represents feature extraction through enhanced hierarchical feature fusion blocks, X n represents the feature output extracted by the mth (m = {1, 2, ..., m, m + 1}) EHFFB module, {·} is the cascade operation, C 1×1 represents a convolution kernel of size 1×1, C 3×3 represents a convolution kernel of size 3×3, X′ is the shallow feature output result, and X out It is the output result of HFN module.

[0053] An electronic device adopts the above method to realize image reconstruction.

[0054] A computer storage medium stores at least one program instruction, and the at least one program instruction is loaded and executed by a processor to implement the above-mentioned image reconstruction method.

[0055] The beneficial effects of the present invention are:

[0056] In order to solve the limitations of poor restoration of texture detail information in image reconstruction, generalization ability and model optimization, a retinal bionics-driven super-resolution reconstruction model RBSRNet is proposed, inspired by the neuronal hierarchical structure and information processing mechanism in the retina. Through the lateral inhibition module, the contrast of local texture is enhanced during feature extraction, and then 6 hierarchical fusion networks are constructed to capture and fuse the global structural information and local detail information of the image at multiple levels. Then, an enhanced hierarchical feature fusion block is designed in each hierarchical network, which helps to extract spatial context extension features, perform multi-level fusion of features, and strengthen the retention of structural information. The contribution of feature channels is optimized through the channel attention mechanism, thereby improving the understanding and representation of image content. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0058] Figure 1 It is a structural schematic diagram of a reconstruction model according to an embodiment of the present invention.

[0059] Figure 2 It is a structural schematic diagram of the lateral inhibition module in the reconstruction model of an embodiment of the present invention.

[0060] Figure 3 It is a structural diagram of an enhanced hierarchical feature fusion block EHFFB in the reconstruction model of an embodiment of the present invention.

[0061] Figure 4 It is a schematic diagram of the structure of a hierarchical fusion network HFN in the reconstruction model of an embodiment of the present invention.

[0062] Figure 5 It is a schematic diagram of the structure of a reconstruction module in a reconstruction model according to an embodiment of the present invention.

[0063] Figure 6 It is a visualization diagram of the side suppression module of the reconstruction method according to an embodiment of the present invention.

[0064] Figure 7 3 is a comparison chart of the reconstruction effect of the reconstruction method of an embodiment of the present invention and other algorithms. DETAILED DESCRIPTION

[0065] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0066] Embodiment 1,

[0067] A super-resolution reconstruction model driven by retinal bionics, the structure of which is as follows Figure 1 As shown, it includes shallow layer extraction, lateral inhibition module (LIM), deep feature extraction and reconstruction module (RM) connected in sequence;

[0068] Among them, shallow extraction uses a 3×3 convolution kernel to extract features from low-resolution images.

[0069] The lateral inhibition module LIM simulates the topological structure of horizontal cells in the retina, such as Figure 2 As shown in Figure 1, local information in the input image or feature map is captured. Subsequently, the Sigmoid function is used to convert this information into an inhibition coefficient, simulating the inhibition strength of horizontal cells on surrounding cells.

[0070] Deep feature extraction, by designing a hierarchical structure, obtains feature responses of different modules and different depths. The feature information of the receiving side suppression module is designed to enhance the processing of multiple information, so as to extract features at different levels and fuse and filter them. Then, 6 EHFFBs are used to design a hierarchical fusion network (HFN), and the enhanced features are connected by splicing and residual connection to enhance the feature extraction and fusion capabilities. Deep feature extraction simulates the role of bipolar cells in signal relay and processing, providing high-quality feature input for subsequent image reconstruction.

[0071] like Figure 3 As shown in the figure, in EHFFB, dilated convolution is used to process signals, and the network's perception of objects of different scales in the image is improved by increasing the receptive field, so as to simulate the bipolar cells dividing visual signals into two categories: light on (ON) and light off (OFF). Multi-layer convolution operations gradually extract feature information at different levels, simulating the function of bipolar cells to gradually extract and integrate features in visual signal processing. The outputs of these layers, after being activated by ReLU, help to strengthen the edge and contour information in the image. By performing the Concat operation on the features, the network can fuse multiple types of features at different levels. This multi-level fusion helps the network retain details and information at different levels when reconstructing the image. The fused features are further processed and integrated by 1×1 convolution, which is similar to the weight adjustment in biological synapses. The ECA layer performs weighted processing on the features, which is similar to the selective enhancement of signal features by bipolar cells. The residual connection retains the original signal information while enhancing the feature representation, which is similar to the feature enhancement of bipolar cells on the basis of retaining the original information.

[0072] like Figure 4 As shown in the figure, HFN consists of 6 levels of processing units EHFFB, each of which processes image features at a different level of abstraction. This multi-level design enables the module to capture the global structural information and local detail information of the image at the same time, thereby improving the understanding and representation of image content. The stacking of multiple EHFFB blocks further simulates the multi-level feature extraction and processing process of bipolar cells. By learning the association and interaction between features, this connection method enables the transmission and communication of feature information at different levels, similar to the diversity of bipolar cell connection methods to adapt to different lighting conditions. The use of the Concat operation promotes the communication and integration of features at different levels, similar to the integration of signals from multiple photoreceptor cells by bipolar cells.

[0073] The reconstruction module is used to reconstruct the integrated extracted features and restore the contour and detail features of the low-resolution image. Figure 5 As shown in FIG, the reconstruction module first uses a convolution kernel of size 3×3 for basic reconstruction, and then uses a convolution kernel of size 3×3 for deep reconstruction after upsampling to obtain a reconstructed image.

[0074] The design of network models based on deep learning is based on the application background. For example, image classification and image segmentation focus on the analysis and understanding of image content. In model design, they tend to focus on the recognition of targets, and separate target information from the whole for identification and classification. Image super-resolution reconstruction focuses on the nonlinear mapping relationship between low-resolution images and high-resolution images. By extracting each feature information in the image, the original weak contour features are enhanced, and the details and textures are improved. Using low-resolution images to infer all missing high-frequency details is the key to reconstruction. In order to fully extract feature information from low-resolution images and minimize high-frequency details, the embodiment of the present invention proposes a retinal bionics-driven super-resolution reconstruction model and its implementation method. The method uses a 3×3 convolution kernel to extract features from low-resolution images. The lateral inhibition module LIM simulates the topological structure of horizontal cells in the retina and captures local information in the input image or feature map. Subsequently, the Sigmoid function is used to convert this information into an inhibition coefficient, simulating the inhibition strength of horizontal cells on surrounding cells. Deep feature extraction obtains feature responses of different modules and different depths by designing a hierarchical structure. The feature information of the receiving side suppression module is used to design an enhanced hierarchical feature fusion block (EHFFB) to enhance the processing of multiple information, so as to extract features at different levels and fuse and filter them. Six EHFFBs are then used to design a hierarchical fusion network (HFN), and the enhanced features are concatenated and connected with residuals to enhance the ability of feature extraction and fusion. Deep feature extraction simulates the role of bipolar cells in signal relay and processing, providing high-quality feature input for subsequent image reconstruction. In the EHFFB, dilated convolution is used to process the signal, and the network's perception of targets of different scales in the image is improved by increasing the receptive field to simulate the bipolar cells dividing the visual signals into two categories: light on (ON) and light off (OFF). Multi-layer convolution operations gradually extract feature information at different levels, simulating the function of bipolar cells to gradually extract and integrate features in visual signal processing. The outputs of these layers, after being activated by ReLU, help to strengthen the edge and contour information in the image. By performing the Concat operation on the features, the network can fuse multiple types of features at different levels. This multi-level fusion helps the network retain details and information at different levels when reconstructing the image. The fused features are further processed and integrated through 1×1 convolution, which is similar to the weight adjustment in biological synapses. The ECA layer performs weighted processing on the features, which is similar to the selective enhancement of signal features by bipolar cells. The residual connection retains the original signal information while enhancing the feature representation, which is similar to the feature enhancement of bipolar cells while retaining the original information. HFN consists of 6 levels of processing units EHFFB, each of which processes image features at different levels of abstraction.This multi-level design enables the module to capture both the global structural information and local detail information of the image, thereby improving the understanding and representation of the image content. The stacking of multiple EHFFB blocks further simulates the process of multi-level feature extraction and processing of bipolar cells. By learning the association and interaction between features, this connection method enables the transmission and communication of feature information at different levels, similar to the diversity of bipolar cell connection methods to adapt to different lighting conditions. The Concat operation is then used to promote the communication and integration of features at different levels, similar to the integration of signals from multiple photoreceptor cells by bipolar cells. The reconstruction module is used to reconstruct the integrated extracted features to restore the contour and detail features of the low-resolution image. The reconstruction module first uses a convolution kernel of size 3×3 for basic reconstruction, and then uses a convolution kernel of size 3×3 for deep reconstruction after upsampling to obtain a reconstructed image.

[0075] Embodiment 2,

[0076] A retinal bionics-driven super-resolution reconstruction model and its implementation method are reconstructed according to the following steps:

[0077] S1, extract low-level feature information from low-resolution images to achieve preliminary processing of image signals;

[0078] S2, lateral inhibition module LIM receives shallow layer extraction information, integrates the information of local area cells, expands the receptive field, and enhances the contour contrast of the visual image;

[0079] The side inhibition module performs the inhibition feedback according to the following formula:

[0080] X 0 =Sigmoid(C 3×3 (X′))*X′

[0081] Among them, X′ represents the output of shallow feature extraction, C 3×3 represents a convolution kernel of size 3×3, * represents element-by-element multiplication, X 0 Represents the output result of the lateral inhibition module;

[0082] S3, deep extraction extracts multiple information through enhanced hierarchical feature fusion block EHFFB for enhanced processing, so as to extract features at different levels and fuse and filter them. The hierarchical fusion network HFN simultaneously captures the global structural information and local details of the image and integrates them in itself, providing high-quality feature input for subsequent image reconstruction;

[0083] S4, the reconstruction module uses the low-level features extracted from the shallow layer and the high-level features extracted from the deep layer to perform super-resolution reconstruction on the low-resolution image according to the integrated feature map;

[0084] The reconstruction module is reconstructed according to the following formula:

[0085] Y=C 3×3 (f PixelShuffle (C 3×3 (X out )))

[0086] Among them, f PixelShuffle is a sub-pixel convolution upsampling operation, and Y represents the reconstructed output result.

[0087] Furthermore, in the deep feature extraction of S3, the enhanced hierarchical feature fusion block EHFFB performs feature information enhancement at different levels according to the following formula:

[0088]

[0089] F 2 =ReLU(C 3×3 (F 1 ))

[0090] F 3 =ReLU(C 3×3 (F 2 +X i-1 ))

[0091] F 4 =C 1×1 ({F 2 ,F 3})

[0092]

[0093] F 6 =C 1×1 ({F 1 ,F 5})

[0094] F 7 =ReLU(C 3×3 (F 6 ))

[0095] F 8 =C 1×1 ({F 2 ,F 3 ,F 7})

[0096] X i =ECA(F 8 )+X i-1

[0097] Among them, F krepresents the output of the kth (k = {1, 2, 3, 4, 5, 6, 7, 8}) convolution kernel, C 3×3 represents the convolution kernel of size 3×3, C 2 3×3 indicates that the convolution kernel size is 3×3, the dilation rate is 2, {·} indicates the concatenation operation, and X i-1 Is the input of the i-th (i=1, ..., n) EHFFB module. i It is the output of the i-th (i=1, ..., n) EHFFB module.

[0098] Furthermore, in the deep feature extraction of S3, the hierarchical fusion network HFN performs multi-level information fusion according to the following formula:

[0099] X 1 =f EHFFB (X 0 )

[0100] X n =f EHFFB (X n-1 )

[0101] X n+1 =f EHFFB (C 1×1 {X 1 ,…,X n})

[0102] X out =C 3×3 (X n+1 )+X′

[0103] Among them, X 0 represents the output of the lateral inhibition module, f EHFFB represents feature extraction through enhanced hierarchical feature fusion blocks, X n represents the feature output extracted by the mth (m = {1, 2, ..., m, m + 1}) EHFFB module, {·} is the cascade operation, C 1×1 represents a convolution kernel of size 1×1, C 3×3 represents a convolution kernel of size 3×3, X′ is the shallow feature output result, and X out It is the output result of HFN module.

[0104] The ultimate goal of constructing a super-resolution reconstruction network is to construct X and X HR For a given training data set X and X are established by the following formula HR The mapping relationship between The value reaches the minimum:

[0105]

[0106] Where θ={W 1 ,W 2 ,W 3 ...W m ,b 1 ,b 2 ,b 3 ...b m}, represents the weights and biases of the m-layer neural network, represents the least squares solution of θ, F θ () represents the mapping relationship after adding the weights and biases of the m-layer neural network, X i and Respectively represent the low-resolution image and high-resolution image of the i-th pair of training sets, Represents the function used to minimize Y i and The loss function of the difference between the two; this mapping is the upsampling operation that maps the X feature to X HR feature.

[0107] In order to avoid introducing unnecessary training techniques and speed up convergence, we finally choose L 1 Loss function. The loss function is defined as:

[0108]

[0109] Among them, |||| 1 Represents solving the 1-norm operation.

[0110] Use the training set to train the super-resolution reconstruction model based on the retinal bionics principle to minimize the loss and find The optimal parameters of .

[0111] In order to verify the effectiveness of the retinal bionics-driven super-resolution reconstruction model and its implementation method of the embodiment of the present invention, Set5, Set14, Urban100, and BSD100 were selected as test sets, and Keys' algorithm (R.Keys, Cubic convolution interpolation for digital image processing, in IEEE Transactions on Acoustics, Speech, and Signal Processing, vol.29, December 1981, pp.1153-1160.); Dong's algorithm (Dong C, Loy CC, He K, and Tang X 2015 Image super-resolution using deep convolutional networks IEEE transactions on pattern analysis and machine intelligence. 38295-307); Kim's algorithm (Kim J, Lee JK, and Lee KM 2016 Accurate image super-resolution using very deep convolutional networks Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1646-1654); Ketsoi's algorithm (Ketsoi V, RazaM, Chen H, and Yang heterogeneous group CNN for image super-resolutionIEEE transactions on neural networks and learning systems.356507-6519); Zhang's algorithm (Zhang Y, Bai H, Bing Y, and Liang X 2023Lightweight image super-resolution with convnext residual network Neural Processing Letters.559545-9561) and the experimental results of the present invention were compared and analyzed in both subjective and objective aspects. .

[0112] like Figure 6 As shown, a visualization diagram of the lateral inhibition module in the retinal bionics-driven super-resolution reconstruction model and its implementation method provided in an embodiment of the present invention is shown. In order to intuitively understand the role of LIM, the visualization results of its three key layers are displayed: the convolution layer (Conv), the convolution layer with the Sigmoid activation function (Conv+Sigmoid), and the output layer (Output) of LIM. By observing the feature map Conv_image extracted after the convolution of the original image and the output feature map Output_image of LIM, it can be intuitively felt that LIM significantly enhances the contour contrast of the visual image, making the edges and textures in the image more prominent.

[0113] like Figure 7 As shown in the figure, the retinal bionics-driven super-resolution reconstruction model and its implementation method provided by the embodiment of the present invention are compared with other algorithms for image reconstruction experimental effects, and the images are partially compared. Figure 7 (a) is the original image, Figure 7 (b) is the reconstruction result of Keys' bicubic method. Figure 7 (c) is the reconstruction result image of Dong’s SRCNN method. Figure 7 (d) is the reconstruction result image of Kim’s SRGAN method. Figure 7 (e) is the reconstruction result of Ketsoi’s IMDN method. Figure 7 (f) is the reconstruction result of Tian’s RFDN method. Figure 7 (g) is the reconstruction result of Zhang’s EMASRN method. Figure 7(h) is a diagram of the reconstruction result of the method of the embodiment of the present invention. It can be seen from the locally enlarged picture that RBSRNet can recover more high-frequency details and more effectively retain the structural information of the original image compared with SRCNN and CARN. In addition, RBSRNet performs well in suppressing noise, and can generate less distorted and cleaner super-resolution images compared with SREFBN and IRN. In comparison with HGSRCNN, both can complete the task well in restoring the letters "i" and "t", but RBSRNet can better restore the edge and texture information of "w" in the image, maintaining the sharpness and naturalness of the overall image. Therefore, the method of the embodiment of the present invention can better restore texture detail information when reconstructing an image.

[0114] In order to avoid the deviation caused by qualitative analysis, this example uses two objective indicators, peak signal-to-noise ratio (PSNR) and structural similarity (SSIM), for quantitative evaluation. The reconstruction and restoration comparisons with three different magnifications of 2, 3, and 4 times are performed on four test data sets, as shown in Table 1, Table 2, and Table 3 respectively:

[0115] Table 1 Comparison of PSNR / SSIM results of different methods on different datasets at ×2 magnification

[0116]

[0117] Table 2 Comparison of PSNR / SSIM results of different methods on different datasets at ×3 magnification

[0118]

[0119] Table 3 Comparison of PSNR / SSIM results of different methods on different datasets at ×4 magnification

[0120]

[0121] As can be seen from the three tables, for different scaling factors of ×2, ×3 and ×4, the RBSRNet method achieves higher PSNR and SSIM on the four datasets. The effect is the highest on the Urban 100 dataset, with PSNR and SSIM increasing by 0.31dB / 0.0028, 0.18dB / 0.0033 and 0.17dB / 0.0048 respectively compared to IRN, which ranks second in reconstruction effect.

[0122] From the data in Tables 1, 2 and 3, it can be seen that for PSNR and SSIM, the higher the value, the more similar the result is to the real image, and the higher the image quality. Therefore, the method of the embodiment of the present invention has a significant improvement in the peak signal-to-noise ratio and structural similarity of the reconstructed image, and the visual quality of the reconstructed image is improved, and the detailed features are richer.

[0123] If the image reconstruction method described in the embodiment of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the image reconstruction method described in the embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks or optical disks.

[0124] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. Retinal bionics-driven super-resolution reconstruction model, characterized by: It consists of four parts: shallow layer extraction, lateral inhibition module (LIM), deep feature extraction and reconstruction module (RM); The shallow extraction simulates the process of photoreceptor cells in the biological visual system processing light signals, extracts low-level feature information from low-resolution images, and realizes preliminary processing of image signals. The lateral inhibition module LIM simulates the lateral inhibition mechanism of horizontal cells to capture signals and suppress features with different intensities. The core function of the lateral inhibition module is to simulate the lateral inhibition mechanism of horizontal cells. It uses 3×3 convolution to represent the dendrites of horizontal cells, captures and integrates local signals from neighboring photoreceptor cells, and then uses the Sigmoid function to convert this information into an inhibition coefficient to simulate the inhibition strength of horizontal cells on surrounding cells. Finally, the inhibition coefficient is multiplied element by element with the original input to achieve the lateral inhibition effect on the local area. The deep feature extraction adopts multi-layer feature fusion and hierarchical structure, imitating the multiple functions of bipolar cells, gradually extracting and fusing feature information at different levels to improve the naturalness of the overall image; The hierarchical structure of the deep feature extraction is composed of a hierarchical fusion network HFN, a 3×3 convolution and a residual structure, which realizes the integration and enhancement of features and improves the naturalness of the overall image; The hierarchical fusion network HFN includes two parts: signal enhancement and information fusion; The signal enhancement part uses 6 enhanced hierarchical feature fusion blocks (EHFFBs), the output of the previous layer is propagated to the next layer layer by layer to form a dense connection, and high-frequency features at different levels are gradually extracted; The information fusion part uses concat connection and residual connection to connect the high-frequency features extracted by the enhanced hierarchical feature fusion block EHFFB to enhance the feature extraction and fusion capabilities; The reconstruction module RM simulates ganglion cells and is responsible for generating the final high-resolution reconstructed image; The reconstruction module RM consists of different numbers of upsampling blocks and one 3×3 convolution to achieve image reconstruction effects with different magnifications; The upsampling block consists of one 3×3 convolution and one sub-pixel convolution, simulating the highly dense connections between ganglion cells and pyramidal cells in the fovea region to respond to different visual stimuli.

2. The retinal bionics-driven super-resolution reconstruction model according to claim 1, characterized in that: The multi-layer feature fusion of the deep feature extraction is responsible for extracting spatial context extension features, performing multi-level fusion of features and retaining enhanced structural information through the enhanced hierarchical feature fusion block EHFFB, and optimizing the contribution of feature channels through the channel attention mechanism, thereby effectively improving the quality and details of the reconstructed image; The enhanced hierarchical feature fusion block EHFFB is composed of five dilated convolutions with a convolution kernel size of 3×3 and dilation rates of 1, 1, 1, 2, and 2, respectively, to capture spatial features at different levels, and the number of channels of each convolution kernel remains unchanged when outputting; The five convolution kernels in the EHFFB and the convolutions with the same void ratio are fused through Concat, and the 1×1 convolution and ECA are used to realize the effective fusion and enhancement of features, thereby improving the representation ability of the model.

3. A method for reconstructing a super-resolution reconstruction model driven by retinal bionics, characterized in that: Follow these steps: S1, extract low-level feature information from low-resolution images to achieve preliminary processing of image signals; S2, lateral inhibition module LIM receives shallow layer extraction information, integrates the information of local area cells, expands the receptive field, and enhances the contour contrast of the visual image; The side inhibition module performs the inhibition feedback according to the following formula: X0=Sigmoid(C 3×3 (X′))*X′ Among them, X′ represents the output of shallow feature extraction, C 3×3 represents a convolution kernel of size 3×3, * represents element-by-element multiplication, and X0 represents the output result of the side inhibition module; S3, deep extraction extracts multiple information through enhanced hierarchical feature fusion block EHFFB for enhanced processing, so as to extract features at different levels and fuse and filter them. The hierarchical fusion network HFN simultaneously captures the global structural information and local details of the image and integrates them in itself, providing high-quality feature input for subsequent image reconstruction; S4, the reconstruction module uses the low-level features extracted from the shallow layer and the high-level features extracted from the deep layer to perform super-resolution reconstruction on the low-resolution image according to the integrated feature map; The reconstruction module is reconstructed according to the following formula: Y=C 3×3 (f PixelShuffle (C 3×3 (X out ))) Among them, f PixelShuffle is a sub-pixel convolution upsampling operation, and Y represents the reconstructed output result.

4. The reconstruction method of the retinal bionics-driven super-resolution reconstruction model according to claim 3, characterized in that: In the deep feature extraction of S3, the enhanced hierarchical feature fusion block EHFFB performs feature information enhancement at different levels according to the following formula: F2=ReLU(C 3×3 (F1)) <h2 style=";text-align:left;direction:ltr">F3 = ReLU(C<h2 style=";text-align:left;direction:ltr"> 3×3 <h2 style=";text-align:left;direction:ltr"> (F2+X<h2 style=";text-align:left;direction:ltr"> i-1 <h2 style=";text-align:left;direction:ltr"> )) F4=C 1×1 ({F2,F3}) F6=C 1×1 ({F1,F5}) F7=ReLU(C 3×3 (F6)) <h2 style=";text-align:left;direction:ltr">F8=C<h2 style=";text-align:left;direction:ltr"> 1×1 <h2 style=";text-align:left;direction:ltr"> ({F2,F3,F7}) X i =ECA(F8)+X i-1 Among them, F k represents the output of the kth (k = {1, 2, 3, 4, 5, 6, 7, 8}) convolution kernel, C 3×3 represents the convolution kernel of size 3×3, C 2 3×3 indicates that the convolution kernel size is 3×3, the dilation rate is 2, {·} indicates the concatenation operation, and X i-1 is the input of the ith (i=1, ..., n) EHFFB module, X i It is the output of the i-th (i=1, ..., n) EHFFB module.

5. The reconstruction method of the retinal bionics-driven super-resolution reconstruction model according to claim 3, characterized in that: In the deep feature extraction of S3, the hierarchical fusion network HFN performs multi-level information fusion according to the following formula: X1=f EHFFB (X0) X n =f EHFFB (X n-1 ) X n+1 =f EHFFB (C 1×1 {X1,…,X n }) X out =C 3×3 (X n+1 )+X′ Among them, X0 represents the output result of the lateral inhibition module, f EHFFB represents feature extraction through enhanced hierarchical feature fusion blocks, X n represents the feature output extracted by the mth (m = {1, 2, ..., m, m + 1}) EHFFB module, {·} is the cascade operation, C 1×1 represents a convolution kernel of size 1×1, C 3×3 represents a convolution kernel of size 3×3, X′ is the shallow feature output result, and X out It is the output result of HFN module.

6. An electronic device, characterized in that: Image reconstruction is achieved by using the method described in any one of claims 3 to 5.

7. A computer storage medium, characterized in that: The storage medium stores at least one program instruction, and the at least one program instruction is loaded and executed by the processor to implement the image reconstruction method according to any one of claims 3 to 5.

Images