A light-weighted stereoscopic image super-resolution reconstruction method and device

Abstract

The application discloses a kind of light stereoscopic image super-resolution reconstruction method and device, adaptive pruning strategy is proposed, method includes: the teacher network is constructed based on the importance estimation gate unit, the residual block in corresponding gate unit and mutual attention mechanism are pruned, and the optimized teacher network is obtained to construct student network;Cross-view distillation mechanism is proposed, by constructing cross-view hierarchical attention module, the correlation between the left-view multi-level features in the optimized teacher network and the right-view multi-level features is explored, and the correlation between the left-view multi-level features in the student network and the right-view multi-level features is explored, respectively to obtain correlation matrix;Using root mean square error loss function to measure the difference between correlation matrix, providing additional supervision signal for student network, obtaining light student network;Only light student network is used in inference stage, and high-resolution stereoscopic image is reconstructed.The device includes: processor and memory.

Description

Technical Field

[0001] This invention relates to the fields of image processing and deep learning, and in particular to a lightweight method and apparatus for super-resolution reconstruction of stereo images based on adaptive pruning and cross-viewpoint distillation. Background Technology

[0002] In recent years, stereo image super-resolution reconstruction, as a key technology for obtaining high-quality, high-resolution stereo images, has gradually become a research hotspot in academia. This technology aims to reconstruct high-resolution stereo images from low-resolution stereo images captured by binocular cameras. Currently, stereo image super-resolution reconstruction plays an important role in many 3D edge devices and applications, such as smartphones, smart glasses, and tablets. However, existing stereo image super-resolution reconstruction methods mainly focus on the quality of the reconstructed image, neglecting the high computational complexity of deep learning models, thus resulting in high computational costs. Therefore, how to research lightweight stereo image super-resolution reconstruction methods tailored to the characteristics of stereo images to achieve a balance between computational complexity and performance remains to be further explored.

[0003] Early research focused on single-image super-resolution reconstruction methods, which convert low-resolution input images into high-resolution images using pre-defined mathematical models. Typical methods include interpolation-based and sparse representation-based methods, such as linear interpolation, bilinear interpolation, and sparse learning dictionaries. Compared to traditional methods, deep learning has demonstrated superior performance in single-image super-resolution reconstruction. For example, Dai et al. designed a second-order channel attention module to adaptively modulate features from different channels and proposed a non-local enhancement residual group to capture spatial context information, effectively improving the quality of super-resolution reconstruction. Fang et al. proposed a soft-edge-assisted super-resolution reconstruction network that reconstructs high-resolution images by fusing coarse super-resolution feature maps and soft super-resolution edges. Compared to reconstructing left and right views separately using single-image methods, super-resolution reconstruction methods for stereo images can further improve performance by exploring the relationship between viewpoints in the left and right views. For example, Wang et al. integrated left and right viewpoint information by exploring the consistency relationship between viewpoints along the horizontal epipolar lines of the left and right viewpoints, effectively improving the performance of stereo image super-resolution reconstruction. Ying et al. extracted features using existing single-image super-resolution reconstruction methods and combined them with a stereo attention module to mine inter-viewpoint information, thereby reconstructing high-resolution stereo images. Chen et al. overcame the limitation of parallax attention in acquiring information only at the epipolar line by expanding the receptive field and used a single model to achieve super-resolution reconstruction at different magnifications.

[0004] In the process of realizing this invention, the inventors discovered that the prior art has at least the following drawbacks and deficiencies:

[0005] Existing methods typically assign the same number of channels to different layers of the network during network construction. However, considering that different layers in the network have varying degrees of importance to reconstruction quality, assigning the same number of channels to each layer leads to redundancy in the network structure. Furthermore, existing methods often rely on mutual attention mechanisms to explore inter-view information. When calculating the inter-view correlation of features at each layer for both left and right viewpoints, the mutual attention mechanism incurs a huge computational overhead. Summary of the Invention

[0006] This invention provides a lightweight method and apparatus for super-resolution reconstruction of stereo images. Firstly, it proposes an adaptive pruning strategy. A teacher network is constructed by designing gating units based on importance estimation. The calculated importance is used to adaptively prune the structure of the teacher network to build a lightweight student network. Secondly, a cross-viewpoint distillation mechanism is proposed. By constructing a cross-viewpoint hierarchical attention module, the inter-viewpoint correlation of multi-level features from the left and right viewpoints is explored, thereby providing additional supervision signals for the student network. Details are described below.

[0007] Firstly, a lightweight method for super-resolution reconstruction of stereo images, the method comprising:

[0008] An adaptive pruning strategy is proposed. A teacher network is constructed based on gating units with importance estimation. The residual blocks and mutual attention mechanisms in the corresponding gating units are pruned to obtain an optimized teacher network, which is then used to construct the student network.

[0009] A cross-viewpoint distillation mechanism is proposed. By constructing a cross-viewpoint hierarchical attention module, the correlation between left-viewpoint multi-level features and right-viewpoint multi-level features in the optimized teacher network and the correlation between left-viewpoint multi-level features and right-viewpoint multi-level features in the student network are explored, and the correlation matrices are obtained respectively.

[0010] The root mean square error loss function is used to measure the difference between the correlation matrices, providing additional supervision signals to the student network and obtaining a lightweight student network.

[0011] High-resolution stereo images are reconstructed using only a lightweight student network during the inference phase.

[0012] The adaptive pruning strategy is used to obtain a lightweight student network by adaptively pruning residual blocks and mutual attention mechanisms at different levels in the teacher network. The importance estimation gating unit is: the importance of the residual blocks used to extract single-viewpoint features; and the importance β of the mutual attention mechanism. n As weights, soft gating is used to weight and fuse the input and output features of the mutual attention mechanism, as shown below:

[0013]

[0014] Where CrossAttn(·) represents the mutual attention mechanism, f l n+1 and These represent the left and right viewpoint features output by the nth gating unit, respectively.

[0015] in,

[0016]

[0017] Where ReLU(·) represents the ReLU activation function, Cos(·) represents the cosine similarity calculation, and β n ∈[0,1] indicates the importance of the mutual attention mechanism in the nth gated unit obtained by computation. This indicates the importance of the mutual attention mechanism in the nth gating unit of the left viewpoint feature; This indicates the importance of the mutual attention mechanism in the nth gating unit of the right viewpoint feature.

[0018] The left and right viewpoint features output by the gating unit are as follows:

[0019] Importance α n As weights, soft gating is used to perform weighted fusion of the input and output features of the residual block, as shown below:

[0020]

[0021] Where Res(·) represents the residual block, and ⊙ represents pixel-by-pixel multiplication. and These represent the left and right viewpoint features of the residual block output, respectively.

[0022]

[0023] Where Conv(·) represents a convolutional layer, GP(·) represents a global pooling layer, Sigmoid(·) represents the Sigmoid activation function, Avg(·) represents the calculation of the average value, and α n ∈[0,1] indicates the importance of the residual block in the final calculated nth gated unit. This indicates the importance of the residual block in the nth gated unit of the left viewpoint feature; This indicates the importance of the residual block in the nth gated unit of the right viewpoint feature.

[0024] Specifically, the pruning of residual blocks and mutual attention mechanisms in the corresponding gating units yields the optimized teacher network as follows:

[0025]

[0026] in, This represents the number of channels in the residual block of the nth gated unit in the teacher network. This represents the number of channels in the nth residual block of the pruned student network, where × indicates a multiplication operation. This indicates the rounding up operation. A flag indicating whether a mutual attention mechanism should be introduced after the nth residual block in the student network. t and Net s These represent the teacher network and the student network, respectively. Prun(·) represents the process of pruning the network based on the calculated number of channels and flags. N represents the total number of gating units.

[0027] The proposed cross-viewpoint distillation mechanism explores the correlation between left-viewpoint multi-level features and right-viewpoint multi-level features in the optimized teacher network by constructing a cross-viewpoint hierarchical attention module.

[0028] Multi-level features of left and right viewpoints acquired from student networks and By constructing a cross-viewpoint hierarchical attention module, we obtain the inter-viewpoint attention graph M. s ;

[0029] Constructing inter-view relevance distillation constraints, utilizing the inter-view attention graph M of the teacher network. t Inter-view attention graph M of constrained student network s To improve the accuracy of the relationship between viewpoints at each level of features extracted by the student network, the distillation constraint of the correlation between viewpoints is expressed as follows:

[0030]

[0031] in, L represents the mean squared error loss. distillation This indicates the distillation constraint of inter-viewpoint correlation.

[0032] The inter-viewpoint attention map is as follows:

[0033]

[0034] Among them, M t ∈R HW×N×N Viewpoint attention graph representing multi-level features of left and right viewpoints in a teacher network. · represents matrix multiplication T R(·) represents the matrix transpose operation, and R(·) represents the dimension transformation operation, which transforms Q... l and Q r The dimension is determined by R N×C×H×W Transform to R HW×N×C .

[0035] Among them, the

[0036]

[0037] Among them, Q l and Q r Let R represent the query vectors for the left and right viewpoints, respectively, and let R(·) represent the dimension transformation operation. and Transform them into dimension R respectively N×C×H×W The feature vector, where C represents the number of channels, H and W represent the width and height of the feature, respectively, and proj(·) represents a linear mapping.

[0038] In a second aspect, a lightweight stereoscopic image super-resolution reconstruction apparatus, the apparatus comprising: a processor and a memory, the memory storing program instructions, the processor calling the program instructions stored in the memory to cause the apparatus to perform the method described in any of the first aspects.

[0039] Third aspect, a computer-readable storage medium storing a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the method described in any one of the first aspects.

[0040] The beneficial effects of the technical solution provided by this invention are:

[0041] 1. This method can achieve better performance in stereo image super-resolution reconstruction while effectively reducing the computational complexity of the network;

[0042] 2. In the process of reconstructing high-quality, high-resolution stereo images, this method can maintain the consistency of the matching relationship between viewpoints while focusing on pixel-level accuracy within each view. Attached Figure Description

[0043] Figure 1 This is a schematic diagram of the network structure of a gating unit based on importance estimation.

[0044] Figure 2 This is a schematic diagram showing the qualitative comparison results of the super-resolution reconstruction of stereo images.

[0045] Figure 3 This is a flowchart of a lightweight method for super-resolution reconstruction of stereo images. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below.

[0047] Example 1

[0048] This invention presents a lightweight stereo image super-resolution reconstruction method, see [link to relevant documentation]. Figures 1-3 The method includes the following steps:

[0049] 101: An adaptive pruning strategy is proposed. The teacher network is constructed using the designed gating unit based on importance estimation. By calculating the importance of each layer in the teacher network, the residual blocks and mutual attention mechanisms in the corresponding gating units are pruned, thereby obtaining a more efficient network structure to construct the student network.

[0050] 102: Proposes a cross-viewpoint distillation mechanism, and explores the multi-level features of the left viewpoint in the teacher network by constructing a cross-viewpoint hierarchical attention module. With right viewpoint multi-level features The correlation between them, and the multi-level features of the left viewpoint in the student network. With right viewpoint multi-level features The correlation between them is used to obtain the correlation matrix M. t and M s ;

[0051] 103: Measure matrix M by using the root mean square error loss function t and M s The differences between them provide additional supervisory signals to the student network, thereby further improving the stereo image super-resolution reconstruction performance of the student network;

[0052] 104: High-resolution stereo images with high quality can be reconstructed using only a lightweight student network during the inference phase.

[0053] In summary, the embodiments of the present invention, through steps 101-104 above, can effectively reduce the computational complexity of the network while achieving superior performance in stereo image super-resolution reconstruction, thus meeting various needs in practical applications.

[0054] Example 2

[0055] The scheme in Example 1 will be further described below with specific calculation formulas and examples. See the description below for details:

[0056] I. Adaptive Pruning Strategy

[0057] Given that different layers in a network contribute differently to overall performance, layers with a more significant impact on performance should be allocated more channels to enhance their feature representation capabilities. Conversely, layers with relatively low importance and less contribution to performance should have their channel count appropriately reduced to optimize the allocation of computational resources and improve the overall efficiency of the network. Furthermore, using mutual attention mechanisms to calculate the inter-view correlation of features at each level in the network requires a massive amount of computation. Therefore, it is considered to introduce mutual attention mechanisms only at specific layers of the network to achieve a balance between computational complexity and performance. Thus, this invention proposes an adaptive pruning strategy to obtain a lightweight student network by adaptively pruning residual blocks and mutual attention mechanisms at different levels in the teacher network.

[0058] To achieve adaptive pruning of the teacher network, a gating unit based on importance estimation was designed, and the teacher network was constructed using the designed gating unit. Figure 1 The network structure of a gating unit based on importance estimation is shown. Taking the nth gating unit as an example, for the input left viewpoint feature f... l n and right viewpoint features First, the importance of the residual blocks used for extracting single-viewpoint features in the gating unit is estimated. This process can be expressed by the following formula:

[0059]

[0060] Where Conv(·) represents a convolutional layer, GP(·) represents a global pooling layer, Sigmoid(·) represents the Sigmoid activation function, and Avg(·) represents calculating the average value. α n ∈[0,1] indicates the importance of the residual block in the final calculated nth gated unit. This indicates the importance of the residual block in the nth gated unit of the left viewpoint feature; This indicates the importance of the residual block in the nth gated unit of the right viewpoint feature.

[0061] The importance of calculating the residual block α n Then, the importance α n As weights, soft gating is used to perform weighted fusion of the input and output features of the residual block. This process can be expressed by the following formula:

[0062]

[0063] Where Res(·) represents the residual block, and ⊙ represents pixel-by-pixel multiplication. and These represent the left and right viewpoint features of the residual block output, respectively.

[0064] Subsequently, based on the features of the left and right viewpoints... and Estimate the importance of the mutual attention mechanism in this gating unit. This process can be expressed by the following formula:

[0065]

[0066] Where ReLU(·) represents the ReLU activation function, and Cos(·) represents the cosine similarity calculation. β n ∈[0,1] indicates the importance of the mutual attention mechanism in the nth gated unit obtained by computation. This indicates the importance of the mutual attention mechanism in the nth gating unit of the left viewpoint feature; This indicates the importance of the mutual attention mechanism in the nth gating unit of the right viewpoint feature.

[0067] The importance of calculating the mutual attention mechanism β n Then, the importance β n As weights, soft gating is used to weight and fuse the input and output features of the mutual attention mechanism. This process can be expressed by the following formula:

[0068]

[0069] Here, CrossAttn(·) represents the mutual attention mechanism. l n+1 and These represent the left and right viewpoint features output by the nth gating unit, respectively.

[0070] Finally, after training the teacher network, the residual blocks and mutual attention mechanisms in the gating units are pruned based on the importance calculated for each gating unit in the teacher network to obtain a lightweight student network. This process is expressed by the following formula:

[0071]

[0072] in, This represents the number of channels in the residual block of the nth gated unit in the teacher network. This represents the number of channels in the nth residual block of the pruned student network. × indicates a multiplication operation. This indicates the rounding up operation. This flag indicates whether a mutual attention mechanism should be introduced after the nth residual block in the student network. If the flag is False, a mutual attention mechanism is not introduced; if the flag is True, a mutual attention mechanism is introduced. t and Net sThese represent the teacher network and the student network, respectively. Prun(·) represents the process of pruning the network based on the calculated number of channels and flags. N represents the total number of gating units.

[0073] II. Cross-viewpoint distillation mechanism

[0074] Considering the parallax between different viewpoints in stereo images, reconstructing high-quality, high-resolution stereo images requires attention to pixel-level accuracy within each view while maintaining consistency in the matching relationships between viewpoints. Therefore, this invention proposes a cross-viewpoint distillation mechanism, which further improves the reconstruction performance of the student network by constructing feature-level inter-viewpoint correlation distillation constraints.

[0075] Because the residual blocks used to extract left and right viewpoint features in the teacher network have a higher number of channels and utilize a mutual attention mechanism to explore the inter-viewpoint information at each layer of features, the left and right viewpoint features obtained by the teacher network have more accurate inter-viewpoint relationships compared to the student network. Therefore, by calculating the inter-viewpoint correlation matrix of different viewpoint features in the teacher network, guiding the student network to extract more accurate left and right viewpoint features helps improve the performance of the student network. Specifically, for the left viewpoint multi-level features obtained by the teacher network... and right viewpoint multi-level features A cross-viewpoint hierarchical attention module was constructed to explore inter-viewpoint correlations. In this module, the first step is to... and Perform a dimensionality transformation and then use a linear mapping to project the transformed features into a query vector. This process can be expressed by the following formula:

[0076]

[0077] Among them, Q l and Q r Let R represent the query vectors for the left and right viewpoints, respectively. R(·) represents the dimension transformation operation, which transforms the left and right viewpoints... and Transform them into dimension R respectively N×C×H×W The feature vector is defined by C, where C represents the number of channels, and H and W represent the width and height of the feature, respectively. proj(·) represents a linear mapping.

[0078] Then, the query vectors of the left and right views are transformed and multiplied by matrix multiplication to calculate the inter-view attention map of the multi-level features of the left and right views in the teacher network. This process can be expressed by the following formula:

[0079]

[0080] Among them, M t ∈R HW×N×NA viewpoint attention graph representing the multi-level features of left and right viewpoints in a teacher network. · represents matrix multiplication T R represents the matrix transpose operation. R(·) represents the dimension transformation operation, transforming Q... l and Q r The dimension is determined by R N×C×H×W Transform to R HW×N×C .

[0081] Similarly, for the left and right viewpoint multi-level features acquired from the student network... and By constructing a cross-viewpoint hierarchical attention module, the inter-viewpoint attention graph M can be calculated. s Finally, a distillation constraint for inter-view relevance is constructed, utilizing the inter-view attention graph M of the teacher network. t Inter-view attention graph M of constrained student network s This improves the accuracy of the relationship between viewpoints at each level of features extracted by the student network. The distillation constraint of viewpoint correlation can be expressed by the following formula:

[0082]

[0083] in, L represents the mean squared error loss. distillation This indicates the distillation constraint of inter-viewpoint correlation.

[0084] III. Loss Function

[0085] For the teacher network, this embodiment of the invention uses mean squared error loss to measure the difference between the reconstructed image and the high-resolution ground truth. The loss function of the teacher network can be expressed as follows:

[0086]

[0087] Among them, L SR This represents the loss function of the teacher network. and These represent the true values ​​of the left and right views at high resolution in the super-resolution reconstruction task, respectively. and These represent the left and right views of the teacher network reconstruction, respectively.

[0088] For the student network, in order to ensure the training stability of the student network, the images reconstructed by the network using high-resolution ground truth constraints are only based on mean squared error loss in the first 15 epochs; thereafter, the inter-view correlation distillation constraint is introduced on the basis of mean squared error loss and trained together with the student network, where the weight of the inter-view correlation distillation constraint is set to 0.1.

[0089] In both the teacher and student networks, the residual blocks employ an intra-block structure similar to that in nonlinear, inactive networks, sharing weights between the left and right viewpoints. The reconstruction layer utilizes sub-pixel convolutional layers to acquire high-resolution images. The teacher network contains 32 gating units based on importance estimation, with each gating unit having 96 channels in its residual blocks (N=32). For the student network, 1×1 convolutional layers are inserted between residual blocks to ensure concatenation between them. For the proposed cross-viewpoint distillation mechanism, the cross-viewpoint hierarchical attention modules of the teacher and student networks share weights.

[0090] Figure 2 The qualitative comparison results of the proposed method and comparative methods for 4x super-resolution reconstruction are presented. The comparison algorithms include the StereoSR method and the CVCnet method. Figure 2 As shown, compared with other lightweight methods, the image reconstructed by the method proposed in this invention is closer to the real high-resolution image. It can be seen from the magnified area that the left and right views reconstructed by the method proposed in this embodiment of the invention both have higher quality. Figure 2 Taking the first and second rows as examples, the stereoscopic image reconstructed by the method proposed in this embodiment of the invention has clearer edges; Figure 2 Taking the third and fourth lines as examples, the method proposed in this embodiment of the invention can reconstruct more accurate texture details. This is because the method proposed in this embodiment of the invention adaptively prunes different network layers and utilizes the inter-view correlations obtained from the teacher network to provide additional constraints for the student network, thus significantly reducing the computational complexity of the model while maintaining the performance of stereo image super-resolution reconstruction. Figure 3 A technical flowchart of an embodiment of the present invention is provided, which mainly includes: adaptive pruning strategy, cross-viewpoint distillation mechanism, and loss function.

[0091] Example 3

[0092] A lightweight stereo image super-resolution reconstruction apparatus includes a processor and a memory. The memory stores program instructions, and the processor invokes the program instructions stored in the memory to cause the apparatus to perform the following method steps in Embodiment 1:

[0093] An adaptive pruning strategy is proposed. A teacher network is constructed based on gating units with importance estimation. The residual blocks and mutual attention mechanisms in the corresponding gating units are pruned to obtain an optimized teacher network, which is then used to construct the student network.

[0094] A cross-viewpoint distillation mechanism is proposed. By constructing a cross-viewpoint hierarchical attention module, the correlation between left-viewpoint multi-level features and right-viewpoint multi-level features in the optimized teacher network and the correlation between left-viewpoint multi-level features and right-viewpoint multi-level features in the student network are explored, and the correlation matrices are obtained respectively.

[0095] The root mean square error loss function is used to measure the difference between the correlation matrices, providing additional supervision signals to the student network and obtaining a lightweight student network.

[0096] High-resolution stereo images are reconstructed using only a lightweight student network during the inference phase.

[0097] Among them, the adaptive pruning strategy is used to obtain a lightweight student network by adaptively pruning the residual blocks and mutual attention mechanisms at different levels in the teacher network.

[0098] The gating unit for importance estimation is the importance of the residual block used to extract single-viewpoint features.

[0099] The importance of the mutual attention mechanism (β) n As weights, soft gating is used to weight and fuse the input and output features of the mutual attention mechanism, as shown below:

[0100]

[0101] Where CrossAttn(·) represents the mutual attention mechanism, f l n+1 and These represent the left and right viewpoint features output by the nth gating unit, respectively.

[0102] in,

[0103]

[0104] Where ReLU(·) represents the ReLU activation function, Cos(·) represents the cosine similarity calculation, and β n ∈[0,1] indicates the importance of the mutual attention mechanism in the nth gated unit obtained by computation. This indicates the importance of the mutual attention mechanism in the nth gating unit of the left viewpoint feature; This indicates the importance of the mutual attention mechanism in the nth gating unit of the right viewpoint feature.

[0105] The left and right viewpoint features output by the gating unit are as follows:

[0106] Importance α n As weights, soft gating is used to perform weighted fusion of the input and output features of the residual block, as shown below:

[0107]

[0108] Where Res(·) represents the residual block, and ⊙ represents pixel-by-pixel multiplication. and These represent the left and right viewpoint features of the residual block output, respectively.

[0109]

[0110] Where Conv(·) represents a convolutional layer, GP(·) represents a global pooling layer, Sigmoid(·) represents the Sigmoid activation function, Avg(·) represents the calculation of the average value, and α n ∈[0,1] indicates the importance of the residual block in the final calculated nth gated unit. This indicates the importance of the residual block in the nth gated unit of the left viewpoint feature; This indicates the importance of the residual block in the nth gated unit of the right viewpoint feature.

[0111] Specifically, by pruning the residual blocks and mutual attention mechanisms in the corresponding gating units, the optimized teacher network is obtained as follows:

[0112]

[0113] in, This represents the number of channels in the residual block of the nth gated unit in the teacher network. This represents the number of channels in the nth residual block of the pruned student network, where × indicates a multiplication operation. This indicates the rounding up operation. A flag indicating whether a mutual attention mechanism should be introduced after the nth residual block in the student network. t and Net s These represent the teacher network and the student network, respectively. Prun(·) represents the process of pruning the network based on the calculated number of channels and flags. N represents the total number of gating units.

[0114] Among these, a cross-viewpoint distillation mechanism is proposed. By constructing a cross-viewpoint hierarchical attention module, the correlation between left-viewpoint multi-level features and right-viewpoint multi-level features in the optimized teacher network is explored as follows:

[0115] Multi-level features of left and right viewpoints acquired from student networks and By constructing a cross-viewpoint hierarchical attention module, we obtain the inter-viewpoint attention graph M. s ;

[0116] Construct inter-view relevance distillation constraints, and use the inter-view attention graph M of the teacher network to constrain the inter-view attention graph M of the student network. s To improve the accuracy of the relationship between viewpoints at each level of features extracted by the student network, the distillation constraint of the correlation between viewpoints is expressed as follows:

[0117]

[0118] in, L represents the mean squared error loss. distillation This indicates the distillation constraint of inter-viewpoint correlation.

[0119] The inter-viewpoint attention map is as follows:

[0120]

[0121] Among them, M t ∈R HW×N×N Viewpoint attention graph representing multi-level features of left and right viewpoints in a teacher network. · represents matrix multiplication T R(·) represents the matrix transpose operation, and R(·) represents the dimension transformation operation, which transforms Q... l and Q r The dimension is determined by R N×C×H×W Transform to R HW×N×C .

[0122] in,

[0123]

[0124]

[0125] Among them, Q l and Q r Let R represent the query vectors for the left and right viewpoints, respectively, and let R(·) represent the dimension transformation operation. and Transform them into dimension R respectively N×C×H×W The feature vector, where C represents the number of channels, H and W represent the width and height of the feature, respectively, and proj(·) represents a linear mapping.

[0126] It should be noted that the device descriptions in the above embodiments correspond to the method descriptions in the embodiments, and the embodiments of the present invention will not be repeated here.

[0127] The execution entities of the aforementioned processor and memory can be devices with computing functions such as computers, microcontrollers, and single-chip microcomputers. In specific implementations, the embodiments of the present invention do not limit the execution entities and can select them according to the needs of actual applications.

[0128] Data signals are transmitted between the memory and the processor via a bus, which will not be elaborated upon in this embodiment of the invention.

[0129] Based on the same inventive concept, embodiments of the present invention also provide a computer-readable storage medium, the storage medium including a stored program, which, when the program is running, controls the device where the storage medium is located to execute the method steps in the above embodiments.

[0130] The computer-readable storage medium includes, but is not limited to, flash memory, hard disk, solid-state drive, etc.

[0131] It should be noted that the description of the readable storage medium in the above embodiments corresponds to the description of the method in the embodiments, and the embodiments of the present invention will not be repeated here.

[0132] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of the present invention is generated.

[0133] A computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in or transmitted through a computer-readable storage medium. A computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be magnetic or semiconductor, etc.

[0134] Unless otherwise specified, the model numbers of the various devices in this embodiment of the invention are not limited, and any device that can perform the above functions is acceptable.

[0135] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0136] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A light-weighted stereoscopic image super-resolution reconstruction method, characterized in that, The method comprises: An adaptive pruning strategy is proposed, a teacher network is constructed based on an importance estimation gating unit, residual blocks and mutual attention mechanisms in the corresponding gating unit are pruned, an optimized teacher network is obtained to construct a student network; A cross-viewpoint distillation mechanism is proposed, a cross-viewpoint hierarchical attention module is constructed, the correlation between the multi-level features of the left viewpoint and the multi-level features of the right viewpoint in the optimized teacher network and the correlation between the multi-level features of the left viewpoint and the multi-level features of the right viewpoint in the student network are explored, and correlation matrices are obtained respectively; The root mean square error loss function is used to measure the difference between the correlation matrices, an additional supervision signal is provided for the student network, and a lightweight student network is obtained; In the inference stage, only the lightweight student network is used to reconstruct high-resolution stereo images; The importance estimation gating unit is used to extract the importance of the residual blocks of the single-viewpoint features: Importance of the mutual attention mechanism As the weight, the input features and the output features of the mutual attention mechanism are weighted and fused by using the soft gating, which is expressed as follows: , , wherein, denotes the mutual attention mechanism, and denote the left-view feature and the right-view feature of the i-th gating unit output, respectively; denote the left-view feature and the right-view feature of the i-th gating unit output, respectively; The left-viewpoint features and the right-viewpoint features output by the gating unit are: , , , in, Represents the ReLU activation function. This indicates the calculation of cosine similarity. Indicates the calculated first... The importance of mutual attention mechanisms in gating units Indicates the left viewpoint feature. The importance of mutual attention mechanisms in gating units; Indicates the right viewpoint feature. The importance of mutual attention mechanisms in gating units; The residual blocks and mutual attention mechanisms in the corresponding gating unit are pruned to obtain the optimized teacher network: Importance As the weight, the input features and the output features of the residual block are weighted and fused by the soft gate, which is expressed as follows: , , wherein, denotes a residual block, denotes a pixel-wise multiplication, and denotes a left-view feature and a right-view feature of the residual block output, respectively; , , wherein, denotes a convolutional layer, denotes a global pooling layer, denotes a Sigmoid activation function, denotes a computation of the mean, denotes the importance of the residual block in the th gating unit of the final computation, denotes the importance of the residual block in the th gating unit of the left-view feature; denotes the importance of the residual block in the th gating unit of the right-view feature. The adaptive pruning strategy is used to adaptively prune the residual blocks and mutual attention mechanisms in different levels of the teacher network to obtain a lightweight student network. , , , wherein, represents the number of channels of the residual block in the i-th gating unit in the teacher network, represents the number of channels of the i-th residual block in the pruned student network, represents a multiplication operation, represents a rounding up operation, represents a flag indicating whether to introduce the attention mechanism after the i-th residual block in the student network, and respectively represent the teacher network and the student network, represents a process of pruning the network according to the calculated number of channels and the flag, and N represents the total number of gating units.​​​ 2. The method of claim 1, wherein, The cross-viewpoint distillation mechanism is proposed, and the cross-viewpoint hierarchical attention module is constructed to explore the correlation between the multi-level features of the left viewpoint and the multi-level features of the right viewpoint in the optimized teacher network:

3. The method of claim 1, wherein, The inter-viewpoint attention map is: For student network to obtain left and right view multi-level features And By constructing a cross-view hierarchical attention module, an inter-view attention map is obtained ; The viewpoint inter-relation distillation constraint is constructed by using the viewpoint inter-attention map of the teacher network The viewpoint inter-attention map of the student network is constrained The accuracy of the viewpoint inter-relation of each level of features extracted by the student network is improved, and the viewpoint inter-relation distillation constraint is as follows: , wherein, denotes the mean squared error loss, denotes the inter-view correlation distillation constraint.

4. The method of claim 3, wherein, The , wherein, represents the inter-view attention map of the multi-level features of left and right views in the teacher network, represents matrix multiplication, represents a matrix transpose operation, represents a dimension transformation operation that transforms and from to .

5. The method of claim 4, wherein, The device comprises a processor and a memory, the memory stores program instructions, and the processor invokes the program instructions stored in the memory to enable the device to execute the method of any one of claims 1-5. , , wherein, and denote the query vector for the left and right view, respectively, denotes a dimension transformation operation that transforms and into feature vectors of dimension denotes the number of channels, and denote the width and height of the feature, respectively, denotes a linear mapping.​ 6. A lightweight stereoscopic image super-resolution reconstruction apparatus, characterized by comprising: The computer readable storage medium stores a computer program, the computer program comprises program instructions, and the program instructions are executed by the processor to enable the processor to execute the method of any one of claims 1-5.

7. A computer readable storage medium characterized in that, ​

Images