Stereoscopic image reconstruction method, device, equipment, medium and program product

Abstract

The invention provides a three-dimensional image reconstruction method and device, equipment, a medium and a program product. The method comprises the following steps: acquiring single views of two symmetrical view angles under a target high resolution; obtaining a multi-scale visual feature map of the single view based on the feature extraction network, and obtaining a multi-scale semantic feature map and a global semantic vector of the single view based on a visual encoder of a pre-trained visual language model; performing cross-view fusion on the multi-scale visual feature maps of all the visual angles and the multi-scale semantic feature maps based on a semantic guidance parallax attention network to obtain a target visual enhanced feature map of each visual angle; the method is applied to a semantic-guided reconstruction network, and comprises the following steps of: performing feature modulation reconstruction on a visual enhancement feature map in a spatial dimension and a channel dimension based on a global semantic vector to obtain a super-resolution stereo image of each visual angle; therefore, through a visual semantic double-feature extraction framework, a semantic-guided parallax attention mechanism and spatial channel double-dimensional feature modulation reconstruction, the reconstruction accuracy is improved.

Description

Technical Field

[0001] This invention relates to the field of image reconstruction technology, and in particular to a method, apparatus, device, medium, and program product for stereoscopic image reconstruction. Background Technology

[0002] With the continuous development of stereoscopic imaging technology, people's needs for images are no longer limited to traditional two-dimensional images. Stereoscopic images, because they can provide parallax information as well as richer structural and texture information, can significantly enhance the human eye's stereoscopic visual perception and are gradually being widely used.

[0003] To improve image resolution, related technologies propose reconstructing high-resolution super-resolution stereo images from low-resolution stereo images, thereby enhancing image quality. However, most current stereo image super-resolution reconstruction methods rely solely on directly learning the mapping between low-resolution and super-resolution stereo images, depending on low-level pixel space features and lacking high-level semantic prior constraints, resulting in inaccurate reconstructed super-resolution stereo images. Summary of the Invention

[0004] This invention provides a method, apparatus, device, medium, and program product for stereo image reconstruction, which solves the technical problem that existing stereo image super-resolution reconstruction methods rely only on low-level features in pixel space and lack high-level semantic prior constraints, resulting in inaccurate super-resolution stereo images after reconstruction.

[0005] This invention provides a method for stereoscopic image reconstruction, the method comprising: Acquire a single view of the target at high resolution from two symmetrical perspectives in a stereo vision system; For each single view, a multi-scale visual feature map corresponding to the single view is obtained based on a feature extraction network, and a multi-scale semantic feature map and a global semantic vector corresponding to the single view are obtained based on a pre-trained visual language model visual encoder. A semantically guided parallax attention network is used to perform cross-view fusion of the multi-scale visual feature maps and the multi-scale semantic feature maps of all viewpoints to obtain the target visual enhancement feature map for each viewpoint. For each viewpoint, the target visual enhancement feature map is applied to a semantically guided reconstruction network. Based on the global semantic vector, the visual enhancement feature map is modulated and reconstructed in both spatial and channel dimensions to obtain a super-resolution stereo image reconstructed for each viewpoint.

[0006] According to a stereo image reconstruction method provided by the present invention, both the multi-scale visual feature map and the multi-scale semantic feature map include I levels of feature maps ordered from low resolution to high resolution, and the resolution of the semantic feature map at the same level is smaller than the resolution of the visual feature map. The semantically guided parallax attention network comprises I cascaded semantically guided parallax attention units; The target visual enhancement feature map for each viewpoint is obtained in the following way: Obtain the input visual feature map of the i-th level; The resolution of the semantic feature map at the i-th level is upsampled to be the same as the resolution of the input visual feature map at the i-th level to obtain the input semantic feature map; The input visual feature map and the input semantic feature map of the i-th level of all viewpoints are input into the semantically guided disparity attention unit corresponding to the i-th level to obtain the visual enhancement feature map of the i-th level of each viewpoint, and the visual enhancement feature map of the last level is used as the target visual enhancement feature map. Wherein, when i=1, the visual feature map of the first level is used as the input visual feature map; when 1<i≤1, the visual enhancement feature map of the (i-1)th level is fused with the visual feature map of the ith level to obtain the input visual feature map of the ith level.

[0007] According to a stereo image reconstruction method provided by the present invention, the semantically guided disparity attention unit includes a feature transformation subunit, an attention calculation subunit, and an orthogonal projection fusion subunit; The visual enhancement feature maps for each level are obtained as follows: Based on the feature transformation subunit, the semantic query vector corresponding to the first input semantic feature map of the current view, the visual query vector corresponding to the first input visual feature map, the semantic key vector corresponding to the second input semantic feature map of another view, and the visual key vector corresponding to the second input visual feature map of the other view are obtained respectively. Determine a first similarity score map between the semantic query vector and the semantic key vector, and determine a second similarity score map between the visual query vector and the visual key vector; The first similarity score map and the second similarity score map are input into the attention calculation subunit to obtain the disparity attention map of the current view. Based on the disparity attention map of the current view, a cross-view feature map aligned with the first input visual feature map is aggregated from the second input visual feature map. Based on the orthogonal projection fusion subunit, an orthogonal feature map orthogonal to the first input visual feature map is extracted from the cross-view feature map, and the orthogonal feature map is fused with the first input visual feature map to obtain the visual enhancement feature map of the current view.

[0008] According to a stereoscopic image reconstruction method provided by the present invention, the semantically guided reconstruction network includes J cascaded semantically guided reconstruction units corresponding to each viewpoint, and each semantically guided reconstruction unit includes a semantically guided subunit and a reconstruction subunit. The super-resolution stereo images for each viewpoint are obtained as follows: Obtain the input visual enhancement feature map of the j-th semantic guidance subunit; The input visual enhancement feature map and the global semantic vector are input to the j-th semantic guidance subunit, and feature modulation is performed in the spatial dimension and the channel dimension respectively to obtain the spatial modulation feature map and the channel modulation feature map. The spatial modulation feature map and the channel modulation feature map are then added pixel by pixel to obtain the fused modulation feature map output by the j-th semantic guidance subunit. The fused modulation feature map output by the j-th semantic guidance subunit is input into the j-th reconstruction subunit for feature reconstruction to obtain a reconstructed feature map. A convolution operation is then performed on the reconstructed feature map output by the last reconstruction subunit to obtain the super-resolution stereo image. Wherein, when j=1, the target visual enhancement feature map is used as the input visual enhancement feature map of the first semantic guidance subunit, and when 1<j≤J, the reconstructed feature map output by the (j-1)th reconstruction subunit is used as the input visual enhancement feature map of the jth semantic guidance subunit.

[0009] According to a stereo image reconstruction method provided by the present invention, the semantic guidance subunit includes a first fully connected layer and a convolutional parameter generation layer; The spatial modulation feature map is obtained in the following way: The global semantic vector is input into the first fully connected layer for feature mapping to obtain intermediate semantic features; The semantic intermediate features are input into the convolutional parameter generation layer, and the semantic intermediate features are converted into kernel parameters of the deep convolutional kernel through a reshaping operation; A dynamic convolution kernel is constructed based on the kernel parameters, and the input visual enhancement feature map is convolved using the dynamic convolution kernel to obtain the spatial modulation feature map.

[0010] According to a stereo image reconstruction method provided by the present invention, the semantic guidance subunit includes a second fully connected layer and a feature scaling layer; The channel modulation feature map is obtained in the following way: The global semantic vector is input into the second fully connected layer to obtain the channel modulation coefficients; Based on the channel modulation coefficients, the input visual enhancement feature map is weighted channel by channel through the feature scaling layer to obtain the channel modulation feature map.

[0011] According to a stereo image reconstruction method provided by the present invention, the network parameters of the feature extraction network, the semantically guided disparity attention network, and the semantically guided reconstruction network are obtained by minimizing the total loss, which is obtained in the following manner: Obtain the real super-resolution stereo image and the reconstructed predicted super-resolution stereo image corresponding to each viewpoint; The super-resolution loss is obtained based on the average absolute error between the real super-resolution stereo image and the predicted super-resolution stereo image corresponding to all viewpoints. Based on the real super-resolution stereo images and the predicted super-resolution stereo images corresponding to all viewpoints, the semantic consistency loss is obtained by applying it to the visual encoder of the pre-trained visual language model. The total loss is obtained based on the super-resolution loss and the semantic consistency loss.

[0012] The present invention also provides a stereoscopic image reconstruction apparatus, the apparatus comprising: The view acquisition module is used to acquire a single view of the target at high resolution from two symmetrical perspectives in a stereo vision system. A semantic feature extraction network is used to obtain a multi-scale visual feature map corresponding to each single view based on the feature extraction network, and to obtain a multi-scale semantic feature map and a global semantic vector corresponding to each single view based on a pre-trained visual language model. The view enhancement module is used to perform cross-view fusion of the multi-scale visual feature maps and the multi-scale semantic feature maps of all viewpoints based on a semantically guided parallax attention network to obtain the target visual enhancement feature map for each viewpoint. The reconstruction module is used to apply the target visual enhancement feature map to a semantically guided reconstruction network for each viewpoint. Based on the global semantic vector, the visual enhancement feature map is modulated and reconstructed in the spatial and channel dimensions to obtain a super-resolution stereo image reconstructed for each viewpoint.

[0013] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the stereoscopic image reconstruction method as described above.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the stereoscopic image reconstruction method as described above.

[0015] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the stereoscopic image reconstruction method as described above.

[0016] The stereo image reconstruction method, apparatus, device, medium, and program products provided by this invention, after acquiring single views of two symmetrical perspectives under a target high resolution in a stereo vision system, firstly obtain multi-scale visual structural features and semantic representation features through a visual semantic dual-branch feature extraction architecture. Then, based on a semantically guided disparity attention mechanism, semantic information is integrated into the cross-view disparity calculation and feature fusion process, improving the accuracy of cross-view information capture. Finally, through global semantic vector feature modulation reconstruction in two dimensions of spatial channels, the geometric structure and semantic description of the super-resolution image are highly consistent, improving the accuracy of the reconstructed super-resolution stereo image. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the stereoscopic image reconstruction method provided by the present invention.

[0019] Figure 2 This is one of the scene illustrations of the stereoscopic image reconstruction method provided by the present invention.

[0020] Figure 3 This is the second scene illustration of the stereoscopic image reconstruction method provided by the present invention.

[0021] Figure 4 This is a schematic diagram of the structure of the densely voided block provided by the present invention.

[0022] Figure 5 This is the third scenario illustration of the stereoscopic image reconstruction method provided by the present invention.

[0023] Figure 6 This is the fourth scenario illustration of the stereoscopic image reconstruction method provided by the present invention.

[0024] Figure 7 This is a schematic diagram of the structure of the semantic guidance subunit provided by the present invention.

[0025] Figure 8 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0027] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same class, not limited in number; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0028] With the continuous development of stereoscopic imaging technology, people's needs for images are no longer limited to traditional two-dimensional images. Stereoscopic images, because they can provide parallax information and richer structural and texture information, can significantly enhance the human eye's stereoscopic visual perception and are gradually being widely used. However, in practice, stereoscopic images are inevitably affected by various factors during the acquisition and display process, such as motion transformation, blurring, noise, and compression, which leads to a decrease in image quality, affecting not only visual perception but also limiting the effects of subsequent image processing.

[0029] To improve image resolution, stereo image super-resolution reconstruction technology has emerged. It aims to reconstruct a high-resolution stereo image with clear texture from a low-resolution stereo image through specific algorithmic processing, thereby improving image quality. Unlike single-image super-resolution reconstruction, which recovers a super-resolution image from a single low-resolution image, stereo image super-resolution reconstruction aims to reconstruct a low-resolution stereo image. Stereo images consist of left and right views or more views with certain differences, often caused by horizontal parallax. Therefore, in the process of stereo image super-resolution reconstruction, it is necessary not only to utilize the information of each individual view but also to fully explore the complementary information between views, i.e., cross-view information.

[0030] Traditional stereo image super-resolution reconstruction methods first estimate the disparity, then use this disparity to register two views, and use the registered images as complementary information to help reconstruct the super-resolution left and right views. However, their performance is limited due to the inaccuracy of the estimation algorithm. In recent years, with the rapid development of deep learning, deep learning-based stereo image super-resolution reconstruction methods have been widely studied. Existing deep learning-based methods mainly focus on exploring the acquisition and utilization of cross-view information to improve reconstruction accuracy.

[0031] Based on the different methods of acquiring cross-view information, existing methods can be broadly categorized into methods based on translational copy stacking, methods based on disparity estimation, methods based on disparity attention, and methods based on interactive modules. Among these, methods based on disparity attention can more accurately establish correspondences between views, and therefore have received the most widespread research attention. The core idea is to calculate the similarity between different positions along the horizontal epipolar direction in the left and right views, and use these similarities to effectively acquire cross-view information, thereby assisting in image reconstruction.

[0032] While parallax attention-based methods have achieved good reconstruction performance, they primarily focus on low-level features in pixel space, such as edges and basic textures, during cross-view information acquisition and image reconstruction, generally neglecting deeper semantic information, such as object categories and scene structures. However, relying solely on low-level features for stereo correspondence matching can easily lead to matching errors in low-texture or texture-similar regions, resulting in inaccurate cross-view information capture and ultimately artifacts in the reconstructed image. Furthermore, due to the lack of semantic constraints, the reconstructed super-resolution image struggles to maintain semantic consistency with the real super-resolution image, potentially generating semantically unreasonable details.

[0033] Based on the above problems, this embodiment proposes a stereo image reconstruction method, which improves the detail accuracy and semantic consistency of the reconstructed image by introducing semantic information to optimize feature learning and cross-view matching process.

[0034] Figure 1 This is a flowchart illustrating the stereoscopic image reconstruction method provided in this embodiment, as shown below. Figure 1 As shown, the method includes steps 110, 120, 130 and 140.

[0035] Step 110: Obtain a single view of the target at high resolution from two symmetrical perspectives using a stereo vision system.

[0036] Here, stereo vision system usually refers to a binocular imaging system that includes left and right cameras, or a binocular image acquisition device with a fixed baseline; while two symmetrical viewpoints refer to the left and right viewpoints for the same target scene.

[0037] It should be noted that the original images acquired by stereo vision systems are usually in a low-resolution format with inconsistent resolution, and the resolution is relatively low. Therefore, in this embodiment, to ensure the consistency of the input image size for subsequent super-resolution reconstruction, an interpolation algorithm can be used in advance. For example, bicubic interpolation can be used to normalize and upsample the original single views of different sizes to a single view of a user-preset high-resolution size, so that the subsequent feature extraction process can be completed directly in the high-resolution space, thereby more accurately restoring image details and avoiding the problem of feature information loss in low-dimensional space.

[0038] Step 120: For each single view, obtain the multi-scale visual feature map corresponding to the single view based on the feature extraction network, and obtain the multi-scale semantic feature map and global semantic vector corresponding to the single view based on the visual encoder of the pre-trained visual language model.

[0039] Specifically, a feature extraction network is used to extract pixel-level features from a single viewpoint. In practical applications, this feature extraction network can employ a convolutional neural network architecture, performing multi-level feature encoding on a single view through different levels of convolution, pooling, and upsampling operations. For example, shallow networks capture low-level visual features such as edges and textures, while deep networks extract high-level semantic visual features such as shape and contours. Feature fusion generates multi-scale visual feature maps of different resolutions, preserving the spatial structural information of the image to extract a variety of visual information from subtle textures to local structures.

[0040] In addition, refer to Figure 2 As shown, in order to introduce high-level semantic priors, this embodiment also utilizes the visual encoder of a pre-trained visual language model to mine the semantic-level features of a single view from each perspective.

[0041] In one example, a pre-trained visual language model is used as the Contrastive Language-Image Pre-training (CLIP) model. CLIP is pre-trained on large-scale image-text pairs and possesses powerful semantic understanding capabilities. In this embodiment, by inputting a single view into CLIP's visual encoder, CLIP's visual encoder maps the single view to a semantic space. On the one hand, it outputs a multi-scale semantic feature map that matches the scale of the multi-scale visual feature map through hierarchical encoding. On the other hand, it compresses the semantic information of the entire image into a low-dimensional global semantic vector through global pooling or attention aggregation operations. This vector contains the overall semantic description of the image.

[0042] Step 130: Based on the semantically guided parallax attention network, perform cross-view fusion on the multi-scale visual feature maps and the multi-scale semantic feature maps of all viewpoints to obtain the target visual enhancement feature map for each viewpoint.

[0043] In this embodiment, reference Figure 2 As shown, the explanation uses left and right views as an example. The semantically guided disparity attention network receives four sets of inputs: a multi-scale left visual feature map, a multi-scale left semantic feature map, a multi-scale right visual feature map, and a multi-scale right semantic feature map. During the internal computation of the semantically guided disparity attention network, visual similarity is first calculated using the multi-scale left and right visual feature maps to generate a visual attention matrix reflecting the left and right visual images. Semantic similarity is then calculated using the multi-scale left and right semantic feature maps to generate a semantic attention matrix reflecting the left and right semantic images.

[0044] Continuing with the left-view example, a parallax attention mechanism is constructed. Based on the visual attention matrix and semantic attention matrix, feature information matching the left view is extracted from the multi-scale right-view feature map, and this extracted feature information is fused into the original multi-scale left-view feature map. After this cross-view interaction and fusion operation, the feature map output by the semantically guided parallax attention network is the target visual enhancement feature map of the left view containing binocular complementary information.

[0045] Step 140: For the target visual enhancement feature map of each viewpoint, apply it to the semantically guided reconstruction network, and perform feature modulation reconstruction on the visual enhancement feature map in the spatial and channel dimensions based on the global semantic vector to obtain the super-resolution stereo image reconstructed for each viewpoint.

[0046] Specifically, the semantically guided reconstruction network contains several processing units. In the data processing of each perspective, the global semantic vector is first mapped to two sets of modulation parameters through the fully connected layers and other transformation layers of the semantically guided reconstruction network: one set is the dynamic convolution kernel parameters for the spatial dimension, and the other set is the scaling coefficients for the channel dimension.

[0047] Next, in the spatial dimension, the generated dynamic convolution kernel is used to perform convolution operations on the target visual enhancement feature map of this viewpoint, or the spatial attention map is multiplied element-wise with the feature map, thereby adjusting the response values ​​of the feature map at different spatial locations according to the semantic category. In the channel dimension, the generated scaling factor is multiplied channel-wise with each channel of the feature map, thereby redistributing the weights of different feature channels according to the global semantics.

[0048] After the feature map is modulated as described above, it is finally mapped back to the RGB pixel space through a reconstruction layer, such as a 3×3 convolutional layer, to obtain the super-resolution stereo image reconstructed from this viewpoint.

[0049] The stereo image reconstruction method proposed in this embodiment first obtains single views of the target at high resolution from two symmetrical perspectives under a stereo vision system. Then, it acquires multi-scale visual structural features and semantic representation features through a visual semantic dual-branch feature extraction architecture. Next, based on a semantically guided disparity attention mechanism, semantic information is integrated into the cross-view disparity calculation and feature fusion process, which improves the accuracy of cross-view information capture. Finally, it reconstructs the super-resolution image by modulating features in two dimensions of the spatial channel using global semantic vectors, so that the geometric structure and semantic description of the super-resolution image are highly consistent, thereby improving the accuracy of the reconstructed super-resolution stereo image.

[0050] In some embodiments, reference Figure 3 As shown, both the multi-scale visual feature map and the multi-scale semantic feature map include I levels of feature maps ordered from low resolution to high resolution, and the resolution of the semantic feature map at the same level is smaller than that of the visual feature map; the semantically guided disparity attention network includes I cascaded semantically guided disparity attention units.

[0051] In this embodiment, to fully utilize the detailed and semantic information of the image at different resolutions, both the multi-scale visual feature map and the multi-scale semantic feature map are designed to include I levels of feature maps ordered from low resolution to high resolution. Typically, I is an integer greater than 1, for example, I=3.

[0052] Specifically, the feature extraction network comprises multiple layers of feature extraction units, and the visual feature maps extracted by different feature extraction units have different resolutions. For example, taking the left-hand view as an example, three layers of feature extraction units are used to extract visual feature maps of the left-hand view from low resolution to high resolution. , and ,in, and These represent the required image height and width, respectively. This indicates the number of channels in the feature map.

[0053] Each feature extraction unit in this embodiment consists of a 3×3 convolutional layer and two dense dilated blocks (D2B). (Continue to refer to...) Figure 4As shown, each D2B layer contains three dilated convolutional layers and two 1×1 kernel convolutional layers. The dilated convolutional layers are used to extract hierarchical features, and the convolutional layers are used to reduce the channel dimension. Furthermore, except for the first dilated convolutional layer, each dilated convolutional layer has multiple dilation rates, processing feature maps from different layers. For example, continuing with the left-hand perspective, the first dilated convolutional layer has a dilation rate of 1, used to process the visual feature map of the first layer. The second multi-hole convolutional layer has a dilation rate of 1 and 2, used to process the visual feature maps of the second layer. The third multi-hole convolutional layer has a dilation rate of 1, 2, and 4, used to process the visual feature maps of the third layer. .

[0054] Compared to using standard convolutional layers, this embodiment uses multi-hole convolutional layers in densely holed blocks to obtain a larger receptive field, and a larger receptive field helps to learn accurate stereo correspondences.

[0055] Continuing with the left-view example, while extracting the three levels of visual feature maps from the left view, the single view from the left is input into the visual encoder of the pre-trained visual language model to extract the three levels of semantic feature maps. It should be noted that because visual language models typically downsample images to obtain higher-level semantics, the resolution of semantic feature maps at the same level is usually smaller than the resolution of the visual feature maps. For example, the three levels of semantic feature maps from the left view are as follows: , and ,in, The resolution is 1 / 16 of the single view from the left perspective. The resolution is 1 / 8 of the single view from the left perspective. The resolution is 1 / 4 of the single view from the left perspective.

[0056] Correspondingly, the semantic-guided parallax attention network also includes I cascaded semantic-guided parallax attention modules (SGPAMs) in terms of structure. These modules are connected sequentially from low-resolution to super-resolution image processing to form a cascaded structure.

[0057] Based on this, the target visual enhancement feature map for each viewpoint is obtained as follows: Obtain the input visual feature map of the i-th level; The resolution of the semantic feature map at the i-th level is upsampled to be the same as the resolution of the input visual feature map at the i-th level to obtain the input semantic feature map; The input visual feature map and the input semantic feature map of the i-th level of all viewpoints are input into the semantically guided disparity attention unit corresponding to the i-th level to obtain the visual enhancement feature map of the i-th level of each viewpoint, and the visual enhancement feature map of the last level is used as the target visual enhancement feature map. Wherein, when i=1, the visual feature map of the first level is used as the input visual feature map; when 1<i≤1, the visual enhancement feature map of the (i-1)th level is fused with the visual feature map of the ith level to obtain the input visual feature map of the ith level.

[0058] In this cascaded structure, the generation process of the target visual enhancement feature map from the left perspective will be explained as an example. The generation process of the target visual enhancement feature map from the right perspective is the same as that from the left perspective, and will not be described in detail here.

[0059] First, the input visual feature map of the i-th level from the left perspective is obtained. Here, when i=1, i.e., the lowest resolution level, the original visual feature map of the first level from the left perspective extracted by the feature extraction network is directly used as the input visual feature map of this level. When 1<i≤i, the visual enhancement feature map output by the previous level (i.e., the (i-1)-th level) is fused with the original visual feature map extracted by the current level (i.e., the i-th level), for example, by concatenation or addition, to obtain the input visual feature map of the current i-th level. This cascading method allows higher levels to utilize the prior information already aligned by lower levels.

[0060] Secondly, since the semantic feature map has a small resolution, before performing semantic visual interaction, it is necessary to upsample the resolution of the semantic feature map of the i-th level, for example, by using bilinear interpolation or deconvolution to make it the same as the resolution of the input visual feature map of the i-th level, so as to obtain a spatially aligned input semantic feature map.

[0061] Finally, cross-view interaction is performed. The input visual feature maps and input semantic feature maps of the i-th level from all views are input together into the semantically guided disparity attention unit corresponding to the i-th level. This unit outputs the visual enhancement feature map of the i-th level for each view.

[0062] The stereo image reconstruction method proposed in this embodiment uses a cascading interaction strategy from low to high resolution. It can first capture a wide range of disparities at low resolution using strong semantics, and then gradually correct details at super-resolution, thereby improving the accuracy of stereo matching.

[0063] In some embodiments, the semantically guided disparity attention unit includes a feature transformation subunit, an attention calculation subunit, and an orthogonal projection fusion subunit; The visual enhancement feature maps for each level are obtained as follows: Based on the feature transformation subunit, the semantic query vector corresponding to the first input semantic feature map of the current view, the visual query vector corresponding to the first input visual feature map, the semantic key vector corresponding to the second input semantic feature map of another view, and the visual key vector corresponding to the second input visual feature map of the other view are obtained respectively. Determine a first similarity score map between the semantic query vector and the semantic key vector, and determine a second similarity score map between the visual query vector and the visual key vector; The first similarity score map and the second similarity score map are input into the attention calculation subunit to obtain the disparity attention map of the current view. Based on the disparity attention map of the current view, a cross-view feature map aligned with the first input visual feature map is aggregated from the second input visual feature map. Based on the orthogonal projection fusion subunit, an orthogonal feature map orthogonal to the first input visual feature map is extracted from the cross-view feature map, and the orthogonal feature map is fused with the first input visual feature map to obtain the visual enhancement feature map of the current view.

[0064] In this embodiment, the semantically guided disparity attention unit mainly includes a feature transformation subunit, an attention calculation subunit, and an orthogonal projection fusion subunit.

[0065] The following example illustrates the generation process of the visual enhancement feature map for the left viewpoint, with the current viewpoint being the left viewpoint and the other viewpoint being the right viewpoint. The generation process of the visual enhancement feature map for the right viewpoint is the same as that for the left viewpoint, and will not be elaborated upon here.

[0066] refer to Figure 5 As shown, the semantically guided disparity attention unit has four inputs, namely the first input semantic feature map. First input visual feature map Second input semantic feature map Second input visual feature map First, feature transformation is performed using residual blocks in the feature transformation subunit and a convolutional layer with a kernel size of 1×1 to generate the semantic query vector corresponding to the first input semantic feature map. The visual query vector corresponding to the first input visual feature map The semantic key vector corresponding to the second input semantic feature map The visual key vector corresponding to the second input visual feature map Then , Transpose , and respectively in and , and Batch matrix multiplication is performed between the nodes to generate the first similarity score graph. Second similarity score map .

[0067] The two score maps are then combined, and the softmax function in the attention computation sub-unit is applied to obtain the disparity attention map. And utilize this parallax attention map and the second input visual feature map To capture cross-view feature maps that assist in the reconstruction of the left image. .

[0068] Furthermore, to utilize cross-view information more effectively, after obtaining the cross-view features, an orthogonal projection fusion subunit is used to perform orthogonal projection operations, generating the cross-view feature map. Projected onto the first input visual feature map In space, calculate the projection components and from the cross-view feature map Subtracting the projection component from the input image extracts the feature map that matches the first input visual feature map. Orthogonal feature maps orthogonal feature map This represents complementary information that is present in the right view but not in the left view. The specific operation is shown in the following formula: ; in, express The c-th channel, express The c-th channel, This indicates pixel-by-pixel multiplication.

[0069] Finally, the first input visual feature map Orthogonal feature map The layers are cascaded along the channel dimension and fed into a 1×1 convolutional layer to obtain the visual enhancement feature map from the left perspective. .

[0070] The stereo image reconstruction method proposed in this embodiment achieves cross-view feature alignment and complementary enhancement through the above methods, effectively improving the semantic consistency of visual feature maps and the accuracy of disparity matching.

[0071] In some embodiments, reference Figure 6As shown, the semantically guided reconstruction network includes J cascaded semantically guided reconstruction units corresponding to each viewpoint, and each semantically guided reconstruction unit includes a semantically guided subunit and a reconstruction subunit.

[0072] In this embodiment, the semantic guidance subunit is used to guide image reconstruction based on the global semantic vector, thereby reconstructing semantically consistent image details. The reconstruction subunit is used to perform secondary feature reconstruction on the feature map output by the semantic guidance subunit to reconstruct a super-resolution stereo image. In this embodiment, the reconstruction subunit may include D2B; similarly, the hole ratio of D2B at different levels is different, which will not be described in detail here.

[0073] Based on this, the super-resolution stereo image for each viewpoint is obtained as follows: Obtain the input visual enhancement feature map of the j-th semantic guidance subunit; The input visual enhancement feature map and the global semantic vector are input to the j-th semantic guidance subunit, and feature modulation is performed in the spatial dimension and the channel dimension respectively to obtain the spatial modulation feature map and the channel modulation feature map. The spatial modulation feature map and the channel modulation feature map are then added pixel by pixel to obtain the fused modulation feature map output by the j-th semantic guidance subunit. The fused modulation feature map output by the j-th semantic guidance subunit is input into the j-th reconstruction subunit for feature reconstruction to obtain a reconstructed feature map. A convolution operation is then performed on the reconstructed feature map output by the last reconstruction subunit to obtain the super-resolution stereo image. Wherein, when j=1, the target visual enhancement feature map is used as the input visual enhancement feature map of the first semantic guidance subunit, and when 1<j≤J, the reconstructed feature map output by the (j-1)th reconstruction subunit is used as the input visual enhancement feature map of the jth semantic guidance subunit.

[0074] In this embodiment, the semantically guided reconstruction network comprises J stages. For the j-th stage, the input visual enhancement feature map of the j-th semantically guided subunit is first obtained. Here, if it is the first stage (j=1), the target visual enhancement feature map output by the aforementioned SGPAM is directly used as the input. If it is a subsequent stage (1<j≤J), the reconstruction feature map output by the reconstruction subunit of the previous stage (i.e., the (j-1)-th) is used as the input of this stage.

[0075] Next, the input visual enhancement feature map and the previously extracted global semantic vector are input to the j-th semantic guidance subunit. Within this semantic guidance subunit, two-dimensional modulation is performed in parallel: spatial modulation parameters are generated using the global semantic vector to modulate the input features spatially, resulting in a spatial modulation feature map; channel modulation parameters are generated using the global semantic vector to modulate the input features channel-wise, resulting in a channel modulation feature map. Then, the spatial modulation feature map and the channel modulation feature map are added pixel by pixel to obtain the fused modulation feature map output by the semantic guidance subunit.

[0076] Subsequently, the fused modulation feature map is input into the last reconstruction subunit, for example, by performing deep feature extraction and reconstruction through the j-th DB, to obtain the reconstructed feature map of the j-th stage.

[0077] After completing the cascaded processing of all J stages, the reconstructed feature map output by the Jth reconstruction subunit is subjected to a final convolution operation, such as a 3×3 convolution, to map the number of feature channels to the number of image channels, thereby obtaining a super-resolution stereo image.

[0078] The stereo image reconstruction method proposed in this embodiment uses J cascaded semantically guided reconstruction units to modulate and fuse the input features layer by layer in both spatial and channel dimensions using global semantic vectors, thereby achieving a final output of a super-resolution stereo image with rich details and high semantic consistency.

[0079] In some embodiments, the semantic guidance subunit includes a first fully connected layer and a convolutional parameter generation layer; The spatial modulation feature map is obtained in the following way: The global semantic vector is input into the first fully connected layer for feature mapping to obtain intermediate semantic features; The semantic intermediate features are input into the convolutional parameter generation layer, and the semantic intermediate features are converted into kernel parameters of the deep convolutional kernel through a reshaping operation; A dynamic convolution kernel is constructed based on the kernel parameters, and the input visual enhancement feature map is convolved using the dynamic convolution kernel to obtain the spatial modulation feature map.

[0080] In this embodiment, the semantic guidance subunit includes a first fully connected layer and a convolutional parameter generation layer, as referenced. Figure 7 As shown, the first fully connected layer includes fully connected layers FC1 and FC2, and the convolution parameter generation layer includes a reshape operation layer, a depthwise convolutional layer, and a convolutional layer with a kernel size of 1×1.

[0081] Here, we will continue to use the generation process of the spatial modulation feature map of the left view in the j-th stage as an example for explanation. The generation process of the spatial modulation feature map of the right view in the j-th stage is the same as that of the spatial modulation feature map of the right view, and will not be described in detail here.

[0082] First, the one-dimensional global semantic vector Input fully connected layers FC1 and FC2. This layer maps semantic vectors into semantic intermediate features with specific dimensions through matrix multiplication and non-linear activation functions. , Next, this semantic intermediate feature... The input convolutional parameter generation layer transforms the one-dimensional feature vector into a four-dimensional tensor through a dimension reshaping operation; these values ​​serve as the kernel parameters for the depthwise convolutional kernel. Finally, a dynamic convolutional kernel is constructed using these semantically generated kernel parameters, and this dynamic convolutional kernel is used to enhance the input visual feature map of the left-view perspective in the (j-1)th stage. Perform a convolution operation to obtain the spatial modulation feature map of the left viewpoint in the j-th stage. , .

[0083] The stereo image reconstruction method proposed in this embodiment constructs a dynamic convolution kernel based on a global semantic vector using a fully connected layer mapping and a convolution parameter generation layer. The dynamic convolution is used to perform spatial dimension semantic modulation on the input visual enhancement feature map, accurately endowing the feature map with spatial semantic constraints and improving the matching degree between the spatial structure of the feature map and the global semantics.

[0084] In some embodiments, the semantic guidance subunit includes a second fully connected layer and a feature scaling layer; The channel modulation feature map is obtained in the following way: The global semantic vector is input into the second fully connected layer to obtain the channel modulation coefficients; Based on the channel modulation coefficients, the input visual enhancement feature map is weighted channel by channel through the feature scaling layer to obtain the channel modulation feature map.

[0085] In this embodiment, the semantic guidance subunit also includes a second fully connected layer and an activation layer, and continues to refer to... Figure 7 As shown, the second fully connected layer includes fully connected layers FC3 and FC4, and the feature scaling layer performs pixel-by-pixel multiplication.

[0086] Here, we will continue to use the generation process of the channel modulation feature map from the left perspective in the j-th stage as an example for explanation. The generation process of the channel modulation feature map from the right perspective in the j-th stage is the same as that of the channel modulation feature map from the right perspective, and will not be described in detail here.

[0087] First, the one-dimensional global semantic vector Input fully connected layers FC3 and FC4. Channel-level modulation coefficients are generated using the Sigmoid activation function. , Finally, the channel-level modulation coefficients are used in the feature scaling layer. Input visual enhancement feature map for the left viewpoint in the (j-1)th stage Scaling adjustments are made to further enhance the feature channels that match semantics, resulting in the channel modulation feature map of the left-view perspective at the j-th stage. , .

[0088] The stereo image reconstruction method proposed in this embodiment transforms the global semantic vector into channel modulation coefficients using a second fully connected layer, and then performs a channel-wise weighted operation on the input visual enhancement feature map through a feature scaling layer. This accurately achieves semantic modulation of the feature channel dimensions, strengthens feature channels related to global semantics, and suppresses irrelevant channels, effectively improving the channel semantic recognition of the feature map.

[0089] In some embodiments, the network parameters of the feature extraction network, the semantically guided disparity attention network, and the semantically guided reconstruction network are obtained by minimizing the total loss, which is obtained in the following manner: Obtain the real super-resolution stereo image and the reconstructed predicted super-resolution stereo image corresponding to each viewpoint; The super-resolution loss is obtained based on the average absolute error between the real super-resolution stereo image and the predicted super-resolution stereo image corresponding to all viewpoints. Based on the real super-resolution stereo images and the predicted super-resolution stereo images corresponding to all viewpoints, the semantic consistency loss is obtained by applying it to the visual encoder of the pre-trained visual language model. The total loss is obtained based on the super-resolution loss and the semantic consistency loss.

[0090] First, obtain the ground truth (HR) stereo image corresponding to each viewpoint and the reconstructed predicted super-resolution stereo image (SR) output by the network.

[0091] Next, based on the mean absolute error between the real super-resolution stereo images and the predicted super-resolution stereo images corresponding to all viewpoints, the super-resolution loss is obtained. , This super-resolution loss The constrained reconstructed image approximates the real image in terms of pixel values.

[0092] Furthermore, this embodiment also calculates semantic consistency loss, which constrains the reconstructed image to maintain semantic understanding consistency with the real image. Specifically, the real super-resolution stereo image and the predicted super-resolution stereo image are input into the visual encoder of a pre-trained visual language model, such as the visual encoder of CLIP, to extract their high-level features or embedding vectors. The distance between these two sets of features is calculated to obtain the semantic consistency loss. , .

[0093] Finally, the total loss is obtained by weighted summation based on the super-resolution loss and semantic consistency loss. , Here, These are pre-set conventional coefficients used to balance the two losses, which will not be elaborated on here.

[0094] The stereo image reconstruction method proposed in this embodiment jointly optimizes the super-resolution loss and semantic consistency loss. Using real super-resolution stereo images as a benchmark, it both uses the mean absolute error to constrain the pixel-level differences between the predicted image and the real image, ensuring the visual fidelity of the reconstructed image, and uses the visual encoder of a pre-trained visual language model to constrain the semantic representation matching degree between the predicted image and the real image, strengthening the semantic consistency of the reconstructed image. Finally, it optimizes the network parameters by minimizing the total loss, thereby ensuring the accuracy of the super-resolution stereo image reconstructed based on this network.

[0095] Based on any of the above embodiments, the present invention also provides a stereoscopic image reconstruction apparatus, the apparatus comprising: The view acquisition module is used to acquire a single view of the target at high resolution from two symmetrical perspectives in a stereo vision system. A semantic feature extraction network is used to obtain a multi-scale visual feature map corresponding to each single view based on the feature extraction network, and to obtain a multi-scale semantic feature map and a global semantic vector corresponding to each single view based on a pre-trained visual language model. The view enhancement module is used to perform cross-view fusion of the multi-scale visual feature maps and the multi-scale semantic feature maps of all viewpoints based on a semantically guided parallax attention network to obtain the target visual enhancement feature map for each viewpoint. The reconstruction module is used to apply the target visual enhancement feature map to a semantically guided reconstruction network for each viewpoint. Based on the global semantic vector, the visual enhancement feature map is modulated and reconstructed in the spatial and channel dimensions to obtain a super-resolution stereo image reconstructed for each viewpoint.

[0096] The stereo image reconstruction apparatus provided in this invention, after acquiring single views of the target at high resolution from two symmetrical perspectives in a stereo vision system, first obtains multi-scale visual structural features and semantic representation features through a visual semantic dual-branch feature extraction architecture. Then, based on a semantically guided disparity attention mechanism, semantic information is integrated into the cross-view disparity calculation and feature fusion process, improving the accuracy of cross-view information capture. Finally, through feature modulation reconstruction of global semantic vectors in two dimensions of spatial channels, the geometric structure and semantic description of the super-resolution image are highly consistent, improving the accuracy of the reconstructed super-resolution stereo image.

[0097] The stereoscopic image reconstruction device described in this embodiment can be referred to in correspondence with the stereoscopic image reconstruction method provided by the present invention described above, and will not be described in detail here.

[0098] Figure 8 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 8 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute a stereoscopic image reconstruction method, which includes: Acquire a single view of the target at high resolution from two symmetrical perspectives in a stereo vision system; For each single view, a multi-scale visual feature map corresponding to the single view is obtained based on a feature extraction network, and a multi-scale semantic feature map and a global semantic vector corresponding to the single view are obtained based on a pre-trained visual language model visual encoder. A semantically guided parallax attention network is used to perform cross-view fusion of the multi-scale visual feature maps and the multi-scale semantic feature maps of all viewpoints to obtain the target visual enhancement feature map for each viewpoint. For each viewpoint, the target visual enhancement feature map is applied to a semantically guided reconstruction network. Based on the global semantic vector, the visual enhancement feature map is modulated and reconstructed in both spatial and channel dimensions to obtain a super-resolution stereo image reconstructed for each viewpoint.

[0099] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0100] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer is able to execute the stereoscopic image reconstruction method provided by the above methods, the method comprising: Acquire a single view of the target at high resolution from two symmetrical perspectives in a stereo vision system; For each single view, a multi-scale visual feature map corresponding to the single view is obtained based on a feature extraction network, and a multi-scale semantic feature map and a global semantic vector corresponding to the single view are obtained based on a pre-trained visual language model visual encoder. A semantically guided parallax attention network is used to perform cross-view fusion of the multi-scale visual feature maps and the multi-scale semantic feature maps of all viewpoints to obtain the target visual enhancement feature map for each viewpoint. For each viewpoint, the target visual enhancement feature map is applied to a semantically guided reconstruction network. Based on the global semantic vector, the visual enhancement feature map is modulated and reconstructed in both spatial and channel dimensions to obtain a super-resolution stereo image reconstructed for each viewpoint.

[0101] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the stereoscopic image reconstruction methods provided by the methods described above, the method comprising: Acquire a single view of the target at high resolution from two symmetrical perspectives in a stereo vision system; For each single view, a multi-scale visual feature map corresponding to the single view is obtained based on a feature extraction network, and a multi-scale semantic feature map and a global semantic vector corresponding to the single view are obtained based on a pre-trained visual language model visual encoder. A semantically guided parallax attention network is used to perform cross-view fusion of the multi-scale visual feature maps and the multi-scale semantic feature maps of all viewpoints to obtain the target visual enhancement feature map for each viewpoint. For each viewpoint, the target visual enhancement feature map is applied to a semantically guided reconstruction network. Based on the global semantic vector, the visual enhancement feature map is modulated and reconstructed in both spatial and channel dimensions to obtain a super-resolution stereo image reconstructed for each viewpoint.

[0102] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0103] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0104] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical coding feature maps; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for reconstructing a stereo image, characterized in that, The method includes: Acquire a single view of the target at high resolution from two symmetrical perspectives in a stereo vision system; For each single view, a multi-scale visual feature map corresponding to the single view is obtained based on a feature extraction network, and a multi-scale semantic feature map and a global semantic vector corresponding to the single view are obtained based on a pre-trained visual language model visual encoder. A semantically guided parallax attention network is used to perform cross-view fusion of the multi-scale visual feature maps and the multi-scale semantic feature maps of all viewpoints to obtain the target visual enhancement feature map for each viewpoint. For each viewpoint, the target visual enhancement feature map is applied to a semantically guided reconstruction network. Based on the global semantic vector, the visual enhancement feature map is modulated and reconstructed in both spatial and channel dimensions to obtain a super-resolution stereo image reconstructed for each viewpoint.

2. The stereoscopic image reconstruction method according to claim 1, characterized in that, Both the multi-scale visual feature map and the multi-scale semantic feature map include I levels of feature maps ordered from low resolution to high resolution, and the resolution of the semantic feature map at the same level is smaller than that of the visual feature map. The semantically guided parallax attention network comprises I cascaded semantically guided parallax attention units; The target visual enhancement feature map for each viewpoint is obtained in the following way: Obtain the input visual feature map of the i-th level; The resolution of the semantic feature map at the i-th level is upsampled to be the same as the resolution of the input visual feature map at the i-th level to obtain the input semantic feature map; The input visual feature map and the input semantic feature map of the i-th level of all viewpoints are input into the semantically guided disparity attention unit corresponding to the i-th level to obtain the visual enhancement feature map of the i-th level of each viewpoint, and the visual enhancement feature map of the last level is used as the target visual enhancement feature map. Wherein, when i=1, the visual feature map of the first level is used as the input visual feature map; When 1 < i ≤ I, the visual enhancement feature map of the (i-1)th level is fused with the visual feature map of the ith level to obtain the input visual feature map of the ith level.

3. The stereoscopic image reconstruction method according to claim 2, characterized in that, The semantically guided disparity attention unit includes a feature transformation subunit, an attention calculation subunit, and an orthogonal projection fusion subunit; The visual enhancement feature maps for each level are obtained as follows: Based on the feature transformation subunit, the semantic query vector corresponding to the first input semantic feature map of the current view, the visual query vector corresponding to the first input visual feature map, the semantic key vector corresponding to the second input semantic feature map of another view, and the visual key vector corresponding to the second input visual feature map of the other view are obtained respectively. Determine a first similarity score map between the semantic query vector and the semantic key vector, and determine a second similarity score map between the visual query vector and the visual key vector; The first similarity score map and the second similarity score map are input into the attention calculation subunit to obtain the disparity attention map of the current view. Based on the disparity attention map of the current view, a cross-view feature map aligned with the first input visual feature map is aggregated from the second input visual feature map. Based on the orthogonal projection fusion subunit, an orthogonal feature map orthogonal to the first input visual feature map is extracted from the cross-view feature map, and the orthogonal feature map is fused with the first input visual feature map to obtain the visual enhancement feature map of the current view.

4. The stereoscopic image reconstruction method according to claim 1, characterized in that, The semantically guided reconstruction network includes J cascaded semantically guided reconstruction units corresponding to each viewpoint, and each semantically guided reconstruction unit includes a semantically guided subunit and a reconstruction subunit. The super-resolution stereo images for each viewpoint are obtained as follows: Obtain the input visual enhancement feature map of the j-th semantic guidance subunit; The input visual enhancement feature map and the global semantic vector are input to the j-th semantic guidance subunit, and feature modulation is performed in the spatial dimension and the channel dimension respectively to obtain the spatial modulation feature map and the channel modulation feature map. The spatial modulation feature map and the channel modulation feature map are added pixel by pixel to obtain the fused modulation feature map output by the j-th semantic guidance subunit. The fused modulation feature map output by the j-th semantic guidance subunit is input into the j-th reconstruction subunit for feature reconstruction to obtain a reconstructed feature map. A convolution operation is then performed on the reconstructed feature map output by the last reconstruction subunit to obtain the super-resolution stereo image. Wherein, when j=1, the target visual enhancement feature map is used as the input visual enhancement feature map of the first semantic guidance subunit, and when 1<j≤J, the reconstructed feature map output by the (j-1)th reconstruction subunit is used as the input visual enhancement feature map of the jth semantic guidance subunit.

5. The stereoscopic image reconstruction method according to claim 4, characterized in that, The semantic guidance subunit includes a first fully connected layer and a convolutional parameter generation layer; The spatial modulation feature map is obtained in the following way: The global semantic vector is input into the first fully connected layer for feature mapping to obtain intermediate semantic features; The semantic intermediate features are input into the convolutional parameter generation layer, and the semantic intermediate features are converted into kernel parameters of the deep convolutional kernel through a reshaping operation; A dynamic convolution kernel is constructed based on the kernel parameters, and the input visual enhancement feature map is convolved using the dynamic convolution kernel to obtain the spatial modulation feature map.

6. The stereoscopic image reconstruction method according to claim 4, characterized in that, The semantic guidance subunit includes a second fully connected layer and a feature scaling layer; The channel modulation feature map is obtained in the following way: The global semantic vector is input into the second fully connected layer to obtain the channel modulation coefficients; Based on the channel modulation coefficients, the input visual enhancement feature map is weighted channel by channel through the feature scaling layer to obtain the channel modulation feature map.

7. The stereoscopic image reconstruction method according to any one of claims 1 to 6, characterized in that, The network parameters of the feature extraction network, the semantically guided disparity attention network, and the semantically guided reconstruction network are obtained by minimizing the total loss, which is obtained in the following way: Obtain the real super-resolution stereo image and the reconstructed predicted super-resolution stereo image corresponding to each viewpoint; The super-resolution loss is obtained based on the average absolute error between the real super-resolution stereo image and the predicted super-resolution stereo image corresponding to all viewpoints. Based on the real super-resolution stereo images and the predicted super-resolution stereo images corresponding to all viewpoints, the semantic consistency loss is obtained by applying it to the visual encoder of the pre-trained visual language model. The total loss is obtained based on the super-resolution loss and the semantic consistency loss.

8. A stereoscopic image reconstruction device, characterized in that, The device includes: The view acquisition module is used to acquire a single view of the target at high resolution from two symmetrical perspectives in a stereo vision system. A semantic feature extraction network is used to obtain a multi-scale visual feature map corresponding to each single view based on the feature extraction network, and to obtain a multi-scale semantic feature map and a global semantic vector corresponding to each single view based on a pre-trained visual language model. The view enhancement module is used to perform cross-view fusion of the multi-scale visual feature maps and the multi-scale semantic feature maps of all viewpoints based on a semantically guided parallax attention network to obtain the target visual enhancement feature map for each viewpoint. The reconstruction module is used to apply the target visual enhancement feature map to a semantically guided reconstruction network for each viewpoint. Based on the global semantic vector, the visual enhancement feature map is modulated and reconstructed in the spatial and channel dimensions to obtain a super-resolution stereo image reconstructed for each viewpoint.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the stereo image reconstruction method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the stereo image reconstruction method as described in any one of claims 1 to 7.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the stereo image reconstruction method as described in any one of claims 1 to 7.

Images