Depth-guided inter-frame-intra-frame stereo image redirection method and device

Abstract

The present invention discloses a depth-guided inter-frame and intra-frame stereoscopic image retargeting method and device. The method comprises: constructing a stereoscopic image retargeting network, the network comprising: an inter-frame and intra-frame feature extraction module; a depth-guided feature integration module; the inter-frame and intra-frame feature extraction module obtains deep features of long-range dependency information between and within views; the depth-guided feature integration module utilizes a disparity attention learning mechanism based on deep features to interactively learn cross-view information of stereo features, aggregates the stereo correspondence between left and right views, and then uses channel attention to generate a stereo attention map for the left and right views; a hybrid loss function is used to optimize the stereoscopic image retargeting network to maintain the structural integrity and disparity consistency of the stereoscopic image retargeting result; and retargeting the stereoscopic image based on the optimized stereoscopic image retargeting network. The device comprises: a processor and a memory. The present invention enhances the consistency of disparity changes, thereby achieving high-quality stereoscopic image retargeting.

Description

Technical Field

[0001] The present invention relates to the field of image redirection, and in particular to a depth-guided inter-frame and intra-frame stereoscopic image redirection method and device. Background Art

[0002] Stereoscopic image retargeting is a hotly debated topic in computer graphics and multimedia 3D display. This technology involves converting input stereo images into high-quality stereo images at arbitrary resolutions. Stereoscopic image retargeting plays a vital role in many 3D devices and applications, including 3D movies and television, 3D games, mobile phones, and iPads. However, due to the diversity of 3D smart devices, directly displaying the same 3D content on devices of varying sizes and resolutions can result in an uncomfortable visual experience. Therefore, intelligently retargeting stereoscopic image content to displays of varying aspect ratios while ensuring compatibility with human visual perception is a challenging task.

[0003] In recent years, researchers have proposed a variety of 2D image retargeting methods to improve human visual quality. Traditional 2D image retargeting methods can be categorized as discrete, continuous, and hybrid. Examples include saliency-based, depth-based, edge-based, and seam-cutting-assisted methods. Compared to traditional methods using handcrafted features, deep learning offers outstanding performance in extracting salient objects and detail information from images. Researchers have proposed 2D image retargeting methods based on convolutional neural networks (CNNs). For example, WSSDCNN (weakly supervised and self-supervised deep convolutional neural networks) and CycleIR (Cycle Image Retargeting) address issues such as warping distortion and algorithmic complexity in image retargeting. Due to the rapid development of stereoscopic devices, researchers have proposed methods for retargeting stereo images to adjust the aspect ratio of 3D content. Inspired by the success of deep learning in 2D image retargeting, researchers have proposed using convolutional neural networks for stereo image retargeting. For example, Fan et al. proposed a deep Siamese network for stereo image resizing. This method uses a cross-attention extraction module to obtain an attention map and a disparity generation module to generate a 3D importance map. In addition, Imani et al. proposed using mobile layers to deform video frames and constructed a stereo video Transformer to assist in the redirection process. The above method can redirect stereo images without obvious content distortion.

[0004] In the process of realizing the present invention, the inventors found that the prior art has at least the following shortcomings and deficiencies:

[0005] Existing methods utilize the local receptive field of convolutional neural networks to capture the correspondence between left and right views, but this results in an inability to model the long-range dependencies of stereo images and, consequently, to accurately extract the deep features of salient objects in stereo images. Furthermore, existing methods often employ cost-volume methods to estimate disparity relationships, which results in an inability to model the varying depth information of stereo images and, consequently, to effectively preserve depth information. Summary of the Invention

[0006] The present invention provides a depth-guided inter-intra stereo image retargeting method and apparatus. Leveraging deep learning techniques, the present invention employs an inter-intra feature extraction module to capture long-range inter-view and intra-view dependencies in stereo images. Furthermore, a depth-guided feature integration module is employed to learn the stereo correspondence between left and right views, thereby enhancing the consistency of parallax changes and achieving high-quality stereo image retargeting. Details are described below:

[0007] In the first aspect, a depth-guided inter-frame-intra-frame stereo image redirection method, the method comprising: constructing

[0008] A stereo image redirection network, comprising: an inter-frame-intra-frame feature extraction module, a depth-guided feature integration module, wherein:

[0009] The inter-frame and intra-frame feature extraction module obtains deep features of long-range dependency information between views and within views;

[0010] The depth-guided feature integration module uses a disparity attention learning mechanism based on deep features to interact with cross-view information of stereo features and aggregate the stereo correspondence between the left and right views, and then uses channel attention to generate a stereo attention map for the left and right views;

[0011] A hybrid loss function is used to optimize the stereo image redirection network to maintain the structural integrity and disparity consistency of the stereo image redirection results; stereo images are redirected based on the optimized stereo image redirection network.

[0012] The inter-frame-intra-frame feature extraction module includes: P cascaded Swin Transformer sub-modules, each sub-module includes: an intra-frame feature learning layer, an inter-frame-intra-frame feature enhancement layer and an inter-frame feature interaction layer.

[0013] Among them, the intra-frame feature learning layer is:

[0014] Using a non-overlapping N×N window, the left view F of size H×W×C is l and right view F r The basic features are adjusted to the size of (H×W / N 2 )×N 2×C blocks;

[0015] Each local window is layer normalized, and the feature X of each local window is generated with a learnable weight matrix W Q 、W K and W V The query Q, key K and value V;

[0016] Windowed multi-head self-attention is adopted to capture information from different views, and layer normalization and multi-layer perception are applied to refine feature labels.

[0017] The calculation process of the intra-frame feature learning layer is as follows:

[0018]

[0019]

[0020] Among them, X l and X r are the input features of the left and right views of the intra-frame feature learning layer, Z l and Z r are the output features of the left and right views of the intra-frame feature learning layer, W-MSA(·) is the window multi-head self-attention, LN(·) and MLP(·) are the normalization layer and the multi-layer perception layer, The output features and input features of the window multi-head self-attention of the left view are X l The feature map obtained by adding, The output features and input features X of the window multi-head self-attention of the right view r The feature map obtained by addition.

[0021] The inter-frame-intra-frame feature enhancement layer is:

[0022] Three dilated convolutional layers are used to expand the inter-view features of the left and right views by three scales respectively, and the expanded features are cascaded together;

[0023] Three convolutional layers with gradually increasing kernel sizes are stacked on the common features. The unique features of the left and right views are obtained by extracting the common features from the cascaded features and then added to the original in-view features.

[0024] The calculation process of the inter-frame-intra-frame feature enhancement layer is as follows:

[0025] R l =Z l +U l

[0026] R r =Z r +U r

[0027] and

[0028]

[0029] Among them, Z l and Z r is the output in-view feature of the left and right views, U l and U r is the only feature of the right and left views, M self is a self-refining feature, and There are three dilated convolutional layers with a kernel size of 3×3. Conv1, Conv3, and Conv5 are convolutional layers with three different kernel sizes. is a multiplication operation.

[0030] The inter-frame feature interaction layer adopts offset window multi-head interactive attention, given the enhanced features of the left and right views, and the overall process of the inter-frame feature interaction layer of each local window is defined as follows:

[0031]

[0032]

[0033] Among them, R l and R r are the output features of the left and right views of the inter-intra feature enhancement layer, D l and D r are the output features of the left and right views of the inter-frame feature interaction layer, SW-MSA(·) is the offset window multi-head interaction attention, The output features of the offset window multi-head interactive attention of the left view and the output feature R l The feature map obtained by adding, The output features of the offset window multi-head interactive attention of the right view and the output feature R r The feature map obtained by addition.

[0034] The depth-guided feature integration module is as follows: the output features of the three ISTBs of the left and right views in the inter-frame and intra-frame feature extraction module are connected respectively, the cross-view information of the stereo features of the bidirectional disparity attention interaction between the stereo images is used, and the left and right features are converted to the other side, and the intra-frame feature learning layer and the inter-frame feature interaction layer are used again to aggregate the depth information of the left and right views, and the channel attention is used to generate the stereo attention map G of the left and right views. l and G r .

[0035] Among them, the stereo attention map G land G r for:

[0036] G l =CA(C(IFIL(IFLL(Q l ,F′ l-r ))))

[0037] G r =CA(C(IFIL(IFLL(Q r ,F′ r-l ))))

[0038] and

[0039]

[0040] Among them, M l-r and M r-l is the disparity attention map of the left and right views, Q l and Q r is the cascade feature of the left view, F′ l-r and F′ r-l is the conversion feature of the left and right views, conv1 is a convolutional layer with a kernel size of 1×1, C(·) is the cascade, and CA(·) is the channel attention.

[0041] In a second aspect, a depth-guided inter-frame to intra-frame stereoscopic image redirection device is provided, the device comprising: a processor and a memory, the memory storing program instructions, the processor calling the program instructions stored in the memory to enable the device to execute any one of the methods described in the first aspect.

[0042] The beneficial effects of the technical solution provided by the present invention are:

[0043] 1. This method can preserve the structure and texture of salient objects in stereo image redirection results, reduce binocular parallax distortion, and obtain high-quality stereo image redirection results;

[0044] 2. This method is the first to use the long-range inter-frame and intra-frame dependency relationship between the left and right views to solve the stereo image redirection problem, which can accurately extract the salient information of the stereo image. At the same time, it uses the depth information of the left and right views to constrain the parallax change, thereby reducing the parallax distortion of the stereo image redirection result. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 Schematic diagram of the inter-frame-intra-frame feature extraction module;

[0046] Figure 2 Schematic diagram of the parallax attention learning mechanism;

[0047] Figure 3Schematic diagram of the depth distortion score comparison results of stereo image redirection results;

[0048] Figure 4 Flowchart of a depth-guided inter-intra stereo image retargeting method. DETAILED DESCRIPTION

[0049] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention are described in further detail below.

[0050] In order to overcome the shortcomings of the existing technology, an embodiment of the present invention designs an inter-intra stereo image redirection method based on depth guidance. The method consists of three modules: the inter-intra feature extraction module uses the inter-intra Swin Transformer submodule to obtain deep features of long-range dependency information between views and within views; the depth-guided feature integration module uses the disparity attention learning mechanism to interact with the cross-view information of stereo features and aggregate the stereo correspondence between the left and right views; the hybrid loss function is used to maintain the structural integrity and disparity consistency of the stereo image redirection result. This stereo image redirection method achieves accurate scaling of the image size, reduces content distortion and maintains the binocular parallax relationship, as described below:

[0051] Example 1

[0052] A depth-guided inter-frame to intra-frame stereo image retargeting method, the method comprising the following steps:

[0053] 1. Inter-frame and intra-frame feature extraction module

[0054] For stereo image retargeting, maintaining visually salient areas in the input stereo image and preventing content artifacts and distortion in the retargeted stereo image are fundamental tasks when changing the target aspect ratio and resolution. To preserve information about the same salient areas in both the left and right views, an inter-intra feature extraction module is designed in this embodiment to extract deep features. The inter-intra Swin Transformer is designed to learn long-range dependencies between and within views.

[0055] Given a stereo image pair, the backbone network is first used to extract the basic shallow features of the left and right views. The backbone network consists of two 3×3 convolutional layers and a parameterized Relu activation function. Then, the inter-intra feature extraction module is used to extract deep features. The inter-intra feature extraction module consists of P cascaded Swin Transformer submodules (ISTB). Specifically, each ISTB includes: an intra feature learning layer (IFLL), an inter-intra feature enhancement layer (IFRL), and an inter feature interaction layer (IFIL), as shown in Figure 2. Figure 1 shown.

[0056] Intra-frame feature learning layer: In order to integrate the global information and local information in the left and right views respectively, the intra-frame feature learning layer is designed to extract the intra-view features of the stereo image. Specifically, first, a non-overlapping N×N window is used to transform the left view F of size H×W×C into l and right view F r The basic features are adjusted to the size of (H×W / N 2 )×N 2 ×C blocks. Then, each local window is layer normalized, where the feature X of each local window generates a learnable weight matrix W Q 、W K and W V The query Q, key K and value V are constructed. Next, considering the visual attention distribution in stereo images, windowed multi-head self-attention is adopted to capture information from different views. Finally, layer normalization and multi-layer perception are applied to refine the feature labels, where residual connections are applied to the above stages.

[0057] The process of the intra-frame feature learning layer of each local window is defined as follows:

[0058]

[0059] Among them, X l and X r Z is the input feature of the left and right views of the intra-frame feature learning layer. l and Z r are the output features of the left and right views of the intra-frame feature learning layer. W-MSA(·) is the windowed multi-head self-attention. LN(·) and MLP(·) are the normalization layer and the multi-layer perception layer.

[0060] Inter-Intra Feature Enhancement Layer: Due to binocular parallax in stereo images, the left and right views contain both common and unique information. To distinguish the common and unique information between the left and right views and enhance the unique information of the left and right views, an inter-intra feature enhancement layer is designed to extract inter-view and intra-view features using different receptive fields. Specifically, to capture multi-scale contextual information without losing image resolution, the inter-view features of the left and right views are first dilated by three scales respectively using three dilated convolutional layers. The dilated features are then concatenated. Furthermore, considering the complementarity and interaction between the left and right views at different scales, a self-refinement strategy is applied to the common features by increasing the receptive field size. This involves stacking three convolutional layers with gradually increasing kernel sizes. Finally, the unique features of the left and right views are obtained by extracting the common features from the concatenated features. These unique features are then added to the original intra-view features for feature enhancement.

[0061] Among them, the overall process of the inter-frame-intra-frame feature enhancement layer is expressed as:

[0062] R l =Z l +U l

[0063] R r =Z r +U r

[0064] and

[0065]

[0066] Among them, Z l and Z r is the output in-view feature of the left and right views, U l and U r is the only feature of the right and left views, M self is a self-refining feature, and There are three dilated convolutional layers with a kernel size of 3×3, and the dilation rates are set to 1, 2, and 4. Conv1, Conv3, and Conv5 are three convolutional layers with different kernel sizes, namely 1×1, 3×3, and 5×5. is a multiplication operation.

[0067] Inter-frame feature interaction layer: To further promote information association and interaction between the left and right views, an embodiment of the present invention designs an inter-frame feature interaction layer to extract inter-view features between the left and right views. The inter-frame feature interaction layer and the intra-frame feature learning layer have similar network architectures. However, to further achieve long-range dependency exchange between different views, offset window multi-head interactive attention is adopted in the inter-frame feature interaction layer. Therefore, given the enhanced features of the left and right views, the overall process of the inter-frame feature interaction layer for each local window is defined as follows:

[0068]

[0069] Among them, R l and R r are the output features of the left and right views of the inter-intra feature enhancement layer, D l and D r are the output features of the left and right views of the inter-frame feature interaction layer. SW-MSA(·) is the offset window multi-head interactive attention.

[0070] 2. Depth-guided Feature Integration Module

[0071] The depth information of stereo images provides 3D scene information between the left and right views, which helps maintain the original disparity relationship when retargeting the stereo images. Therefore, the embodiment of the present invention designs a depth-guided feature integration module to learn stereo correspondences and enhance the consistency of disparity changes.

[0072] In the depth-guided feature integration module, the present invention proposes a disparity attention learning mechanism to exchange cross-view information of stereo features and aggregate the depth information of left and right views. Figure 2 As shown. Specifically, considering that deep hierarchical features are helpful for stereo relationship learning, the output features of the three ISTBs of the left and right views in the inter-intra feature extraction module are first connected separately. Then, the cross-view information of the stereo features is exchanged using the bidirectional disparity attention between the stereo images, and the left and right features are converted to the other side. Then, the intra-frame feature learning layer and the inter-frame feature interaction layer are used again to aggregate the depth information of the left and right views. Finally, the channel attention is used to generate the stereo attention map G of the left and right views. l and G r , which is defined as:

[0073] G l =CA(C(IFIL(IFLL(Q l ,F′ l-r ))))

[0074] G r =CA(C(IFIL(IFLL(Q r,F′ r-l ))))

[0075] and

[0076]

[0077] Among them, M l-r and M r-l is the disparity attention map of the left and right views, Q l and Q r is the cascade feature of the left view, F′ l-r and F′ r-l is the conversion feature of the left and right views, conv1 is a convolutional layer with a kernel size of 1×1, C(·) is the cascade, and CA(·) is the channel attention.

[0078] 3. Mixed Loss

[0079] The proposed network aims to explore the deep feature information of stereo images and promote the interaction of long-range dependencies between the left and right views, thereby improving the quality of stereo image retargeting. Therefore, a hybrid loss function is designed to optimize the proposed stereo image retargeting network, which includes structural similarity loss and disparity consistency loss. The hybrid loss is defined as follows:

[0080] L hybrid =αL str +βL dis

[0081] Among them, α and β are the control structure similarity loss L str and parallax consistency loss L dis The trade-off parameters between .

[0082] Structural similarity loss: Structural similarity can reflect the contextual information of stereo images from three aspects: light, contrast, and structure. Therefore, the embodiments of the present invention use the structural similarity loss between the reconstructed stereo image and the input stereo image to ensure that the redirected left and right views retain more of the structure and texture of the corresponding views. The structural similarity loss is defined as:

[0083]

[0084] in, and are the input left and right views, and are the reconstructed left and right views obtained by re-feeding the re-redirected left and right views into the proposed stereo image re-direction network, and ssim(·) denotes the structural similarity operation.

[0085] Disparity consistency loss: Considering the difference in perspective between the left and right views can describe the depth information of the 3D scene. In order to ensure the disparity consistency of the stereo image redirection result and obtain better depth perception, a disparity consistency loss is designed to constrain the disparity difference between the reconstructed stereo image and the input stereo image. The disparity consistency loss is defined as:

[0086]

[0087] Here, |·| represents the absolute value operator, and ||·|| represents the L1 norm.

[0088] Example 2

[0089] Figure 3 This article compares the depth distortion scores of stereo image retargeting results. The comparison algorithms include the DPS method (a traditional stereo image retargeting algorithm) and the SIRCNN method (a deep learning-based stereo image retargeting algorithm). The lower the depth distortion score, the higher the 3D visual perception quality. Figure 3 It can be seen that compared with the DPS method, the depth distortion score obtained by the method proposed in the embodiment of the present invention is smaller, indicating that the depth guidance information integration module can effectively reduce the depth distortion of stereo image redirection. In addition, the depth distortion score obtained by the SIRCNN method is also worse than that of the method proposed in the embodiment of the present invention. The reason is that the embodiment of the present invention uses multiple cascaded Swin Transformer submodules to preparatory modeling of the long-range dependency of stereo images, effectively learns the depth feature information of the left and right images, and reduces the depth distortion of the stereo image redirection result. Figure 4 A technical flow chart of an embodiment of the present invention is given, which mainly includes an inter-frame-intra-frame feature extraction module, a depth-guided feature integration module, and a hybrid loss.

[0090] Example 3

[0091] A depth-guided inter-frame to intra-frame stereoscopic image retargeting device includes: a processor and a memory, wherein the memory stores program instructions, and the processor calls the program instructions stored in the memory to cause the device to perform the following method steps in Example 1:

[0092] Construct a stereo image redirection network, which includes: inter-frame and intra-frame feature extraction module, depth-guided feature integration module, among which,

[0093] Inter-frame and intra-frame feature extraction module to obtain deep features of long-range dependency information between and within views;

[0094] The depth-guided feature integration module uses the disparity attention learning mechanism based on deep features to interact with the cross-view information of stereo features, aggregate the stereo correspondences of the left and right views, and then use channel attention to generate a stereo attention map for the left and right views;

[0095] A hybrid loss function is used to optimize the stereo image redirection network to maintain the structural integrity and disparity consistency of the stereo image redirection results; stereo images are redirected based on the optimized stereo image redirection network.

[0096] The inter-intra feature extraction module includes P cascaded Swin Transformer submodules, each of which includes an intra-frame feature learning layer, an inter-intra feature enhancement layer, and an inter-frame feature interaction layer.

[0097] Among them, the intra-frame feature learning layer is:

[0098] Using a non-overlapping N×N window, the left view F of size H×W×C is l and right view F r The basic features are adjusted to the size of (H×W / N 2 )×N 2 ×C blocks;

[0099] Each local window is layer normalized, and the feature X of each local window is generated with a learnable weight matrix W Q 、W K and W V The query Q, key K and value V;

[0100] Windowed multi-head self-attention is adopted to capture information from different views, and layer normalization and multi-layer perception are applied to refine feature labels.

[0101] Among them, the calculation process of the intra-frame feature learning layer is:

[0102]

[0103] Among them, X l and X r are the input features of the left and right views of the intra-frame feature learning layer, Z l and Z r are the output features of the left and right views of the intra-frame feature learning layer, W-MSA(·) is the window multi-head self-attention, LN(·) and MLP(·) are the normalization layer and the multi-layer perception layer.

[0104] Among them, the inter-frame-intra-frame feature enhancement layer is:

[0105] Three dilated convolutional layers are used to expand the inter-view features of the left and right views by three scales respectively, and the expanded features are cascaded together;

[0106] Three convolutional layers with gradually increasing kernel sizes are stacked on the common features. The unique features of the left and right views are obtained by extracting the common features from the cascaded features and then added to the original in-view features.

[0107] Among them, the calculation process of the inter-frame-intra-frame feature enhancement layer is:

[0108] R l =Z l +U l

[0109] R r =Z r +U r

[0110] and

[0111]

[0112] Among them, Z l and Z r is the output in-view feature of the left and right views, U l and U r is the only feature of the right and left views, M self is a self-refining feature, and There are three dilated convolutional layers with a kernel size of 3×3. Conv1, Conv3, and Conv5 are convolutional layers with three different kernel sizes. is a multiplication operation.

[0113] Among them, the inter-frame feature interaction layer is: using offset window multi-head interactive attention, given the enhanced features of the left and right views, the overall process of the inter-frame feature interaction layer of each local window is defined as follows:

[0114]

[0115] Among them, R l and R r are the output features of the left and right views of the inter-intra feature enhancement layer, D l and D r are the output features of the left and right views of the inter-frame feature interaction layer, and SW-MSA(·) is the offset window multi-head interactive attention.

[0116] Among them, the depth-guided feature integration module is as follows: the output features of the three ISTBs of the left and right views in the inter-frame and intra-frame feature extraction module are connected respectively, the cross-view information of the stereo features is interactively obtained by using the bidirectional disparity attention between the stereo images, and the left and right features are converted to the other side. The intra-frame feature learning layer and the inter-frame feature interaction layer are used again to aggregate the depth information of the left and right views, and the channel attention is used to generate the stereo attention map G of the left and right views. l and G r .

[0117] Among them, the stereo attention map G l and G r for:

[0118] G l =CA(C(IFIL(IFLL(Q l ,F′ l-r ))))

[0119] G r =CA(C(IFIL(IFLL(Q r ,F′ r-l ))))

[0120] and

[0121]

[0122] Among them, M l-r and M r-l is the disparity attention map of the left and right views, Q l and Q r is the cascade feature of the left view, F′ l-r and F′ r-l is the conversion feature of the left and right views, conv1 is a convolutional layer with a kernel size of 1×1, C(·) is the cascade, and CA(·) is the channel attention.

[0123] It should be noted here that the device description in the above embodiment corresponds to the method description in the embodiment, and the embodiment of the present invention will not be described in detail here.

[0124] The execution subjects of the above-mentioned processor and memory can be computers, single-chip microcomputers, microcontrollers and other devices with computing functions. In specific implementation, the embodiment of the present invention does not limit the execution subject and it can be selected according to the needs of actual application.

[0125] Data signals are transmitted between the memory and the processor via a bus, which will not be described in detail in the embodiment of the present invention.

[0126] Based on the same inventive concept, an embodiment of the present invention further provides a computer-readable storage medium, which includes a stored program, and when the program is running, controls the device where the storage medium is located to execute the method steps in the above embodiment.

[0127] The computer-readable storage medium includes but is not limited to a flash memory, a hard disk, a solid-state drive, and the like.

[0128] It should be noted here that the description of the readable storage medium in the above embodiment corresponds to the description of the method in the embodiment, and the embodiment of the present invention will not be described in detail here.

[0129] In the above embodiments, all or part of the embodiments may be implemented using software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions according to the embodiments of the present invention are generated in whole or in part.

[0130] The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in or transmitted via computer-readable storage media. Computer-readable storage media can be any available medium that can be accessed by a computer, or a data storage device such as a server or data center that integrates one or more available media. Available media can include magnetic media or semiconductor media, etc.

[0131] Unless otherwise specified, the embodiments of the present invention do not limit the models of the components. Any component that can perform the above functions may be used.

[0132] Those skilled in the art will understand that the accompanying drawings are only a schematic diagram of a preferred embodiment, and the serial numbers of the embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0133] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A depth-guided inter-intra stereo image redirection method, characterized in that: The method comprises: Construct a stereo image redirection network, the network includes: an inter-frame-intra-frame feature extraction module, a depth-guided feature integration module, wherein, The inter-frame and intra-frame feature extraction module obtains deep features of long-range dependency information between views and within views; The depth-guided feature integration module uses a disparity attention learning mechanism based on deep features to interact with cross-view information of stereo features and aggregate the stereo correspondence between the left and right views, and then uses channel attention to generate a stereo attention map for the left and right views; A hybrid loss function is used to optimize the stereo image redirection network to maintain the structural integrity and disparity consistency of the stereo image redirection result; stereo images are redirected based on the optimized stereo image redirection network; Among them, the hybrid loss function includes structural similarity loss and disparity consistency loss; the structural similarity loss between the reconstructed stereo image and the input stereo image is used to keep the redirected left and right views retaining more structures and textures of the corresponding views; the disparity consistency loss is used to constrain the disparity difference between the reconstructed stereo image and the input stereo image.

2. The depth-guided inter-intra stereo image redirection method according to claim 1, characterized in that: The inter-frame-intra-frame feature extraction module includes: P cascaded Swin Transformer submodules ISTB, each submodule includes: an intra-frame feature learning layer IFLL, an inter-frame-intra-frame feature enhancement layer IFRL and an inter-frame feature interaction layer IFIL.

3. The depth-guided inter-intra stereo image redirection method according to claim 2, characterized in that: The intra-frame feature learning layer is: Using a non-overlapping N×N window, the left view F of size H×W×C is l and right view F r The basic features are adjusted to the size of (H×W / N 2 )×N 2 ×C blocks; Each local window is layer normalized, and the feature X of each local window is generated with a learnable weight matrix W Q 、W K and W V The query Q, key K and value V; Windowed multi-head self-attention is adopted to capture information from different views, and layer normalization and multi-layer perception are applied to refine feature labels.

4. The depth-guided inter-intra stereo image redirection method according to claim 3, wherein: The calculation process of the intra-frame feature learning layer is: {Q l ,K l ,V l }={X l W l Q ,X l W l K ,X l W l V } Among them, X l and X r are the input features of the left and right views of the intra-frame feature learning layer, Z l and Z r are the output features of the left and right views of the intra-frame feature learning layer, W-MSA(·) is the window multi-head self-attention, LN(·) and MLP(·) are the normalization layer and the multi-layer perception layer, The output features and input features of the window multi-head self-attention of the left view are X l The feature map obtained by adding, The output features and input features X of the window multi-head self-attention of the right view r The feature map obtained by addition.

5. The depth-guided inter-intra stereo image redirection method according to claim 2, wherein: The inter-frame-intra-frame feature enhancement layer is: Three dilated convolutional layers are used to expand the inter-view features of the left and right views by three scales respectively, and the expanded features are cascaded together; Three convolutional layers with gradually increasing kernel sizes are stacked on the common features. The unique features of the left and right views are obtained by extracting the common features from the cascaded features and then added to the original in-view features.

6. The depth-guided inter-intra stereo image redirection method according to claim 5, characterized in that: The calculation process of the inter-frame-intra-frame feature enhancement layer is: R l =Z l +U l R r =Z r +U r and Among them, Z l and Z r is the output in-view feature of the left and right views, U l and U r is the only feature of the right and left views, M self is a self-refining feature, and There are three dilated convolutional layers with a kernel size of 3×3. Conv1, Conv3, and Conv5 are convolutional layers with three different kernel sizes. is a multiplication operation.

7. The depth-guided inter-intra stereo image redirection method according to claim 6, characterized in that: The inter-frame feature interaction layer adopts offset window multi-head interactive attention, given the enhanced features of the left and right views, and the overall process of the inter-frame feature interaction layer of each local window is defined as follows: {Q r ,K r ,V r }={R r W r Q ,R r W l K ,R r W l V } Among them, R l and R r are the output features of the left and right views of the inter-intra feature enhancement layer, D l and D r are the output features of the left and right views of the inter-frame feature interaction layer, SW-MSA(·) is the offset window multi-head interaction attention, The output features of the offset window multi-head interactive attention of the left view and the output feature R l The feature map obtained by adding, The output features of the offset window multi-head interactive attention of the right view and the output feature R r The feature map obtained by addition.

8. The depth-guided inter-intra stereo image redirection method according to claim 2, wherein: The depth-guided feature integration module is as follows: the output features of the three ISTBs of the left and right views in the inter-frame and intra-frame feature extraction module are connected respectively, the cross-view information of the stereo features is interactively obtained by using the bidirectional disparity attention between the stereo images, and the left and right features are converted to the other side. The intra-frame feature learning layer and the inter-frame feature interaction layer are used again to aggregate the depth information of the left and right views, and the channel attention is used to generate the stereo attention map G of the left and right views. l and G r .

9. The depth-guided inter-intra stereo image redirection method according to claim 8, characterized in that: The stereo attention map G l and G r for: G l =CA(C(IFIL(IFLL(Q l ,F' l-r )))) G r =CA(C(IFIL(IFLL(Q r ,F' r-l )))) and Among them, M l-r and M r-l is the disparity attention map of the left and right views, Q l and Q r is the cascade feature of the left view, F' l-r and F' r-l is the conversion feature of the left and right views, conv1 is a convolutional layer with a kernel size of 1×1, C(·) is the cascade, and CA(·) is the channel attention.

10. A depth-guided inter-frame to intra-frame stereo image redirection device, characterized in that: The device includes: a processor and a memory, wherein program instructions are stored in the memory, and the processor calls the program instructions stored in the memory to enable the device to execute the method according to any one of claims 1 to 9.

Images