Depth guidance-based inter-frame and intra-frame three-dimensional image redirection method and device

Abstract

The invention discloses an inter-frame and intra-frame three-dimensional image redirection method and device based on depth guidance, and the method comprises the steps: constructing a three-dimensional image redirection network which comprises an inter-frame and intra-frame feature extraction module, a depth guidance feature integration module and an inter-frame and intra-frame feature extraction module, obtaining deep features of long-range dependency information between the views and in the views; the deep guide feature integration module is used for interacting cross-view information of stereoscopic features based on deep features by utilizing a parallax attention learning mechanism, aggregating a stereoscopic corresponding relation of left and right views, and further generating stereoscopic attention maps of the left and right views by adopting channel attention; the mixed loss function is used for optimizing the stereo image redirection network and keeping the structural integrity and parallax consistency of a stereo image redirection result; and redirecting the stereo image based on the optimized stereo image redirection network. The device comprises a processor and a memory. According to the invention, the consistency of parallax change is enhanced, and then high-quality stereo image redirection is realized.

Description

Technical Field

[0001] The present invention relates to the field of image redirection, and in particular to a depth-guided inter-frame and intra-frame stereoscopic image redirection method and device. Background Art

[0002] Currently, stereoscopic image redirection is a hotly discussed topic in the field of computer graphics and multimedia 3D display. This technology refers to the generation of high-quality stereoscopic images with arbitrary resolution from input stereoscopic images. At present, stereoscopic image redirection plays an important role in many 3D devices and applications. For example: 3D movies and TV, 3D games, mobile phones and iPads, etc. However, due to the diversity of 3D smart terminals, directly displaying the same 3D content on devices of different sizes and resolutions may result in an uncomfortable visual experience. Therefore, how to intelligently redirect stereoscopic image content to display devices with different aspect ratios and conform to human visual perception is a challenging task.

[0003] In recent years, researchers have proposed a variety of 2D image retargeting methods to improve the visual quality of human perception. Traditional 2D image retargeting methods are divided into discrete methods, continuous methods, and hybrid methods. For example: saliency-based methods, depth-based methods, edge-based methods, and seam cutting-assisted methods. Compared with traditional methods using manual features, researchers have proposed 2D image retargeting methods based on convolutional neural networks because deep learning has outstanding performance in extracting image salient objects and detail information. For example: WSSDCNN (weakly supervised and self-supervised deep convolutional neural network) and CycleIR (cycle image retargeting) methods are used to solve problems such as bending distortion and algorithm time consumption in image retargeting. Due to the rapid development of stereoscopic devices, researchers have proposed methods for retargeting stereo images to adjust the aspect ratio of 3D content. Inspired by the performance of deep learning in 2D image retargeting tasks, researchers have proposed using convolutional neural networks for stereo image retargeting. For example: Fan et al. proposed a deep twin network for resizing stereo images, which uses a cross-attention extraction module to obtain an attention map and a disparity generation module to generate a 3D importance map. In addition, Imani et al. proposed to use mobile layers to deform video frames and built a stereo video Transformer to help the redirection process. The above method can redirect stereo images without obvious content distortion.

[0004] In the process of implementing the present invention, the inventors found that the prior art has at least the following shortcomings and deficiencies:

[0005] Existing methods use the local receptive field of convolutional neural networks to capture the correspondence between left and right views, which makes it impossible to model the long-range dependencies of stereo images, and thus cannot accurately extract the deep features of salient objects in stereo images. In addition, existing methods often use cost-volume methods to estimate the disparity relationship, which makes it impossible to model the changing depth information of stereo images, and thus cannot effectively maintain the depth information of stereo images. Summary of the invention

[0006] The present invention provides a method and device for inter-frame-intra stereo image redirection based on depth guidance. The present invention uses deep learning technology, adopts an inter-frame-intra feature extraction module to capture the long-range dependency information between views and within views in a stereo image, and uses a depth-guided feature integration module to learn the stereo correspondence between left and right views, enhance the consistency of parallax changes, and thus achieve high-quality stereo image redirection. See the following description for details:

[0007] In the first aspect, a depth-guided inter-frame-intra-frame stereo image redirection method is provided, the method comprising: constructing

[0008] A stereo image redirection network, the network comprising: an inter-frame-intra-frame feature extraction module, a depth-guided feature integration module, wherein:

[0009] The inter-frame-intra-frame feature extraction module obtains deep features of long-range dependency information between views and within views;

[0010] The depth-guided feature integration module uses a disparity attention learning mechanism based on deep features to interact with cross-view information of stereo features and aggregate the stereo correspondence between the left and right views, and then uses channel attention to generate a stereo attention map of the left and right views;

[0011] A hybrid loss function is used to optimize the stereo image redirection network to maintain the structural integrity and disparity consistency of the stereo image redirection result; and the stereo image is redirected based on the optimized stereo image redirection network.

[0012] The inter-frame-intra-frame feature extraction module includes: P cascaded Swin Transformer sub-modules, each of which includes: an intra-frame feature learning layer, an inter-frame-intra-frame feature enhancement layer and an inter-frame feature interaction layer.

[0013] Among them, the intra-frame feature learning layer is:

[0014] Using a non-overlapping N×N window, the left view F of size H×W×C is l and right view F r The basic features are adjusted to a size of (H×W / N 2 )×N 2×C blocks;

[0015] Each local window is layer normalized, and the feature X of each local window generates a learnable weight matrix W Q , W K and W V The query Q, key K and value V of

[0016] Windowed multi-head self-attention is adopted to capture information from different views, and layer normalization and multi-layer perception are applied to refine feature markers.

[0017] The calculation process of the intra-frame feature learning layer is as follows:

[0018]

[0019]

[0020] Among them, X l and X r are the input features of the left and right views of the intra-frame feature learning layer, Z l and Z r are the output features of the left and right views of the intra-frame feature learning layer, W-MSA(·) is the window multi-head self-attention, LN(·) and MLP(·) are the normalization layer and the multi-layer perception layer, The output features and input features X of the window multi-head self-attention of the left view l The feature map obtained by adding, The output features and input features X of the window multi-head self-attention of the right view r The feature map obtained by addition.

[0021] Wherein, the inter-frame-intra-frame feature enhancement layer is:

[0022] Three dilated convolutional layers are used to expand the inter-view features of the left and right views by three scales respectively, and the expanded features are cascaded together;

[0023] Three convolutional layers with gradually increasing kernel sizes are stacked on the common features. The unique features of the left and right views are obtained by extracting the common features from the cascaded features and added to the original in-view features.

[0024] The calculation process of the inter-frame-intra-frame feature enhancement layer is as follows:

[0025] R l =Z l +U l

[0026] R r =Z r +U r

[0027] and

[0028]

[0029] Among them, Z l and Z r is the output in-view feature of the left and right views, U l and U r is the only feature of the right and left views, M self is a self-refining feature, and There are three dilated convolutional layers with a kernel size of 3×3. Conv1, Conv3, and Conv5 are convolutional layers with three different kernel sizes. is a multiplication operation.

[0030] Among them, the inter-frame feature interaction layer is: using offset window multi-head interactive attention, given the enhanced features of the left and right views, the overall process of the inter-frame feature interaction layer of each local window is defined as follows:

[0031]

[0032]

[0033] Among them, R l and R r are the output features of the left and right views of the inter-frame feature interaction layer, K l and K r is the input feature of the left or right view of the inter-frame feature interaction layer, SW-MSA(·) is the offset window multi-head interactive attention, The output features of the offset window multi-head interactive attention of the left view and the output feature R l The feature map obtained by adding, The output features of the offset window multi-head interactive attention of the right view and the output feature R r The feature map obtained by addition.

[0034] The depth-guided feature integration module is as follows: the output features of the three ISTBs of the left and right views in the inter-frame-intra-frame feature extraction module are connected respectively, the cross-view information of the stereo features of the bidirectional parallax attention interaction between the stereo images is used, and the left and right features are converted to the other side, and the intra-frame feature learning layer and the inter-frame feature interaction layer are used again to aggregate the depth information of the left and right views, and the channel attention is used to generate the stereo attention map G of the left and right views. l and G r .

[0035] Among them, the stereo attention map G land G r for:

[0036] G l =CA(C(IFIL(IFLL(Q l ,F l ' -r ))))

[0037] G r =CA(C(IFIL(IFLL(Q r ,F r ' -l ))))

[0038] and

[0039]

[0040] Among them, M l-r and M r-l is the disparity attention map of the left and right views, Q l and Q r is the cascade feature of the left view, F l ' -r and F r ' -l are the conversion features of the left and right views, conv1 is a convolutional layer with a kernel size of 1×1, C(·) is the cascade, and CA(·) is the channel attention.

[0041] In the second aspect, a depth-guided inter-frame to intra-frame stereoscopic image redirection device is provided, the device comprising: a processor and a memory, the memory storing program instructions, the processor calling the program instructions stored in the memory to enable the device to execute any one of the methods described in the first aspect.

[0042] The beneficial effects of the technical solution provided by the present invention are:

[0043] 1. This method can maintain the structure and texture of salient objects in the stereo image redirection results, reduce binocular parallax distortion, and obtain high-quality stereo image redirection results;

[0044] 2. This method uses the long-range inter-frame and intra-frame dependency relationship between the left and right views for the first time to solve the stereoscopic image redirection problem, and can accurately extract the salient information of the stereoscopic image. At the same time, it uses the depth information of the left and right views to constrain the parallax change, thereby reducing the parallax distortion of the stereoscopic image redirection result. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 is a schematic diagram of an inter-frame-intra-frame feature extraction module;

[0046] Figure 2Schematic diagram of the parallax attention learning mechanism;

[0047] Figure 3 A schematic diagram showing the comparison results of depth distortion scores of stereo image redirection results;

[0048] Figure 4 A flowchart of a depth-guided inter-intra stereo image redirection method. DETAILED DESCRIPTION

[0049] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention are described in further detail below.

[0050] In order to overcome the shortcomings of the prior art, an embodiment of the present invention designs a depth-guided inter-intra stereo image redirection method. The method consists of three modules: the inter-intra feature extraction module uses the inter-intra Swin Transformer submodule to obtain deep features of long-range dependency information between views and within views; the depth-guided feature integration module uses the disparity attention learning mechanism to interact with the cross-view information of stereo features and aggregate the stereo correspondence between the left and right views; the hybrid loss function is used to maintain the structural integrity and disparity consistency of the stereo image redirection result. The stereo image redirection method realizes accurate scaling of the image size, reduces content distortion and maintains the binocular parallax relationship, as described below:

[0051] Example 1

[0052] A depth-guided inter-frame-intra-frame stereo image redirection method, the method comprising the following steps:

[0053] 1. Inter-frame and intra-frame feature extraction module

[0054] For stereo image retargeting, one of the most basic tasks is to maintain visually salient areas in the input stereo image and prevent content artifacts and distortion in the stereo image retargeting result when changing the target aspect ratio and resolution. In order to simultaneously retain the information of the same salient area in the left view and the right view, an inter-intra feature extraction module is designed in the embodiment of the present invention to extract deep features, wherein the inter-intra Swin Transformer is designed to learn the long-range dependency information between views and within views.

[0055] Given a stereo image pair, the backbone network is first used to extract the basic shallow features of the left and right views. The backbone network consists of two 3×3 convolutional layers and a parameterized Relu activation function. Then, the inter-intra feature extraction module is used to extract deep features. The inter-intra feature extraction module consists of P cascaded Swin Transformer submodules (ISTB). Specifically, each ISTB includes: an intra feature learning layer (IFLL), an inter-intra feature enhancement layer (IFRL) and an inter-feature interaction layer (IFIL), as shown in Figure 1 shown.

[0056] Intra-frame feature learning layer: In order to integrate the global information and local information in the left view and the right view respectively, the intra-frame feature learning layer is designed to extract the intra-view features of the stereo image. Specifically, firstly, a non-overlapping N×N window is used to transform the left view F of size H×W×C into l and right view F r The basic features are adjusted to a size of (H×W / N 2 )×N 2 ×C blocks. Then, each local window is layer normalized, where the feature X of each local window generates a learnable weight matrix W Q , W K and W V The query Q, key K and value V are given by . Next, considering the visual attention distribution in stereo images, windowed multi-head self-attention is adopted to capture information from different views. Finally, layer normalization and multi-layer perception are applied to refine the feature labeling, where residual connections are applied to the above stages.

[0057] Among them, the process of intra-frame feature learning layer of each local window is defined as follows:

[0058]

[0059] Among them, X l and X r Z is the input feature of the left and right views of the intra-frame feature learning layer. l and Z r are the output features of the left and right views of the intra-frame feature learning layer. W-MSA(·) is the window multi-head self-attention. LN(·) and MLP(·) are the normalization layer and the multi-layer perception layer.

[0060] Inter-intra feature enhancement layer: Since stereo images have binocular parallax, the left and right views contain common and unique information. In order to distinguish the common and unique information in the left and right views and enhance the unique information of the left and right views, an inter-intra feature enhancement layer is designed to extract inter-view and intra-view features using different receptive fields. Specifically, in order to capture multi-scale contextual information without losing image resolution, three dilated convolutional layers are first used to expand the inter-view features of the left and right views by three scales respectively. Then, the expanded features are cascaded together. At the same time, considering the complementarity and interaction between the left and right views of different scales, a self-refinement strategy by increasing the receptive field size is applied to the common features, that is, three convolutional layers with gradually increasing convolution kernel sizes are stacked. Finally, the unique features of the left and right views are obtained by extracting the common features from the cascaded features, and the unique features of the left and right views are further added to the original intra-view features to achieve feature enhancement.

[0061] Among them, the overall process of the inter-frame-intra-frame feature enhancement layer is expressed as:

[0062] R l =Z l +U l

[0063] R r =Z r +U r

[0064] and

[0065]

[0066] Among them, Z l and Z r is the output in-view feature of the left and right views, U l and U r is the only feature of the right and left views, M self is a self-refining feature, and are three dilated convolutional layers with a kernel size of 3×3, and the dilation rates are set to 1, 2, and 4. conv1, conv3, and conv5 are three convolutional layers with different kernel sizes, namely 1×1, 3×3, and 5×5. is a multiplication operation.

[0067] Inter-frame feature interaction layer: In order to further promote the information association and interaction between the left and right views, the embodiment of the present invention designs an inter-frame feature interaction layer to extract the inter-view features between the left and right views. The inter-frame feature interaction layer and the intra-frame feature learning layer have similar network architectures. However, in order to further realize the exchange of long-range dependencies between different views, offset window multi-head interactive attention is adopted in the inter-frame feature interaction layer. Therefore, given the enhanced features of the left and right views, the overall process of the inter-frame feature interaction layer of each local window is defined as follows:

[0068]

[0069] Among them, R l and R r K is the output feature of the left and right views of the inter-frame feature interaction layer. l and K r is the input feature of the left or right view of the inter-frame feature interaction layer. SW-MSA(·) is the offset window multi-head interaction attention.

[0070] 2. Depth-guided Feature Integration Module

[0071] The depth information of the stereo image provides 3D scene information between the left and right views, which helps to maintain the original disparity relationship when the stereo image is redirected. Therefore, the embodiment of the present invention designs a depth-guided feature integration module to learn the stereo correspondence and enhance the consistency of disparity changes.

[0072] In the depth-guided feature integration module, the present invention proposes a disparity attention learning mechanism to exchange cross-view information of stereo features and aggregate the depth information of left and right views. Figure 2 As shown. Specifically, considering that the deep hierarchical features are helpful for stereo relationship learning, the output features of the three ISTBs of the left and right views in the inter-intra feature extraction module are first connected respectively. Then, the cross-view information of the stereo features is interacted by the bidirectional disparity attention between the stereo images, and the left and right features are converted to the other side. Then, the intra-frame feature learning layer and the inter-frame feature interaction layer are used again to aggregate the depth information of the left and right views. Finally, the channel attention is used to generate the stereo attention map G of the left and right views. l and G r , which is defined as:

[0073] G l =CA(C(IFIL(IFLL(Q l ,F l ' -r ))))

[0074] G r =CA(C(IFIL(IFLL(Qr ,F r ' -l ))))

[0075] and

[0076]

[0077] Among them, M l-r and M r-l is the disparity attention map of the left and right views, Q l and Q r is the cascade feature of the left view, F l ' -r and F r ' -l are the conversion features of the left and right views, conv1 is a convolutional layer with a kernel size of 1×1, C(·) is the cascade, and CA(·) is the channel attention.

[0078] 3. Mixed Loss

[0079] The proposed network aims to explore the deep feature information of stereo images and promote the interaction of long-range dependencies between left and right views, thereby improving the quality of stereo image retargeting. Therefore, a hybrid loss function is designed to optimize the proposed stereo image retargeting network, which includes structural similarity loss and disparity consistency loss. The hybrid loss is defined as follows:

[0080] L hybrid =αL str +βL dis

[0081] Among them, α and β are the control structure similarity loss L str and parallax consistency loss L dis The trade-off parameters between .

[0082] Structural similarity loss: Structural similarity can reflect the contextual information of the stereo image from three aspects: light, contrast, and structure. Therefore, the embodiment of the present invention uses the structural similarity loss between the reconstructed stereo image and the input stereo image to keep the redirected left and right views retaining more structures and textures of the corresponding views. The structural similarity loss is defined as:

[0083]

[0084] in, and are the input left and right views, and are the reconstructed left and right views obtained by re-feeding the re-targeted left and right views into the proposed stereo image re-targeting network, and ssim(·) denotes the structural similarity operation.

[0085] Disparity consistency loss: Considering the difference in perspective between the left view and the right view can describe the depth information of the 3D scene. In order to ensure the disparity consistency of the stereo image redirection result and obtain better depth perception, a disparity consistency loss is designed to constrain the disparity difference between the reconstructed stereo image and the input stereo image. The disparity consistency loss is defined as:

[0086]

[0087] Here, |·| represents the absolute value operator, and ||·|| represents the L1 norm.

[0088] Example 2

[0089] Figure 3 The depth distortion score comparison results of the stereo image retargeting results are listed. The comparison algorithms include: DPS method and SIRCNN method. The DPS method is a traditional stereo image retargeting algorithm, and the SIRCNN method is a stereo image retargeting algorithm based on deep learning. The smaller the depth distortion score, the higher the 3D visual perception quality. Figure 3 It can be seen that compared with the DPS method, the depth distortion score obtained by the method proposed in the embodiment of the present invention is smaller, indicating that the depth guidance information integration module can effectively reduce the depth distortion of stereo image redirection. In addition, the depth distortion score obtained by the SIRCNN method is also worse than that of the method proposed in the embodiment of the present invention. The reason is that the embodiment of the present invention uses multiple cascaded Swin Transformer sub-modules to prepare and model the long-range dependency of stereo images, effectively learns the depth feature information of the left and right images, and reduces the depth distortion of the stereo image redirection result. Figure 4 A technical flow chart of an embodiment of the present invention is given, which mainly includes an inter-frame-intra-frame feature extraction module, a depth-guided feature integration module, and a hybrid loss.

[0090] Example 3

[0091] A depth-guided inter-frame-intra-frame stereoscopic image redirection device, the device comprising: a processor and a memory, the memory storing program instructions, the processor calling the program instructions stored in the memory to enable the device to perform the following method steps in embodiment 1:

[0092] Construct a stereo image redirection network, which includes: an inter-frame-intra-frame feature extraction module and a depth-guided feature integration module, wherein:

[0093] Inter-frame and intra-frame feature extraction module to obtain deep features of long-range dependency information between and within views;

[0094] The depth-guided feature integration module uses the disparity attention learning mechanism based on deep features to interact with the cross-view information of stereo features and aggregate the stereo correspondence between the left and right views, and then uses channel attention to generate the stereo attention map of the left and right views;

[0095] A hybrid loss function is used to optimize the stereo image redirection network to maintain the structural integrity and disparity consistency of the stereo image redirection result; and the stereo image is redirected based on the optimized stereo image redirection network.

[0096] The inter-frame-intra-frame feature extraction module includes: P cascaded Swin Transformer sub-modules, each of which includes: an intra-frame feature learning layer, an inter-frame-intra-frame feature enhancement layer and an inter-frame feature interaction layer.

[0097] Among them, the intra-frame feature learning layer is:

[0098] Using a non-overlapping N×N window, the left view F of size H×W×C is l and right view F r The basic features are adjusted to a size of (H×W / N 2 )×N 2 ×C blocks;

[0099] Each local window is layer normalized, and the feature X of each local window generates a learnable weight matrix W Q , W K and W V The query Q, key K and value V of

[0100] Windowed multi-head self-attention is adopted to capture information from different views, and layer normalization and multi-layer perception are applied to refine feature markers.

[0101] Among them, the calculation process of the intra-frame feature learning layer is:

[0102]

[0103] Among them, X l and X r are the input features of the left and right views of the intra-frame feature learning layer, Z l and Z r are the output features of the left and right views of the intra-frame feature learning layer, W-MSA(·) is the window multi-head self-attention, LN(·) and MLP(·) are the normalization layer and the multi-layer perception layer.

[0104] Among them, the inter-frame-intra-frame feature enhancement layer is:

[0105] Three dilated convolutional layers are used to expand the inter-view features of the left and right views by three scales respectively, and the expanded features are cascaded together;

[0106] Three convolutional layers with gradually increasing kernel sizes are stacked on the common features. The unique features of the left and right views are obtained by extracting the common features from the cascaded features and added to the original in-view features.

[0107] Among them, the calculation process of the inter-frame-intra-frame feature enhancement layer is:

[0108] R l =Z l +U l

[0109] R r =Z r +U r

[0110] and

[0111]

[0112] Among them, Z l and Z r is the output in-view feature of the left and right views, U l and U r is the only feature of the right and left views, M self is a self-refining feature, and There are three dilated convolutional layers with a kernel size of 3×3. Conv1, Conv3, and Conv5 are convolutional layers with three different kernel sizes. is a multiplication operation.

[0113] Among them, the inter-frame feature interaction layer is: using offset window multi-head interactive attention, given the enhanced features of the left and right views, the overall process of the inter-frame feature interaction layer of each local window is defined as follows:

[0114]

[0115] Among them, R l and R r are the output features of the left and right views of the inter-frame feature interaction layer, K l and K r is the input feature of the left or right view of the inter-frame feature interaction layer, and SW-MSA(·) is the offset window multi-head interactive attention.

[0116] The depth-guided feature integration module is as follows: the output features of the three ISTBs of the left and right views in the inter-frame-intra-frame feature extraction module are connected respectively, the cross-view information of the stereo features is interactively obtained by using the bidirectional disparity attention between the stereo images, and the left and right features are converted to the other side. The intra-frame feature learning layer and the inter-frame feature interaction layer are used again to aggregate the depth information of the left and right views, and the channel attention is used to generate the stereo attention map G of the left and right views. l and G r .

[0117] Among them, the stereo attention map G l and G r for:

[0118] G l =CA(C(IFIL(IFLL(Q l ,F l ' -r ))))

[0119] G r =CA(C(IFIL(IFLL(Q r ,F r ' -l ))))

[0120] and

[0121]

[0122] Among them, M l-r and M r-l is the disparity attention map of the left and right views, Q l and Q r is the cascade feature of the left view, F l ' -r and F r ' -l are the conversion features of the left and right views, conv1 is a convolutional layer with a kernel size of 1×1, C(·) is the cascade, and CA(·) is the channel attention.

[0123] It should be pointed out here that the device description in the above embodiment corresponds to the method description in the embodiment, and the embodiment of the present invention will not be described in detail here.

[0124] The execution subjects of the above-mentioned processor and memory can be devices with computing functions such as computers, single-chip microcomputers, and microcontrollers. In specific implementation, the embodiments of the present invention do not limit the execution subjects and are selected according to the needs of actual applications.

[0125] The data signal is transmitted between the memory and the processor via a bus, which is not described in detail in the embodiment of the present invention.

[0126] Based on the same inventive concept, an embodiment of the present invention further provides a computer-readable storage medium, the storage medium includes a stored program, and when the program is running, the device where the storage medium is located is controlled to execute the method steps in the above embodiment.

[0127] The computer-readable storage medium includes but is not limited to a flash memory, a hard disk, a solid-state drive, and the like.

[0128] It should be pointed out here that the description of the readable storage medium in the above embodiment corresponds to the description of the method in the embodiment, and the embodiment of the present invention will not be described in detail here.

[0129] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to the embodiments of the present invention are generated.

[0130] The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions may be stored in a computer-readable storage medium or transmitted via a computer-readable storage medium. The computer-readable storage medium may be any available medium that can be accessed by the computer or a data storage device such as a server or a data center that includes one or more available media. The available medium may be a magnetic medium or a semiconductor medium, etc.

[0131] Unless otherwise specified, the models of the components in the embodiments of the present invention are not limited, and any device that can perform the above functions may be used.

[0132] Those skilled in the art will appreciate that the accompanying drawing is only a schematic diagram of a preferred embodiment, and the serial numbers of the embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0133] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A depth-guided inter-intra stereo image redirection method, characterized in that: The method comprises: A stereo image redirection network is constructed, the network comprising: an inter-frame-intra-frame feature extraction module and a depth-guided feature integration module, wherein: The inter-frame-intra-frame feature extraction module obtains deep features of long-range dependency information between views and within views; The depth-guided feature integration module uses a disparity attention learning mechanism based on deep features to interact with cross-view information of stereo features and aggregate the stereo correspondence between the left and right views, and then uses channel attention to generate a stereo attention map of the left and right views; A hybrid loss function is used to optimize the stereo image redirection network to maintain the structural integrity and disparity consistency of the stereo image redirection result; and the stereo image is redirected based on the optimized stereo image redirection network.

2. The depth-guided inter-intra stereo image redirection method according to claim 1, characterized in that: The inter-frame-intra-frame feature extraction module includes: P cascaded Swin Transformer sub-modules, each of which includes: an intra-frame feature learning layer, an inter-frame-intra-frame feature enhancement layer and an inter-frame feature interaction layer.

3. The depth-guided inter-intra stereo image redirection method according to claim 2, characterized in that: The intra-frame feature learning layer is: Using a non-overlapping N×N window, the left view F of size H×W×C is l and right view F r The basic features are adjusted to a size of (H×W / N 2 )×N 2 ×C blocks; Each local window is layer normalized, and the feature X of each local window generates a learnable weight matrix W Q , W K and W V The query Q, key K and value V of Windowed multi-head self-attention is adopted to capture information from different views, and layer normalization and multi-layer perception are applied to refine feature markers.

4. The depth-guided inter-intra stereo image redirection method according to claim 3, characterized in that: The calculation process of the intra-frame feature learning layer is: {Q l ,K l ,V l }={X l W l Q ,X l W l K ,X l W l V } {Q r ,K r ,V r }={X r W r Q ,X r W r K ,X r W r V } Among them, X l and X r are the input features of the left and right views of the intra-frame feature learning layer, Z l and Z r are the output features of the left and right views of the intra-frame feature learning layer, W-MSA(·) is the window multi-head self-attention, LN(·) and MLP(·) are the normalization layer and the multi-layer perception layer, The output features and input features X of the window multi-head self-attention of the left view l The feature map obtained by adding, The output features and input features X of the window multi-head self-attention of the right view r The feature map obtained by addition.

5. The depth-guided inter-intra stereo image redirection method according to claim 2, characterized in that: The inter-frame-intra-frame feature enhancement layer is: Three dilated convolutional layers are used to expand the inter-view features of the left and right views by three scales respectively, and the expanded features are cascaded together; Three convolutional layers with gradually increasing kernel sizes are stacked on the common features. The unique features of the left and right views are obtained by extracting the common features from the cascaded features and added to the original in-view features.

6. The depth-guided inter-intra stereo image redirection method according to claim 5, characterized in that: The calculation process of the inter-frame-intra-frame feature enhancement layer is: R l =Z l +U l R r =Z r +U r and Among them, Z l and Z r is the output in-view feature of the left and right views, U l and U r is the only feature of the right and left views, M self is a self-refining feature, and There are three dilated convolutional layers with a kernel size of 3×3. Conv1, Conv3, and Conv5 are convolutional layers with three different kernel sizes. is a multiplication operation.

7. The depth-guided inter-intra stereo image redirection method according to claim 6, characterized in that: The inter-frame feature interaction layer adopts offset window multi-head interactive attention, given the enhanced features of the left and right views, and the overall process of the inter-frame feature interaction layer of each local window is defined as follows: {Q l ,K l ,V l }={R l W l Q ,R l W r K ,R l W r V } {Q r ,K r ,V r }={R r W r Q ,R r W l K ,R r W l V } Among them, R l and R r are the output features of the left and right views of the inter-frame feature interaction layer, K l and K r is the input feature of the left or right view of the inter-frame feature interaction layer, SW-MSA(·) is the offset window multi-head interactive attention, The output features of the offset window multi-head interactive attention of the left view and the output feature R l The feature map obtained by adding, The output features of the offset window multi-head interactive attention of the right view and the output feature R r The feature map obtained by addition.

8. The depth-guided inter-intra stereo image redirection method according to claim 1, characterized in that: The depth-guided feature integration module is as follows: the output features of the three ISTBs of the left and right views in the inter-frame-intra-frame feature extraction module are connected respectively, the cross-view information of the stereo features is interactively obtained by using the bidirectional parallax attention between the stereo images, and the left and right features are converted to the other side, and the intra-frame feature learning layer and the inter-frame feature interaction layer are used again to aggregate the depth information of the left and right views, and the channel attention is used to generate the stereo attention map G of the left and right views. l and G r .

9. The depth-guided inter-intra stereo image redirection method according to claim 8, characterized in that: The stereo attention map G l and G r for: G l =CA(C(IFIL(IFLL(Q l ,F′ l-r )))) G r =CA(C(IFIL(IFLL(Q r ,F′ r-l )))) and Among them, M l-r and M r-l is the disparity attention map of the left and right views, Q l and Q r is the cascade feature of the left view, F′ l-r and F′ r-l are the conversion features of the left and right views, conv1 is a convolutional layer with a kernel size of 1×1, C(·) is the cascade, and CA(·) is the channel attention.

10. A depth-guided inter-frame to intra-frame stereoscopic image redirection device, characterized in that: The device comprises: a processor and a memory, wherein program instructions are stored in the memory, and the processor calls the program instructions stored in the memory to enable the device to execute the method according to any one of claims 1 to 9.

Images