A stereoscopic image retargeting method based on collaborative mamba and disparity enhancement network

Abstract

The application discloses a kind of stereoscopic image redirection methods based on collaborative manba and parallax enhancement network, the method includes the following steps: step S1, obtains training dataset, wherein the training dataset includes multiple input stereoscopic images, the input stereoscopic image includes left image I L And right image I R ;Step S2, construct stereoscopic image redirection depth model, for obtaining redirected left image and right image;Step S3, based on the training dataset and preset overall loss function for the stereoscopic image redirection depth model is trained, obtains stereoscopic image redirection target depth model;Step S4, using the stereoscopic image redirection target depth model for the stereoscopic image to be redirected is handled, obtains stereoscopic image redirection result.The application carries out feature extraction and parallax learning to stereoscopic image, learns the long-distance dependence between image and reduces binocular parallax distortion, realizes the redirection to stereoscopic image.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a stereo image retargeting method based on a cooperative Mamba and parallax enhancement network. Background Technology

[0002] In recent years, stereoscopic image repositioning technology has become a hot topic in the fields of multimedia 3D display and computer vision. This technology aims to adapt stereoscopic images to target aspect ratios of different resolutions while maintaining consistency in content and binocular parallax between the repositioned image and the source image. Due to the complexity and diversity of scenes depicted in different stereoscopic images, directly repositioning stereoscopic image content on devices of varying sizes inevitably introduces visual artifacts and incorrect parallax. Therefore, adjusting stereoscopic images to different resolutions while ensuring that their image content and binocular parallax conform to a true 3D viewing experience remains a challenging task.

[0003] Currently, researchers have proposed numerous stereo image resizing methods, mainly categorized into traditional stereo image resizing methods and deep learning-based stereo image resizing methods. Traditional stereo image resizing methods typically resize the stereo image by modifying pixel values ​​or constructing a mesh deformation function, such as discrete methods, continuous methods, and hybrid methods. However, these methods often heavily rely on handcrafted features or prior constraints, resulting in limited resizing accuracy when processing stereo images with multiple foreground objects or complex backgrounds.

[0004] Unlike traditional methods, some deep learning-based stereo image retargeting methods utilize convolutional neural networks (CNNs) to adjust the resolution of stereo images, such as SIRCNN and USIR-Net. These methods extract salient object features from images using CNNs. However, the fixed local receptive field of CNNs limits their performance in capturing global context. Therefore, some research has explored stereo image retargeting methods based on visual Transformers. These methods utilize self-attention mechanisms to simulate visual context information and learn long-range dependencies. For example, Fan et al. proposed a depth-guided Transformer-based stereo image retargeting method that captures long-range dependency information through an intra-to-inter-frame feature extraction module. Le et al. proposed an offset layer network based on stereo Transformers for deformable video frames.

[0005] In the process of realizing this invention, the inventors discovered that the prior art has at least the following drawbacks and deficiencies:

[0006] First, existing methods typically employ deep learning networks based on self-attention mechanisms for feature extraction, but their computational cost increases exponentially with the square of the stereo image size, thus limiting the practical application of stereo image retargeting techniques. Second, existing methods fail to adequately consider the preservation and enhancement of binocular disparity between the left and right images, especially when disparity variations are significant, leading to inconsistencies in binocular disparity in the retargeted stereo image. Summary of the Invention

[0007] The technical objective of this invention is to provide a stereo image retargeting method based on a Cooperative Mamba and disparity enhancement network. This invention designs a Cooperative Mamba feature extraction sub-model, utilizing the Cooperative Mamba feature extraction module to extract multi-view depth features with long-range dependencies. Furthermore, a disparity enhancement feature aggregation sub-model is designed, employing an occlusion-assisted mask module to reduce disparity differences and perform binocular disparity enhancement. This stereo image retargeting method can obtain high-quality stereo image retargeting results while reducing content distortion and disparity differences.

[0008] Based on the above technical objectives, this invention provides a stereo image retargeting method based on a cooperative Mamba and disparity enhancement network, the method comprising the following steps:

[0009] Step S1: Obtain the training dataset, wherein the training dataset includes multiple input stereo images, and the input stereo images include the left image I. L And right image I R ;

[0010] Step S2: Construct a stereo image retargeting depth model, wherein the stereo image retargeting depth model is used to obtain the retargeted left and right images;

[0011] Step S3: Train the stereo image repositioning depth model based on the training dataset and the preset overall loss function to obtain the stereo image repositioning target depth model;

[0012] Step S4: Use the stereo image redirection target depth model to process the stereo image to be redirected to obtain the stereo image redirection result.

[0013] In one embodiment, the stereo image retargeting depth model includes a co-Mamba feature extraction sub-model, a disparity enhancement feature aggregation sub-model, and a uniform grid layer, wherein:

[0014] The Collaborative Mamba Feature Extraction Submodel is used to extract features from the left image I. L And right image I R Depth feature extraction is performed to obtain the viewpoint features of the left and right images. and and inter-viewpoint correlation features and ;

[0015] The disparity enhancement feature aggregation sub-model is used to utilize the viewpoint features of the left and right images. and and inter-viewpoint correlation features and The left and right images are then subjected to disparity preservation and enhancement to obtain stereo saliency maps of the left and right images. and ;

[0016] The uniform grid map is used for stereo saliency mapping based on the left and right images. and And the target scaling ratio η is used to adjust the size of the left and right images simultaneously, resulting in the retargeted left and right images.

[0017] In one embodiment, the collaborative Mamba feature extraction sub-model includes a base convolutional module and a collaborative Mamba feature extraction module group connected in sequence, wherein:

[0018] The basic convolutional module consists of two convolutional layers, a normalization layer, and a PReLU activation function, used to extract initial shallow features from the left and right images. and ;

[0019] The Cooperative Mamba Feature Extraction Module Group comprises three cascaded Cooperative Mamba Feature Extraction Modules, used to extract features based on the initial shallow features of the left and right images. and Deep features with global context were extracted from the left and right images: viewpoint features of the left and right images. and and inter-viewpoint correlation features and Among the three cascaded collaborative Mamba feature extraction modules, the viewpoint correlation features output by the first collaborative Mamba feature extraction module are... and The input to the next Cooperative Mamba Feature Extraction module is the initial shallow features of the left and right images, while the input to the first Cooperative Mamba Feature Extraction module is the initial shallow features of the left and right images. and .

[0020] In one embodiment, the collaborative Mamba feature extraction module includes two parallel self-viewpoint feature learning Mambas and one inter-viewpoint correlation learning Mamba, wherein:

[0021] The two parallel viewpoint feature learning Mambas are used based on the initial shallow features of the left and right images, respectively. and The left and right images are processed to extract their viewpoint features. and The self-viewpoint feature learning Mamba includes layer normalization, two parallel branches and a linear layer. One of the two parallel branches includes a linear layer, a depthwise separable convolution, a Sigmoid linear unit, a 2D selective scan layer and layer normalization in sequence, and the other branch includes a linear layer and a Sigmoid linear unit in sequence.

[0022] The inter-viewpoint correlation learning Mamba is used to capture long-range dependencies and is based on the self-viewpoint features of the left and right images. and By learning the inter-view correlations from the left and right images, the viewpoint correlation features between the left and right images are obtained. and The viewpoint correlation learning Mamba includes a left image processing unit group, a right image processing unit group, and a common hidden state branch. Each image processing unit group includes layer normalization, two parallel processing unit branches, and a linear layer. The two parallel processing unit branches include a first processing unit branch consisting of a cascaded linear layer, a depthwise separable convolution, a sigmoid linear unit, a 2D selective scan layer, and layer normalization, and a second processing unit branch consisting of a cascaded linear layer and a sigmoid linear unit. The common hidden state branch includes a cascaded 2D selective scan layer and layer normalization.

[0023] In one embodiment, the disparity enhancement feature aggregation sub-model includes three occlusion-assisted mask modules, which are used to obtain an enhanced disparity map based on self-viewpoint features and inter-viewpoint correlation features. The self-viewpoint features and inter-viewpoint correlation features used by the three occlusion-assisted masking modules are the self-viewpoint features and inter-viewpoint correlation features output by the three cascaded collaborative Mamba feature extraction modules, respectively. The three occlusion-assisted masking modules obtain three enhanced disparity maps. Correlation features between viewpoints of the left and right images after element-wise summation. and Linear combination yields the stereo saliency map of the left and right images. and .

[0024] In one embodiment, the occlusion-assisted mask module includes a disparity and occlusion estimation Transformer and a disparity enhancement block, wherein the disparity and occlusion estimation Transformer includes multi-head self-attention and multi-head cross-attention, used to perform self-viewpoint features based on the left and right images. and An initial disparity map and occlusion mask are estimated; the disparity enhancement block utilizes the initial disparity map and occlusion mask, as well as the viewpoint correlation features of the left and right images. and To further enhance the binocular parallax in the occluded area, an enhanced parallax map is obtained. .

[0025] In one embodiment, in the disparity and occlusion estimation Transformer:

[0026] First, multi-head self-attention is used to aggregate viewpoint features from the left and right images. and The self-attention features of the left and right images are obtained respectively. and :

[0027]

[0028] in, , and These represent the query vector, key vector, and value vector of the left and right images based on viewpoint features, respectively. ⊕ represents multi-head self-attention and feedforward networks, and ⊕ represents element-wise addition;

[0029] Then, multi-head cross-attention is used to capture the inherent spatial relationship between the left and right images to obtain the cross-attention features of the left and right images. and :

[0030]

[0031]

[0032]

[0033]

[0034] in, , and Let represent the query vector, key vector, and value vector of the self-attention features of the left and right images, respectively. This represents the softmax function. This indicates the transpose operation. It is a cascading operation. ⊕ indicates pixel-wise multiplication, and ⊕ indicates element-wise addition. This indicates element-wise subtraction. To convert the value vectors of the self-attention features of the left and right images and The shared value obtained by adding elements one by one, which has the positional relationship of the cross view;

[0035] Finally, based on the cross-attention features of the left and right images and The initial disparity map is estimated using disparity and occlusion estimation methods. and masking .

[0036] In one embodiment, the enhanced disparity map is obtained in the disparity enhancement block using the following formula. :

[0037]

[0038] in, and This represents residual attention and convolutional layers. This indicates a transformation operation. Indicates the relevant layer, , and This represents three dilated convolutional layers with kernel sizes of 3×3 and expansion rates of 1, 2, and 4, respectively. This represents the ReLU activation function.

[0039] In one embodiment, the preset overall loss function includes a stereo similarity loss function and a disparity occlusion consistency loss function:

[0040] L multi =L lpips +L po

[0041] Among them, L multi L represents the preset overall loss function. lpips L represents the stereo similarity loss function. po This represents the parallax occlusion consistency loss function.

[0042] In one embodiment, the stereo similarity loss function L lpips Represented as:

[0043]

[0044] in, and This refers to the reconstructed left and right images obtained by processing the left and right images in the training dataset using the stereo image repositioning depth model, and then feeding them back into the stereo image repositioning depth model. and This represents the feature extractor that performs feature extraction on the image corresponding to the i-th and j-th convolutional layers in VGG. and These are the weights of the i-th and j-th convolutional layers in VGG that process the image. It is the square L2 norm;

[0045] The disparity-occlusion consistency loss function includes a disparity supervision term and an occlusion supervision term:

[0046] ,

[0047]

[0048] in, and These represent the parallax supervision term and the occlusion supervision term, respectively. Indicates occlusion mask. This represents the actual value of the occlusion mask obtained directly using the left-right consistency check. It is a very small number. and Indicates the weight.

[0049] The beneficial effects of the technical solution provided by this invention are:

[0050] 1. This invention can accurately redirect stereoscopic images, preserving image content while reducing parallax distortion, thereby obtaining high-quality stereoscopic image redirection results.

[0051] 2. This invention utilizes deep learning technology to solve the problem of stereo image retargeting. It reduces stereo image content distortion by using a collaborative Mamba feature extraction sub-model and a stereo similarity loss function, and reduces disparity distortion by using a disparity enhancement feature aggregation sub-model and a disparity occlusion consistency loss function.

[0052] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the description, claims, and drawings. Attached Figure Description

[0053] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with the embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0054] Figure 1 This is a flowchart of a stereo image retargeting method based on a cooperative Mamba and parallax enhancement network according to an embodiment of the present invention;

[0055] Figure 2 This is a schematic diagram illustrating the structural relationship between the self-viewpoint feature learning Mamba and the viewpoint correlation learning Mamba according to an embodiment of the present invention.

[0056] Figure 3 This is a schematic diagram of the disparity and occlusion estimation Transformer, the multi-head cross attention processing mechanism, and the structural relationship of the disparity enhancement block according to an embodiment of the present invention.

[0057] Figure 4 This is a schematic diagram showing the comparison results of disparity difference scores for different stereoscopic image retargeting methods according to an embodiment of the present invention. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings.

[0059] Figure 1 This is a flowchart of a stereo image retargeting method based on a cooperative Mamba and parallax enhancement network according to an embodiment of the present invention. The following is an example... Figure 1 The following example illustrates some specific implementation processes of the present invention. Figure 1 As shown, the stereo image retargeting method based on the Cooperative Mamba and disparity enhancement network proposed in this invention includes the following steps:

[0060] Step S1: Obtain the training dataset, wherein the training dataset includes multiple input stereo images, and the input stereo images include the left image I. L And right image I R ;

[0061] The left image refers to the image acquired based on the left viewpoint, and the right image refers to the image acquired based on the right viewpoint.

[0062] Step S2: Construct a stereo image retargeting depth model, wherein the stereo image retargeting depth model is used to obtain the retargeted left and right images;

[0063] In one embodiment of the present invention, the stereo image retargeting depth model includes a co-Mamba feature extraction sub-model, a disparity enhancement feature aggregation sub-model, and a uniform grid layer.

[0064] The collaborative Mamba feature extraction sub-model is used to extract features from the left image I. L And right image I R Depth feature extraction is performed to obtain the viewpoint features of the left and right images. and and inter-viewpoint correlation features and .

[0065] Furthermore, the Cooperative Mamba Feature Extraction Sub-model includes a basic convolutional module and a Cooperative Mamba Feature Extraction module group connected in sequence. The Cooperative Mamba Feature Extraction module group comprises three cascaded Cooperative Mamba Feature Extraction modules. The basic convolutional module is used to extract initial shallow features from the stereo image, and the extracted left image I... L The initial shallow features are denoted as The extracted right image I R The initial shallow features are denoted as The basic convolutional module sequentially comprises two convolutional layers, one normalization layer, and one PReLU activation function. The collaborative Mamba feature extraction module group is used to extract initial shallow features from the left and right images. and The deep features with global context obtained from the left and right images are extracted, namely the self-viewpoint features of the left and right images. and and inter-viewpoint correlation features and Among the three cascaded collaborative Mamba feature extraction modules in the collaborative Mamba feature extraction module group, the viewpoint correlation features output by the first collaborative Mamba feature extraction module are... and The input to the next Cooperative Mamba Feature Extraction module is the initial shallow features X of the left and right images extracted by the basic convolutional module. L and X R The output of the last Cooperative Mamba Feature Extraction module is the viewpoint features of the left and right images obtained by the Cooperative Mamba Feature Extraction module group. and and inter-viewpoint correlation features and Each collaborative Mamba feature extraction module in the collaborative Mamba feature extraction module group includes two parallel self-viewpoint feature learning Mambas and one viewpoint correlation learning Mamba, such as... Figure 2 As shown.

[0066] Furthermore, the two parallel self-viewpoint feature learning Mambas are used to learn the initial shallow features of the left and right images, respectively. and The left and right images are processed to extract their viewpoint features. and Its structure is as follows Figure 2As shown, the self-viewpoint feature learning Mamba includes layer normalization, two parallel branches, and a linear layer. One of the two parallel branches sequentially includes a linear layer, a depthwise separable convolution, a sigmoid linear unit, a 2D selective scan layer, and layer normalization; the other branch sequentially includes a linear layer and a sigmoid linear unit. Specifically, for processing the self-viewpoint feature learning Mamba on the left image, the initial shallow features of the left image are first processed... Layer normalization is performed; then the output of the layer normalization is input into two parallel branches. In one branch, the output of the layer normalization is processed sequentially using a linear layer, a depthwise separable convolution, a sigmoid linear unit, a 2D selective scan layer, and layer normalization to obtain the left feature with global context information. In another branch, the normalized output of the layer is processed sequentially using a linear layer and a Sigmoid linear unit to obtain the left feature with the original information. Finally, the output features of the two branches are... and Element-wise multiplication is performed, and a linear layer is used to generate the viewpoint features of the left image. Similarly, for the viewpoint feature learning Mamba algorithm for processing the right image, the initial shallow features of the right image are first processed. Layer normalization is performed; then the output of the layer normalization is input into two parallel branches. In one branch, the output of the layer normalization is processed sequentially using a linear layer, a depthwise separable convolution, a sigmoid linear unit, a 2D selective scan layer, and layer normalization to obtain the right feature with global context information. In another branch, the normalized output of the layer is processed sequentially using a linear layer and a Sigmoid linear unit to obtain the right feature with the original information. Finally, the output features of the two branches are... and Element-wise multiplication is performed, and a linear layer is used to generate the viewpoint features of the right image. .

[0067] The above processing can be represented by the following formula:

[0068]

[0069] and

[0070] in, This indicates pixel-by-pixel multiplication. Indicates a linear layer. Representation layer normalization, This represents depthwise separable convolution. Represents the Sigmoid linear unit. This indicates a 2D selective scanning layer.

[0071] Furthermore, the viewpoint correlation learning Mamba is used to capture long-range dependencies and is based on the self-viewpoint features of the left and right images. and By learning the inter-view correlations from the left and right images, the viewpoint correlation features between the left and right images are obtained. and ,like Figure 2 As shown, the viewpoint correlation learning Mamba includes features for processing left image features from the viewpoint. The left image processing unit group is used to process the viewpoint features of the right image. The image processing unit group comprises a right image processing unit group and a common hidden state branch. Each image processing unit group includes layer normalization, two parallel processing unit branches, and a linear layer. The two parallel processing unit branches include a first processing unit branch consisting of a cascaded linear layer, a depthwise separable convolution, a sigmoid linear unit, a 2D selective scan layer, and layer normalization, and a second processing unit branch consisting of a cascaded linear layer and a sigmoid linear unit. The common hidden state branch includes a cascaded 2D selective scan layer and layer normalization. The input of the 2D selective scan layer is the output of the sigmoid linear unit of the first processing unit branch of the two image processing unit groups, the pixel-wise multiplication result of the output of the depthwise separable convolution of the first processing unit branch of the two image processing unit groups, and the pixel-wise summation result of the output of the sigmoid linear unit of the first processing unit branch of the two image processing unit groups.

[0072] More specifically, for the viewpoint correlation learning Mamba, firstly, the self-viewpoint features of the left image are... Self-viewpoint features of the right image The layers are processed by layer normalization in the left and right image processing unit groups, respectively. Then, the processed features are cross-fed from one view to the other to facilitate complementary feature learning between the two images. Specifically, the viewpoint features of the left image are... The corresponding layer normalized output is input to the second processing unit branch of the left image processing unit group to obtain features. Simultaneously, the data is also input to the first processing unit branch of the right image processing unit group to obtain features. Viewpoint features of the right image The corresponding layer normalized output is input to the second processing unit branch in the right image processing unit group to obtain features. Simultaneously, the data is also input to the first processing unit branch of the left image processing unit group to obtain features. Furthermore, to effectively model the correlation between views, a common hidden state branch is established to integrate information from two views, thereby extracting more discriminative features. Specifically, after the depthwise separable convolution of the two first processing unit branches, element-wise multiplication and element-wise addition are used to enhance complementary feature fusion. Subsequently, 2D selective scanning layers and layer normalization are used to promote the correlation interaction between views and model long-distance dependencies to obtain common view features Y. More specifically, the depthwise separable convolution output of the first processing unit branch of the left image processing unit group is... Depth-separable convolution output of the first processing unit branch of the right image processing unit group After pixel-by-pixel multiplication, the outputs of the Sigmoid linear units of the first processing unit branch of the left image processing unit group and the first processing unit branch of the right image processing unit group are added element-by-elementally. The resulting features are then input into the 2D selective scan layer of the common hidden state branch. Finally, after layer normalization processing by the common hidden state branch, the common view feature Y is obtained.

[0073]

[0074] Here, ⊕ represents element-wise addition.

[0075] Then, the common view feature Y is compared with the feature respectively. and characteristics By multiplying the elements of the Hadamard product, we can obtain the characteristic. and characteristics , will feature and characteristics After pixel-by-pixel multiplication, the output obtained by inputting it into the final linear layer of the left image processing unit group is compared with the viewpoint features of the left image. By adding pixels one by one, the correlation features between viewpoints are obtained. , will feature and After pixel-by-pixel multiplication, the output obtained by inputting it into the final linear layer of the right image processing unit group is compared with the viewpoint features of the right image. By adding pixels one by one, the correlation features between viewpoints are obtained. .

[0076] The above processing can be represented by the following formula:

[0077]

[0078] and

[0079]

[0080] in, and ☉ represents the viewpoint correlation features between the left and right images, and ☉ represents the element-wise multiplication of the Hadamard product.

[0081] As mentioned above, the Cooperative Mamba Feature Extraction Module Group comprises three cascaded Cooperative Mamba Feature Extraction Modules. It is also mentioned above that the input to the Cooperative Mamba Feature Extraction Module is the initial shallow features X of the left and right images. L and X R The output is the viewpoint features of the left and right images. and and inter-viewpoint correlation features and Among the three cascaded collaborative Mamba feature extraction modules included in the collaborative Mamba feature extraction module, the viewpoint correlation features output by the previous collaborative Mamba feature extraction module are... and The input to the next Cooperative Mamba Feature Extraction module is the initial shallow features X of the left and right images extracted by the basic convolutional module. L and X R The output of the last Cooperative Mamba Feature Extraction module is the viewpoint feature of the left and right images obtained by the Cooperative Mamba Feature Extraction module group. and and inter-viewpoint correlation features and Specifically, for the first collaborative Mamba feature extraction module of the three cascaded collaborative Mamba feature extraction modules, its input is the initial shallow features X of the left and right images extracted by the basic convolution module. L and X R The output is the viewpoint features of the first obtained left and right images. and and inter-viewpoint correlation features and For the second collaborative Mamba feature extraction module of the three cascaded collaborative Mamba feature extraction modules, the viewpoint correlation features of the left image output by the first collaborative Mamba feature extraction module are used. Replace the initial shallow feature X L As an input to the second Co-Mamba Feature Extraction module, the viewpoint correlation features of the right image output by the first Co-Mamba Feature Extraction module are used. Replace the initial shallow feature X R As another input to the second collaborative Mamba feature extraction module, the second collaborative Mamba feature extraction module again outputs the self-viewpoint features of the left and right images. and and inter-viewpoint correlation features and For the third collaborative Mamba feature extraction module of the three cascaded collaborative Mamba feature extraction modules, the viewpoint correlation features of the left image output by the second collaborative Mamba feature extraction module are used. Replace the initial shallow feature X L As an input to the third Co-Mamba Feature Extraction module, the viewpoint correlation features of the right image output by the second Co-Mamba Feature Extraction module are used. Replace the initial shallow feature X R As another input to the third collaborative Mamba feature extraction module, the viewpoint features of the left and right images output by the third collaborative Mamba feature extraction module are... and and inter-viewpoint correlation features and The viewpoint features of the left and right images output by the final Collaborative Mamba Feature Extraction Module Group and the Collaborative Mamba Feature Extraction Sub-model. and and inter-viewpoint correlation features and .

[0082] The disparity enhancement feature aggregation sub-model is used to integrate the viewpoint features of the left and right images. and and inter-viewpoint correlation features and The left and right images are then subjected to disparity preservation and enhancement to obtain stereo saliency maps of the left and right images. and .

[0083] The disparity enhancement feature aggregation sub-model includes three occlusion-assisted mask modules, which are used to obtain an enhanced disparity map based on self-viewpoint features and inter-viewpoint correlation features. The three occlusion-assisted masking modules use self-viewpoint features and inter-viewpoint correlation features, respectively, the output self-viewpoint features and inter-viewpoint correlation features of the three collaborative Mamba feature extraction modules in the collaborative Mamba feature extraction module group. Specifically, the first occlusion-assisted masking module uses the same self-viewpoint features and inter-viewpoint correlation features as the first collaborative Mamba feature extraction module in the collaborative Mamba feature extraction module group; the second occlusion-assisted masking module uses the same self-viewpoint features and inter-viewpoint correlation features as the second collaborative Mamba feature extraction module in the collaborative Mamba feature extraction module group; and the third occlusion-assisted masking module uses the same self-viewpoint features and inter-viewpoint correlation features as the third collaborative Mamba feature extraction module in the collaborative Mamba feature extraction module group. Thus, each of the three occlusion-assisted masking modules can obtain an enhanced disparity map. Finally, these three enhanced disparity maps By adding element-wise, the final enhanced disparity map can be obtained. This final enhanced disparity map is then compared with the viewpoint correlation features of the left and right images obtained from the collaborative Mamba feature extraction sub-model. and By performing a linear combination, we obtain the stereo saliency map of the left and right images. and .

[0084] The structure and workflow of the occlusion-assisted mask module will be described below using one example. The occlusion-assisted mask module includes a disparity and occlusion estimation Transformer and a disparity enhancement block. The disparity and occlusion estimation Transformer includes multi-head self-attention and multi-head cross-attention, based on the self-viewpoint features of the left and right images. and An initial disparity map and occlusion mask are estimated from the left and right viewpoints. Then, the disparity enhancement block uses the initial disparity map and occlusion mask estimated by the disparity and occlusion estimation Transformer, as well as the viewpoint correlation features of the left and right images. and To further enhance the binocular parallax in the occluded area, an enhanced parallax map is obtained. ,like Figure 3 As shown.

[0085] The disparity and occlusion estimation Transformer includes multi-head self-attention and multi-head cross-attention, respectively. Figure 3As shown. In the disparity and occlusion estimation Transformer, firstly, multi-head self-attention is used to aggregate self-viewpoint features from the left and right images. and The self-attention features of the left and right images are obtained respectively. and :

[0086]

[0087] in, This represents the self-attention features of the left image. This represents the self-attention features of the right image. , and These represent the query vector, key vector, and value vector of the left and right images based on viewpoint features, respectively. This represents a multi-head self-attention and feedforward network.

[0088] Then, in order to handle the binocular parallax between the left and right images, multi-head cross-attention is used to capture the inherent spatial relationship between the left and right images, so as to obtain the cross-attention features of the left and right images. and The specific processing mechanism of multi-head cross-attention is as follows: Figure 3 As shown, specifically, considering that the value vector contains the location information of the features, the value vectors of the self-attention features of the left and right images are first... and Perform element-wise addition to obtain shared values ​​that have cross-view positional relationships. Simultaneously, the transposes of the query vector and key vector of the single-view image are subjected to softmax calculation to obtain the single-view attention map. and The query vector of the self-attention features of the left image. and key vector The attention map of the left image is obtained by performing softmax calculation on the transpose of the image. The query vector of the self-attention features of the right image and key vector The attention map of the right image is obtained by performing softmax calculation on the transpose of the image. And the single-view attention map and respectively with Perform element-wise multiplication to obtain single-view related features. and To capture the positional differences between the cross views, the key vector of the self-attention feature of the left image is used. The query vector of the transpose and self-attention features of the right image Perform softmax calculation to obtain the cross-view attention map. The key vector of the self-attention feature of the right image The query vector of the transpose and self-attention features of the left image. Perform softmax calculation to obtain the cross-view attention map. and cross-view attention map The value vector of the self-attention features of the left image Element-wise multiplication yields the cross-view difference features of the left image. Cross-view attention map The value vector of the self-attention features of the right image Element-wise multiplication yields the cross-view difference features of the right image. Finally, the self-attention features of a single view are... and Each is associated with the corresponding single-view features. and Element-wise addition, the self-attention feature of a single view. and The differences between the cross views and the corresponding single views are as follows: and By subtracting element-wise, the two features obtained from the element-wise addition and subtraction of the single view are concatenated to obtain the cross-attention features of the left and right images. and That is, the self-attention features of the left image Relevant features of the left image Element-wise addition, applying the self-attention features of the left image. Cross-view difference features with the left image Element-wise subtraction is performed, and the two features obtained from element-wise addition and element-wise subtraction of the left image are concatenated to obtain the cross-attention features of the left image. The self-attention features of the right image Relevant features of the right image Element-wise addition, applying the self-attention features of the right image. Cross-view difference features with the right image By subtracting element-wise, the two features obtained from the element-wise addition and subtraction of the right image are concatenated to obtain the cross-attention features of the right image. The above process can be represented by the following formula:

[0089]

[0090]

[0091]

[0092]

[0093]

[0094]

[0095] in, , and Let represent the query vector, key vector, and value vector of the self-attention features of the left and right images, respectively. This represents the softmax function. This indicates the transpose operation. It is a cascading operation. ⊕ indicates pixel-wise multiplication, and ⊕ indicates element-wise addition. This represents element-wise subtraction. Finally, it utilizes the cross-attention features of the left and right images. and The initial disparity map is estimated using disparity and occlusion estimation methods. and masking The methods for obtaining the initial disparity map and occlusion mask using disparity and occlusion estimation are techniques that should be mastered by those skilled in the art, and will not be elaborated upon in this invention.

[0096] like Figure 3 As shown, the input to the disparity enhancement block includes an initial disparity map. , masking Viewpoint correlation features between left and right images and The output is an enhanced disparity map. In the disparity enhancement block, firstly, an initial disparity map is used. The initial disparity map obtained after the convolutional layer The features of viewpoint correlation in the right image Perform deformation to generate synthetic left features Simultaneously, residual attention with skip connections is used to analyze the initial disparity map. Extracting residual features from features Then, the masking layer will be applied. With synthetic left features Perform element-wise multiplication and then further combine with residual characteristics. Element-by-element addition yields the reconstructed left features. Subsequently, the inter-viewpoint correlation features of the left image were calculated using the correlation layer. Reconstructing left features Correlation characteristics between ; then, the relevant features and residual characteristics Cascaded together, we obtain composite features. Then, composite features The three features obtained by convolution with three dilated convolutional layers Conv3-1, Conv3-2, and Conv3-4 are concatenated, and the concatenated output features are then compared with the initial disparity map. The features are added pixel by pixel; finally, the ReLU activation function is used to process the features obtained by pixel-by-pixel addition to obtain the enhanced disparity map. To maintain the enhanced parallax map Non-negative. The above process can be represented by the following formula:

[0097]

[0098] in, and This represents residual attention and convolutional layers. This indicates a transformation operation. Indicates the relevant layer, , and This represents three dilated convolutional layers with a kernel size of 3×3 and expansion rates of 1, 2, and 4, respectively. This represents the ReLU activation function.

[0099] Finally, the stereo saliency maps of the left and right images obtained from the disparity enhancement feature aggregation sub-model are... and The target scaling ratio η is input into the uniform grid map to simultaneously adjust the size of the left and right images, resulting in retargeted left and right images. The target scaling ratio η can be set according to the needs of the actual application.

[0100] The technique of using a uniform grid layer to obtain a retargeted image is a technique that should be mastered by those skilled in the art, and will not be described in detail in this invention.

[0101] Step S3: Train the stereo image repositioning depth model based on the training dataset and the preset overall loss function to obtain the stereo image repositioning target depth model;

[0102] In one embodiment of the present invention, the preset overall loss function includes a stereo similarity loss function and a disparity occlusion consistency loss function, so as to maintain the consistency of content and binocular disparity between the retargeted stereo image and the source stereo image.

[0103] Wherein, the preset overall loss function L multi Represented as:

[0104] L multi =L lpips +L po

[0105] Among them, L lpips L represents the stereo similarity loss function. po This represents the parallax occlusion consistency loss function.

[0106] The stereo similarity loss function is used to maintain the perceptual quality of stereo images and preserve high-level semantic features.

[0107] Furthermore, the stereo similarity loss function L lpips It can be represented as:

[0108]

[0109] in, and This refers to the reconstructed left and right images obtained by processing the left and right images in the training dataset using the stereo image repositioning depth model, and then feeding them back into the stereo image repositioning depth model. and This represents the feature extractor that performs feature extraction on the image corresponding to the i-th and j-th convolutional layers in VGG. and These are the weights of the i-th and j-th convolutional layers in VGG that process the image. It is the square L2 norm.

[0110] The disparity occlusion consistency loss function includes a disparity supervision term and an occlusion supervision term. The disparity supervision term is used to constrain the disparity consistency between the stereo image retargeting result and the reconstructed stereo image, and the occlusion supervision term is used to constrain the occlusion consistency between the stereo image retargeting result and the reconstructed stereo image.

[0111] Furthermore, the disparity occlusion consistency loss function L po It can be represented as:

[0112]

[0113] and

[0114]

[0115] in, and These represent the parallax supervision term and the occlusion supervision term, respectively. This represents the true value of the occlusion mask obtained by directly utilizing the left-right consistency check. It is an extremely small number, for example, it can be set to 10. -6 To prevent the denominator from being zero, and Indicates the weight.

[0116] Step S4: Use the stereo image redirection target depth model to process the stereo image to be redirected to obtain the stereo image redirection result.

[0117] Figure 4 This paper presents a comparison of disparity scores for stereo image retargeting results obtained using different methods. The algorithms compared include Shao's method and Fan's method. Shao's method is a traditional stereo image retargeting method, while Fan's method is a deep learning-based stereo image retargeting method. A lower disparity score indicates less disparity distortion in the stereo image retargeting result. Figure 4 As can be seen, compared with Shao's method, the disparity difference score of this invention is smaller. The main reason is that Shao's method uses traditional handcrafted features to learn salient object features, ignoring high-level semantic information, which leads to shape distortion in the retargeted image. Furthermore, Fan's method also performs worse than this invention in terms of disparity difference score, mainly because Fan's method does not adequately learn the long-distance dependencies between stereo images. In contrast, this invention, by constructing a collaborative Mamba feature extraction sub-model and a disparity enhancement feature aggregation sub-model, combined with a stereo similarity loss function and a disparity occlusion consistency loss function, maintains the salient structure of the stereo image and reduces disparity distortion, achieving good stereo image retargeting performance.

[0118] Unless otherwise specified, the model numbers of the various devices in this embodiment of the invention are not limited, and any device that can perform the above functions is acceptable.

[0119] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0120] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A stereo image retargeting method based on a cooperative Mamba and parallax enhancement network, characterized in that, The method includes the following steps: Step S1: Obtain the training dataset, wherein the training dataset includes multiple input stereo images, and the input stereo images include the left image I. L And right image I R ; Step S2: Construct a stereo image retargeting depth model, wherein the stereo image retargeting depth model is used to obtain the retargeted left and right images; Step S3: Train the stereo image repositioning depth model based on the training dataset and the preset overall loss function to obtain the stereo image repositioning target depth model; Step S4: Use the stereo image redirection target depth model to process the stereo image to be redirected to obtain the stereo image redirection result.

2. The method according to claim 1, characterized in that, The stereo image retargeting depth model includes a cooperative Mamba feature extraction sub-model, a disparity enhancement feature aggregation sub-model, and a uniform grid layer, wherein: The Collaborative Mamba Feature Extraction Submodel is used to extract features from the left image I. L And right image I R Depth feature extraction is performed to obtain the viewpoint features of the left and right images. and and inter-viewpoint correlation features and ; The disparity enhancement feature aggregation sub-model is used to utilize the viewpoint features of the left and right images. and and inter-viewpoint correlation features and The left and right images are then subjected to disparity preservation and enhancement to obtain stereo saliency maps of the left and right images. and ; The uniform grid layer is used for stereo saliency mapping based on the left and right images. and And the target scaling ratio η is used to adjust the size of the left and right images simultaneously, resulting in the retargeted left and right images.

3. The method according to claim 2, characterized in that, The collaborative Mamba feature extraction sub-model comprises a basic convolutional module and a collaborative Mamba feature extraction module group connected in sequence, wherein: The basic convolutional module consists of two convolutional layers, a normalization layer, and a PReLU activation function, used to extract initial shallow features from the left and right images. and ; The Cooperative Mamba Feature Extraction Module Group comprises three cascaded Cooperative Mamba Feature Extraction Modules, used to extract features based on the initial shallow features of the left and right images. and Deep features with global context were extracted from the left and right images: viewpoint features of the left and right images. and and inter-viewpoint correlation features and Among the three cascaded collaborative Mamba feature extraction modules, the viewpoint correlation features output by the first collaborative Mamba feature extraction module are... and The input to the next Cooperative Mamba Feature Extraction module is the initial shallow features of the left and right images, while the input to the first Cooperative Mamba Feature Extraction module is the initial shallow features of the left and right images. and .

4. The method according to claim 3, characterized in that, The collaborative Mamba feature extraction module includes two parallel self-viewpoint feature learning Mambas and one viewpoint correlation learning Mamba, wherein: The two parallel viewpoint feature learning Mambas are used based on the initial shallow features of the left and right images, respectively. and The left and right images are processed to extract their viewpoint features. and The self-viewpoint feature learning Mamba includes layer normalization, two parallel branches and a linear layer. One of the two parallel branches includes a linear layer, a depthwise separable convolution, a Sigmoid linear unit, a 2D selective scan layer and layer normalization in sequence, and the other branch includes a linear layer and a Sigmoid linear unit in sequence. The viewpoint correlation learning Mamba is used to capture long-range dependencies and is based on the self-viewpoint features of the left and right images. and By learning the inter-view correlations from the left and right images, the viewpoint correlation features between the left and right images are obtained. and The viewpoint correlation learning Mamba includes a left image processing unit group, a right image processing unit group, and a common hidden state branch. Each image processing unit group includes layer normalization, two parallel processing unit branches, and a linear layer. The two parallel processing unit branches include a first processing unit branch consisting of a cascaded linear layer, a depthwise separable convolution, a sigmoid linear unit, a 2D selective scan layer, and layer normalization, and a second processing unit branch consisting of a cascaded linear layer and a sigmoid linear unit. The common hidden state branch includes a cascaded 2D selective scan layer and layer normalization.

5. The method according to claim 3, characterized in that, The disparity enhancement feature aggregation sub-model includes three occlusion-assisted mask modules, which are used to obtain enhanced disparity maps based on self-viewpoint features and inter-viewpoint correlation features. The self-viewpoint features and inter-viewpoint correlation features used by the three occlusion-assisted masking modules are the self-viewpoint features and inter-viewpoint correlation features output by the three cascaded collaborative Mamba feature extraction modules, respectively. The three occlusion-assisted masking modules obtain three enhanced disparity maps. Correlation features between viewpoints of the left and right images after element-wise summation. and Linear combination yields the stereo saliency map of the left and right images. and .

6. The method according to claim 5, characterized in that, The occlusion-assisted mask module includes a disparity and occlusion estimation Transformer and a disparity enhancement block. The disparity and occlusion estimation Transformer includes multi-head self-attention and multi-head cross-attention, used to perform self-viewpoint features based on the left and right images. and An initial disparity map and occlusion mask are estimated; the disparity enhancement block utilizes the initial disparity map and occlusion mask, as well as the viewpoint correlation features of the left and right images. and To further enhance the binocular parallax in the occluded area, an enhanced parallax map is obtained. .

7. The method according to claim 6, characterized in that, In the disparity and occlusion estimation Transformer: First, multi-head self-attention is used to aggregate viewpoint features from the left and right images. and The self-attention features of the left and right images are obtained respectively. and : ; in, , and These represent the query vector, key vector, and value vector of the left and right images based on viewpoint features, respectively. ⊕ represents multi-head self-attention and feedforward networks, and ⊕ represents element-wise addition; Then, multi-head cross-attention is used to capture the inherent spatial relationship between the left and right images to obtain the cross-attention features of the left and right images. and : ; ; ; ; ; ; in, , and Let represent the query vector, key vector, and value vector of the self-attention features of the left and right images, respectively. This represents the softmax function. This indicates the transpose operation. It is a cascading operation. ⊕ indicates pixel-wise multiplication, and ⊕ indicates element-wise addition. This indicates element-wise subtraction. To convert the value vectors of the self-attention features of the left and right images and The shared value obtained by adding elements one by one, which has the positional relationship of the cross view; Finally, based on the cross-attention features of the left and right images and The initial disparity map is estimated using disparity and occlusion estimation methods. and masking .

8. The method according to claim 7, characterized in that, In the parallax enhancement block, the enhanced parallax map is obtained using the following formula. : ; in, and This represents residual attention and convolutional layers. This indicates a transformation operation. Indicates the relevant layer, , and This represents three dilated convolutional layers with kernel sizes of 3×3 and expansion rates of 1, 2, and 4, respectively. This represents the ReLU activation function.

9. The method according to claim 1, characterized in that, The preset overall loss function includes a stereo similarity loss function and a disparity occlusion consistency loss function: L multi =L lpips +L po Among them, L multi L represents the preset overall loss function. lpips L represents the stereo similarity loss function. po This represents the parallax occlusion consistency loss function.

10. The method according to claim 9, characterized in that, The stereo similarity loss function L lpips Represented as: ; in, and This refers to the reconstructed left and right images obtained by processing the left and right images in the training dataset using the stereo image repositioning depth model, and then feeding them back into the stereo image repositioning depth model. and This represents the feature extractor that performs feature extraction on the image corresponding to the i-th and j-th convolutional layers in VGG. and These are the weights of the i-th and j-th convolutional layers in VGG that process the image. It is the square L2 norm; The disparity-occlusion consistency loss function includes a disparity supervision term and an occlusion supervision term: ， ; in, and These represent the parallax supervision term and the occlusion supervision term, respectively. Indicates occlusion mask. This represents the actual value of the occlusion mask obtained directly using the left-right consistency check. It is a very small number. and Indicates the weight.

Images