A stereoscopic image redirection method and apparatus for resource-constrained terminals

Abstract

The present application relates to the technical field of computer vision, and provides a stereoscopic image redirection method and device for resource-limited terminals. The present application extracts the stereoscopic image features of an original stereoscopic image to obtain a disparity map, a left-view saliency map and a right-view saliency map, and fuses to obtain a global saliency map; left and right binary saliency maps are obtained through binarization processing to calculate the boundary values of a protection area; the target width of a cropping window is determined according to the boundary values and a target scaling ratio; a cropped image is obtained by cropping the non-salient area of the edge part; if the cropped image meets the target size requirement, it is determined as a redirection image; otherwise, an axis-aligned image is obtained by performing axis alignment scaling on the cropped image, and the axis-aligned image is determined as a redirection image; the problem that resource-limited mobile terminals in the prior art cannot balance the calculation complexity and redirection quality of stereoscopic image redirection, resulting in the inability to flexibly adapt to the resolution and aspect ratio of different display screens, is solved.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method and apparatus for stereoscopic image retargeting for resource-constrained terminals. Background Technology

[0002] Three-dimensional images provide depth information about a scene and are the core visual content carrier in applications such as Virtual Reality (VR) and Augmented Reality (AR).

[0003] In practical applications, stereoscopic images need to be presented on various terminal devices with different display resolutions and aspect ratios. For example, VR headsets, AR glasses, and smartphones from different brands have significant differences in screen physical size and pixel resolution; even on the same mobile terminal, the aspect ratio of the displayed image will change when the user rotates the screen (e.g., from 16:9 to 9:16). To achieve the best visual presentation on various mobile terminals, the original stereoscopic image must be adaptively resized, i.e., stereoscopic image redirection. Stereoscopic image redirection is one of the key technologies in stereoscopic image processing. The core objective of stereoscopic image redirection is to effectively maintain the visual saliency information and depth features of the stereoscopic image while adjusting the image size to adapt to different display devices, thereby providing users with a high-quality immersive visual experience.

[0004] However, VR, AR, and smartphones are resource-constrained systems with limited processor and memory resources, and are highly sensitive to image processing latency. For example, high latency in displaying stereoscopic images on mobile screens can directly cause visual dizziness in users. While existing content-aware, high-complexity methods for stereoscopic image repositioning can guarantee image quality, they are difficult to implement in real-time on these devices, severely impacting the user experience. Furthermore, stereoscopic perception distortion occurs during screen aspect ratio adaptation. When the aspect ratio of a mobile screen changes (e.g., screen rotation), existing solutions often result in asynchronous row heights or column widths in the left and right views to adapt to the new size. This alters the original horizontal and vertical parallax, disrupting depth perception and reducing the realism and comfort of 3D displays. In addition, deformation accumulates in insignificant edge areas. Even with smoothing constraints, when using axis alignment or mesh deformation operators, severe compression or stretching deformation still occurs in insignificant areas such as image edges. This deformation is particularly noticeable in VR or AR scenes with large field of view, disrupting the structural coherence of the image.

[0005] In summary, on resource-constrained mobile terminals, for different screen resolutions and aspect ratios (including dynamic changes in aspect ratio due to screen rotation), existing technologies cannot effectively maintain the depth perception of stereo images and avoid edge structure distortion while achieving high-quality stereo image adaptive redirection with low computational complexity and low memory consumption.

[0006] Therefore, overcoming the shortcomings of the existing technology is an urgent problem to be solved in this technical field. Summary of the Invention

[0007] The technical problem that this invention aims to solve is that existing technologies, for resource-constrained mobile terminals, cannot flexibly adapt to the resolutions and aspect ratios of different displays because they cannot balance the computational complexity and redirection quality of stereoscopic image redirection.

[0008] Firstly, a method for stereoscopic image retargeting for resource-constrained terminals is provided, including: Stereo image features are extracted from the original stereo image to obtain a disparity map, a left-view saliency map, and a right-view saliency map, which are then fused to obtain a global saliency map. The left-view saliency map is binarized to obtain a left binary saliency map, and the right-view saliency map is binarized to obtain a right binary saliency map. Based on the left and right binary saliency maps, the boundary values ​​of the protected area are calculated; the target width of the cropping window is determined based on the boundary values ​​and the target scaling ratio; the cropping operator is subjected to dual optimization constraints based on the boundary values, the global saliency map, and the target width to crop the non-salient areas of the edge portion of the original stereoscopic image, resulting in a cropped image; if the cropped image meets the target size requirements, the cropped image is determined as a retargeting image. If the cropped image does not meet the target size requirement, then the cropped image is axially aligned and scaled to obtain an axially aligned image; the axially aligned image is then identified as the retargeting image.

[0009] Further, calculating the boundary values ​​of the protected area based on the left binary saliency map and the right binary saliency map includes: Extract the minimum and maximum column values ​​of all salient pixels in the left binary saliency map; extract the minimum and maximum column values ​​of all salient pixels in the right binary saliency map; The minimum value among all the minimum column values ​​is determined as the left boundary value, and the maximum value among all the maximum column values ​​is determined as the right boundary value, thus obtaining the boundary values.

[0010] Further, determining the target width of the cropping window based on the boundary value and the target scaling ratio includes: The difference between the column coordinates corresponding to the right boundary value and the column coordinates corresponding to the left boundary value is determined as the intermediate value; the sum of the reference value and the mitigation coefficient is determined as the coefficient value; the product of the intermediate value and the coefficient value is determined as the optimal width. The planned width is determined by the product of the specified scaling ratio and the original image width; if the optimal width is less than the planned width, the planned width is determined as the target width; otherwise, the optimal width is determined as the target width.

[0011] Furthermore, when the optimal width is greater than or equal to the original image width, the original stereoscopic image is not cropped, but directly axially aligned and scaled to make the axially aligned image the retargeting image.

[0012] Further, the step of applying dual optimization constraints to the cropping operator based on the boundary value, the global saliency map, and the target width to crop the non-salient regions of the edge portion of the original stereo image, resulting in a cropped image, includes: Based on the target width, determine all pixels outside the cropping window in the original stereo image; determine a first accumulated value of the global saliency of all pixels outside the cropping window, and determine a second accumulated value of the global saliency of all pixels in the original stereo image; The ratio of the first accumulated value to the second accumulated value is determined as the significant energy term; The sum of the salient energy term and the weighting parameter is determined as the weighted value; the sum of the weighted value and the distance energy term is minimized to obtain the cropped image; wherein, the distance energy term is used to constrain the cropping range from intruding into the depth key region around the salient object.

[0013] Further, the step of performing axis-aligned scaling on the cropped image to obtain an axis-aligned image includes: Overlay a grid of N columns and M rows onto the cropped image, and initialize the column width and row height variables of each grid cell; The similarity energy term constrains the aspect ratio variation between adjacent grid cells; the rigidity energy term constrains each grid cell to maintain similar transformation; the regularization energy term penalizes the size difference between adjacent rows or columns; and the disparity energy term constrains the column width variation of corresponding columns at the same horizontal index in the left and right views of the cropped image to be equal. The product of the disparity energy term and the weight parameter is determined as the weighting term; the sum of the similarity energy term, the rigidity energy term, the regularization energy term, and the weighting term is determined as the total energy function. Using the column width variable and the row height variable, the total energy function is used as the objective function of the optimization problem to solve for the optimal width of each column and the optimal height of each row in the required axis-aligned image. The cropped image is then scaled according to the optimal width and the optimal height to obtain the axis-aligned image.

[0014] Furthermore, the constraint by which the column width variation of corresponding columns at the same horizontal index in the left and right views of the cropped image is equal through the disparity energy term is as follows: For each pair of left and right views with the same horizontal index, calculate the square of the difference between the column width change in the left view and the column width change in the right view to obtain the disparity change. The sum of the disparity changes in all corresponding columns is used to determine the disparity energy term.

[0015] Furthermore, the step of using the column width variable and the row height variable, and taking the total energy function as the objective function of the optimization problem, to solve for the optimal width of each column and the optimal height of each row in the required axis-aligned image includes: Each term in the total energy function is rewritten as a quadratic form in terms of the column width and row height variables, and then combined into a standard quadratic programming expression in the form of a positive semidefinite matrix. Constraints are imposed on the standard quadratic programming expression, including: each column width variable is non-negative, each row height variable is non-negative, the sum of all row height variables is equal to the target image height, the sum of all column width variables is equal to the target image width, and the corresponding row height variables in the left and right views are equal. Under the constraints, the quadratic programming standard form is minimized using a convex optimization solver to obtain the optimal width and optimal height.

[0016] In a second aspect, a stereoscopic image redirection device for a resource-constrained terminal is provided, the stereoscopic image redirection device for a resource-constrained terminal comprising: a processor and a memory for storing processor-executable instructions; The processor is configured to execute the stereoscopic image retargeting method for resource-constrained terminals.

[0017] Thirdly, a non-volatile computer storage medium is provided, the computer storage medium storing computer-executable instructions, which are executed by one or more processors to perform the stereoscopic image retargeting method for resource-constrained terminals described in the first aspect.

[0018] Fourthly, a computer program product containing instructions is provided that, when executed on a computer or processor, causes the computer or processor to perform the stereoscopic image repositioning method for a resource-constrained terminal as described in the first aspect.

[0019] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention extracts only the most important spatial and disparity features from the original stereo image, thereby reducing the computational load in the feature extraction stage. Then, based on the aforementioned feature image, a cropping operator is used to remove background information from non-salient areas outside salient regions. This directly reduces the pixel size required for subsequent axis alignment and fundamentally avoids the image quality degradation caused by compression deformation in edge regions through Laplacian regularization in existing technologies. It also avoids the severe structural distortion problems that may occur when using axis alignment operators later. Finally, the retargeting task is completed using the axis alignment operator. By using coarse cropping to quickly reduce complexity and preserve the integrity of objects in the original stereo image, and fine axis alignment to smoothly supplement the target size, an optimal balance between computational resources and visual fidelity can be achieved on resource-constrained mobile terminals. Furthermore, by adjusting the cropping width and axis alignment scaling ratio, it can flexibly adapt to the resolution and aspect ratio of different displays. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating a stereoscopic image redirection method for resource-constrained terminals provided in an embodiment of the present invention. Figure 2 This is a schematic diagram illustrating a specific example of a low-complexity multi-operator stereo image retargeting framework provided in this embodiment of the invention. Figure 3 This is a schematic diagram illustrating a specific example of structural distortion in a planar image axis alignment method provided by an embodiment of the present invention; Figure 4 This is a schematic diagram illustrating a specific example of the optimization comparison of the effect of regularized energy terms on axis alignment redirection provided by an embodiment of the present invention; Figure 5 This is a flowchart illustrating the first step 20 provided in an embodiment of the present invention; Figure 6 This is a flowchart illustrating the second step 20 provided in an embodiment of the present invention; Figure 7 This is a flowchart illustrating the third step 20 provided in an embodiment of the present invention; Figure 8 This is a flowchart illustrating step 30 provided in an embodiment of the present invention; Figure 9 This is a flowchart illustrating step 302 provided in an embodiment of the present invention; Figure 10 This is a flowchart illustrating step 304 provided in an embodiment of the present invention; Figure 11 This is a schematic diagram of a stereoscopic image redirection device for a resource-constrained terminal provided in an embodiment of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0023] Unless the context otherwise requires, throughout the specification and claims, the term "comprising" is interpreted as openly inclusive, meaning "including, but not limited to." In the description of the specification, terms such as "one embodiment," "some embodiments," "exemplary embodiment," "example," "specific example," or "some examples" are intended to indicate that a particular feature, structure, material, or characteristic associated with that embodiment or example is included in at least one embodiment or example of this disclosure. The illustrative representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics mentioned may be included in any suitable manner in any one or more embodiments or examples; that is, although they may be incorporated into embodiments or examples using the above terms for reasons such as order and position, it does not limit them to be incorporated in combination by a single embodiment or example.

[0024] In the description of this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined with "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of this disclosure, unless otherwise stated, "a plurality of" means two or more. Furthermore, for example, the description may use the prefix "A" or "B" to describe the same type of nouns as two independent entities. In this case, the corresponding features defined with "A" and "B" are used only to distinguish between similar entities and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features.

[0025] In describing some embodiments, the terms "coupled," "coupled," and "connected," and their derivative expressions, may be used. For example, the term "connected" may be used in describing some embodiments to indicate that two or more components have direct physical or electrical contact with each other. Similarly, the term "coupled" may be used in describing some embodiments to indicate that two or more components have direct physical or electrical contact. However, the terms "connected" or "coupled" may also refer to two or more components that do not have direct contact with each other but still cooperate or interact with each other, such as "optical coupling," "wireless connection," etc. The embodiments disclosed herein are not necessarily limited to the scope of this invention.

[0026] Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0027] Existing stereo image retargeting methods generally suffer from high computational complexity and high resource consumption, making it difficult to meet the needs of real-time processing and limiting their deployment and promotion in practical applications.

[0028] As the current mainstream technical solution, existing content-aware stereo image retargeting methods mainly include three typical methods: mesh deformation, slit cropping, and cropping. The computational process of these methods can be decomposed into two key parts: feature extraction and dynamic programming. In the feature extraction stage, the method needs to extract depth features (e.g., disparity features) and spatial features (e.g., salient features, edge features, etc.) from the original image. The time complexity of this stage is positively correlated with the number of features to be extracted, while the spatial complexity increases quadratically with the feature map resolution. Dynamic programming is mainly used to compute and optimize the deformation or cropping path during the retargeting process; different methods exhibit significant differences in computational efficiency in this part. The cropping method achieves retargeting through simple region selection and cropping operations, resulting in high computational efficiency and low memory usage. The mesh deformation method controls image deformation by adjusting mesh vertices, involving a large number of matrix operations and optimizations. Memory usage is linearly related to mesh resolution, resulting in low computational efficiency, and the computational complexity increases exponentially with mesh refinement. The slit cropping method achieves retargeting by iteratively detecting and deleting the least important slits in the image. Slit detection and deletion operations are computationally intensive and consume a significant amount of memory. Furthermore, the entire energy matrix needs to be recalculated for each deleted slit, making it the most complex of the three methods.

[0029] In recent years, deep learning technology has made significant progress in the field of image processing, providing new solutions for stereo image retargeting. However, existing deep learning-based image retargeting methods are still in the exploratory stage, mainly divided into two categories: one type directly embeds dynamic programming methods into the training process to solve for the optimal solution and performs image scaling based on the optimal parameters. Although this can guarantee the quality of retargeting, the complex network structure leads to excessive memory consumption and low computational efficiency. The other type migrates the retargeting operation to be completed separately outside of training. Although this simplifies the training process, it does not achieve end-to-end retargeting and still faces high memory requirements, making it difficult to deploy on resource-constrained devices.

[0030] Overall, existing deep learning solutions are generally limited by the large number of parameters and high computational complexity. Even in the deployment mode of offline training and online inference, they still cannot meet the stringent requirements of mobile devices for memory usage and real-time performance.

[0031] Mobile terminals with limited resources, such as VR and AR, need to cope with differences in screen resolution and dynamic changes in aspect ratio (e.g., screen rotation). The practical technical problem to be solved by this invention is how to complete the adaptive redirection of stereo images with low complexity under such low computing power and low power consumption hardware constraints, while effectively maintaining depth perception and avoiding edge deformation.

[0032] To address the problems of existing technologies, this embodiment proposes a stereoscopic image retargeting method for resource-constrained terminals. In one embodiment, such as... Figure 1 As shown, it includes: Step 10: Extract the stereo image features from the original stereo image to obtain a disparity map, a left-view saliency map, and a right-view saliency map, and fuse them to obtain a global saliency map; perform binarization processing on the left-view saliency map to obtain a left binary saliency map; perform binarization processing on the right-view saliency map to obtain a right binary saliency map.

[0033] The original stereoscopic images include the original images from the left and right perspectives.

[0034] One of the main challenges facing current stereoscopic image retargeting technology is the inability to achieve fast processing speeds while maintaining high-quality visual effects; this limitation restricts its widespread use in resource-constrained and real-time application scenarios. To address this issue, this invention proposes a stereoscopic image retargeting method for resource-constrained terminals. The following section uses reducing image width as an example to systematically illustrate the method of this invention. Figure 2 The overall framework of the method according to the embodiments of the present invention is shown.

[0035] It should be noted that, Figure 2The simultaneous appearance of "Left Image," "Right Image," and "Redirected Image" on the output side does not represent three parallel optional results, but rather serves to accurately express the data structure of the stereoscopic image output. A stereoscopic image consists of a left view and a right view paired together. The redirection task must output both the processed left and right images simultaneously; these two together form a complete stereoscopic image pair that can be used in a 3D display device. "Left Image" and "Right Image" are data entities: they are two specific components of the final output "Redirected Image," each output independently after cropping (or subsequent axis alignment). "Redirected Image" is a functional term: it is the overall abstract name for the combined "Left Image" and "Right Image" pair, representing the final stereoscopic image product delivered to the user.

[0036] This invention first extracts stereo image features from the original image to obtain a disparity map and left and right view saliency maps. In one embodiment, during the feature extraction and optimization stage, this invention generates a disparity map based on the original left and right view images of the original stereo image. Significant left-angle view and right-angle salient image Then, by using parallax information, the saliency map of the left viewpoint is fused. and right-angle salient image The global saliency map is obtained. Simultaneously, the saliency maps from the left perspective were analyzed. and right-angle salient image Perform binarization to generate a left binary saliency map. And right binary saliency map This provides precise saliency guidance for subsequent processing.

[0037] Step 20: Calculate the boundary value of the protected area based on the left binary saliency map and the right binary saliency map; determine the target width of the cropping window based on the boundary value and the target scaling ratio; apply dual optimization constraints to the cropping operator based on the boundary value, the global saliency map, and the target width to crop the non-salient areas of the edge portion of the original stereoscopic image to obtain the cropped image; if the cropped image meets the target size requirements, then the cropped image is determined as the retargeting image.

[0038] Then, based on the aforementioned feature image, a cropping operator is used to remove background information outside the salient region, while also avoiding the severe structural distortion problem that may occur when using the axis alignment operator later. In one embodiment, in the adaptive cropping process, this embodiment of the invention uses the left binary saliency map... And right binary saliency map The boundary values ​​of the protected area are calculated, and the width of the cropping window is determined by combining the target scaling ratio. The cropping operator is subjected to dual optimization constraints using salient energy and distance energy terms, accurately cropping non-salient areas at the edges of the stereo image, minimizing image content loss while protecting salient objects. If the cropped image meets the target size requirements, it is output as a retargeted image.

[0039] Step 30: If the cropped image does not meet the target size requirement, then the cropped image is axially aligned and scaled to obtain an axially aligned image; the axially aligned image is then identified as the retargeting image.

[0040] Finally, the retargeting task is completed using an axis alignment operator. In one embodiment, during the axis alignment scaling stage, if the cropped image still does not meet the target size requirement, further axis alignment scaling is performed, and the axis-aligned image is output as the retargeted image. Optimization is achieved through constraints from similarity energy, rigidity energy, regularization energy, and disparity energy, transforming the complex dynamic programming problem into a simple quadratic programming problem. This ensures high-quality stereo retargeted images while maintaining real-time processing performance.

[0041] This invention extracts only the most important spatial and disparity features from the original stereo image, thereby reducing the computational load in the feature extraction stage. Then, based on the aforementioned feature image, a cropping operator is used to remove background information from non-salient areas outside salient regions. This directly reduces the pixel size required for subsequent axis alignment and fundamentally avoids the image quality degradation caused by compression deformation in edge regions through Laplacian regularization in existing technologies. It also avoids the severe structural distortion problems that may occur when using axis alignment operators later. Finally, the retargeting task is completed using the axis alignment operator. By using coarse cropping to quickly reduce complexity and preserve the integrity of objects in the original stereo image, and fine axis alignment to smoothly supplement the target size, an optimal balance between computational resources and visual fidelity can be achieved on resource-constrained mobile terminals. Furthermore, by adjusting the cropping width and axis alignment scaling ratio, it can flexibly adapt to the resolution and aspect ratio of different displays.

[0042] To address the challenge of balancing computational complexity and repositioning quality in existing stereo image repositioning methods, this invention proposes a stereo image repositioning method for resource-constrained terminals. This method innovatively combines cropping and axis alignment operators, employing a dual optimization mechanism to ensure the integrity of salient objects and maintain stereo perception while achieving low-complexity stereo image repositioning. Experimental results demonstrate that the method of this invention improves computational efficiency while generating stereo repositioned images with excellent visual quality. The following is a further description of the method: In existing technologies, content-aware stereo image retargeting methods mainly rely on three methods: mesh deformation, slit clipping, and clipping.

[0043] Mesh deformation methods preserve important image content by dividing the image into regular meshes and controlling mesh deformation. This effectively protects the global structure and important regions of the image and offers high flexibility, allowing for adaptation to different retargeting needs by adjusting the mesh size and deformation strategy. However, this method is prone to local distortion in non-critical areas, and its computational efficiency is low due to the large number of matrix operations and optimizations involved. The computational complexity increases exponentially, especially as the mesh becomes finer.

[0044] The slit clipping method achieves retargeting by dynamically deleting or inserting the least important slits in the image, which can better maintain the parallax consistency of the stereo image and reduce distortion. However, the slit detection and deletion operations are computationally intensive, and each slit deletion requires recalculation, resulting in high computational complexity.

[0045] This invention provides specific examples of structural distortion in two sets of planar image axis alignment methods, such as... Figure 3 (a) in the image is a raw image, such as... Figure 3 (b) in the middle is Figure 3 The axis alignment result of (a) in the image; such as Figure 3 (c) in the image is a raw image, such as... Figure 3 (d) in the middle is Figure 3 The axis alignment result of (c) in the diagram.

[0046] Cropping methods achieve retargeting by identifying and protecting salient regions and selectively retaining the most important image patches. While computationally efficient due to their simple region selection and cropping operations, precise cropping is difficult to achieve in complex scenes where salient and non-salient content are often interspersed.

[0047] In summary, due to the complex optimization calculations involved, the mesh deformation method and the slit clipping method have low computational efficiency and are difficult to meet the requirements of real-time processing. Furthermore, the clipping method often leads to the loss of important content when processing images with scattered or large proportions of salient objects, limiting its applicability. Using any of these three operators alone cannot achieve efficient and high-quality stereo image retargeting.

[0048] Therefore, embodiments of the present invention select low-complexity retargeting operators such as axis alignment operators and cropping operators to solve the above-mentioned problems in stereo image retargeting.

[0049] The axis alignment operator in this invention divides the image into a regular grid and uniformly adjusts the width or height of the grid along the horizontal axis (i.e., column) or vertical axis (i.e., row) to achieve image size transformation. That is, the grid width of the same column changes synchronously, and the grid height of the same row changes synchronously. Compared with existing grid optimization or vertex optimization methods, the axis alignment operator in this invention does not require individual processing of each grid cell, but rather performs a uniform adjustment on the entire column or row, significantly reducing the number of variables involved, thus offering the advantages of simplicity and efficiency. Based on this characteristic, this invention considers extending the axis alignment method commonly used in planar image retargeting to the field of stereo image retargeting. However, this extension faces two key challenges: First, after extending the two-dimensional axis alignment method to three-dimensional space, how to effectively maintain the depth perception and stereoscopic effect of the stereo image while keeping computationally low is a problem that urgently needs to be solved. Stereo image retargeting not only needs to maintain the integrity of planar visual information but also needs to ensure the consistency of depth information, which places higher demands on the design and optimization of the method.

[0050] To address the aforementioned issues, this invention proposes a depth preservation strategy based on disparity energy terms and integrates it into the energy optimization framework of the axis alignment method. This strategy, by introducing a disparity energy term, explicitly constrains the disparity consistency between the left and right views during retargeting, thereby effectively maintaining depth perception and stereoscopic effect in stereoscopic images without significantly increasing computational costs.

[0051] Secondly, during scaling, the axis alignment method can easily lead to noticeable boundaries or breaks between adjacent regions due to inconsistent scaling ratios in different areas, resulting in image structure distortion. Figure 4 As shown, where, as Figure 4 (a1) in the image is a raw image, such as... Figure 4 (b1) in the middle is Figure 4 The unregularized image in (a1), such as Figure 4 (c1) in the middle is Figure 4 (a1) contains a regularized image; such as Figure 4 (a2) in the image is a raw image, such as... Figure 4 (b2) in the middle is Figure 4 The unregularized image in (a2), such as Figure 4 (c2) in the middle is Figure 4 (a2) contains a regularized image; such as Figure 4 (a3) in the image is a raw image, such as... Figure 4 (b3) in the middle is Figure 4 The unregularized image in (a3), such as Figure 4 (c3) in the middle is Figure 4(a3) contains a regularized image. This structural distortion problem has not been effectively solved in planar image axis alignment methods, seriously affecting the visual coherence and naturalness of the redirected image. Therefore, if axis alignment technology is to be applied to stereo image redirection, its inherent structural distortion problem must first be solved to ensure the structural integrity and visual rationality of the redirected stereo image in three-dimensional space.

[0052] To address the structural distortion problem in axis alignment methods, existing research has proposed an optimization method based on Laplacian regularization. This method introduces a regularization term into the optimization objective function to constrain the width and height differences between adjacent rows or columns, thereby improving the smoothness and coherence of the retargeted image to some extent. Figure 4 As shown in (c1).

[0053] However, while Laplacian regularization improves image smoothness to some extent, it fails to completely resolve the problem of image structural distortion. For example... Figure 4 (b1) Figure 4 As shown in (b2), without a regularization energy term, obvious tortuosity and structural distortion appear between adjacent regions in the image; when a regularization energy term is added, although the structural distortion in some images is alleviated, it also leads to unnatural deformation and compression in the image edge regions, such as... Figure 4 As shown in (c2), it severely damages the visual realism and structural integrity of the image.

[0054] Furthermore, the Laplacian regularization energy function carries the risk of overcorrection during optimization: images that originally exhibit natural visual characteristics may become distorted after regularization, such as... Figure 4 As shown in (c2). This phenomenon indicates that there is a significant trade-off between smoothness constraints and geometry preservation in Laplace regularization.

[0055] Therefore, although Laplacian regularization provides an effective approach to solving structural distortion problems, it still faces a significant challenge in practical applications: how to minimize structural distortion while maximally suppressing deformation in image edge regions, thereby achieving a better balance between image smoothness and structural integrity. Solving this challenge is crucial for improving the applicability and visual effects of axis alignment methods in stereo image retargeting.

[0056] The applicant's research found that the retargeted image exhibited significant compression distortion after the addition of the regularization energy function. The fundamental reason for this is that the axis alignment method tends to compress non-salient regions (especially the image edges with low salientity) to minimize interference with salient objects.

[0057] However, because the Laplacian regularization function imposes constraints on the width and height differences between adjacent rows or columns, the edge regions, although compressed, still retain a certain width, resulting in uneven compression deformation of the image, which seriously affects the visual appearance and structural consistency of the image.

[0058] To address this issue, this invention proposes an improved strategy: when retargeting a stereo image, non-salient edge regions that need to be deleted are completely cropped to avoid retaining parts of these regions due to Laplacian regularization, thereby effectively reducing compression deformation of edge regions. Simultaneously, Laplacian regularization is applied to constrain non-salient regions between salient objects, limiting the width and height differences between adjacent rows or columns to reduce structural distortion and ensure smooth scaling transitions. Through this region-based optimization, this invention can minimize structural distortion while maximally suppressing edge deformation, achieving a better balance between image smoothness, structural integrity, and visual appeal.

[0059] To achieve the above strategy, this invention proposes an optimization strategy combining a cropping operator and Laplacian regularization. The core idea of ​​this strategy is to crop the stereo image before performing axis alignment, precisely removing unwanted, non-salient edge regions. Through this preprocessing step, when the axis alignment method is applied later, Laplacian regularization will no longer apply to the cropped edge regions, effectively avoiding the compression and deformation problem of edge regions. Furthermore, if the cropping operator can independently complete the retargeting task without damaging salient content, there is no need to further use the axis alignment method, thus achieving high-quality stereo retargeted images with low complexity.

[0060] Specifically, the advantages of using the cropping operator are reflected in the following two aspects. First, it avoids edge deformation in stereoscopic images: by pre-deleting non-salient edge regions through the cropping operator, the constraint of Laplacian regularization on edge regions is eliminated, fundamentally solving the edge compression deformation problem and improving the visual naturalness of the image. Second, it balances efficiency and high quality: the cropping operator can independently complete part of the redirection task without damaging salient content, reducing the computational overhead of axis alignment methods. While ensuring high-quality redirection results, it further reduces the complexity of the method, meeting the design goal of efficiency.

[0061] By organically combining the cropping operator and the axis alignment method, the method of this invention achieves an effective balance between reducing structural distortion, suppressing edge deformation, and reducing computational complexity, providing an efficient and high-quality solution for stereoscopic image retargeting.

[0062] Based on this, the feature selection optimization and clipping window calculation of the embodiments of the present invention will be described below: One of the main time-consuming aspects of existing content-aware stereo image retargeting methods lies in the image feature extraction process. To reduce the overall runtime of the method, this invention employs a selective approach, choosing only the most important image features to efficiently complete the retargeting task. The following section details the processing and optimization of feature images in the stereo image retargeting task, and provides a specific example of a method for calculating the cropping window size.

[0063] First, the process of extracting key features is explained. In step 10, in one embodiment, the key features of the stereo image mainly include salient features and disparity features. The saliency map is used to identify the most visually attractive regions in the image, preventing them from being cropped or excessively distorted during retargeting. In this embodiment, PiCANet is used to detect the spatial saliency of the left and right views, generating left-view saliency maps respectively. and right-angle salient image The disparity map represents the displacement difference between corresponding pixels in the left and right views of a stereoscopic image, directly reflecting the depth information of the scene. The specific method for generating the disparity map is determined by those skilled in the art based on the specific application scenario. In one embodiment, the disparity map is generated using optical flow. .

[0064] Then, in generating the global saliency map, in one embodiment, to more comprehensively consider the spatial distribution and depth information of salient objects when calculating the clipping window, this embodiment of the invention employs a saliency map generation method based on multi-feature fusion. This is achieved by fusing the left-view saliency map. Right-angle salient image and parallax map Finally, a global saliency map is generated. : ; Global saliency map By integrating saliency and depth information from both left and right views, the method can more comprehensively reflect salient objects in the image and their spatial distribution. Based on the global saliency map, the method of this invention can more accurately calculate the cropping window width, ensuring that salient objects are fully protected during the cropping process.

[0065] This invention utilizes binary saliency maps to guide the determination of salient protection regions, providing high-quality feature support for subsequent redirection tasks. In one embodiment, when generating the binary saliency map, it is set... They represent Pixel values ​​of the saliency map at the left and right viewpoints. Then they represent the binarized values ​​respectively. Value. If the gray level in the saliency map is The binarization process can then be described as follows: ; in, This is a threshold parameter used to determine whether a pixel is significant, and it can take any value between [0,1]. To better conform to the perceptual characteristics of the human eye, in one embodiment, it can be... Set it to 0.3.

[0066] The process of determining the size of the cropping window is described below. In one embodiment, such as... Figure 5 As shown, in step 20, calculating the boundary values ​​of the protected area based on the left binary saliency map and the right binary saliency map includes: Step 201a: Extract the minimum and maximum column values ​​of all salient pixels in the left binary saliency map; extract the minimum and maximum column values ​​of all salient pixels in the right binary saliency map.

[0067] Step 202a: Determine the minimum value among all the minimum column values ​​as the left boundary value, and determine the maximum value among all the maximum column values ​​as the right boundary value, in order to obtain the boundary values.

[0068] In one embodiment, the boundary value of the significantly protected area is defined as the left boundary value. Right boundary value .in, The minimum column value among all salient pixels in the left binary saliency map (BL). This represents the maximum column value among all salient pixels in the left binary saliency map (BL). This represents the smallest column value among all salient pixels in the right binary saliency map BR. It represents the maximum column value among all salient pixels in the right binary saliency map BR.

[0069] In one embodiment, such as Figure 6 As shown, in step 20, determining the target width of the cropping window based on the boundary value and the target scaling ratio includes: Step 201b: Determine the difference between the column coordinates corresponding to the right boundary value and the column coordinates corresponding to the left boundary value as the intermediate value; determine the sum of the reference value and the mitigation coefficient as the coefficient value; determine the product of the intermediate value and the coefficient value as the optimal width.

[0070] Based on this, the optimal width for cropping the window is determined. It can be calculated using the following formula: ; Among them, the mitigation coefficient To protect the background area immediately surrounding prominent content and prevent loss of contextual information related to the prominent object due to excessive cropping, in one embodiment, a method can be set... The aforementioned left boundary value and right boundary value For numerical values, The column coordinates corresponding to the right boundary value are used to locate the specific position of the right boundary of the salient region in the horizontal direction of the image. The column coordinates corresponding to the left boundary value are used to locate the specific position of the left boundary of the salient region in the horizontal direction of the image.

[0071] Step 202b: Determine the planned width as the product of the specified scaling ratio and the original image width; if the optimal width is less than the planned width, then determine the planned width as the target width; otherwise, determine the optimal width as the target width.

[0072] Despite optimal width It can be used as the target width of the final cropping window. The candidate values, but in determining The following two abnormal situations still need to be avoided: First, if the width of the cropping window exceeds the width of the original image, the cropping operation cannot be performed and the method fails; second, if the width of the cropping window is smaller than the target redirection size, the redirection effect will be affected.

[0073] In one embodiment, when the optimal width is greater than or equal to the original image width, the original stereoscopic image is not cropped, but the original stereoscopic image is directly axially aligned and scaled so that the axially aligned image can be determined as the retargeting image.

[0074] Based on the above constraints, the target width of the cropping window is determined. The value is determined by Specify scaling ratio and original image width As determined together, as shown in the following formula: .

[0075] The energy function construction for the clipping method and the axis alignment method is explained below: To protect the depth perception of stereo repositioned images, existing stereo repositioning methods typically aim to maintain parallax invariance. When extending axis alignment methods to stereo images, this invention also follows this principle: by constructing an axis-aligned parallax energy term, the horizontal parallax before and after stereo repositioning is kept as consistent as possible; and the height changes of the corresponding horizontal axes in the left and right views are rigidly constrained to be consistent, avoiding alteration of the original vertical parallax, thereby further maintaining the depth perception and visual consistency of the stereo image.

[0076] In one embodiment, the total energy function of the pruning operator includes a distance energy term and a salient energy term; such as Figure 7 As shown, in step 20, the cropping operator is subjected to dual optimization constraints based on the boundary value, the global saliency map, and the target width to crop the non-salient regions of the edge portion of the original stereoscopic image, resulting in a cropped image, including: Step 201c: Based on the target width, determine all pixels outside the cropping window in the original stereo image; determine the first accumulated value of the global saliency of all pixels outside the cropping window, and determine the second accumulated value of the global saliency of all pixels in the original stereo image.

[0077] Step 202c: Determine the ratio of the first accumulated value to the second accumulated value as the significant energy term.

[0078] To accurately protect salient content, embodiments of the present invention utilize a global saliency map. Construct significant energy terms This guides the selection of the cropping window. Significant energy terms. Defined as the ratio of the sum of importance of pixels outside the cropping window to the sum of importance of pixels throughout the entire image space, a saliency energy term is calculated using the following formula: ; in, Indicates the location The global saliency value of the pixel at that location. and These represent the set of pixels within the cropping window and the set of all pixels in the entire space of the original stereoscopic image, respectively.

[0079] Step 203c: The sum of the salient energy term and the weighting parameter is determined as the weighted value; the sum of the weighted value and the distance energy term is minimized to obtain the cropped image; wherein, the distance energy term is used to constrain the cropping range from intruding into the depth key region around the salient object.

[0080] This invention embodiment utilizes a distance energy term based on a pruning operator. Optimize stereo perception preservation. The distance energy term is determined by those skilled in the art based on the specific application scenario and with reference to existing technologies, and is not limited here. By constraining the cropping range on the background side of the salient region, it is ensured that the depth-critical region is not excessively deleted, maintaining the continuity of depth information around the salient content, thereby effectively preserving stereo perception during size transformation.

[0081] Total energy function of the pruning operator It consists of two parts: a significant energy term. and distance energy term It can be calculated using the following formula: ; in, These are weighting parameters used to control significant energy terms. The weights in the total energy function. In one embodiment, as verified by the applicant's experiments, it can be... Set it to 10000.

[0082] To obtain the optimal clipping window, this embodiment of the invention uses the least squares method to solve for the total energy function of the clipping operator. The minimum value is obtained to obtain the optimal clipping window.

[0083] The total energy function of the axis alignment operator will now be explained: In one embodiment, such as Figure 8 As shown, in step 30, the process of performing axis-aligned scaling on the cropped image to obtain an axis-aligned image includes: Step 301: Overlay a grid of N columns and M rows onto the cropped image, and initialize the column width and row height variables of each grid cell.

[0084] Let the width and height of the cropped 3D image be respectively... and The expected width and expected height of the retargeting image are respectively and To achieve precise deformation control of the image, this embodiment of the invention overlays a grid onto the cropped image. The grid consists of N columns and M rows, with each column having a width of [missing information]. The height of each row is To balance computational efficiency and deformation accuracy, this embodiment of the invention sets the size of each mesh cell to 30×30 pixels.

[0085] set up Indicates the height of each line. Indicating the width of each column, the deformation of the 3D image axis alignment can be determined by... It means that, among them, and Corresponding to the row height and column width in the left image, and This corresponds to the row height and column width in the right figure.

[0086] In one embodiment, in the axis alignment method, the image is divided into several grid cells. Therefore, it is necessary to calculate the grid importance of the cropped image to ensure that salient objects are effectively protected during deformation. Assuming the grid consists of N columns and M rows, the formula for calculating the left-view grid saliency map is as follows: ; in, This is the cropped left-side salient image. Indicates the first The region of a grid cell, right-view grid saliency map The calculation is similar.

[0087] In one embodiment, the left and right views of the cropped image are axially aligned and scaled according to the optimal width and optimal height to generate a redirected stereoscopic image with the target image size; wherein, during scaling, the grid width of the same column changes synchronously, and the grid height of the same row changes synchronously.

[0088] Step 302: Constrain the aspect ratio change between adjacent grid cells using the similarity energy term; constrain each grid cell to maintain similar transformation using the rigidity energy term; penalize the size difference between adjacent rows or columns using the regularization energy term; constrain the column width change of corresponding columns at the same horizontal index in the left and right views of the cropped image to be equal using the disparity energy term.

[0089] During axis alignment deformation, to ensure that the deformation of each grid cell (especially in visually important regions) is similar to the deformation of its surrounding grid cells, thereby making the retargeted image smoother and more natural, this embodiment of the invention introduces a similarity energy term. This energy term ensures the continuity of the deformation process within local regions by constraining the aspect ratio variation between adjacent mesh cells, thereby effectively reducing visual structural distortion.

[0090] In one embodiment, the similarity energy term is expressed as follows: ; To transform the similarity energy term into a general optimization problem and ultimately achieve a fast solution, a matrix is ​​defined. : ; in, ,and It can be seen that... The similarity energy term represents the vector in each row of this energy term. .make , Then the similarity energy term can be expressed as a standard quadratic programming form. Obviously, It is a positive semi-definite matrix, therefore It is a convex function, which can be solved quickly using an efficient convex optimization solver.

[0091] To achieve smooth geometric deformation while preserving local details, embodiments of the present invention introduce a rigidity energy term. In one embodiment, it is as follows: ; To solve quickly, a matrix is ​​defined. : ; ; in, ,and To re-express the rigid energy term using matrix notation, embodiments of the present invention define a vector. ,but It can be expressed by the following formula: ; To solve quickly, it is transformed into a QP expression: ; ; matrix The form clearly indicates that it is positive semidefinite, therefore The energy term is also expressed as a convex function that can be solved quickly.

[0092] To avoid severe structural distortion, this embodiment of the invention employs Laplacian regularization to distribute deformation more evenly across the image, resulting in a smoother and more natural retargeting result. Laplacian regularization ensures consistency in the deformation process within local regions by penalizing significant differences in size between adjacent rows or columns. (Regularization energy term) The definition is as follows: ; To incorporate the regularization energy term into the quadratic programming expression, it is only necessary to simply add the term... Added to energy, in which It is related to the regularization energy term The corresponding Laplace matrix.

[0093] A specific example of the parallax energy term will be provided below, and will not be elaborated upon here.

[0094] Step 303: Determine the product of the disparity energy term and the weight parameter as the weighting term; determine the sum of the similarity energy term, the rigidity energy term, the regularization energy term, and the weighting term as the total energy function.

[0095] Total energy function of axis alignment operator It consists of four parts: similarity energy term Rigid energy term Regularized energy term Parallax energy term express: ; In one embodiment, the applicant has experimentally verified that the weighting parameters can be... Set it to 0.0005.

[0096] Step 304: Using the column width variable and the row height variable, take the total energy function as the objective function of the optimization problem, solve for the optimal width of each column and the optimal height of each row in the required axis-aligned image, and then perform axis-aligned scaling on the cropped image according to the optimal width and the optimal height to obtain the axis-aligned image.

[0097] The specific method for scaling the cropped image according to the optimal width and optimal height is determined by those skilled in the art based on the specific application scenario.

[0098] In one embodiment, such as Figure 9 As shown, in step 302, the constraint by which the column width changes of corresponding columns at the same horizontal index in the left and right views of the cropped image are equal through the disparity energy term is as follows: Step 3021: For each pair of left and right views with the same horizontal index, calculate the square of the difference between the column width change in the left view and the column width change in the right view to obtain the disparity change.

[0099] Step 3022: The sum of the disparity changes of all corresponding columns is determined as the disparity energy term.

[0100] In one embodiment, the total energy function also includes a hard constraint: the height variables of the corresponding rows at the same horizontal index in the left and right views are equal to keep the vertical parallax zero before and after redirection.

[0101] Parallax is the horizontal displacement of corresponding pixels in the left and right views of a stereo image, directly reflecting the image's depth information. Maintaining parallax consistency is crucial for preserving depth perception and visual realism in stereo images. Therefore, when extending axis alignment methods from two-dimensional to three-dimensional, the most critical task is to ensure that the parallax of the stereo image remains as constant as possible before and after repositioning.

[0102] Parallax is divided into horizontal parallax and vertical parallax. Horizontal parallax is the difference in horizontal displacement of corresponding pixels in the left and right views of a stereoscopic image, reflecting the depth information of a scene point in the horizontal direction. Vertical parallax is the difference in vertical displacement of corresponding pixels in the left and right views, usually caused by factors such as camera misalignment or lens distortion. It is worth noting that during the process of capturing stereoscopic images, strict camera calibration and image correction are usually used to avoid the occurrence of vertical parallax; therefore, the vertical parallax of the original stereoscopic image can be considered to be 0. Specifically, for a scene point in a stereoscopic image, let its pixel coordinates in the left view be... The pixel coordinates in the right view are Then horizontal parallax The change in parallax. It can be defined as follows: ; in, , and These represent the horizontal position (column coordinates) of the scene point in the redirected left view, the horizontal position in the right view, and their horizontal disparity values, respectively. Therefore, changes in disparity can be converted into changes in the horizontal position of a pixel. Since column coordinates are equal to the sum of column widths, by constraining the changes in column widths of corresponding columns in the left and right views to be the same, the disparity before and after redirection can be kept as constant as possible. Based on the above analysis, this embodiment of the invention introduces a disparity energy term, as follows: ; Similarly, you only need to simply put the item Join In this way, a fast solution can be achieved, among which It is related to the parallax energy term The corresponding positive semidefinite matrix.

[0103] To improve efficiency, the method in this embodiment of the invention converts the axis-aligned total energy function into a standard quadratic programming form and performs minimization using a convex function solver. Specifically, in one embodiment, as... Figure 10 As shown, step 304 includes: Step 3041: Rewrite each term in the total energy function as a quadratic form in terms of the column width variable and the row height variable, and combine them into a standard quadratic programming expression in the form of a semi-positive definite matrix.

[0104] To achieve a fast solution, the total energy function of the axis alignment operator is transformed into a standard quadratic programming form and minimized using the efficient CVX solver. ; Step 3042: Apply constraints to the quadratic programming standard formula. The constraints include: each column width variable is non-negative, each row height variable is non-negative, the sum of all row height variables is equal to the target image height, the sum of all column width variables is equal to the target image width, and the corresponding row height variables in the left and right views are equal.

[0105] To ensure that the solved mesh results meet the requirements of the retargeting size and to avoid altering the vertical parallax of the original image, this embodiment of the invention imposes constraints on the solution process. Specific constraints are as follows: first, a non-negative row height constraint ensures that the height of each row is not less than 0; second, the row height is subject to left and right parallax constraints. Figure 1 The system employs five constraints: consistency constraint to ensure that the height of corresponding rows in the left and right views is the same to avoid changes in vertical parallax; total height constraint to ensure that the sum of the heights of all rows equals the target height; non-negative column width constraint to ensure that the width of each column is not less than 0; and total width constraint to ensure that the sum of the widths of all columns equals the target width.

[0106] ; Step 3043: Under the constraints, the quadratic programming standard form is minimized using a convex optimization solver to obtain the optimal width and optimal height.

[0107] To achieve a fast solution, in one embodiment, the total energy function of the axis alignment method can be converted into a standard quadratic programming form and minimized using an efficient CVX solver.

[0108] This invention proposes a low-complexity multi-operator stereo image retargeting method, utilizing low-complexity cropping and axis alignment operators to achieve retargeting. The axis alignment energy function is modeled as a quadratic programming problem and solved quickly using an efficient convex optimization solver, effectively avoiding the complex dynamic programming problems of traditional methods and achieving high-quality stereo image retargeting under low complexity conditions. This invention also proposes a novel optimization strategy combining the cropping operator with Laplacian regularization, significantly reducing structural distortion in traditional planar axis alignment methods and minimizing global image deformation. Furthermore, a new disparity energy term is proposed and integrated into the energy optimization framework of the axis alignment method, extending the axis alignment method from a two-dimensional plane to three-dimensional space.

[0109] Compared with other content-based retargeting methods, axis alignment not only avoids the problem of cropping salient objects that may occur with clipping methods, but also overcomes the high time consumption of mesh deformation and slit clipping methods. Experimental results show that, compared with existing methods, the method of this invention performs excellently in terms of visual quality, depth perception, and computational efficiency, providing an efficient and high-quality solution for stereo image retargeting with broad practical application potential.

[0110] By employing an efficient feature selection strategy, the method of this invention retains only key features such as disparity and saliency, avoiding the computation of redundant information such as edge features. This results in feature extraction efficiency that is significantly better than or at least comparable to existing content-based stereo image retargeting methods.

[0111] Through experiments, the applicant has verified that the method of this invention significantly outperforms existing methods in terms of computational resource consumption while ensuring image retargeting quality. Its average processing time is approximately 1.5 seconds, with the dynamic programming module's average processing time being only 13% of the most efficient existing method. It also demonstrates excellent performance in both subjective quality evaluation and user survey results. In objective quality evaluation, the method performs well in aspect ratio similarity and viewpoint transformation grid deformation indices, with scores approaching the level of the optimal method.

[0112] The foregoing embodiments provided a stereoscopic image redirection method for resource-constrained terminals. In this embodiment, another stereoscopic image redirection device for resource-constrained terminals will be proposed. The stereoscopic image redirection device for resource-constrained terminals includes: a processor and a memory for storing processor-executable instructions; wherein the processor is configured to execute the stereoscopic image redirection method for resource-constrained terminals described in the foregoing embodiments.

[0113] like Figure 11 As shown, the stereoscopic image repositioning device for a resource-constrained terminal includes a processor 21 and a memory 22, wherein the processor 21 and the memory 22 can be connected by a bus or other means.

[0114] Processor 21 can be a CPU. Processor 21 can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or combinations of the above types of chips.

[0115] The memory 22, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the stereoscopic image retargeting method for resource-constrained terminals in the aforementioned embodiments. The processor executes various functional applications and training processes by running the non-transitory software programs, instructions, and modules stored in the memory.

[0116] The memory 22 may include a program storage area and a training storage area. The program storage area may store the operating system and applications required for at least one function; the training storage area may store training data created by the processor. Furthermore, the memory may include high-speed random access memory and non-transitory memory, such as at least one disk storage device, flash memory, or other non-transitory solid-state storage device. In some embodiments, the memory 22 may optionally include memory remotely located relative to the processor, which can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. The one or more modules stored in the memory 22, when executed by the processor 21, perform the stereoscopic image redirection method for resource-constrained terminals as shown in the embodiments of the present invention. Specific details of the stereoscopic image redirection method for resource-constrained terminals can be understood by referring to the corresponding descriptions and effects in the embodiments of the present invention, and will not be repeated here.

[0117] This embodiment also provides a computer storage medium storing a computer program that can be executed by a processor to perform the stereoscopic image retargeting method for resource-constrained terminals described in the foregoing embodiments.

[0118] The computer storage medium stores computer-executable instructions, which can execute the stereoscopic image redirection method for resource-constrained terminals in any of the above method embodiments. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk drive (HDD), or solid-state drive (SSD), etc.; the storage medium may also include combinations of the above types of memory.

[0119] The specific steps of the stereoscopic image retargeting method for resource-constrained terminals are described in the foregoing embodiments and will not be repeated in this embodiment.

[0120] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for stereoscopic image retargeting in resource-constrained terminals, characterized in that, include: Stereo image features are extracted from the original stereo image to obtain a disparity map, a left-view saliency map, and a right-view saliency map, which are then fused to obtain a global saliency map. The left-view saliency map is binarized to obtain a left binary saliency map, and the right-view saliency map is binarized to obtain a right binary saliency map. Calculate the boundary value of the protected area based on the left and right binary saliency maps; determine the target width of the clipping window based on the boundary value and the target scaling ratio; The cropping operator is subjected to dual optimization constraints based on the boundary value, the global saliency map, and the target width to crop the non-salient regions of the edge portion of the original stereo image, thereby obtaining the cropped image. If the cropped image meets the target size requirement, then the cropped image is determined as the retargeted image; If the cropped image does not meet the target size requirement, then the cropped image is axially aligned and scaled to obtain an axially aligned image; the axially aligned image is then identified as the retargeting image.

2. The stereoscopic image redirection method for resource-constrained terminals according to claim 1, characterized in that, The calculation of the boundary value of the protected area based on the left binary saliency map and the right binary saliency map includes: Extract the minimum and maximum column values ​​of all salient pixels in the left binary saliency map; extract the minimum and maximum column values ​​of all salient pixels in the right binary saliency map; The minimum value among all the minimum column values ​​is determined as the left boundary value, and the maximum value among all the maximum column values ​​is determined as the right boundary value, thus obtaining the boundary values.

3. The stereoscopic image redirection method for resource-constrained terminals according to claim 2, characterized in that, Determining the target width of the cropping window based on the boundary value and the target scaling ratio includes: The difference between the column coordinates corresponding to the right boundary value and the column coordinates corresponding to the left boundary value is determined as the intermediate value; the sum of the reference value and the mitigation coefficient is determined as the coefficient value; the product of the intermediate value and the coefficient value is determined as the optimal width. The planned width is determined by the product of the specified scaling ratio and the original image width; if the optimal width is less than the planned width, the planned width is determined as the target width; otherwise, the optimal width is determined as the target width.

4. The stereoscopic image redirection method for resource-constrained terminals according to claim 3, characterized in that, When the optimal width is greater than or equal to the original image width, the original stereoscopic image is not cropped, but directly axially aligned and scaled to make the axially aligned image the retargeting image.

5. The stereoscopic image redirection method for resource-constrained terminals according to claim 1, characterized in that, The cropping operator is subjected to dual optimization constraints based on the boundary value, the global saliency map, and the target width to crop the non-salient regions of the edge portion of the original stereo image, resulting in a cropped image including: Based on the target width, determine all pixels outside the cropping window in the original stereo image; determine a first accumulated value of the global saliency of all pixels outside the cropping window, and determine a second accumulated value of the global saliency of all pixels in the original stereo image; The ratio of the first accumulated value to the second accumulated value is determined as the significant energy term; The sum of the salient energy term and the weighting parameter is determined as the weighted value; the sum of the weighted value and the distance energy term is minimized to obtain the cropped image; wherein, the distance energy term is used to constrain the cropping range from intruding into the depth key region around the salient object.

6. The stereoscopic image redirection method for resource-constrained terminals according to claim 1, characterized in that, The process of scaling the cropped image to obtain an axis-aligned image includes: Overlay a grid of N columns and M rows onto the cropped image, and initialize the column width and row height variables of each grid cell; The similarity energy term constrains the aspect ratio variation between adjacent grid cells; the rigidity energy term constrains each grid cell to maintain similar transformation; the regularization energy term penalizes the size difference between adjacent rows or columns; and the disparity energy term constrains the column width variation of corresponding columns at the same horizontal index in the left and right views of the cropped image to be equal. The product of the disparity energy term and the weight parameter is determined as the weighting term; the sum of the similarity energy term, the rigidity energy term, the regularization energy term, and the weighting term is determined as the total energy function. Using the column width variable and the row height variable, the total energy function is used as the objective function of the optimization problem to solve for the optimal width of each column and the optimal height of each row in the required axis-aligned image. The cropped image is then scaled according to the optimal width and the optimal height to obtain the axis-aligned image.

7. The stereoscopic image redirection method for resource-constrained terminals according to claim 6, characterized in that, The constraint, which uses the disparity energy term, ensures that the column width changes of corresponding columns at the same horizontal index in the left and right views of the cropped image are equal: For each pair of left and right views with the same horizontal index, calculate the square of the difference between the column width change in the left view and the column width change in the right view to obtain the disparity change. The sum of the disparity changes in all corresponding columns is used to determine the disparity energy term.

8. The stereoscopic image redirection method for resource-constrained terminals according to claim 6, characterized in that, The process of using the column width variable and the row height variable, and taking the total energy function as the objective function of the optimization problem, to solve for the optimal width of each column and the optimal height of each row in the required axis-aligned image includes: Each term in the total energy function is rewritten as a quadratic form in terms of the column width and row height variables, and then combined into a standard quadratic programming expression in the form of a positive semidefinite matrix. Constraints are imposed on the standard quadratic programming expression, including: each column width variable is non-negative, each row height variable is non-negative, the sum of all row height variables is equal to the target image height, the sum of all column width variables is equal to the target image width, and the corresponding row height variables in the left and right views are equal. Under the constraints, the quadratic programming standard form is minimized using a convex optimization solver to obtain the optimal width and optimal height.

9. A stereoscopic image redirection device for a resource-constrained terminal, characterized in that, The stereoscopic image retargeting device for a resource-constrained terminal includes: a processor and a memory for storing processor-executable instructions; The processor is configured to execute the stereoscopic image retargeting method for a resource-constrained terminal as described in any one of claims 1 to 8.

10. A non-volatile computer storage medium, characterized in that, The computer storage medium stores computer-executable instructions, which are executed by one or more processors to perform the stereoscopic image retargeting method for a resource-constrained terminal as described in any one of claims 1 to 8.

Images