Multi-aspect-ratio image redirection method and system based on key semantic reservation

By introducing semantic gap engraving algorithm and stable diffusion model in image redirection technology, combined with ControlNet and IP-Adapter, the problems of semantic deletion and pixel discontinuity in image redirection are solved, and high-quality multiple aspect ratio image heaviness is achieved Direction, ensuring semantic retention and visual coherence of generated results.

CN118941469AActive Publication Date: 2024-11-12SOUTHEAST UNIV

Patent Information

Application Number
CN202410935101.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-12
Publication Date
2024-11-12
Estimated Expiration
2044-07-12

AI Technical Summary

Technical Problem

The existing image redirection technology has problems such as semantic deletion, pixel discontinuity and inconsistent generation results with the original image, especially when processing complex images, it is difficult to maintain the semantic information and visual coherence of the image.

Method used

The multiple aspect ratio image redirection method based on key semantic retention is adopted, including the preliminary redirection module, the redraw area selection module and the image guidance generation module. The semantic gap engraving algorithm is combined with the significance graph and the gradient operator for preliminary redirection. Repaint areas with large pixel shifts and use ControlNet and IP-Adapter-assisted stable diffusion models to perform local redrawing to ensure that the generated results meet the expectations of human aesthetics.

Benefits of technology

It significantly improves the semantic retention and visual effects of image redirection, ensures that the generation results are consistent with the original image, and is suitable for various aspect ratio requirements, reducing the problems of pixel discontinuity and loss of important objects.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118941469A_ABST
    Figure CN118941469A_ABST
Patent Text Reader

Abstract

The invention discloses a multi-aspect-ratio image redirection method and system based on key semantic reservation. The system at least comprises a preliminary redirection module, a redrawing area selection module and an image guide generation module. The preliminary redirection module is based on a semantic gap engraving algorithm, introduces a saliency map with space prior to be combined with a gradient operator, and deletes a linear region with low energy to obtain a preliminary redirection result; the redrawing area selection module takes an output result of the preliminary redirection module as input, regenerates an area with large pixel shift in the preliminary redirection image, and performs image expansion on a result that semantic gap carving does not reach a target aspect ratio; and the picture guide generation module guides local redrawing by using the original picture as a control condition based on a result output by the redrawing area selection module, and outputs an image. According to the method, the problem that a traditional redirection method lacks semantics is solved, abrupt pixels are regenerated in a self-adaptive mode, and an expected aspect ratio picture which retains key semantics and conforms to the human aesthetic feeling is output.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the technical field of computer vision and AIGC, and mainly relates to a multi-aspect ratio image redirection method and system based on key semantics retention. Background Art

[0002] Image redirection technology is widely used in various digital media and display devices, including mobile phones, tablets, computer monitors and smart TVs. As the display screen ratios of these devices diversify, how to display images of uniform quality on different devices has become an urgent problem to be solved.

[0003] Traditional image retargeting methods, such as scaling, cropping, and seam carving, can meet the needs to a certain extent, but they all have the problem of semantic loss. Although scaling technology can maintain the integrity of the image, it will cause the image content to be out of proportion and severely deformed. Cropping technology can maintain the local clarity of the image, but it will lose important image information. The more complex seam carving technology achieves retargeting by removing low-energy paths in the image, which can maintain the important content of the image to a certain extent. However, due to the lack of understanding of the deep semantic information of the image, seam carving will lead to the loss of important objects and image discontinuity when processing complex images. With the development of deep learning, the generation method based on the adversarial generative network (GAN) has alleviated the problems of semantics and pixel discontinuity to a certain extent. However, the instability of the generative adversarial network in training and the inconsistency of the generated results also limit its application in practical scenarios.

[0004] The Stable Diffusion Model is a popular image generation technology that generates high-quality image content by gradually denoising the image noise. This method can provide highly consistent and delicate image details during the image generation and redrawing process, ensuring the visual coherence and naturalness of the final result. However, in practical applications, the Stable Diffusion Model often has difficulty in achieving effective control of complex scenes by using text as a control condition. Summary of the invention

[0005] The present invention is aimed at the problems of semantic loss, pixel discontinuity and inconsistency between generated results and original images in existing image redirection technologies, and provides a multi-aspect ratio image redirection method and system based on key semantics preservation, which at least includes a preliminary redirection module, a redrawing area selection module and a picture guide generation module; the preliminary redirection module is based on a semantic gap carving algorithm, and removes the linear areas with low energy by introducing a saliency map with spatial prior and combining it with a gradient operator to obtain a preliminary redirection result; the redrawing area selection module uses the output result of the preliminary redirection module as input, regenerates the area with large pixel shift in the preliminary redirection map, and expands the result of semantic gap carving that does not reach the target aspect ratio; the picture guide generation module: based on the output result of the redrawing area selection module, uses the original image as a control condition to guide local redrawing and output an image. The present invention solves the problem of missing semantics in traditional redirection methods, adaptively regenerates abrupt pixels, and outputs an image with the desired aspect ratio that retains key semantics and conforms to human aesthetics.

[0006] In order to achieve the above-mentioned object, the technical solution adopted by the present invention is: a multi-aspect ratio image redirection system based on key semantics retention at least includes a preliminary redirection module, a redrawing area selection module and a picture guide generation module;

[0007] The preliminary redirection module: based on the semantic gap carving algorithm, by introducing a saliency map with spatial priors and combining it with a gradient operator, deletes the linear areas with low energy to obtain a preliminary redirection result;

[0008] The redrawing area selection module takes the output result of the preliminary redirection module as input, regenerates the area with large pixel shift in the preliminary redirection map, and expands the result of semantic gap carving that does not reach the target aspect ratio;

[0009] The image guide generation module: based on the result output by the redrawing area selection module, uses the original image as a control condition to guide the local redrawing and output the image.

[0010] As an improvement of the present invention, the picture-guided generation module at least includes ControlNet and IP-Adapter auxiliary stable diffusion model, wherein ControlNet is used to control the redrawing area, and IP-Adapter is used to control the generated content.

[0011] In order to achieve the above object, the present invention also adopts a technical solution: a multi-aspect ratio image redirection method based on key semantics preservation, which specifically includes the following steps:

[0012] S1, determine the target image: input RGB image and set target aspect ratio;

[0013] S2, preliminary redirection: Based on the semantic gap carving algorithm, the saliency detection algorithm is used to obtain the saliency map, which is combined with the gradient operator after attaching the spatial prior to calculate the energy of each area of ​​the image, and starts to crop from the lines with low energy. After deleting the linear areas with low energy, the preliminary redirection result O is obtained. i ;

[0014] S3, find out the unnatural pixel connection area: use the sliding window to find out the initial redirection result of step S2 i The number of pixels deleted before and after each pixel in the image is determined, and the area with large pixel shift is found, and the result of semantic gap carving that does not reach the target aspect ratio is expanded; the pixel shift is large, and the number of pixels deleted around it is large, and the retained pixels are obtained through the deletion-retention binary image of the semantic analysis cutting algorithm;

[0015] S4, redrawing of unnatural regions: regenerating the unnatural regions obtained in step S3 using an image-guided stable diffusion model, and then pasting the unmasked regions back onto the generated regions;

[0016] S5, image output: according to the target aspect ratio, output the corresponding redirection result to obtain the target aspect ratio image.

[0017] As an improvement of the present invention, the energy calculation formula in step S2 is:

[0018]

[0019] Where I(x,y) represents the pixel value of image I at position (x,y), S(x,y) represents the saliency of its saliency map at position (x,y), x_0 represents the abscissa of the centroid of the saliency region, and W is the width of the image;

[0020] After obtaining the energy of each pixel in the image, use A vertical line describing a set of pixels in the image, for any I, has |x(i)-x(i-1)|≤1; the energy of a vertical line is defined as To be deleted * The line is obtained by minimizing the energy:

[0021]

[0022] Using the dynamic programming algorithm, several lines with the least energy are obtained for deletion.

[0023] As another improvement of the present invention, in step S2, when one third of the salient area is cropped, the semantic seam cutting algorithm automatically stops cropping the image.

[0024] As another improvement of the present invention, in step S3, the number of pixels deleted before and after each pixel exceeds a threshold and is determined to be a region with a large pixel shift. The shift region mask M is a one-dimensional sliding window with a length of l that counts the number of deleted pixels in the neighborhood of each pixel, and a one-dimensional convolution is performed on the deletion-retention binary map S:

[0025]

[0026] Among them, conv1d represents one-dimensional convolution, K is a convolution kernel with a length of l and a value of 1, and the area in M ​​that exceeds the threshold η is the area with large pixel shift.

[0027] As another improvement of the present invention, in step S3, the size of the convolution kernel K is 25, the value of each position is 1, and the threshold η is 12.

[0028] As another improvement of the present invention, in step S4, the mask is covered on the preliminary redirection result and then input into the Controlnet model; the original image is input into the IP-Adapter model as a control condition, and the two together assist the Stablediffusion stable diffusion model to generate an image.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] (1) The present invention proposes a multi-aspect ratio image redirection method based on key semantics preservation, which significantly improves the semantic preservation and visual effects of image redirection under various aspect ratio requirements.

[0031] (2) The semantic gap cutting algorithm proposed in the present invention further explores the energy difference within the foreground based on the energy distinction between the significant foreground and background, and distinguishes the energy within the foreground through spatial position priors, making it more consistent with human visual perception.

[0032] (3) The present invention proposes a redrawing area selection module, which can efficiently find out the areas with inconsistent pixels in the results in the image redirection method based on pixel shifting, thus providing conditions for post-processing of redirection. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 is a flowchart of the steps of the multi-aspect ratio image redirection method based on key semantics preservation of the present invention;

[0034] Figure 2 is a working principle diagram of the multi-aspect ratio image redirection system based on key semantics preservation of the present invention;

[0035] Figure 3 It is a structural schematic diagram of a picture guide generation module in a multi-aspect ratio image redirection system based on key semantics preservation of the present invention;

[0036] Figure 4 It is a comparison diagram of the effects of the method of the present invention and other methods in Example 2 of the present invention. DETAILED DESCRIPTION

[0037] The present invention will be further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0038] Example 1

[0039] A multi-aspect ratio image retargeting method based on key semantics preservation, such as Figure 1 As shown, the specific steps include:

[0040] Step S1, input RGB image and target aspect ratio: input an image I and target aspect ratio r;

[0041] Step S2, semantic gap cutting algorithm realizes preliminary redirection: use the saliency detection algorithm to obtain the saliency map, attach the spatial prior and combine it with the gradient operator, calculate the energy of each area of ​​the image, and start cutting from the lines with lower energy to obtain the preliminary redirection result equal to or close to the expected aspect ratio. i ;

[0042] Semantic gap carving selects lines with lower energy for deletion based on the energy map. The energy calculation formula is:

[0043]

[0044] The energy formula introduces semantic information with spatial priors based on traditional image operators. Among them, s(x,y) represents the saliency of the position (x,y) in the saliency map, and x0 represents the horizontal coordinate of the centroid of the saliency region. In addition, when one-third of the saliency region is deleted, the semantic gap cutting algorithm will automatically stop cropping the image.

[0045] After obtaining the energy of each pixel in the image, use A vertical line describing a set of pixels in the image has |x(i)-x(i-1)|≤1 for any I. The energy of a vertical line can be defined as To be deleted * The line can be obtained by minimizing the energy:

[0046]

[0047] Using the dynamic programming algorithm, we can get several lines with the least energy for deletion.

[0048] Step S3, the redrawing area selection module finds the area where the pixels are not naturally connected: use the sliding window to find the initial redirection result of step S2 i The number of pixels deleted before and after each pixel in the image. If the number exceeds the threshold, the pixel is judged to be over-shifted and needs to be redrawn.

[0049] When selecting the redraw area, not only will the area with severe pixel shift be found adaptively, but the image will also be expanded according to the current image aspect ratio so that the processed image aspect ratio is equal to the target aspect ratio. Among them, the area with severe pixel shift is obtained through the deletion-retention binary image of the semantic analysis cutting algorithm. The retained pixels with more surrounding deleted pixels will be considered as severely shifted pixels and need to be regenerated.

[0050] The shifted region mask M is a one-dimensional sliding window of length l that counts the number of deleted pixels in the neighborhood of each pixel. The specific implementation is to perform a one-dimensional convolution on the deletion-retention binary map S:

[0051]

[0052] Where conv1d represents a one-dimensional convolution, and K is a convolution kernel with a length of l and a value of 1. The area in M ​​that exceeds the threshold η is the area where the pixel shift is serious. The size of the convolution kernel K is 25, the value of each position is 1, and the threshold η is set to 12.

[0053] Step S4, the image-guided generation module redraws the unnatural area: the masked pixel unnatural area is regenerated using the image-guided stable diffusion model, and then the unmasked area is pasted back to the generated area.

[0054] After the mask is covered on the preliminary redirection result, it is input into the Controlnet model; the original image is input into the IP-Adapter model as a control condition, and the two together assist the Stable diffusion model to generate. The generation process uses the original image as a control condition to avoid generating semantics that are inconsistent with the original image and improve the consistency between the generated area and the original image.

[0055] Step S5, outputting the target aspect ratio image: outputting the corresponding redirection result according to the target aspect ratio.

[0056] Example 2

[0057] A multi-aspect ratio image redirection system based on key semantics preservation, comprising at least a preliminary redirection module, a redrawing area selection module and an image guide generation module;

[0058] The preliminary redirection module: based on the semantic gap carving algorithm, introduces a saliency map with spatial prior and combines it with a gradient operator to delete the linear area with low energy to obtain a preliminary redirection result; the redrawing area selection module: takes the output result of the preliminary redirection module as input, regenerates the area with large pixel displacement in the preliminary redirection map, and expands the result of the semantic gap carving that does not reach the target aspect ratio; the picture guidance generation module: based on the result output by the redrawing area selection module, uses the original image as a control condition to guide local redrawing and output an image.

[0059] The working principle of the above system is as follows Figure 2 As shown, the multi-aspect ratio image retargeting method based on key semantics preservation using the above system includes the following steps:

[0060] Step S1: Input RGB image and target aspect ratio: Input an image I and target aspect ratio r.

[0061] The image redirection experiment in this embodiment is implemented on the public dataset RetargetMe. After investigation, the mainstream aspect ratios of current devices are: 16:9, 4:3, 1:1 and 9:16. Figure 4 As shown, the target aspect ratio selected in this embodiment is 16:9.

[0062] Step S2: adopting the semantic gap cutting algorithm.

[0063] This embodiment, as described in the steps of Embodiment 1, uses the Visual Saliency Transformer, a salient object detection model, as a salient energy extraction method, and uses the Sobel operator as a manual operator energy extraction method. The two are organically combined based on the spatial prior principle that the center is strong and the sides are weak. The energy of each line is calculated using a dynamic programming algorithm, and the lines with lower energy are deleted. The number of lines is determined by the smaller median of the target aspect ratio r and the saliency loss tolerance λ that allows the number of lines to be deleted. After deleting the low-energy lines, the result of the preliminary redirection is output. The saliency loss tolerance λ represents the maximum salient area that is allowed to be lost during the redirection process, and is set to 0.33 in this embodiment.

[0064] Step S3: The redrawing area selection module finds out the area where the pixels are connected unnaturally.

[0065] The redrawing area selection module proposed in the present invention is applied to the preliminary redirection result outputted in step S2 to find out the area where the pixels are discontinuous and generate a mask. At the same time, the module compares the aspect ratio of the preliminary redirection result with the target aspect ratio. If they are inconsistent, the desired aspect ratio is achieved by adding masks on the upper and lower sides. The mask area generated by this module will be regenerated in step S4.

[0066] Step S4: The image guidance generation module redraws the unnatural area.

[0067] The masked pixel artifacts are regenerated using an image-guided stable diffusion model, and the unmasked regions are subsequently pasted back onto the generated image. Figure 3 The generated model structure shown is generated by ControlNet and IP-Adapter assisted stable diffusion model, where ControlNet controls the redrawing area and IP-Adapter controls the generated content.

[0068] Step S5: Output the corresponding redirection result according to the target aspect ratio.

[0069] In this embodiment, the model uses a single RTX3090 graphics card for inference sampling, the saliency detection model encoder is selected as the t2t-vit model pre-trained on imagenet, the redrawing process sampling step is 50 steps, the sampling method is ddim, and the IP-Adapter version is selected as the plus version.

[0070] Figure 4 The redirection effect of the method of the present invention and other methods under the aspect ratio of 16:9 is shown. The first column is the input RGB image, the second, third and fourth columns are the redirection results obtained by scaling, cropping and gap cutting, and the fifth column is the result of the present invention. Figure 4 It can be seen that the image retargeting results obtained by the traditional method often easily lead to deformation or loss of important areas due to the lack of semantic information, while the present invention solves this problem by introducing saliency prior. In addition, by expanding the image and redrawing it, the present invention solves the problem of important objects exceeding the target resolution, making the present invention applicable to any aspect ratio.

[0071] Test Case

[0072] In order to verify the reliability of the redirection results of the present invention under different aspect ratios, the performance of the method of the present invention is compared with the most common existing image redirection methods: this test example is compared with the other three commonly used image redirection methods on the public dataset RetargetMe at four aspect ratios of 16:9, 4:3, 1:1 and 9:16.

[0073] The objective evaluation index of this test case is the Saliency Discard Ratio (SDR), which is calculated as follows:

[0074]

[0075] Among them, W s ori Indicates the maximum value of the salient pixels on the width axis of the saliency map in the original image, Ws out The maximum value of the salient pixels on the width axis of the saliency map of the redirection result. This indicator measures the proportion of salient areas lost before and after redirection. The smaller the indicator, the better the method effect.

[0076] The quantitative comparison results of the method proposed in the present invention and other three existing image redirection methods are shown in the following table:

[0077]

[0078] It can be seen from the above table that the image redirection method proposed in the present invention significantly reduces the loss of salient areas at all aspect ratios, fully demonstrating the high efficiency and versatility of the present invention in retaining key semantics.

[0079] The subjective evaluation indicators of this test case include content integrity score, deformation degree score, local smoothness score and aesthetic score, with a score range of 0 to 3 points. The subjective evaluation results are averaged by 20 volunteers on the 16 to 9 results of the RetargetMe dataset. The results are shown in the following table:

[0080]

[0081] It can be seen from the above table that the image redirection method proposed in the present invention has achieved extremely high scores in various subjective indicators, and the average result is the best among all methods.

[0082] The multi-aspect ratio image redirection method based on key semantics retention provided by the present invention can widely assist in image redirection tasks of various ratios. Specifically in the industry, it is crucial to ensure the consistency and aesthetics of images on different devices and screen sizes. In the fields of digital media, advertising, and e-commerce, the quality of images often directly affects the user's experience and decision-making. Therefore, product images need to adapt to the different screen ratios of various display devices to ensure that they can maintain the best display effect on different platforms. The present invention can achieve excellent redirection results, reduce the need for enterprises to repeatedly produce images, and improve enterprise operating efficiency.

[0083] It should be noted that the above content only illustrates the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications all fall within the protection scope of the claims of the present invention.

Claims

1. A multi-aspect ratio image retargeting system based on key semantics preservation, Features: At least including a preliminary redirection module, a redrawing area selection module and an image guide generation module; The preliminary redirection module: based on the semantic gap carving algorithm, by introducing a saliency map with spatial priors and combining it with a gradient operator, deletes the linear areas with low energy to obtain a preliminary redirection result; The redrawing area selection module takes the output result of the preliminary redirection module as input, regenerates the area with large pixel shift in the preliminary redirection map, and expands the result of semantic gap carving that does not reach the target aspect ratio; The image guide generation module: based on the result output by the redrawing area selection module, uses the original image as a control condition to guide the local redrawing and output the image.

2. The multi-aspect ratio image retargeting system based on key semantics preservation as claimed in claim 1, Features: The image guided generation module at least includes ControlNet and IP-Adapter auxiliary stable diffusion model, wherein ControlNet is used to control the redrawing area, and IP-Adapter is used to control the generated content.

3. A method for retargeting multiple aspect ratio images based on key semantics preservation using the system as claimed in claim 1, It is characterized in that The specific steps include: S1, determine the target image: input RGB image and set target aspect ratio; S2, preliminary redirection: Based on the semantic gap carving algorithm, the saliency detection algorithm is used to obtain the saliency map, which is combined with the gradient operator after attaching the spatial prior to calculate the energy of each area of ​​the image, and starts to crop from the lines with low energy. After deleting the linear areas with low energy, the preliminary redirection result O is obtained. i ; S3, find out the unnatural pixel connection area: use the sliding window to find out the initial redirection result of step S2 i The number of pixels deleted before and after each pixel in the image is determined, and the area with large pixel shift is found, and the result of semantic gap carving that does not reach the target aspect ratio is expanded; the pixel shift is large, and the number of pixels deleted around it is large, and the retained pixels are obtained through the deletion-retention binary image of the semantic analysis cutting algorithm; S4, redrawing of unnatural regions: regenerating the unnatural regions obtained in step S3 using an image-guided stable diffusion model, and then pasting the unmasked regions back onto the generated regions; S5, image output: according to the target aspect ratio, output the corresponding redirection result to obtain the target aspect ratio image.

4. The method for retargeting images with multiple aspect ratios based on key semantics preservation as claimed in claim 3, Features: The energy calculation formula in step S2 is: Among them, I(x,y) represents the pixel value of image I at position (x,y), S(x,y) represents the saliency size of its saliency map at position (x,y), and x 0 Represents the abscissa of the centroid of the salient region, and W is the width of the image; After obtaining the energy of each pixel in the image, use A vertical line describing a set of pixels in the image, for any I, has |x(i)-x(i-1)|≤1; the energy of a vertical line is defined as To be deleted * The line is obtained by minimizing the energy: Using the dynamic programming algorithm, several lines with the least energy are obtained for deletion.

5. The method for retargeting images with multiple aspect ratios based on key semantics preservation as claimed in claim 4, Features: In step S2, when one third of the salient area is cropped, the semantic gap cutting algorithm automatically stops cropping the image.

6. The method for retargeting images with multiple aspect ratios based on key semantics preservation as claimed in claim 3, Features: In step S3, the number of pixels deleted before and after each pixel exceeds the threshold and is determined to be a region with large pixel shift. The shift region mask M is a one-dimensional sliding window with a length of l that counts the number of deleted pixels in the neighborhood of each pixel, and a one-dimensional convolution is performed on the deletion-retention binary map S: Among them, conv1d represents one-dimensional convolution, K is a convolution kernel with a length of l and a value of 1, and the area in M ​​that exceeds the threshold η is the area with large pixel shift.

7. The method for retargeting images with multiple aspect ratios based on key semantics preservation as claimed in claim 6, Features: In step S3, the size of the convolution kernel K is 25, the value of each position is 1, and the threshold η is 12.

8. The method for retargeting images with multiple aspect ratios based on key semantics preservation as claimed in claim 6, Features: In the step S4, the mask is covered on the preliminary redirection result and then input into the Controlnet model; The original image is input into the IP-Adapter model as a control condition, and the two together assist the Stable diffusion model to generate images.

Citation Information

Patent Citations

  • Single-scale motion blurred image frame restoration method for cab environment

    CN112488946A

  • Monocular view depth estimation method based on neural radiation field and semantic segmentation

    CN115393410A

  • Image redirection method based on character region perception

    CN118057484A

  • Image retargeting method and device

    US20220207653A1

Cited By

  • Method, device, equipment and medium for performing commodity material fission on commodity image

    CN120070662A