Image expansion method and device, readable storage medium and program product

Abstract

The invention relates to an image expansion method and device, a readable storage medium and a program product. The method comprises the following steps: expanding a to-be-expanded image into a first image, wherein the first image comprises an original image area which retains the content in the to-be-expanded image and an expansion area of which the content semantics is empty; performing image generation according to a downsampling image obtained after downsampling of the first image and a description text representing content semantics in the to-be-expanded image to obtain a second image; the second image comprises a first area corresponding to the position of the original image area and a second area corresponding to the position of the expansion area; the content in the first area is consistent with the content in the original image area, and the content in the second area extends based on the content in the first area and is matched with the description text; up-sampling the second image to the image size of the first image to obtain an up-sampled image; and fusing an image region corresponding to the position of the expansion region in the up-sampling image with the to-be-expanded image to obtain a target image. By adopting the method, the quality of the expanded image can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image expansion method, device, readable storage medium and program product. Background Art

[0002] With the development of computer technology, the demand for image processing is becoming increasingly diverse. In some scenarios, there is a need to expand images. For example, in virtual reality scenarios, images can be expanded to simulate multi-view cameras. In photography post-processing or film production scenarios, images can be expanded to convert to widescreen mode to optimize the composition. In the prior art, users usually input the image to be expanded and specify the size of the expanded image, and the image is expanded through an artificial intelligence model.

[0003] However, in the prior art, the image quality obtained after image expansion is low. Summary of the invention

[0004] Based on this, the present application provides an image expansion method, device, readable storage medium and program product, which can improve the quality of the image obtained after image expansion.

[0005] On the one hand, the present application provides an image expansion method, comprising:

[0006] Expanding the image to be expanded into a first image, wherein the first image includes an original image region retaining the content of the image to be expanded and an expanded region whose content semantics is empty;

[0007] Downsampling the first image to obtain a downsampled image;

[0008] An image is generated according to the downsampled image and the description text representing the semantics of the content in the image to be expanded, so as to obtain a second image; the second image comprises a first area corresponding to the position of the original image area and a second area corresponding to the position of the expanded area; the content in the first area is consistent with the content in the original image area, and the content in the second area is content extended from the content in the first area and matches the description text;

[0009] Upsampling the second image to an image size of the first image to obtain an upsampled image;

[0010] The image area corresponding to the position of the extended area in the up-sampled image is fused with the image to be extended to obtain a target image.

[0011] In one aspect, the present application further provides an image expansion device, comprising:

[0012] A region expansion module, used for expanding the image to be expanded into a first image, wherein the first image includes an original image region retaining the content of the image to be expanded and an expansion region whose content semantics is empty;

[0013] A downsampling module, used for downsampling the first image to obtain a downsampled image;

[0014] A generating module, configured to generate an image according to the downsampled image and the description text representing the semantics of the content in the image to be expanded, so as to obtain a second image; the second image comprises a first area corresponding to the position of the original image area and a second area corresponding to the position of the expanded area; the content in the first area is consistent with the content in the original image area, and the content in the second area is content extended from the content in the first area and matches the description text;

[0015] an upsampling module, configured to upsample the second image to an image size of the first image to obtain an upsampled image;

[0016] The fusion module is used to fuse the image area corresponding to the position of the extended area in the up-sampled image with the image to be extended to obtain a target image.

[0017] In one aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0018] Expanding the image to be expanded into a first image, wherein the first image includes an original image region retaining the content of the image to be expanded and an expanded region whose content semantics is empty;

[0019] Downsampling the first image to obtain a downsampled image;

[0020] An image is generated according to the downsampled image and the description text representing the semantics of the content in the image to be expanded, so as to obtain a second image; the second image comprises a first area corresponding to the position of the original image area and a second area corresponding to the position of the expanded area; the content in the first area is consistent with the content in the original image area, and the content in the second area is content extended from the content in the first area and matches the description text;

[0021] Upsampling the second image to an image size of the first image to obtain an upsampled image;

[0022] The image area corresponding to the position of the extended area in the up-sampled image is fused with the image to be extended to obtain a target image.

[0023] In one aspect, the present application further provides a computer program product, including a computer program, which, when executed by a processor, implements the following steps:

[0024] Expanding the image to be expanded into a first image, wherein the first image includes an original image region retaining the content of the image to be expanded and an expanded region whose content semantics is empty;

[0025] Downsampling the first image to obtain a downsampled image;

[0026] An image is generated according to the downsampled image and the description text representing the semantics of the content in the image to be expanded, so as to obtain a second image; the second image comprises a first area corresponding to the position of the original image area and a second area corresponding to the position of the expanded area; the content in the first area is consistent with the content in the original image area, and the content in the second area is content extended from the content in the first area and matches the description text;

[0027] Upsampling the second image to an image size of the first image to obtain an upsampled image;

[0028] The image area corresponding to the position of the extended area in the up-sampled image is fused with the image to be extended to obtain a target image.

[0029] The above-mentioned image expansion method, device, readable storage medium and program product, for a first image obtained by expanding an image to be expanded, including an original image area retaining the content of the image to be expanded and an expansion area whose content semantics is empty, an image is generated according to a downsampled image obtained by downsampling the first image and a description text representing the content semantics of the image to be expanded, to obtain a second image, the downsampled image can reduce the consumption of computing resources in the image generation process, and in the obtained second image, the content of the image to be expanded is retained in a first area corresponding to the position of the original image area, and the generated content in a second area corresponding to the position of the expansion area is based on the extension of the content in the first area and matches the description text, so that the expanded second image matches the content semantics of the image to be expanded, is visually coherent, and has high image quality; and after upsampling the second image to the image size of the first image, the image area corresponding to the position of the expansion area in the obtained upsampled image and the image to be expanded are fused to obtain a target image, so that the influence of downsampling on the image resolution can be reduced to a large extent, so that the target image contains the content of the expanded image and retains the high-frequency details of the image to be expanded to a large extent, thereby improving the image quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0031] Figure 1 is a schematic flow chart of an image expansion method in one embodiment;

[0032] Figure 2 A schematic diagram of an example of an extended mask in an embodiment;

[0033] Figure 3 A schematic diagram of an image expansion framework in one embodiment;

[0034] Figure 4 A schematic diagram of a pre-processing step in an embodiment;

[0035] Figure 5 A schematic diagram of an expansion interface in an embodiment;

[0036] Figure 6 A schematic diagram of an image generation framework in one embodiment;

[0037] Figure 7 A schematic diagram of the image fusion process in one embodiment;

[0038] Figure 8 A schematic diagram of an example of an image expansion result in an embodiment;

[0039] Fig. 9 is a structural block diagram of an image expansion device in one embodiment;

[0040] Fig.10 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solution and beneficial effects of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0042] In one embodiment, Figure 1As shown, an image expansion method is provided. This embodiment uses the method applied to a computer device as an example. The computer device can be a terminal or a server. It can be understood that the method can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. Among them, the terminal can be a personal computer, a laptop, a smart phone, a tablet computer or others. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services. In this embodiment, the method includes the following steps:

[0043] Step 102: Expand the image to be expanded into a first image, wherein the first image includes an original image region retaining the content of the image to be expanded and an expanded region whose content semantics is empty.

[0044] Among them, the image to be expanded is an image that needs to be expanded in content. It can be understood that the image size of the first image is larger than the image size of the image to be expanded. The empty content semantics can represent that the content of the extended area has no substantial semantics. For example, each pixel in the extended area can be a zero pixel, and a zero pixel is a pixel with a pixel value of zero. In this case, the extended area can be displayed in black. Each pixel in the extended area can also be a non-zero and identical preset pixel. The same preset pixel is a preset pixel with the same pixel value. The preset pixel refers to a pixel whose pixel value is a preset value. The preset value is a positive integer greater than zero, for example, it can be a pixel value that makes the extended area appear white, specifically 1 or 255.

[0045] Exemplarily, the computer device may expand the image to be expanded into the first image according to the input expansion size information. The expansion size information may be the image size of the first image or the expansion ratio. The expansion ratio is the ratio of the image size of the expanded first image to the image size of the image to be expanded. The image size may include image width and image height. For example, the expansion ratio may be 50%, then the image width of the first image may be expanded by 50% compared to the image width of the image to be expanded, and the image height of the first image may be expanded by 50% compared to the image height of the image to be expanded, that is, the image width of the first image is 150% of the image width of the image to be expanded, and the image height of the first image is 150% of the image height of the image to be expanded. The expansion ratio may also be 100%, 200%, 300% or others. When expanding the image to be expanded into the first image, it may be expanded to all sides with the center of the image to be expanded as the center, or it may be expanded outward from at least one edge of the image to be expanded.

[0046] In one embodiment, the computer device may expand the image to be expanded into the first image according to a default expansion ratio.

[0047] In one embodiment, the computer device may obtain an input original image and input expansion size information, determine the original image as the image to be expanded, and expand the image to be expanded into the first image according to the expansion size information.

[0048] Step 104: downsample the first image to obtain a downsampled image.

[0049] Herein, downsampling is a process of reducing the size of the first image to reduce the resolution of the first image.

[0050] Exemplarily, the computer device may downsample the first image to an image size supported by the image generation model to obtain a downsampled image. The image generation model is used to generate images in subsequent steps. The image generation model may be a generative adversarial network (GANs), a diffusion model, or others. In the process of downsampling the first image, the first image may be proportionally reduced according to the image size supported by the image generation model. The image size supported by the image generation model is, for example, 512*512.

[0051] In one embodiment, the computer device may downsample the first image to a preset image size to obtain a downsampled image, wherein the preset image size is a pre-set image size, such as 640*640.

[0052] Step 106, generate an image based on the downsampled image and the description text representing the semantics of the content in the image to be expanded, and obtain a second image; the second image includes a first area corresponding to the position of the original image area and a second area corresponding to the position of the expanded area; the content in the first area is consistent with the content in the original image area, and the content in the second area is content extended based on the content in the first area and matches the description text.

[0053] Among them, the content semantics in the image to be expanded refers to the meaning of the content of the image to be expanded. The relative position of the first area with respect to the second image is the same as the relative position of the original image area with respect to the first image. The relative position of the second area with respect to the second image is the same as the relative position of the expanded area with respect to the first image. The content in the first area is consistent with the content in the original image area, which can be understood as the image content in the first area is the same as the image content in the image to be expanded. It can be understood that in the image generation process, the content of the area corresponding to the position of the original image area in the downsampled image can be kept unchanged, so that the content of the first area in the obtained second image can be consistent with the content in the original image area.

[0054] Exemplarily, the computer device may extract the descriptive text representing the semantics of the content in the image to be expanded, and generate the image through the trained image generation model according to the downsampled image and the descriptive text to obtain the second image. The image generation model may be a generative adversarial network, a diffusion model or others. The diffusion model may be a latent diffusion model (Latent Diffusion Models, LDMs for short, the diffusion model provided in the paper High-Resolution Image Synthesis with Latent Diffusion Models), a stable diffusion model (Stable Diffusion Models, SD for short) or others.

[0055] In one embodiment, a computer device may obtain a downsampling mask that matches the image size of the downsampled image, wherein in the downsampling mask, each pixel in an area corresponding to the position of the original image area is a zero pixel, and each pixel in an area corresponding to the position of the extended area is a non-zero and identical preset pixel; the downsampled image is denoised to obtain a noisy feature, a feature obtained by performing a matrix multiplication operation on the downsampling mask and the noisy feature is combined with the noisy feature and the downsampling mask to obtain a combined feature, and an image is generated using a trained diffusion model based on the combined feature and a description text representing the semantics of the content in the image to be extended to obtain a second image.

[0056] The diffusion model generates data by simulating the forward diffusion process and the reverse diffusion process. The forward diffusion process starts from a clear image without adding noise, and gradually adds noise, so that the clear image gradually transforms into a noise distribution. The reverse diffusion process starts from the noise distribution transformed by the forward diffusion process, and gradually restores the original clear image or generates a new image. In the reverse diffusion process, the diffusion model learns how to restore the original clear image or generate a new image from the noise data, which can be achieved by training the diffusion model. Adding noise to the downsampled image can be understood as a forward diffusion process, and image generation can be understood as a reverse diffusion process.

[0057] The downsampling mask is a binary image. The image size of the downsampling mask can be the same as the image size of the downsampling image. When adding noise, Gaussian noise can be added at multiple time steps to obtain noisy features. The combination can be a cascade process of the features obtained by matrix multiplication of the downsampling mask and the noisy features, the noisy features, and the downsampling mask, which can be understood as a process of feature splicing in the channel direction.

[0058] Step 108: upsample the second image to the image size of the first image to obtain an upsampled image.

[0059] The upsampling is a process of enlarging the size of the second image.

[0060] For example, the computer device may use a preset interpolation algorithm to upsample the second image to the image size of the first image to obtain an upsampled image, wherein the preset interpolation algorithm may be a nearest neighbor interpolation algorithm, a bilinear interpolation algorithm, a bicubic interpolation algorithm or others.

[0061] Step 110: fuse the image area corresponding to the position of the extended area in the upsampled image with the image to be extended to obtain a target image.

[0062] The relative position of the image area corresponding to the position of the extended area in the upsampled image relative to the upsampled image is the same as the relative position of the extended area relative to the first image.

[0063] Exemplarily, the computer device may fuse the image area corresponding to the position of the expansion area in the upsampled image with the image to be expanded using a preconfigured image fusion algorithm to obtain a target image, wherein the image fusion algorithm may be a Poisson fusion algorithm, an Alpha fusion algorithm or others.

[0064] In the above-mentioned image expansion method, the first image obtained by expanding the image to be expanded includes an original image area retaining the content of the image to be expanded and an expansion area whose content semantics is empty. According to the downsampled image obtained by downsampling the first image and the description text representing the content semantics of the image to be expanded, an image is generated to obtain a second image. The downsampled image can reduce the consumption of computing resources in the image generation process. In the obtained second image, the content of the image to be expanded is retained in the first area corresponding to the position of the original image area, and the generated content in the second area corresponding to the position of the expansion area is based on the extension of the content in the first area and matches the description text. In this way, the expanded second image matches the content semantics of the image to be expanded, is visually coherent, and has high image quality. In addition, after upsampling the second image to the image size of the first image, the image area corresponding to the position of the expansion area in the obtained upsampled image and the image to be expanded are fused to obtain a target image. In this way, the influence of downsampling on the image resolution can be reduced to a large extent, so that the target image retains the high-frequency details of the image to be expanded to a large extent while containing the image content obtained by expansion, thereby improving the image quality.

[0065] In an exemplary embodiment, before step 102, the image expansion method further includes the following steps: acquiring the image to be processed, performing edge detection on the image to be processed, and obtaining an edge image; performing line segment extraction on the edge image, and obtaining a plurality of candidate line segments in the edge image; when there is a target line segment parallel to the edge of the edge image among the plurality of candidate line segments, determining a border in the image to be processed based on the target line segment; and cropping the border in the image to be processed, and obtaining the image to be expanded.

[0066] The image to be processed is an image that needs to be subjected to border detection. The image to be processed can be the original input image, or it can be an image obtained after content review and rotation correction of the original image. The candidate line segment is a line segment extracted from the edge image. The border is used to visually highlight the image area with semantic content in the image to be processed, while the content semantics of the border is empty.

[0067] In this embodiment, since the border is set around the image area with content semantics in the image to be processed, and the content semantics of the border itself is empty, if the expansion is directly based on the image to be processed containing the border, it may cause the content semantics of the second area in the generated second image to be empty, and the obtained image quality is poor. Through edge detection and line segment extraction, the border can be determined more accurately, and then the border can be cropped and then expanded to ensure image quality.

[0068] In one embodiment, the computer device may use a preset edge detection algorithm to perform edge detection on the image to be processed to obtain an edge image. The preset edge detection algorithm may be a Canny algorithm (a multi-level edge detection algorithm proposed by John F. Canny), a Sobel algorithm (an algorithm for detecting edges by calculating the first-order gradient of an image), or others.

[0069] In one embodiment, the computer device may use a Hough Transform algorithm to extract line segments from the edge image to obtain a plurality of candidate line segments in the edge image.

[0070] In one embodiment, when there is a target line segment parallel to the edge of the edge image among multiple candidate line segments, determining the border in the image to be processed based on the target line segment includes: when there are multiple target line segments parallel to the edge of the edge image among multiple candidate line segments, and the multiple target line segments can form at least one rectangle, then determining the border in the image to be processed based on the at least one rectangle.

[0071] Among them, a rectangle is formed with four target line segments among the multiple target line segments as sides. The target rectangle with the largest area can be determined from at least one rectangle, and the area outside the area where the target rectangle is located in the edge image and within the four sides of the edge image is determined as the target area in the edge image, and the area in the image to be processed corresponding to the position of the target area in the edge image is determined as the frame in the image to be processed.

[0072] In one embodiment, when there is a target line segment parallel to the edge of the edge image among multiple candidate line segments, determining the border in the image to be processed based on the target line segment includes: when there is at least one target line segment parallel to the edge of the edge image among multiple candidate line segments, for each target line segment, determining the target edge among the four edges of the edge image that is parallel to the target line segment and closest to the target line segment, determining the rectangular area between the target line segment and the target edge in the edge image, determining the image area corresponding to the position of the rectangular area in the image to be processed as a candidate border, performing semantic analysis on the candidate border to obtain an analysis result, and when the analysis result indicates that the content semantics of the candidate border is empty, determining the candidate border as the border in the image to be processed.

[0073] The rectangular area between the target line segment and the target edge can be understood as extending each target line segment toward both ends of the line segment until an intersection is generated with the edge of the edge image, obtaining two intersections, and constructing a rectangular area with the target edge as one edge and the two intersections as two vertices outside the target edge. Semantic analysis can be implemented using an image semantic segmentation model or a salient object detection model. When no area with a semantic category is detected in the candidate bounding box or no salient object is detected, it can be determined that the analysis result represents that the content semantics is empty.

[0074] In an exemplary embodiment, the step of obtaining an image to be processed includes: obtaining an input original image; when the original image meets a preset quality constraint and the original image does not belong to a preset negative type, obtaining direction information in the metadata of the original image; and performing rotation correction on the original image according to the direction information to obtain the image to be processed.

[0075] Among them, the preset quality constraint condition is used to constrain the image quality of the original image. The preset quality constraint condition may include a preset resolution constraint condition and a preset noise constraint condition. The preset resolution constraint condition is that the resolution of the original image reaches the preset resolution. The preset resolution is, for example, 1280*1280. The preset noise constraint condition is that the noise level evaluation value of the original image meets the preset low noise condition. The noise level evaluation value may be a noise variance, and the noise level evaluation value meets the preset low noise condition, and the noise level evaluation value may be less than the preset noise variance, and the preset noise variance is, for example, 0.03. The noise level evaluation value may also be a PSNR (Peak Signal to Noise Ratio) value between a preset standard image and the original image, and the noise level evaluation value meets the preset low noise condition, and the noise level evaluation value may be greater than a preset peak signal-to-noise ratio value, and the preset peak signal-to-noise ratio value is, for example, 30dB.

[0076] The preset negative type is a preset negative type. The negative type represents the category of images containing negative information. Negative information includes bad information, sensitive information, inappropriate content, etc. Bad information includes illegal information, sensitive information includes political information, and inappropriate content includes violent images. When determining whether the original image belongs to the preset type that produces negative impact, an image recognition algorithm can be used to determine the category of the original image. The image recognition algorithm can specifically extract features from the original image and the image of the preset type, and analyze the extracted features to determine whether the original image belongs to the preset negative type.

[0077] The metadata of the original image is used to record various attribute information of the original image. The metadata may include shooting equipment, shooting parameters, time, location, direction information or others. The metadata may specifically be Exif (Exchangeable Image File Format) metadata. Exif metadata may be extracted by an image processing library, such as ImageMagick (an open source command line tool), PIL (Python Image Library) or others. Specifically, the "_getexif ()" function in PIL may be used to extract Exif metadata.

[0078] The orientation information may be the value of the Orientation tag in the Exif metadata. The value of the Orientation tag may include values ​​representing normal orientation, horizontal flip, clockwise rotation of 180 degrees, vertical flip, clockwise rotation of 90 degrees and horizontal flip, clockwise rotation of 90 degrees, clockwise rotation of 90 degrees and vertical flip, and counterclockwise rotation of 90 degrees. Corresponding one to one to the values ​​of the above-mentioned Orientation tags, the rotation corrections required for the original image are, in order, no correction required, horizontal flip, clockwise rotation of 180 degrees, vertical flip, clockwise rotation of 90 degrees and horizontal flip, clockwise rotation of 90 degrees, clockwise rotation of 90 degrees and vertical flip, and counterclockwise rotation of 90 degrees. It can be understood that when the orientation information is a value representing the normal orientation, there is no need to perform rotation correction on the original image. At this time, the original image can be used as the image to be processed if the original image meets the preset quality constraints and the original image does not belong to the preset negative type.

[0079] In this embodiment, when the original image meets the preset quality constraint conditions and the original image does not belong to the preset negative type, it means that the image quality of the original image is good and will not produce negative effects. At this time, the direction information of the original image is obtained, and the original image is rotated and corrected according to the direction information to obtain the image to be processed, ensuring that the image is in the normal direction, facilitating subsequent image expansion, and improving the quality of the subsequently generated second image.

[0080] In an exemplary embodiment, step 106 includes: obtaining an expansion mask that matches the image size of the first image; in the expansion mask, each pixel in the area corresponding to the position of the original image area is a zero pixel, and each pixel in the area corresponding to the position of the expansion area is a non-zero and identical preset pixel; through a trained conditional control network, feature extraction is performed according to the expansion mask to obtain a control feature; the control feature is used to indicate that an image is generated in the image area corresponding to the position of the expansion area in the downsampled image during the image generation process; according to the control feature, the downsampled image, and the descriptive text representing the content semantics in the image to be expanded, the image is generated through the trained diffusion model to obtain a second image.

[0081] The extended mask is a binary image. The image size of the extended mask may be the same as the image size of the first image. The relative position of the area corresponding to the position of the original image area in the extended mask relative to the extended mask is the same as the relative position of the original image area relative to the first image. The relative position of the area corresponding to the position of the extended area in the extended mask relative to the extended mask is the same as the relative position of the extended area relative to the first image. Since the pixels of the area corresponding to the position of the original image area and the pixels of the area corresponding to the position of the extended area in the extended mask are different, the extended mask can represent the position of the original image area and the position of the extended area.

[0082] A zero pixel is a pixel whose pixel value is zero. An identical preset pixel is a preset pixel with the same pixel value. A preset pixel refers to a pixel whose pixel value is a preset value. The preset value is a positive integer greater than zero, for example, a pixel value that makes the extended area appear white, specifically 1 or 255. Figure 2 In the example diagram of the extended mask shown in FIG. 1 , the black area may represent the area in the extended mask corresponding to the original image area, and the white area may represent the area in the extended mask corresponding to the extended area. Figure 2 The area corresponding to the extended area is white. Figure 2 The dashed lines shown may be used in the application to indicate the edges of the extended mask to make it clear that Figure 2 The dotted line is shown in the extended mask as the location of the area corresponding to the extended area. In actual application, the extended mask may not include the dotted line.

[0083] The conditional control network is used to control the image generation process of the diffusion model. The conditional control network can be ControlNet (the network proposed in the paper Adding Conditional Control to Text-to-Image Diffusion Models). When extracting features through ControlNet, the expansion mask can be used as the Condition information of ControlNet. ControlNet can extract features based on the Condition information, the Input of the diffusion model, and the Prompt of the diffusion model, and output control features. It acts on the Unet network of the diffusion model. It can be understood that the features output by ControlNet to the diffusion model are control features. The Input of the diffusion model can be a downsampled image, or it can be a combined feature obtained by combining as mentioned in the subsequent steps. The Prompt of the diffusion model can be a text prompt, specifically a descriptive text that represents the semantics of the content in the image to be expanded.

[0084] The Unet network of the diffusion model may include multiple Encoder Blocks and Decoder Blocks corresponding to each EncoderBlock. ControlNet may include a convolution layer with a convolution kernel size of 1*1 and multiple trainable Encoder Block copies. In ControlNet, after the Condition information is subjected to feature extraction through a 1*1 convolution layer, it is concatenated with the Input of the diffusion model and the Prompt of the diffusion model. Feature extraction is performed through multiple trainable Encoder Block copies to obtain the control features output by each Encoder Block copy. The control features can be input into the corresponding Decoder Block, thereby controlling the generation process of the diffusion model.

[0085] In this embodiment, control features are extracted through a conditional control network, and the control features can indicate that image generation is performed in an image region corresponding to the position of the extended region in the downsampled image. Therefore, image generation is performed through a trained diffusion model based on the control features, the downsampled image, and a descriptive text representing the semantics of the content in the image to be expanded. This can focus on image generation in the image region corresponding to the position of the extended region, while retaining the image content corresponding to the position of the original image region, thereby improving the accuracy of image generation.

[0086] In an exemplary embodiment, according to the control features, the downsampled image and the descriptive text representing the semantics of the content in the image to be expanded, an image is generated through a trained diffusion model, and the step of obtaining a second image includes: adding noise to the downsampled image to obtain a noisy feature; downsampling the expansion mask to the image size of the downsampled image to obtain a downsampled mask; combining the features obtained by matrix multiplication of the downsampled mask and the noisy feature with the noisy feature and the downsampled mask to obtain a combined feature; according to the control features, the combined features and the descriptive text representing the semantics of the content in the image to be expanded, an image is generated through a trained diffusion model to obtain a second image.

[0087] When adding noise, Gaussian noise can be added in multiple time steps to obtain noisy features. In each time step, after adding Gaussian noise, the noisy feature z_t obtained in that time step is obtained, z_t=sqrt(1-β_t)*z+sqrt(β_t)*ε, ε is the standard Gaussian distribution, β_t is the noise variance at time step t, sqrt represents square root operation, * represents multiplication operation, and z represents the noisy feature obtained in the previous time step.

[0088] The noisy feature can be in the data form of a feature map. Each pixel in the area corresponding to the original image area position in the extended mask is zero pixel, therefore, each pixel in the area corresponding to the original image area position in the downsampled mask is also zero pixel, and the feature obtained by performing matrix multiplication operation on the downsampled mask and the noisy feature can be understood as a feature map, then each pixel in the area corresponding to the original image area position in the feature map is also zero pixel, and the area corresponding to the extended area position in the feature map is the same as the area corresponding to the extended area position in the noisy feature.

[0089] In the process of downsampling the extended mask to the image size of the downsampled image, the extended mask can be proportionally reduced to the image size of the downsampled image. The combination can be a cascade process of the features obtained by matrix multiplication of the downsampled mask and the noisy feature, the noisy feature, and the downsampled mask, which can be understood as a feature splicing process in the channel direction.

[0090] The noisy feature can be a 4-channel feature map, and the downsampling mask can be understood as a single-channel feature map. The feature obtained by performing matrix multiplication operation on the downsampling mask and the noisy feature can be a 4-channel feature map, and the combined feature obtained by combining can be a 9-channel feature map. The combined feature can be used as the input of the diffusion model, and the description text can be used as the prompt of the diffusion model.

[0091] In this embodiment, the features obtained by matrix multiplication of the downsampling mask and the noisy features are combined with the noisy features and the downsampling mask. The obtained combined features provide richer input features for subsequent diffusion model to generate images, which can improve the accuracy of image generation.

[0092] In one embodiment, the computer device may input the downsampled image into an encoder in a trained autoencoder model to obtain four-channel features output by the encoder, and perform noise addition on the four-channel features for multiple time steps to obtain noisy features. The autoencoder model may be an AutoEncoder, and the autoencoder model may include an encoder and a decoder.

[0093] In an exemplary embodiment, an image is generated through a trained diffusion model according to control features, combined features, and a descriptive text representing the semantics of the content in the image to be expanded, and the step of obtaining a second image includes: inputting the image to be expanded into a trained multimodal model to obtain a descriptive text representing the semantics of the content in the image to be expanded; inputting the descriptive text into a trained text encoder to obtain text features; and inputting the control features, combined features, and text features into the trained diffusion model to generate an image to obtain a second image.

[0094] Among them, the multimodal model is a model that converts images into text. The multimodal model can be a BLIP model (the model provided in the paper BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation), a BLIP-2 model (the model provided in the paper BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models), or others.

[0095] The text encoder is used to encode text into text features. The text encoder can be a CLIP model (the model provided in the paper Learning Transferable Visual Models From Natural Language Supervision), a LightCLIP model (the model provided in the paper LightCLIP: Learning Multi-Level Interaction forLightweight Vision-Language Models), or others. In the process of image generation based on text features by the diffusion model, the text features can be injected into the image generation process of the diffusion model through the cross attention layer in the Unet network of the diffusion model through the cross attention mechanism.

[0096] In this embodiment, a description text is extracted from the image to be expanded through a multimodal model, the description text is input into a trained text encoder to obtain text features, and the control features, combined features and text features are input into a trained diffusion model for image generation. In this way, while focusing on image generation in the image area corresponding to the position of the expanded area and retaining the image content corresponding to the position of the original image area, the image content generated in the image area corresponding to the position of the expanded area is matched with the semantics of the image to be expanded, thereby improving the quality of image generation.

[0097] In an exemplary embodiment, step 110 includes: stitching the image area corresponding to the position of the extended area in the upsampled image and the image to be extended to obtain a stitched image; the stitched image is a multi-channel image; for each of the multiple channels of the stitched image, performing Poisson fusion on the image area corresponding to the position of the extended area in the channel and the image area corresponding to the position of the original image area in the channel to obtain a fused image corresponding to the channel; and combining the fused images corresponding to each of the multiple channels to obtain a target image.

[0098] The image size of the stitched image is the same as the size of the upsampled image and the first image. It can be understood that when stitching, the image to be expanded has the same size and shape as the original image area, and the image area corresponding to the position of the expanded area in the upsampled image has the same size and shape as the expanded area, so that the image area corresponding to the position of the expanded area in the upsampled image and the image to be expanded can be stitched into one image.

[0099] A multi-channel image is an image with multiple channels. Each channel in a multi-channel image is a single-channel image. The multiple channels may be an R (Red) channel, a G (Green) channel, and a B (Blue) channel. When performing Poisson fusion processing, the fused image after Poisson fusion processing can be obtained by solving the Poisson equation. The Poisson equation can be found in the following formula (1).

[0100] Formula (1)

[0101] Among them, formula (1) can express that the fused image corresponding to one channel of the stitched image satisfies And meet . The Laplace operator can be characterized as the sum of second-order partial derivatives. In a two-dimensional image, the Laplace operator describes the local curvature of each pixel in the image. The divergence operator can be characterized, which describes the divergence of the gradient field.

[0102] It can represent the fused image corresponding to the channel to be solved. The target image area corresponding to the position of the extended area in the fused image corresponding to the channel can be represented. This can characterize the The gradient field of the reference image region corresponding to the position, Characterizable The boundary conditions, Can represent surrounding The target image area (i.e. ) with respect to the fused image corresponding to the channel is the same as the relative position of the reference image area with respect to the channel, and is the same as the relative position of the extended area with respect to the first image.

[0103] The meaning can be that the fused image corresponding to this channel of the stitched image exist The calculation result of the Laplacian operator in is equal to the channel of the spliced ​​image. The divergence of the gradient field of the image area corresponding to the position, that is, the fused image corresponding to the channel is fused using the Laplace operator The calculation result obtained by calculating the target image area in the channel is equal to the divergence of the gradient field of the reference image area in the channel, which makes the fused target image area maintain the same gradient structure as the reference image area in the channel of the stitched image, thereby maintaining the texture characteristics of the image area corresponding to the position of the extended area in the channel of the stitched image.

[0104] The meaning can be that the fused image corresponding to this channel of the stitched image Boundary conditions The fused image corresponding to this channel of the stitched image is In , and the channel of the stitched image is The reference boundary corresponding to the position is the same. In other words, the fused image corresponding to this channel The boundary of the target image region in the channel is the same as the boundary of the reference image region in the channel, so that the fusion has a smooth transition at the boundary. The relative position of the reference boundary with respect to the channel is The fused image corresponding to this channel The relative positions are the same.

[0105] In this embodiment, image fusion is performed through the Poisson fusion algorithm to obtain a fused image corresponding to each channel, and then a target image is generated according to the fused images corresponding to each of the multiple channels, so that the content of the area corresponding to the original image area position in the target image is the same as the image content of the image to be expanded, and the original resolution and detail texture of the image to be expanded are retained. The image area corresponding to the extended area position in the target image has a smooth transition with the content corresponding to the original image area position in the target image while retaining the gradient structure of the image area corresponding to the extended area position in the upsampled image, and the target image is more realistic and natural.

[0106] In a specific embodiment, see Figure 3 As shown in the schematic diagram of the image expansion framework, the above-mentioned image expansion method may specifically include the following steps.

[0107] See Figure 4In the schematic flow diagram of the preprocessing step shown, the terminal can obtain the input original image, and perform image quality assessment on the original image based on the preset quality constraint condition, and the preset quality constraint condition may include the preset resolution constraint condition and the preset noise constraint condition. When the original image does not meet the preset resolution constraint condition, a prompt message indicating low resolution is generated to prompt the user to re-enter. When the original image does not meet the preset noise constraint condition, a prompt message indicating high noise is generated to prompt the user to re-enter. When the original image meets the preset resolution constraint condition and the preset noise constraint condition, the image content of the original image is reviewed, and the image recognition algorithm can be used to identify whether the original image belongs to the preset negative type. When the original image belongs to the preset negative type, a prompt message indicating that negative information is contained is generated to prompt the user to re-enter, or the original image is sent to the review end for manual review by the reviewer. When the review result indicates that the original image belongs to the preset negative type, a prompt message indicating that negative information is contained is generated to prompt the user to re-enter; when the original image does not belong to the preset negative type, the direction information in the metadata of the original image is obtained. According to the direction information, the original image is rotationally corrected to obtain the image to be processed. Among them, a picture expansion application can be run on the terminal, and the main interface of the picture expansion application can display an image upload control and an image drag control. The user can upload the original image by clicking the image upload control and selecting an image, or drag the selected image to the image drag control to upload it. The terminal can obtain the original image input by the user.

[0108] The terminal can perform border detection and cropping on the image to be processed. Specifically, the terminal can perform edge detection on the image to be processed to obtain an edge image; perform line segment extraction on the edge image to obtain multiple candidate line segments in the edge image; when there is a target line segment parallel to the edge of the edge image among the multiple candidate line segments, determine the border in the image to be processed based on the target line segment; crop the border in the image to be processed to obtain the image to be expanded. It can be understood that the border can be black, white or other colors. When no candidate line segment is obtained after line segment extraction, or the target line segment does not exist in the candidate line segment, or the border in the image to be processed is not found based on the target line segment, it can be determined that the image to be processed does not have a border, and the image to be processed can be determined as the image to be expanded.

[0109] The terminal can display the image to be expanded in the expansion interface of the expansion application, see Figure 5The schematic diagram of the image expansion interface is shown. In the image expansion interface, 501 may represent the image to be expanded, 502 may represent the size adjustment box, and the position of the size adjustment box may represent the position of the edge of the target image to be generated. It can be understood that in the image expansion interface, the size adjustment box can be adjusted according to the size selected in the "Screen Size" at the "Parameter Setting" in the image expansion interface and the image width and image height set at the "Width and Height". The size can also be adjusted by dragging the four corners of the size adjustment box. The user can determine the extended size information by the ratio selected in the screen size and the input width and height. For example, when the user selects "Free" (which can represent a free ratio) and enters a width of 766 and a height of 575, the width of 766*height of 575 can be used as the extended size information to determine the size of the target image to be generated as a width of 766*height of 575. It can be understood that the size of the first image is the same as the size of the target image.

[0110] The "prompt word" in the image expansion interface can be used to input the semantic text of the content you want to generate in the expanded area. The "intelligent association" control can expand the content of the semantic text entered by the user and update the semantic text. In the subsequent steps, the terminal can obtain the semantic text, combine the semantic text with the description text output by the BLIP model, and input the combined text into the CLIP model to form text features to guide the image generation process of the diffusion model.

[0111] This application is generated based on the diffusion model. Compared with the generative adversarial network, the diffusion model has stronger scalability and can support expansion areas with larger areas. The "Number of images generated" displayed in the expansion interface is used to set the number of images of the target image generated at the end, for example, 4 images can be supported. The "Generate Now" displayed in the expansion interface is used to trigger to start image generation, which can be understood as starting the following steps after triggering.

[0112] The terminal may expand the image to be expanded into a first image according to the expansion size information, wherein the first image includes an original image area retaining the content of the image to be expanded, and an expansion area whose content semantics is empty. The pixels of the expansion area may be zero pixels, and the expansion area may be displayed in black. In the expansion mask, each pixel in the area corresponding to the position of the original image area is a zero pixel, and each pixel in the area corresponding to the position of the expansion area is a non-zero and identical preset pixel.

[0113] The terminal may downsample the first image to obtain a downsampled image.

[0114] See Figure 6In the schematic diagram of the image generation framework shown in the figure, the terminal can obtain an expansion mask that matches the image size of the first image; through the trained conditional control network (ControlNet), feature extraction is performed according to the expansion mask to obtain control features; the downsampled image is input into the encoder in the trained autoencoder model to obtain the four-channel features output by the encoder, the four-channel features are denoised for multiple time steps to obtain noisy features, the expansion mask is downsampled to the image size of the downsampled image to obtain the downsampled mask, the features obtained by matrix multiplication of the downsampled mask and the noisy features are combined with the noisy features and the downsampled mask to obtain combined features; the image to be expanded is input into the trained multimodal model (BLIP model) to obtain a description text representing the content semantics in the image to be expanded, the description text is input into the trained text encoder (CLIP model) to obtain text features; the control features, the combined features and the text features are input into the trained diffusion model for image generation to obtain the output features of the diffusion model, the output features are input into the decoder in the autoencoder model to obtain the second image output by the decoder. The second image includes a first area corresponding to the original image area and a second area corresponding to the extended area; the content in the first area is consistent with the content in the original image area, and the content in the second area is an extension of the content in the first area and matches the description text.

[0115] The terminal may upsample the second image to the image size of the first image to obtain an upsampled image. Figure 7 As shown in the flowchart of the image fusion steps, the terminal can segment the image area corresponding to the extended area position from the upsampled image, and splice the image area corresponding to the extended area position in the upsampled image and the image to be expanded to obtain a spliced ​​image; the spliced ​​image is a multi-channel image; for each of the multiple channels of the spliced ​​image, the image area corresponding to the extended area position in the channel and the image area corresponding to the original image area position in the channel are Poisson fused to obtain a fused image corresponding to the channel; the fused images corresponding to each of the multiple channels are combined to obtain a target image.

[0116] See Figure 8 The image expansion result example schematic diagram shown in FIG. 1 may include the image expansion result as shown in FIG. Figure 5 The target image that matches the number of generated images entered in the image expansion interface shown in FIG. Figure 8 Four target images can be represented and generated. By clicking on different thumbnails, the corresponding target images can be displayed.

[0117] In the prior art, the method of directly inputting the original image into the diffusion model may generate an image with an obvious sense of pasting, that is, visually, it is like pasting the original image into a background, without a reasonable fusion visual effect. Compared with the method of directly inputting the original image into the diffusion model, the above-mentioned image expansion method can generate a visually natural and coherent target image with high image quality.

[0118] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0119] Based on the same inventive concept, the embodiment of the present application also provides an image expansion device for implementing the above-mentioned image expansion method. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above-mentioned method, so the specific limitations in one or more image expansion device embodiments provided below can refer to the above-mentioned limitations on the image expansion method, and will not be repeated here.

[0120] In an exemplary embodiment, Fig. 9 As shown, an image expansion device 900 is provided, comprising: a region expansion module 910, a downsampling module 920, a generation module 930, an upsampling module 940 and a fusion module 950, wherein:

[0121] The region expansion module 910 is used to expand the image to be expanded into a first image, where the first image includes an original image region that retains the content of the image to be expanded and an expansion region whose content semantics are empty.

[0122] The downsampling module 920 is configured to downsample the first image to obtain a downsampled image.

[0123] The generation module 930 is used to generate an image based on the downsampled image and the description text representing the semantics of the content in the image to be expanded, so as to obtain a second image; the second image includes a first area corresponding to the position of the original image area and a second area corresponding to the position of the expanded area; the content in the first area is consistent with the content in the original image area, and the content in the second area is content extended based on the content in the first area and matches the description text.

[0124] The up-sampling module 940 is configured to up-sample the second image to the image size of the first image to obtain an up-sampled image.

[0125] The fusion module 950 is used to fuse the image area corresponding to the position of the extended area in the up-sampled image with the image to be extended to obtain a target image.

[0126] In an exemplary embodiment, the image expansion device 900 also includes a preprocessing module, which is used to obtain the image to be processed, perform edge detection on the image to be processed, and obtain an edge image; perform line segment extraction on the edge image to obtain multiple candidate line segments in the edge image; when there is a target line segment parallel to the edge of the edge image among the multiple candidate line segments, determine the border in the image to be processed based on the target line segment; and crop the border in the image to be processed to obtain the image to be expanded.

[0127] In an exemplary embodiment, the preprocessing module is also used to obtain an input original image; when the original image meets a preset quality constraint and the original image does not belong to a preset negative type, obtain the direction information in the metadata of the original image; and perform rotation correction on the original image according to the direction information to obtain an image to be processed.

[0128] In an exemplary embodiment, the generation module 930 is also used to obtain an expansion mask that matches the image size of the first image; in the expansion mask, each pixel in the area corresponding to the original image area position is a zero pixel, and each pixel in the area corresponding to the expansion area position is a non-zero and identical preset pixel; through the trained conditional control network, feature extraction is performed according to the expansion mask to obtain control features; the control features are used to indicate that image generation is performed in the image area corresponding to the expansion area position in the downsampled image during the image generation process; based on the control features, the downsampled image, and the descriptive text representing the content semantics in the image to be expanded, image generation is performed through the trained diffusion model to obtain the second image.

[0129] In an exemplary embodiment, the generation module 930 is also used to add noise to the downsampled image to obtain noisy features; downsample the expansion mask to the image size of the downsampled image to obtain a downsampled mask; combine the features obtained by matrix multiplication of the downsampled mask and the noisy features with the noisy features and the downsampled mask to obtain combined features; generate an image through a trained diffusion model based on the control features, the combined features, and the descriptive text representing the content semantics in the image to be expanded to obtain a second image.

[0130] In an exemplary embodiment, the generation module 930 is also used to input the image to be expanded into a trained multimodal model to obtain a descriptive text that represents the semantics of the content in the image to be expanded; input the descriptive text into a trained text encoder to obtain text features; input the control features, combined features and text features into a trained diffusion model for image generation to obtain a second image.

[0131] In an exemplary embodiment, the fusion module 950 is also used to stitch the image area corresponding to the expansion area position in the upsampled image and the image to be expanded to obtain a stitched image; the stitched image is a multi-channel image; for each of the multiple channels of the stitched image, the image area corresponding to the expansion area position in the channel and the image area corresponding to the original image area position in the channel are Poisson fused to obtain a fused image corresponding to the channel; the fused images corresponding to each of the multiple channels are combined to obtain a target image.

[0132] Each module in the above image expansion device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0133] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Fig.10As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, an image expansion method is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0134] Those skilled in the art will understand that Fig.10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0135] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0136] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0137] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0138] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0139] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0140] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An image expansion method, characterized in that: The method comprises: Expanding the image to be expanded into a first image, wherein the first image includes an original image region retaining the content of the image to be expanded and an expanded region whose content semantics is empty; Downsampling the first image to obtain a downsampled image; An image is generated according to the downsampled image and the description text representing the semantics of the content in the image to be expanded, so as to obtain a second image; the second image comprises a first area corresponding to the position of the original image area and a second area corresponding to the position of the expanded area; the content in the first area is consistent with the content in the original image area, and the content in the second area is content extended from the content in the first area and matches the description text; Upsampling the second image to an image size of the first image to obtain an upsampled image; The image area corresponding to the position of the extended area in the up-sampled image is fused with the image to be extended to obtain a target image.

2. The method according to claim 1, characterized in that The method further comprises: Acquire an image to be processed, perform edge detection on the image to be processed, and obtain an edge image; Performing line segment extraction on the edge image to obtain a plurality of candidate line segments in the edge image; When there is a target line segment parallel to the edge of the edge image among the multiple candidate line segments, determining a border in the image to be processed based on the target line segment; The frame in the image to be processed is cropped to obtain the image to be expanded.

3. The method according to claim 2, characterized in that The obtaining of the image to be processed comprises: Get the input original image; When the original image meets a preset quality constraint condition and the original image does not belong to a preset negative type, obtaining direction information in metadata of the original image; According to the direction information, rotation correction is performed on the original image to obtain an image to be processed.

4. The method according to claim 1, characterized in that The step of generating an image according to the downsampled image and the description text representing the semantics of the content in the image to be expanded to obtain a second image includes: Acquire an extended mask that matches the image size of the first image; in the extended mask, each pixel in the area corresponding to the position of the original image area is a zero pixel, and each pixel in the area corresponding to the position of the extended area is a non-zero and identical preset pixel; Extracting features according to the extended mask through a trained conditional control network to obtain control features; the control features are used to indicate that image generation is performed in an image region corresponding to the position of the extended region in the downsampled image during image generation; According to the control feature, the downsampled image and the description text representing the semantics of the content in the image to be expanded, an image is generated through a trained diffusion model to obtain a second image.

5. The method according to claim 4, characterized in that The step of generating an image by using a trained diffusion model according to the control feature, the downsampled image, and the description text representing the semantics of the content in the image to be expanded to obtain a second image includes: Adding noise to the downsampled image to obtain a noisy feature; Downsampling the extended mask to an image size of the downsampled image to obtain a downsampling mask; Combining a feature obtained by performing a matrix multiplication operation on the downsampling mask and the noisy feature with the noisy feature and the downsampling mask to obtain a combined feature; According to the control feature, the combined feature and the description text representing the semantics of the content in the image to be expanded, an image is generated through a trained diffusion model to obtain a second image.

6. The method according to claim 5, characterized in that The step of generating an image by using a trained diffusion model according to the control feature, the combined feature, and the description text representing the semantics of the content in the image to be expanded to obtain a second image includes: Inputting the image to be expanded into a trained multimodal model to obtain a description text representing the semantics of the content in the image to be expanded; Inputting the description text into a trained text encoder to obtain text features; The control feature, the combined feature and the text feature are input into a trained diffusion model to generate an image, thereby obtaining a second image.

7. The method according to claim 1, characterized in that The step of fusing the image area corresponding to the position of the extended area in the up-sampled image with the image to be extended to obtain a target image includes: splicing the image area corresponding to the position of the extended area in the upsampled image and the image to be extended to obtain a spliced ​​image; the spliced ​​image is a multi-channel image; For each of the multiple channels of the stitched image, perform Poisson fusion on an image region in the channel corresponding to the position of the extended region and an image region in the channel corresponding to the position of the original image region to obtain a fused image corresponding to the channel; The fused images corresponding to the multiple channels are combined to obtain a target image.

8. An image expansion device, characterized in that: The device comprises: A region expansion module, used for expanding the image to be expanded into a first image, wherein the first image includes an original image region retaining the content of the image to be expanded and an expansion region whose content semantics is empty; A downsampling module, used for downsampling the first image to obtain a downsampled image; A generating module, configured to generate an image according to the downsampled image and the description text representing the semantics of the content in the image to be expanded, so as to obtain a second image; the second image comprises a first area corresponding to the position of the original image area and a second area corresponding to the position of the expanded area; the content in the first area is consistent with the content in the original image area, and the content in the second area is content extended from the content in the first area and matches the description text; an upsampling module, configured to upsample the second image to an image size of the first image to obtain an upsampled image; The fusion module is used to fuse the image area corresponding to the position of the extended area in the up-sampled image with the image to be extended to obtain a target image.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images