Image processing method and device, electronic equipment and readable storage medium

Abstract

The invention discloses an image processing method and device, electronic equipment and a readable storage medium, and belongs to the technical field of images. The image processing method comprises the steps of generating a third image based on a first image and a second image; based on the image edge information of the third image and the image semantic information of the third image, expanding the image content of the second image to generate a target image; wherein the first image is an image acquired by a first camera, the second image is an image acquired by a second camera, the first image and the second image are images of the same scene, the focal length of the first camera is smaller than that of the second camera, and the third image is an image generated after first image content in the first image is replaced by second image content; the second image content is the same as the first image content in the second image.

Description

Technical Field

[0001] The present application belongs to the field of image technology, and specifically relates to an image processing method, device, electronic device and readable storage medium. Background Art

[0002] With the development of electronic device technology, users often use camera applications of electronic devices to capture images. Currently, users usually can only capture and obtain images of a specific focal length at a time using electronic devices, such as images of a long focal length or an image of a short focal length.

[0003] In related technologies, in order to break through the limitations of traditional electronic device photography and achieve the effect of obtaining high-quality full-focal-length portraits in one shot, when electronic devices are shooting images, due to the limited number of cameras in electronic devices, electronic devices usually use a telephoto lens to shoot telephoto images with clearer picture quality, and then use a diffusion model (Stable Diffusion) to generate an extended image corresponding to the telephoto lens image, so as to obtain an image with a wider field of view.

[0004] Since the extended image generated by extending only the image taken at the telephoto stage is random, the accuracy of the generated extended image is low. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide an image processing method, device, electronic device and readable storage medium, which can ensure that the extended image corresponding to the image taken at the telephoto end maintains strict physical and semantic consistency with the image taken at the telephoto end, thereby improving the accuracy of the generated extended image.

[0006] In a first aspect, an embodiment of the present application provides an image processing method, which includes: generating a third image based on a first image and a second image; expanding the image content of the second image based on image edge information of the third image and image semantic information of the third image to generate a target image; wherein the first image is an image acquired by a first camera, the second image is an image acquired by a second camera, the first image and the second image are images of the same scene, the focal length of the first camera is smaller than that of the second camera, the third image is an image generated by replacing the first image content in the first image with the second image content, and the second image content is the image content in the second image that is the same as the first image content.

[0007] In a second aspect, an embodiment of the present application provides an image processing device, which includes: a processing module and a generation module; the processing module is used to generate a third image based on a first image and a second image; the generation module is used to expand the image content of the second image based on image edge information of the third image and image semantic information of the third image to generate a target image; wherein the first image is an image acquired by a first camera, the second image is an image acquired by a second camera, the first image and the second image are images of the same scene, the focal length of the first camera is smaller than that of the second camera, the third image is an image generated by replacing the first image content in the first image with the second image content, and the second image content is the image content in the second image that is the same as the first image content.

[0008] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0009] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0010] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0011] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect.

[0012] In an embodiment of the present application, the electronic device expands the image content of the second image based on the image edge information of the third image and the image semantic information of the third image to generate a target image; wherein the first image is an image acquired by the first camera, the second image is an image acquired by the second camera, the first image and the second image are images of the same scene, the focal length of the first camera is smaller than that of the second camera, the third image is an image generated by replacing the first image content in the first image with the second image content, and the second image content is the same image content as the first image content in the second image. Since the focal length of the first camera is smaller than that of the second camera, the first image acquired by the first camera is usually a short focal length image with a wider viewing angle but insufficient clarity, and the second image acquired by the second camera is usually a long focal length image with higher clarity but limited field of view. In this solution, the electronic device generates a third image with a wider field of view by acquiring a short focal length image and a long focal length image, and conditionally constrains the expanded content of the image content of the second image according to the image edge information and image semantic information of the third image, so that the generated target image has a high accuracy and ensures the semantic consistency between the second image and the target image. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is one of the schematic diagrams of an image processing method provided in an embodiment of the present application;

[0014] Figure 2 It is a schematic diagram of the composition of an adapter model provided in an embodiment of the present application;

[0015] Figure 3 This is a second schematic diagram of an image processing method provided in an embodiment of the present application;

[0016] Figure 4 It is a schematic diagram of a diffusion model structure provided in an embodiment of the present application;

[0017] Figure 5 is a specific example flow chart of an image processing method provided in an embodiment of the present application;

[0018] Figure 6 is a structural schematic diagram of an image processing device provided in an embodiment of the present application;

[0019] Figure 7 This is one of the hardware structure diagrams of an electronic device provided in an embodiment of the present application;

[0020] Figure 8 This is the second schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0022] Some terms and nouns involved in the embodiments of the present application are explained below.

[0023] Diffusion model: Stable Diffusion is a text-to-image diffusion model based on deep learning. It uses a method that combines diffusion models and autoencoders. The model uses autoencoders to convert images into representations of latent space, uses text encoders to process text, learns the correlation between the corresponding content of text and image through a cross-attention mechanism, and generates predicted images through a convolutional neural network (U-Net). It is trained on a large dataset containing images of various sizes and shapes. The diversity of this dataset allows Stable Diffusion to generate a variety of realistic images. And it makes it possible to generate new pixels that match the style and content of the original image.

[0024] ControlNet: ControlNet is a neural network designed to control the pre-trained Stable Diffusion model. It locks the parameters of Stable Diffusion and clones them into a trainable copy of ControlNet. It inputs control conditions, such as edge maps and segmentation maps, to control the final result of the Stable Diffusion model. The ControlNet-Outpainting model is trained on a large number of masked images, and the masked areas are generated by inference, which improves the repair ability of Stable Diffusion. Thus, new pixels are generated on the image, achieving seamless image expansion.

[0025] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0026] The terms "at least one (item)", "at least one of" and the like in the specification and claims of the present application refer to any one, any two or a combination of more than two of the objects included therein. For example, at least one (item) of a, b, and c can be represented by: "a", "b", "c", "a and b", "a and c", "b and c" and "a, b and c", where a, b, and c can be single or multiple. Similarly, "at least two (items)" refers to two or more, and its meaning is similar to that of "at least one (item)".

[0027] The logo in this application refers to text, symbols, images, etc. used to indicate information. Logos or other containers can be used as carriers to display information, including but not limited to text logos, image logos, symbol logos, etc.

[0028] It should be noted that the image processing method provided in the embodiments of the present application can be executed by electronic devices such as mobile phones, tablet computers, laptop computers, PDAs, and vehicle-mounted electronic devices. In some embodiments of the present application, the image processing method provided in the embodiments of the present application is described by taking the electronic device as the execution subject to execute the image processing method as an example.

[0029] At present, mainstream electronic devices on the market are equipped with multi-focal length cameras to meet the different shooting needs of users. Among them, the long-focal length camera has a narrower viewing angle and a smaller scene space, which is suitable for shooting distant scenes and can capture clearer details of the scene. The short-focal length camera has a wider viewing angle and scene space, can present a wider range of pictures, and can emphasize the perspective effect and enhance the appeal of the picture, thereby better meeting the diverse requirements of users for shooting needs.

[0030] In the related art, in order to break through the limitations of traditional electronic device photography and achieve the effect of obtaining high-quality full-focal-length portraits in one shot, when shooting images, due to the limited number of cameras in the electronic device, the electronic device will usually use a telephoto length to shoot a telephoto image with clearer picture quality, and then use an image expansion algorithm based on a diffusion model (Stable Diffusion) to generate an extended image corresponding to the telephoto image, so as to obtain an image with a wider field of view.

[0031] However, on the one hand, in the case of limited cameras in electronic devices, current electronic devices need to install multiple high-precision cameras with different focal lengths to directly implement the technology of cutting long focal lengths into short focal lengths, which places extremely high requirements on the hardware of electronic devices. On the other hand, although electronic devices can directly expand images based on the rated generative algorithm in Stable Diffusion, the generative algorithm in Stable Diffusion only directly infers short focal length images through existing long focal length images, and predicts images with a larger viewing angle. In this regard, due to the randomness of the generative algorithm, it is impossible to strictly guarantee the physical and semantic alignment of the content of the expanded area with the original image, resulting in insufficient accuracy of the expanded image.

[0032] In the image processing method provided in the embodiment of the present application, based on the image edge information of the third image and the image semantic information of the third image, the image content of the second image is expanded to generate a target image; wherein, the first image is an image acquired by the first camera, the second image is an image acquired by the second camera, the first image and the second image are images of the same scene, the focal length of the first camera is smaller than that of the second camera, the third image is an image generated by replacing the first image content in the first image with the second image content, and the second image content is the same image content as the first image content in the second image. Since the focal length of the first camera is smaller than that of the second camera, the first image acquired by the first camera is usually a short focal length image with a wider viewing angle but insufficient clarity, and the second image acquired by the second camera is usually a long focal length image with higher clarity but limited field of view. In this solution, the electronic device generates a third image with a wider field of view by acquiring a short focal length image and a long focal length image, and conditionally constrains the expanded content of the image content of the second image according to the image edge information and image semantic information of the third image, so that the accuracy of the generated target image is high, and the semantic consistency between the second image and the target image is ensured.

[0033] The image processing method, device, electronic device and readable storage medium provided by the embodiments of the present application are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0034] The execution subject of the image processing method provided in the embodiment of the present application may be an image processing device. Exemplarily, the image processing device may be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The image processing method provided in the embodiment of the present application will be exemplarily described below using an electronic device as an example.

[0035] The present application provides an image processing method. Figure 1 FIG. 1 is a flowchart of an image processing method provided by an embodiment of the present application, which can be applied to electronic devices. Figure 1As shown, the image processing method provided in the embodiment of the present application may include the following steps 201 and 202.

[0036] Step 201: The electronic device generates a third image based on the first image and the second image.

[0037] In some embodiments of the present application, the first image is an image captured by a first camera.

[0038] In some embodiments of the present application, the second image is an image captured by a second camera.

[0039] In some embodiments of the present application, the first image and the second image are images of the same scene acquired by different cameras.

[0040] In some embodiments of the present application, the focal length of the first camera is smaller than that of the second camera.

[0041] In some embodiments of the present application, the electronic device is an electronic device with a dual-camera system, that is, the electronic device has at least two cameras.

[0042] Exemplarily, if the first camera is a short focal length camera, then the first image is a short focal length image.

[0043] Exemplarily, if the second camera is a telephoto camera, then the second image is a telephoto image.

[0044] Exemplarily, the electronic device obtains a relatively blurry short focal length image and a high-definition long focal length image based on the first camera and the second camera in the camera dual-camera system. The long focal length image can be a 4x image with a clear picture, but the overall frame angle is small. The overall image quality of the short focal length image is blurry, but it has a larger frame angle and contains an extended image corresponding to the short focal length image generated from the long focal length image, that is, the extended picture content required for the target image described below, which provides the possibility for the generation of the target image.

[0045] In some embodiments of the present application, the third image is an image generated by replacing the first image content in the first image with the second image content.

[0046] In some embodiments of the present application, the second image content is the same image content as the first image content in the second image.

[0047] In some embodiments of the present application, the electronic device obtains key points in the first image and the second image, and then matches and aligns the key points of the first image with the key points of the second image one by one to obtain the first image content and the second image content, and then replaces the first image content with the second image content to finally generate a third image.

[0048] Step 202: The electronic device expands the image content of the second image based on the image edge information of the third image and the image semantic information of the third image to generate a target image.

[0049] In some embodiments of the present application, the above-mentioned image edge information is used to represent edge information of each image region in the third image.

[0050] In some embodiments of the present application, the above-mentioned image semantic information is used to represent image information in the third image.

[0051] In some embodiments of the present application, the electronic device inputs image edge information of the third image and image semantic information of the third image into a control condition model, extracts constraint information, and then inputs the second image and the constraint information into a diffusion model, and expands the image content of the second image according to the constraint information through the diffusion model to generate a target image.

[0052] In the image processing method provided in the embodiment of the present application, the electronic device expands the image content of the second image based on the image edge information of the third image and the image semantic information of the third image to generate a target image; wherein the first image is an image acquired by the first camera, the second image is an image acquired by the second camera, the first image and the second image are images of the same scene, the focal length of the first camera is smaller than that of the second camera, the third image is an image generated by replacing the first image content in the first image with the second image content, and the second image content is the image content in the second image that is the same as the first image content. Since the focal length of the first camera is smaller than that of the second camera, the first image acquired by the first camera is usually a short focal length image with a wider viewing angle but insufficient clarity, and the second image acquired by the second camera is usually a long focal length image with higher clarity but limited field of view. In this solution, the electronic device generates a third image with a wider field of view by acquiring a short focal length image and a long focal length image, and conditionally constrains the expanded content of the image content of the second image according to the image edge information and image semantic information of the third image, so that the accuracy of the generated target image is high, and the semantic consistency between the second image and the target image is ensured.

[0053] Optionally, in some embodiments of the present application, the above step 201 can be specifically implemented through the following steps 201a to 201c.

[0054] Step 201a: The electronic device obtains a first key point of the first image and obtains a second key point of the second image.

[0055] In some embodiments of the present application, the electronic device uses a key point detection algorithm to respectively obtain a first key point in the first image and a second key point in the second image.

[0056] Exemplarily, the key points may be key points corresponding to a main character in a character image, or may be key points corresponding to a main scene in a scene image.

[0057] Step 201b: The electronic device matches the first key point with the second key point to determine the first image content and the second image content.

[0058] In some embodiments of the present application, the electronic device matches the same first key point and second key point.

[0059] In other words, the electronic device performs overlapping matching on the first image and the second image, and determines the parts of the first image and the second image with the same image content as the first image content and the second image content.

[0060] For example, assuming that the first image and the second image are half-body images of Xiao Ming, the key points of the first image are the key points of Xiao Ming's face, and the key points of the second image are also the key points of Xiao Ming's face. The electronic device matches the key points of the left eye, the right eye, and the mouth of the two images one by one to determine the content of the first image and the content of the second image.

[0061] Step 201c: The electronic device replaces the first image content with the second image content to generate a third image.

[0062] Exemplarily, the electronic device performs image matching based on corresponding key points of the first image and the second image, and calculates a transformation matrix to determine the corresponding position of the first image in the coordinate system of the second image, such as M(macth_x, match_y, match_width, match_height), and then the electronic device replaces the image content in the first image position area M with the image content of the same coordinate position area in the second image to obtain the above-mentioned third image.

[0063] It can be understood that the third image obtained by the electronic device by completing the second image based on the first image has substantially the same structural information as the first image and the second image. Therefore, in the aligned third image, other field of view ranges of the second image can be completed based on the first image.

[0064] In this way, the electronic device can obtain the magnification ratio of the second image required for enlarging the second image according to the third image, so as to calculate the size of the target image to be finally generated.

[0065] Optionally, in some embodiments of the present application, before the above-mentioned step 202 "the electronic device expands the image content of the second image based on the image edge information of the third image and the image semantic information of the third image to generate a target image", the image processing method provided in some embodiments of the present application also includes the following steps 301 and 302.

[0066] Step 301: The electronic device detects a third image using an edge detection operator to obtain image edge information.

[0067] Exemplarily, the edge detection operator may be a Canny operator.

[0068] It can be understood that when the electronic device uses the AI ​​generative algorithm to predict the expanded area of ​​the second image, in order to ensure that the second image after expansion and filling has physical structure information and semantic information consistent with the first image, this embodiment uses the Canny edge detection operator to extract the edge information of the third image.

[0069] Step 302: The electronic device extracts image semantic information of the third image using the adapter model.

[0070] In some embodiments of the present application, the adapter model includes an IP-Adapter algorithm.

[0071] Exemplarily, the IP-Adapter algorithm is an effective and lightweight adapter for implementing the image prompting function of the pre-trained Stable Diffusion. The high-level semantic information in the image can be used to control the image generation of the Stable Diffusion diffusion model. The decoupled cross-attention mechanism it adopts separates the cross-attention layers of text features and image features, completes multi-modal image generation, and enables the image semantic information to be well used with the text prompt, so that the generated image has the semantic information of the third image.

[0072] It should be noted that the execution order of the above steps 301 and 302 can be step 301 first and then step 302, or step 302 first and then step 301, or they can be executed simultaneously, which is not limited in the embodiment of the present application.

[0073] In this way, the electronic device obtains image edge information and image semantic information to perform image expansion constraints on the process of generating an image, thereby ensuring the consistency of the physical structure information of the generated target image with the first image and the second image, thereby improving the accuracy of target image generation.

[0074] Optionally, in some embodiments of the present application, the above step 302 can be specifically implemented by the following steps 302a to 302d.

[0075] Step 302a: The electronic device inputs the third image and text prompt information into the adapter model.

[0076] In some embodiments of the present application, the above text prompt information is used to instruct the adapter model to extract image semantic information of the third image.

[0077] Step 302b: The electronic device extracts the first image feature information of the third image and the first text feature information of the text prompt information respectively through the cross attention layer in the adapter model.

[0078] In some embodiments of the present application, the above adapter model is used to extract image semantic information of an image.

[0079] In some embodiments of the present application, the above-mentioned adapter model includes at least two cross-attention layers.

[0080] Exemplarily, the electronic device extracts first image feature information of the third image through a first cross-attention layer, and extracts first text feature information of the text prompt information through a second cross-attention layer.

[0081] Exemplarily, after the electronic device inputs the above-mentioned text prompt information and the third image into the adapter model, the first image feature information of the third image and the first text feature information of the text prompt information are respectively extracted through different cross-attention layers in the adapter model.

[0082] Step 302c: The electronic device generates image semantic information based on the first image feature information and the first text feature information through the adapter model.

[0083] Step 302d: The electronic device outputs image semantic information through the adapter model.

[0084] In some embodiments of the present application, the electronic device processes the first image feature information and the second text feature information through the layers in the adapter model to generate image semantic information of the third image, and finally the electronic device outputs the image semantic information through the adapter model.

[0085] For example, Figure 2As shown, the adapter model includes an image encoder, a text encoder, layer normalization, a fully connected layer, and a decoupled cross attention layer, and the third image and text prompt information are input into the adapter model. The image feature information of the image is extracted through the image encoder, layer normalization, and the fully connected layer, and the first image feature information is further extracted through the decoupled cross attention layer. At the same time, the text feature information is extracted through the text encoder, and the first text feature information is further extracted through the decoupled cross attention layer. Among them, the decoupled cross attention layer performs denoising on the first image feature information and the first text feature information through different levels to obtain the final image semantic information.

[0086] In this way, the electronic device processes the cross-attention layers of text features and image features separately through the decoupled cross-attention mechanism in the adapter model, completing multimodal image generation, so that the image semantic information and text prompts can be used well together, thereby extracting the image semantic information of the third image.

[0087] Optionally, in some embodiments of the present application, combined with Figure 1 ,like Figure 3 As shown, the above step 202 "the electronic device performs image expansion processing on the second image based on the image edge information of the third image and the image semantic information of the third image to generate the target image" can be specifically implemented by following the steps 202a to 202d.

[0088] Step 202a: The electronic device inputs the image edge information and the image semantic information into the control condition model for processing, and outputs the first image condition constraint information.

[0089] In some embodiments of the present application, the control condition model may be a ControlNet model, which extracts feature information of image edge information and image semantic information through encoding blocks as the first image condition constraint information.

[0090] In some embodiments of the present application, the above-mentioned extracted image edge information and the image semantic information extracted by the IP-Adapter are used as the input of the ControlNet model, and based on a Stable Diffusion model with strong generation capability, for example, SDXL expands the second image to generate a target image, and makes the target image physically and semantically consistent with the first image and the second image.

[0091] Step 202b: The electronic device inputs the first image condition constraint information and the second image into the diffusion model.

[0092] In some embodiments of the present application, the first image condition constraint information is used to constrain the diffusion model to perform an expansion process on the second image, so that the target image generated by the expansion maintains semantic and physical consistency with the first image and the second image.

[0093] Step 202c: the electronic device performs image expansion processing on the second image based on the first image condition constraint information by using a diffusion model to generate a target image.

[0094] Step 202d: The electronic device outputs the target image through the diffusion model.

[0095] For example, Figure 4 As shown, the embodiment of the present application uses Stable Diffusion (a) as an example to show how ControlNet (b) adds conditional control to a large pre-trained diffusion model. Stable diffusion is essentially a neural network model with an encoder, an intermediate block, and a decoder with jump connections. Both the encoder and the decoder contain 12 blocks, and the complete model contains 25 blocks, including the intermediate blocks. Of the 25 blocks, 8 blocks are downsampling or upsampling convolutional layers, and the other 17 blocks are main blocks, each of which contains 4 resnet layers and 2 visual transformers (ViT). Each ViT contains multiple cross-attention and self-attention mechanisms.

[0096] For example, refer to Figure 3 , “SD Encoder Block A” contains 4 resnet layers and 2 ViTs, and “×3” means that the block is repeated 3 times. The textual cues are encoded using the CLIP text encoder, and the diffusion time steps are encoded using a temporal encoder using positional encoding. The ControlNet structure is applied to each encoder level of the U-Net.

[0097] In particular, ControlNet is used in this embodiment to create trainable copies of 12 encoding blocks and 1 Stable Diffusion intermediate block. The 12 encoding blocks have 4 resolutions (64×64, 32×32, 16×16, 8×8), each of which is replicated 3 times. The output is added to 12 skip connections and 1 intermediate block of U-Net. Since stable diffusion is a typical U-Net structure, this ControlNet architecture may be applicable to other models. The way we connect ControlNet is computationally efficient-since the locked replica parameters are frozen, no gradient calculation is required in the initially locked encoder for fine-tuning. This approach can speed up training and save GPU memory. According to tests on a single NVIDIA A100 PCIE40GB, optimizing stable diffusion with ControlNet only requires about 23% of GPU memory and 34% of memory, and each training iteration takes more time than optimizing stable diffusion without ControlNet. The image diffusion model learns to gradually denoise the image and generate samples from the training domain. The denoising process can occur in pixel space or in the latent space encoded from the training data. Stable Diffusion uses latent images as the training domain because working in this space has been shown to stabilize the training process. Specifically, Stable Diffusion uses a preprocessing method similar to VQ-GAN to convert 512×512 pixel space images into smaller 64×64 latent images. In order to add ControlNet to Stable Diffusion, we first convert each input adjustment image (such as edges, poses, depth, etc.) from a 512×512 input size to a 64×64 feature space vector that matches the size of Stable Diffusion, that is, the first image constraint information mentioned above, and then input this feature space vector into Stable Diffusion so that Stable Diffusion can generate the target image based on the feature space vector.

[0098] In this way, the electronic device introduces the image edge information and image semantic information of the third image to constrain the expansion process of generating the target image, so that the expanded image can better maintain the original edge structure consistency and semantic consistency of the image, so that the target image has a better structure and realism while also better maintaining the overall structural information of the original image, avoiding the generation of characters or debris irrelevant to the original image environment, thereby improving the accuracy of the generated target image.

[0099] The image processing method provided in the embodiments of the present application is exemplarily described below with specific examples.

[0100] Example 1: Figure 5As shown, the image processing method may include the following steps A1 to A3.

[0101] Step A1: First, based on the dual-camera system of the electronic device, a high-definition long-focus main camera image, i.e., the above-mentioned second image, and a relatively blurry short-focus secondary camera image, i.e., the above-mentioned first image, are obtained, and then the main camera image and the secondary camera image are matched and aligned to generate a main-secondary camera aligned image, i.e., the above-mentioned third image.

[0102] Step A2: The electronic device extracts image edge information and image semantic information of the primary and secondary camera aligned images respectively through an edge acquisition method and an adapter model.

[0103] Step A3: The electronic device expands the image content of the main camera image by inputting the image edge information, image semantic information and the main camera image of the main and secondary camera alignment images into the diffusion model to generate a short-long focal length image, that is, the above-mentioned target image.

[0104] Thus, through the embodiment of the present application, the focal length of the photographed photo can be switched without the need for additional hardware support, ensuring the physical and semantic consistency between the extended image predicted by the Stable Diffusion generative algorithm and the original image, and achieving the effect of switching from long focal length to short focal length for the photographed scene. This improves the post-processing limitations of traditional mobile phone photography on the photographed scene, while improving the quality of the generated pictures and the satisfaction with the finished film, and reducing the user's post-processing costs.

[0105] Each of the above-mentioned method embodiments, or various possible implementation methods in each method embodiment can be executed separately, or any two or more of them can be executed in combination with each other. The specific implementation method can be determined according to actual usage requirements, and the embodiments of the present application do not limit this.

[0106] The window fuzzy processing method provided in the embodiment of the present application can be executed by an electronic device or a window fuzzy processing device. In the embodiment of the present application, the window fuzzy processing device executing the window fuzzy processing method is taken as an example to illustrate the window fuzzy processing device provided in the embodiment of the present application.

[0107] Figure 6 A possible structural diagram of an image processing device involved in an embodiment of the present application is shown. Figure 6 As shown, the image processing device 700 may include: a processing module 701 and a generating module 702 .

[0108] Among them, the above-mentioned processing module 701 is used to generate a third image based on the first image and the second image; the above-mentioned generation module 702 is used to expand the image content of the second image based on the image edge information of the third image and the image semantic information of the third image to generate a target image; wherein the first image is an image acquired by the first camera, the second image is an image acquired by the second camera, the first image and the second image are images of the same scene, the focal length of the first camera is smaller than that of the second camera, the third image is an image generated by replacing the first image content in the first image with the second image content, and the second image content is the image content in the second image that is the same as the first image content.

[0109] Optionally, in some embodiments of the present application, the processing module 701 is specifically used to:

[0110] Acquire a first key point of the first image, and acquire a second key point of the second image;

[0111] Matching the first key point with the second key point to determine the first image content and the second image content;

[0112] The first image content is replaced with the second image content to generate a third image.

[0113] Optionally, in some embodiments of the present application, the processing module 701 is also used by the generation module 702 to expand the image content of the second image based on the image edge information of the third image and the image semantic information of the third image, and before generating the target image, use an edge detection operator to detect the third image to obtain the image edge information; the processing module 701 is also used to extract the image semantic information of the third image using an adapter model.

[0114] Optionally, in some embodiments of the present application, the processing module 701 is specifically used to:

[0115] inputting the third image and the text prompt information into the adapter model, wherein the text prompt information is used to instruct the adapter model to extract image semantic information of the third image;

[0116] extracting first image feature information of the third image and first text feature information of the text prompt information respectively through a cross attention layer in the adapter model;

[0117] Generate image semantic information based on the first image feature information and the first text feature information through the adapter model;

[0118] Output image semantic information through the adapter model.

[0119] Optionally, in some embodiments of the present application, the generating module 702 is specifically used to:

[0120] Inputting the image edge information and the image semantic information into the control condition model for processing, and outputting the first image condition constraint information;

[0121] Inputting the first image condition constraint information and the second image into the diffusion model;

[0122] The second image is expanded based on the condition constraint information of the first image by using a diffusion model to generate a target image;

[0123] Output the target image through the diffusion model.

[0124] In the image processing device provided in the embodiment of the present application, the image processing device expands the image content of the second image based on the image edge information of the third image and the image semantic information of the third image to generate a target image; wherein the first image is an image acquired by the first camera, the second image is an image acquired by the second camera, the first image and the second image are images of the same scene, the focal length of the first camera is smaller than that of the second camera, the third image is an image generated by replacing the first image content in the first image with the second image content, and the second image content is the image content in the second image that is the same as the first image content. Since the focal length of the first camera is smaller than that of the second camera, the first image acquired by the first camera is usually a short focal length image with a wider viewing angle but insufficient clarity, and the second image acquired by the second camera is usually a long focal length image with higher clarity but limited field of view. In this solution, the image processing device generates a third image with a wider field of view by acquiring a short focal length image and a long focal length image, and conditionally constrains the expanded content of the image content of the second image according to the image edge information and image semantic information of the third image, so that the accuracy of the generated target image is high, and the semantic consistency between the second image and the target image is ensured.

[0125] The image processing device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0126] The image processing device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0127] The image processing device provided in the embodiment of the present application can implement each process implemented in the image processing method embodiment and achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0128] Alternatively, if Figure 7 As shown, an embodiment of the present application further provides an electronic device 800, including a processor 801 and a memory 802, wherein the memory 802 stores programs or instructions that can be executed on the processor 801, and when the program or instructions are executed by the processor 801, the various steps of the above-mentioned image processing method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0129] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0130] Figure 8 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0131] The electronic device 100 includes but is not limited to components such as a radio frequency unit 101, a network module 102, an audio output unit 103, an input unit 104, a sensor 105, a display unit 106, a user input unit 107, an interface unit 108, a memory 109, and a processor 110.

[0132] Those skilled in the art will appreciate that the electronic device 100 may also include a power source (such as a battery) for supplying power to various components, and the power source may be logically connected to the processor 110 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Figure 8 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0133] The processor 110 is used to generate a third image based on the first image and the second image; the processor 110 is used to expand the image content of the second image based on the image edge information of the third image and the image semantic information of the third image to generate a target image; the first image is an image acquired by a first camera, the second image is an image acquired by a second camera, the first image and the second image are images of the same scene, the focal length of the first camera is smaller than that of the second camera, the third image is an image generated by replacing the first image content in the first image with the second image content, and the second image content is the image content in the second image that is the same as the first image content.

[0134] Optionally, in some embodiments of the present application, the processor 110 is specifically configured to:

[0135] Acquire a first key point of the first image, and acquire a second key point of the second image;

[0136] Matching the first key point with the second key point to determine the first image content and the second image content;

[0137] The first image content is replaced with the second image content to generate a third image.

[0138] Optionally, in some embodiments of the present application, the processor 110 is further used to expand the image content of the second image based on the image edge information of the third image and the image semantic information of the third image, and before generating the target image, use an edge detection operator to detect the third image to obtain the image edge information; the processor 110 is also used to extract the image semantic information of the third image using an adapter model.

[0139] Optionally, in some embodiments of the present application, the processor 110 is specifically configured to:

[0140] inputting the third image and the text prompt information into the adapter model, wherein the text prompt information is used to instruct the adapter model to extract image semantic information of the third image;

[0141] extracting first image feature information of the third image and first text feature information of the text prompt information respectively through a cross attention layer in the adapter model;

[0142] Generate image semantic information based on the first image feature information and the first text feature information through the adapter model;

[0143] Output image semantic information through the adapter model.

[0144] Optionally, in some embodiments of the present application, the processor 110 is specifically configured to:

[0145] Inputting the image edge information and the image semantic information into the control condition model for processing, and outputting the first image condition constraint information;

[0146] Inputting the first image condition constraint information and the second image into the diffusion model;

[0147] The second image is expanded based on the condition constraint information of the first image by using a diffusion model to generate a target image;

[0148] Output the target image through the diffusion model.

[0149] In the electronic device provided in the embodiment of the present application, the electronic device expands the image content of the second image based on the image edge information of the third image and the image semantic information of the third image to generate a target image; wherein the first image is an image acquired by the first camera, the second image is an image acquired by the second camera, the first image and the second image are images of the same scene, the focal length of the first camera is smaller than that of the second camera, the third image is an image generated by replacing the first image content in the first image with the second image content, and the second image content is the same image content as the first image content in the second image. Since the focal length of the first camera is smaller than that of the second camera, the first image acquired by the first camera is usually a short focal length image with a wider viewing angle but insufficient clarity, and the second image acquired by the second camera is usually a long focal length image with higher clarity but limited field of view. In this solution, the electronic device generates a third image with a wider field of view by acquiring a short focal length image and a long focal length image, and conditionally constrains the expanded content of the image content of the second image according to the image edge information and image semantic information of the third image, so that the accuracy of the generated target image is high, and the semantic consistency between the second image and the target image is ensured.

[0150] It should be understood that in the embodiment of the present application, the input unit 104 may include a graphics processor (Graphics Processing Unit, GPU) 1041 and a microphone 1042, and the graphics processor 1041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 106 may include a display panel 1061, and the display panel 1061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 107 includes a touch panel 1071 and at least one of other input devices 1072. The touch panel 1071 is also called a touch screen. The touch panel 1071 may include two parts: a touch detection device and a touch controller. Other input devices 1072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0151] The memory 109 can be used to store software programs and various data. The memory 109 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 109 may include a volatile memory or a non-volatile memory, or the memory 109 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 109 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0152] The processor 110 may include one or more processing units; optionally, the processor 110 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 110.

[0153] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned image processing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0154] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0155] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned image processing method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0156] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0157] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned image processing method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0158] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0159] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0160] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Based on the first image and the second image, generating a third image; Based on the image edge information of the third image and the image semantic information of the third image, the image content of the second image is expanded to generate a target image; Among them, the first image is an image acquired by a first camera, the second image is an image acquired by a second camera, the first image and the second image are images of the same scene, the focal length of the first camera is smaller than that of the second camera, the third image is an image generated by replacing the first image content in the first image with the second image content, and the second image content is the image content in the second image that is the same as the first image content.

2. The method according to claim 1, characterized in that The step of generating a third image based on the first image and the second image comprises: Acquire a first key point of the first image, and acquire a second key point of the second image; Matching the first key point with the second key point to determine the first image content and the second image content; The first image content is replaced with the second image content to generate a third image.

3. The method according to claim 1, characterized in that Before expanding the image content of the second image based on the image edge information of the third image and the image semantic information of the third image to generate the target image, the method further includes: Using an edge detection operator to detect the third image to obtain image edge information of the third image; An adapter model is used to acquire image semantic information of the third image.

4. The method according to claim 3, characterized in that The adopting the adapter model to obtain the image semantic information of the third image includes: generating the image semantic information based on the first image feature information and the first text feature information through the adapter model; The image semantic information is outputted through the adapter model.

5. The method according to claim 3, characterized in that: Expanding the image content of the second image based on the image edge information of the third image and the image semantic information of the third image to generate a target image includes: Inputting the image edge information and the image semantic information into a control condition model for processing, and outputting first image condition constraint information; Inputting the first image condition constraint information and the second image into a diffusion model; Expanding the image content of the second image based on the first image condition constraint information by using the diffusion model to generate the target image; The target image is outputted through the diffusion model.

6. An image processing device, characterized in that: The image processing device comprises: a processing module and a generating module; The processing module is used to generate a third image based on the first image and the second image; The generating module is used to expand the image content of the second image based on the image edge information of the third image processed by the processing module and the image semantic information of the third image processed by the processing module to generate a target image; Among them, the first image is an image acquired by a first camera, the second image is an image acquired by a second camera, the first image and the second image are images of the same scene, the focal length of the first camera is smaller than that of the second camera, the third image is an image generated by replacing the first image content in the first image with the second image content, and the second image content is the image content in the second image that is the same as the first image content.

7. The device according to claim 6, characterized in that The processing module is specifically used for: Acquire a first key point of the first image, and acquire a second key point of the second image; Matching the first key point with the second key point to determine the first image content and the second image content; The first image content is replaced with the second image content to generate the third image.

8. The device according to claim 6, characterized in that The processing module is further used for the generation module to expand the image content of the second image based on the image edge information of the third image and the image semantic information of the third image, and before generating the target image, use an edge detection operator to detect the third image to obtain the image edge information of the third image; The processing module is further configured to extract image semantic information of the third image by using an adapter model.

9. The device according to claim 8, characterized in that The processing module is specifically used for: inputting the third image and text prompt information into the adapter model, wherein the text prompt information is used to instruct the adapter model to extract image semantic information of the third image; respectively extracting first image feature information of the third image and first text feature information of the text prompt information through a cross attention layer in the adapter model; generating the image semantic information based on the first image feature information and the first text feature information through the adapter model; The image semantic information is outputted through the adapter model.

10. The device according to claim 8, characterized in that The generating module is specifically used for: Inputting the image edge information and the image semantic information into a control condition model for processing, and outputting first image condition constraint information; Inputting the first image condition constraint information and the second image into a diffusion model; Expanding the image content of the second image based on the first image condition constraint information by using the diffusion model to generate the target image; The target image is outputted through the diffusion model.

11. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the image processing method according to any one of claims 1 to 5 are implemented.

12. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the image processing method according to any one of claims 1 to 5 are implemented.

13. A computer program product, wherein the computer program product is stored in a storage medium and is executed by at least one processor to implement the steps of the image processing method according to any one of claims 1 to 5.

Images