Image processing method and device, electronic equipment and storage medium

Abstract

The present disclosure relates to an image processing method, device, electronic equipment and storage medium, the method comprising: acquiring a first image; the first image comprising a first object; performing an expansion processing on the periphery of the first image to obtain a second image; adding a second object to a region adjacent to the first object in the second image to obtain a third image; generating a target video with the first image as the first frame image and the third image as the last frame image; the target video reflecting a process of evolving from the first image to the third image, accompanied by dynamic effects of the first object and / or the second object. The essence is to give a brand new special effect: the image content of the first image gradually shrinks, and at the same time gradually changes from a state without second object decoration to a state with second object decoration, and in the changing process, the first object and / or the second object presents a visual effect of a specific action, which can meet the diversified needs of users for special effects and provide a richer visual experience.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to an image processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the rapid development of information technology, image generation technology has made significant progress, which has not only greatly promoted innovation in many industries such as entertainment, advertising, and education, but also provided users with unprecedented creative freedom and ways of expression.

[0003] Despite significant advancements in image generation technology, limitations remain in certain areas, particularly in adding special effects. Specifically, some software and applications on the market currently offer a limited variety of effects, failing to meet the diverse needs of users. Summary of the Invention

[0004] In order to solve the above-mentioned technical problems, or at least partially solve the above-mentioned technical problems, this disclosure provides an image processing method, apparatus, electronic device, and storage medium.

[0005] In a first aspect, this disclosure provides an image processing method, including:

[0006] Obtain a first image; the first image includes a first object:

[0007] The first image is enlarged around its edges to obtain the second image;

[0008] A second object is added to the region adjacent to the first object in the second image to obtain a third image;

[0009] A target video is generated using the first image as the first frame and the third image as the last frame; the target video reflects the process of the first image evolving into the third image, and is accompanied by dynamic effects of the first object and / or the second object.

[0010] Secondly, this disclosure also provides an image processing apparatus, comprising:

[0011] The acquisition module is used to acquire a first image; the first image includes a first object:

[0012] The image expansion module is used to expand the image around the first image to obtain a second image.

[0013] An adding module is used to add a second object to the region adjacent to the first object in the second image to obtain a third image;

[0014] The generation module is used to generate a target video using the first image as the first frame and the third image as the last frame; the target video reflects the process of the first image evolving into the third image, and is accompanied by dynamic effects of the first object and / or the second object.

[0015] Thirdly, this disclosure also provides an electronic device, the electronic device comprising:

[0016] One or more processors;

[0017] Storage device for storing one or more programs;

[0018] When the one or more programs are executed by the one or more processors, the one or more processors implement the image processing method as described above.

[0019] Fourthly, this disclosure also provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the image processing method described above.

[0020] The technical solution provided in this disclosure has the following advantages compared with the prior art:

[0021] The technical solution provided in this disclosure involves setting up a first image; the first image includes a first object; expanding the area around the first image to obtain a second image; adding a second object to the area adjacent to the first object in the second image to obtain a third image; generating a target video using the first image as the first frame and the third image as the last frame; the target video reflects the process of the first image evolving into the third image, accompanied by dynamic effects of the first object and / or the second object. Essentially, it provides a novel special effect. This effect presents the following: the image content of the first image gradually shrinks, while simultaneously transitioning from a state without the second object to a state with the second object, and during the transition, the first object and / or the second object exhibit specific visual effects, which can meet the diverse needs of users for special effects and provide a richer visual experience. Attached Figure Description

[0022] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0023] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0024] Figure 1 A flowchart of an image processing method provided in an embodiment of this disclosure;

[0025] Figure 2 A schematic diagram of a first image provided for an embodiment of this disclosure;

[0026] Figure 3 A schematic diagram of a second image provided for an embodiment of this disclosure;

[0027] Figure 4 A schematic diagram of a third image provided for an embodiment of this disclosure;

[0028] Figure 5 This is a schematic diagram of the structure of an image modification model provided in an embodiment of the present disclosure;

[0029] Figures 6-7 A schematic diagram illustrating the training process of a visual language model and a noise reduction network provided in this embodiment of the disclosure;

[0030] Figure 8 A schematic diagram illustrating the principle of generating original sample images and modified sample images according to embodiments of this disclosure;

[0031] Figure 9 This is a schematic diagram of the structure of an image processing apparatus according to an embodiment of the present disclosure;

[0032] Figure 10 This is a schematic diagram of the structure of an electronic device according to an embodiment of this disclosure. Detailed Implementation

[0033] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0034] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0035] Figure 1 A flowchart illustrating an image processing method provided in this disclosure embodiment. Figure 2This is a data flow diagram of an image processing method provided in an embodiment of this disclosure. This embodiment is applicable to situations where image processing is performed on a client side. The method can be executed by an image processing device, which can be implemented in software and / or hardware. This device can be configured in an electronic device, such as a terminal, including but not limited to smartphones, PDAs, tablets, wearable devices with displays, desktop computers, laptops, all-in-one computers, smart home devices, etc. Alternatively, this embodiment is applicable to situations where image processing is performed on a server side. The method can be executed by an image processing device, which can be implemented in software and / or hardware. This device can be configured in an electronic device, such as a server.

[0036] like Figure 1 The method may specifically include:

[0037] S110. Obtain the first image; the first image includes the first object.

[0038] The first image can be, for example, an image to which special effects need to be added. The first image can be specified by the user; it can be an image taken by the user or an image downloaded from the internet.

[0039] The first object can be, for example, an object in the first image that needs to be focused on or highlighted. This application does not limit the specific things that the first object refers to. In some scenarios, the first object can be, for example, a person, animal, plant, building, or other object. Furthermore, the number of first objects included in the first image of this application is not limited. For example, the number of first objects can be one, two, or more.

[0040] S120. Expand the image around the first image to obtain the second image.

[0041] The first image is expanded around its perimeter, for example, by adding image content in the top, bottom, left, and right directions. The second image is the result of expanding the perimeter of the first image. It is important to emphasize that the added image content during the expansion process is related to the content of the first image, and not to irrelevant content such as frames.

[0042] For example, with Figure 2 For the first image, Figure 3 To Figure 2 The second image after image enlargement processing. (Comparison) Figure 2 and Figure 3 It can be observed that the image enlargement process adds background and parts of the person's limbs to the second image. The added background and... Figure 2The background blends seamlessly with the background, collectively forming the shape of the room where the character is located. Newly added limbs of the character (including forearms, arms, abdomen, etc.) also blend with... Figure 2 The limbs of the figure (including the head, chest, and upper arms) can be naturally connected to form the figure.

[0043] S130. Add a second object to the region adjacent to the first object in the second image to obtain a third image.

[0044] The second object can be, for example, something not included in the first image. Optionally, the second object has a certain relationship with the first object. Specifically, to a certain extent, the second object can be regarded as something that decorates the first object. For example, Figure 4 To Figure 3 The third image obtained after adding the second object. Figure 4 In the text, the second object specifically refers to the wings.

[0045] There are multiple ways to implement this step, and this application does not limit this one. For example, the implementation method of this step may include: obtaining image modification prompt information; the image modification prompt information includes a description of the second object; based on the image modification prompt information, modifying the second image using an image modification model to obtain a third image.

[0046] Image modification prompts can be, for example, information on how to modify the second image. They can be in the form of instructions or descriptions. Instruction-style text, for example, could be a clear instruction to add a second object to a specific area or location in the second image. Description-style text, for example, could be text describing the content or visual effect of the desired modified image. For instance, assuming the second object is wings, if the image modification prompt is in the form of instructions, it could be "Add white wings to the left and right sides of a cartoon character." If the image modification prompt is in the form of description, it could be "A cartoon character with white wings."

[0047] Image modification models can be, for example, pre-trained models designed for modifying images.

[0048] S140. Using the first image as the first frame and the third image as the last frame, generate a target video; the target video reflects the process of evolving from the first image to the third image, and is accompanied by dynamic effects of the first object and / or the second object.

[0049] The target video is the video finally generated using the technical solution of this application, which can be regarded as the result of adding special effects to the original image.

[0050] The target video reflects the process of evolving from a first image to a third image, accompanied by dynamic effects of a first object and / or a second object. Specifically, during the playback of the target video, as time progresses, the first object gradually shrinks, the first object and / or the background gradually become richer, and a second object gradually appears. During this process, the posture, movement, etc., of the first object and / or the second object in the first image may change.

[0051] For example, suppose that with Figure 2 As the first image, with Figure 4 In the third image, within the generated target video, the figure gradually shrinks as time progresses. As the figure shrinks, limbs and parts of the background image, not initially included in the first image, gradually come into view. Since the first image can be viewed as wings folded and hidden behind the figure, as the figure shrinks, the wings gradually unfold from their folded state to the winged state shown in the third image, and the wings flap a few times during this unfolding process. Furthermore, the figure's movements (such as facial and body movements) may also change as the figure shrinks.

[0052] The above technical solution involves obtaining a first image, which includes a first object; expanding the first image around its perimeter to obtain a second image; adding a second object to the area adjacent to the first object in the second image to obtain a third image; and generating a target video using the first image as the first frame and the third image as the last frame. The target video reflects the process of the first image evolving into the third image, accompanied by dynamic effects of the first and / or second objects. Essentially, it provides a novel special effect. This effect presents the following: the first image gradually shrinks, while simultaneously transitioning from a state without the second object to a state with the second object, and during the transition, the first and / or second objects exhibit specific visual effects, satisfying users' diverse needs for special effects and providing a richer visual experience.

[0053] Based on the above technical solution, optionally, S120 may include: cropping the first image to obtain a fourth image; in the fourth image, the area ratio of the first object in the fourth image is greater than or equal to a set threshold; expanding the fourth image around it to obtain a second image; S140 may include: using the fourth image as the first frame image and the third image as the last frame image to generate a target video; the target video reflects the process of the fourth image evolving into the third image.

[0054] The threshold can be a pre-specified reference value. When the area of ​​the first object in the fourth image is greater than or equal to the threshold, it means that the size of the first object is large enough in the fourth image to be very prominent and easily observed by the user. Conversely, when the area of ​​the first object in the fourth image is less than the threshold, it means that the size of the first object is small in the fourth image, making it inconspicuous and difficult for the user to observe quickly.

[0055] By cropping the first image before image expansion, the first object can be made more prominent and easier to identify. Using the cropped fourth image as the first frame to generate the target video further enhances the prominence and ease of identification of the first object within the generated video.

[0056] Based on the above technical solution, optionally, S140 may include: determining a target filter based on the image features of the second object in the third image; adding the target filter to the second object in the third image to obtain a fifth image; and generating a target video with the first image as the first frame image and the fifth image as the last frame image.

[0057] Image features of the second object could be, for example, the color, shape, or contrast between the second object and the background. Here, the background could be, for example, the remaining image in the third image excluding the first and second objects.

[0058] By adding a target filter to the second object, the image quality of the second object can be improved, its prominence enhanced, and its recognizability increased.

[0059] This application does not limit the specific structure of the image modification model mentioned in the above technical solutions. For example, see... Figure 5 The image modification model includes a visual language model and a denoising network; the visual language model is the result of fusing the visual language base model and the visual language fine-tuning model; the denoising network is the result of fusing the denoising base model and the denoising fine-tuning model; both the visual language fine-tuning model and the denoising fine-tuning model correspond to the first style.

[0060] The first style can be, for example, a specific image art style. This application does not limit what style specifically refers to. For example, the style can be comic style, pixel style, cartoon style, clay style, oil painting style, etc. The fact that both the visual language fine-tuning model and the noise reduction fine-tuning model correspond to the first style means that both the visual language fine-tuning model and the noise reduction fine-tuning model correspond to the same style.

[0061] A visual language model is a model that combines computer vision and natural language processing capabilities. It can accept and understand the relationship between visual content (such as images) and linguistic content (such as text), and encode the results of the understanding before outputting them.

[0062] Conventional image generation models include a text encoder and a denoising network. The text encoder can encode any text to obtain corresponding text features. The text features output by the text encoder are used as a control signal input to the denoising network. The denoising network (also called a backdiffusion module or backdiffusion network) uses the text features output by the text encoder as a control signal to flexibly adjust the backdiffusion process (i.e., the process of predicting noise). Then, by removing the predicted noise from the noisy image, the final generated image matches the text description, thus semantically associating the generated image with the textual prompts and making the generated image meet the user's needs.

[0063] Conventional image generation models, particularly text encoders, cannot understand the relationship between visual content (such as images) and linguistic content (such as text). In this application, the image modification model can be viewed as a model obtained by replacing the text encoder in a conventional image generation model with a visual-language model.

[0064] The visual language base model can be, for example, a pre-trained visual language model. The visual language fine-tuning model can be, for example, a model used in conjunction with the visual language base model, essentially a plugin for the visual language base model. The visual language fine-tuning model corresponds to the first style. By setting the visual language model to be a model that fuses the visual language base model and the visual language fine-tuning model, the feature encoding results output from the visual language model can carry the style features corresponding to the visual language fine-tuning model.

[0065] The denoising base model can be, for example, a pre-trained denoising network. The denoising fine-tuning model can be, for example, a model used in conjunction with the denoising base model, essentially a plugin for the denoising base model. The denoising fine-tuning model has a corresponding style; by setting the denoising network to be a model that fuses the denoising base model and the denoising fine-tuning model, the target image output from the denoising model can carry the style features corresponding to the denoising fine-tuning model.

[0066] The visual language fine-tuning model corresponds to the first style, which means that when performing encoding operations on image modification prompts, the visual language model will embed the first style features into the feature encoding results.

[0067] Both the visual language fine-tuning model and the denoising fine-tuning model correspond to the first style, indicating that the style features injected into the feature encoding result by the visual language fine-tuning model are consistent with the style features assigned to the third image by the denoising fine-tuning model. During the generation of the third image, the first style will be enhanced and retained in the modified area of ​​the third image.

[0068] It should be noted that in practice, the second image may not have the first style, but because the second image is modified using an image modification model corresponding to the first style, the modified third image exhibits the first style in its modified area.

[0069] The image modification model with the above structure can make the modified third image exhibit the first style in its modified area, thereby improving the quality of the target video.

[0070] There are various training methods for image modification models, and this application does not limit them. For example, the training method for an image modification model includes: acquiring sample data, which includes a corresponding original sample image, content description information of the original sample image, sample image modification prompt information, sample image modification result, and content description information of the sample image modification result; the sample image modification prompt information includes relevant content reflecting the modification of the original sample image; the sample image modification result is the image obtained after modifying the original sample image according to the sample image modification prompt information; training a visual language basis model and a denoising basis model based on the sample data to make them compatible; then, fusing the visual language basis model with a visual language fine-tuning model, and fusing the denoising basis model with the denoising fine-tuning model; freezing the parameters in the visual language basis model and the denoising basis model, and adjusting the parameters in the visual language fine-tuning model and the denoising fine-tuning model based on the sample data.

[0071] Sample data can be, for example, data used to train an image modification model. The original sample image can be, for example, the image before modification. The sample image modification prompts can be, for example, text information reflecting how the original sample image was modified. The sample image modification result can be, for example, the result after modification based on the sample image modification prompts. The content description information of the original sample image can be, for example, text information obtained by understanding the image content in the original sample image. The content description information of the sample image modification result can be, for example, text information obtained by understanding the image content in the modified sample image.

[0072] Adaptation between the visual language basis model and the denoising basis model refers to their optimal cooperation to achieve the goal of modifying the image based on image modification prompts. Freezing the parameters in both the visual language basis model and the denoising basis model, and adjusting the parameters in the visual language fine-tuning model and the denoising fine-tuning model, allows the visual language fine-tuning model to adapt to the visual language basis model, and the denoising fine-tuning model to the denoising basis model. This enables the effective injection of specific style features into the modified image during the inference phase.

[0073] See Figure 6Based on sample data, the parameters in the visual language fine-tuning model and the denoising fine-tuning model are adjusted. For example, this may include: inputting the original sample image, sample image modification prompts, content description information of the original sample image, and content description information of the modified sample image into the visual language model to obtain sample feature encoding results; the sample image modification prompts include relevant content reflecting the modification of the original sample image; the modified sample image result is the image obtained after modifying the original sample image according to the sample image modification prompts; performing N noise addition processes on the modified sample image result to obtain N sample noise images; different sample noise images undergo different numbers of noise addition processes; N is a positive integer; inputting the original sample image and the sample feature encoding results into the denoising network, and using the denoising network to perform N denoising processes on the sample noise images that have undergone N noise addition processes to obtain N denoising results; different denoising results undergo different numbers of denoising processes; and adjusting the parameters in the visual language fine-tuning model and the denoising fine-tuning model in the image modification model based on the differences between sample noise images with the same noise content and the denoising results.

[0074] Referring to Figure 7, in practice, the modified sample image undergoes N rounds of noise addition, resulting in N noisy sample images. The noisy sample image after N rounds of noise addition is defined as the sixth image. The sixth image undergoes N rounds of denoising, resulting in N denoising results. In practical applications, it can be assumed that the noise content of the noisy sample image obtained after P rounds of noise addition to the modified sample image is equal to the residual noise content of the denoising result after the NP-th denoising process on the sixth image. Here, P is a positive integer less than or equal to N. Figure 7 In the diagram, the noise content of the sample noise images and the noise reduction results located in the same dashed box is the same.

[0075] In practice, if the modified sample image undergoes N rounds of noise addition, and the noisy sample image after N rounds of noise addition undergoes N rounds of noise reduction, this means that multiple sets of noisy sample images with the same noise content and noise reduction results will be obtained. For example, in... Figure 7 The noise content of the sample noise images and the denoising results within each dashed box is the same. In practice, "adjusting the parameters in the visual language fine-tuning model and the denoising fine-tuning model based on the differences between sample noise images with the same noise content and the denoising results" can include adjusting the parameters in the visual language fine-tuning model and the denoising fine-tuning model based on the differences between one or more sets of sample noise images with the same noise content and the denoising results.

[0076] Optionally, "adjusting the parameters in the visual language fine-tuning model and the denoising fine-tuning model based on the differences between sample noise images with the same noise content and the denoising results" includes: determining a first denoising result and a second denoising result from N denoising results; the number of denoising processes performed on the first denoising result is a; the number of denoising processes performed on the second denoising result is b; ba = m, where the value of m is determined by a random number generator; determining a first sample noise image and a second sample noise image from N sample noise images; the number of times the first sample noise image is denoised is Na times; the number of times the second sample noise image is denoised is Nb times; adjusting the parameters in the visual language fine-tuning model and the denoising fine-tuning model based on the differences between the first denoising result and the first sample noise image, and the differences between the second denoising result and the second sample noise image; wherein m, a, and b are all positive integers less than N.

[0077] Furthermore, adjusting the parameters in the visual language fine-tuning model and the denoising fine-tuning model based on the differences between the first denoising result and the first sample noisy image, and the differences between the second denoising result and the second sample noisy image, may include: determining a first loss function based on the differences between the first denoising result and the first sample noisy image; determining a second loss function based on the differences between the second denoising result and the second sample noisy image; determining a third loss function based on the first loss function and the second loss function; and adjusting the parameters in the visual language fine-tuning model and the denoising fine-tuning model based on the third loss function. Optionally, the third loss function may be a weighted calculation result of the first loss function and the second loss function.

[0078] For example, a visual language model and a denoising network are trained using a sample dataset. The sample dataset includes multiple sample data. During training, a random number generator can be used to determine a corresponding random number m for each sample data. Based on each sample data, the visual language model and the denoising network are trained once. Assume that a is set to 1. During each training iteration, the first denoising result can be determined as the denoising result after one denoising process, and the first sample noisy image can be determined as the sample noisy image after N-1 denoising processes. The second denoising result can be determined as the denoising result after 1+m denoising processes, and the second sample noisy image can be determined as the sample noisy image after N-1-m denoising processes. Based on the difference between the first denoising result and the first sample noisy image, a first loss function is obtained; based on the difference between the second denoising result and the second sample noisy image, a second loss function is obtained; based on the first loss function and the second loss function, a third loss function is obtained. Based on the third loss function, the parameters in the visual language fine-tuning model and the denoising fine-tuning model are adjusted.

[0079] The system determines a first denoising result and a second denoising result from N denoising results; the first denoising result undergoes denoising processing number 'a' times; the second denoising result undergoes denoising processing number 'b' times; and determines a first sample noisy image and a second sample noisy image from N sample noisy images; the first sample noisy image undergoes denoising number 'Na' times, and the second sample noisy image undergoes denoising number 'Nb' times. Based on the differences between the first denoising result and the first sample noisy image, and the differences between the second denoising result and the second sample noisy image, the parameters in the visual language fine-tuning model and the denoising fine-tuning model are adjusted. The purpose is to avoid calculating the loss function based on the denoised result after each denoising step, but rather to use a skipping approach, selecting a portion of the denoised results to calculate the loss function. This reduces the complexity of the loss function calculation and, through the skipping selection method, allows the differences between different groups of denoised results and sample noisy images to accumulate, which is beneficial to improving the reliability of the loss function and enabling the trained model to generate higher-quality images.

[0080] Based on the above technical solution, optionally, "adjusting the parameters in the visual language fine-tuning model and the denoising fine-tuning model based on the differences in noise reduction results and sample noise images with the same noise content" can include: adjusting the parameters in the visual-text alignment sub-model (i.e., qformer) of the visual language fine-tuning model, the input layer convolution x_embedder, the attention layer attentionqkv, and the compression layer projfclayer of the denoising fine-tuning model based on the differences in noise reduction results and sample noise images with the same noise content. This setting is beneficial for the rapid convergence of the visual language fine-tuning model and the denoising fine-tuning model.

[0081] Based on the above technical solution, optionally, "acquiring sample data" may include: acquiring content description information of the original sample image and content description information of the modified sample image; generating the original sample image and the modified sample image based on the content description information of the original sample image and the modified sample image; determining sample image modification prompt information based on the original sample image and the modified sample image. Essentially, this setup utilizes an image generation model to generate paired original sample images and modified sample image results. This method enables rapid and large-scale production of sample data, improving the training speed of visual language models and denoising networks.

[0082] Based on the above technical solution, optionally, generating the original sample image and the modified sample image result based on the content description information of the original sample image and the content description information of the modified sample image result includes: obtaining a reference noise image; performing multiple denoising processes on the reference noise image based on the content description information of the original sample image to obtain multiple first candidate images; different first candidate images undergo different numbers of denoising processes; using the first candidate image that has undergone the most denoising processes as the original sample image; performing multiple rounds of denoising on the reference noise image based on the content description information of the modified sample image result to obtain the modified sample image result; wherein, at least some rounds of denoising process in generating the modified sample image result include: determining a second candidate image among the multiple first candidate images; the noise content of the second candidate image is the same as the noise content of the image to be denoised in this round; performing denoising processing on the image to be denoised in this round based on the second candidate image and the content description information of the modified sample image result to obtain the output image of this round of denoising processing.

[0083] The reference noise image can be a pre-specified noise image. The reference noise image used to generate the original sample image and the reference noise image used to generate the modified sample image can be the same noise image or different noise images; this application does not impose any restrictions on this.

[0084] For example, see Figure 8 Assuming that based on the content description information of the original sample image, after E denoising processes on the reference noisy image, E first candidate images will be obtained; the number of denoising processes performed on different first candidate images is different. For example, first candidate image 1 undergoes denoising processing once, and first candidate image 2 undergoes denoising processing twice. The first candidate image E undergoes denoising processing E times, and the first candidate image E is used as the original sample image. E is a positive integer.

[0085] Based on the content description information of the modified sample image, the reference noisy image is denoised E times to finally obtain the modified sample image. At least part of the denoising process in generating the modified sample image may include: if this denoising process is the i-th denoising process, since the first candidate image i-1 has the same noise level as the image to be denoised in this round, the first candidate image i-1 is used as the second candidate image in this round; based on the second candidate image and the content description information of the modified sample image, denoising processing is performed on the image to be denoised in this round to obtain the output image of this round of denoising processing. i is a positive integer greater than 1, and i is less than or equal to E.

[0086] Specifically, see Figure 8In the second denoising process of generating the modified sample image result, denoising is performed on the image to be denoised in this round (i.e., the image after the first denoising) based on the content description information of the first candidate image 1 and the modified sample image result. In the third denoising process of generating the modified sample image result, denoising is performed on the image to be denoised in this round (i.e., the image after the second denoising) based on the content description information of the first candidate image 2 and the modified sample image result. This process continues in the same manner thereafter.

[0087] By setting content description information based on the second candidate image and the modification result of the sample image, the image to be denoised in this round is denoised to obtain the output image of this round of denoising. This ensures that the generated original sample image and the modification result of the sample image are related at the content level, and the modification result of the sample image can be regarded as the product of the modification of the original sample image.

[0088] It should also be noted that, using the training method of the image modification model provided in this application, modifications to the second image can be controlled solely through text-based image modification prompts, and there are no restrictions on the format of the image modification prompts. For example, the image modification prompts are not required to include content layout descriptions of the second image. This makes the writing of image modification prompts less constrained, allowing for greater flexibility and freedom, and helping users quickly master the methods for modifying images.

[0089] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0090] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0091] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0092] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0093] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0094] Figure 9 This is a schematic diagram of the structure of an image processing apparatus according to an embodiment of this disclosure. The image processing apparatus provided in this embodiment can be configured in a client or in a server. See also Figure 9 The image processing device specifically includes:

[0095] Acquisition module 310 is used to acquire a first image; the first image includes a first object:

[0096] The image expansion module 320 is used to expand the image around the first image to obtain a second image;

[0097] Adding module 330 is used to add a second object to the region adjacent to the first object in the second image to obtain a third image;

[0098] The generation module 340 is used to generate a target video using the first image as the first frame and the third image as the last frame; the target video reflects the process of the first image evolving into the third image, and is accompanied by dynamic effects of the first object and / or the second object.

[0099] Furthermore, the map expansion module 320 is used for:

[0100] The first image is cropped to obtain a fourth image; in the fourth image, the area ratio of the first object in the fourth image is greater than or equal to a set threshold.

[0101] The fourth image is expanded around its edges to obtain the second image;

[0102] Module 340 is used for:

[0103] A target video is generated using the fourth image as the first frame and the third image as the last frame; the target video reflects the process of the fourth image evolving into the third image.

[0104] Furthermore, the generation module 340 is used for:

[0105] The target filter is determined based on the image features of the second object in the third image;

[0106] Add a target filter to the second object in the third image to obtain the fifth image;

[0107] The target video is generated using the first image as the first frame and the fifth image as the last frame.

[0108] Furthermore, module 330 is added for:

[0109] Obtain image modification prompt information; the image modification prompt information includes a description of the second object;

[0110] Based on the image modification prompt information, the second image is modified using an image modification model to obtain the third image.

[0111] Furthermore, the image modification model includes a visual language model and a denoising network; the visual language model is the result of fusing a visual language base model and a visual language fine-tuning model; the denoising network is the result of fusing a denoising base model and a denoising fine-tuning model; both the visual language fine-tuning model and the denoising fine-tuning model correspond to a first style.

[0112] The device also includes a training module for:

[0113] The original sample image, sample image modification prompt information, content description information of the original sample image, and content description information of the sample image modification result are input into the visual language model to obtain the sample feature encoding result; the sample image modification prompt information includes relevant content reflecting the modification of the original sample image; the sample image modification result is the image obtained after modifying the original sample image according to the sample image modification prompt information;

[0114] The modified sample image is subjected to N noise addition processes to obtain N sample noise images; the number of noise addition processes performed on different sample noise images is different; N is a positive integer;

[0115] The original sample image and the sample feature encoding result are input into the denoising network. The denoising network is used to perform N denoising processes on the sample noisy image that has undergone N denoising processes, resulting in N denoising results. The number of denoising processes performed on different denoising results is different.

[0116] The parameters in the visual language base model and the denoising base model are frozen. Based on the differences between the sample noise images with the same noise content and the denoising results, the parameters in the visual language fine-tuning model and the denoising fine-tuning model are adjusted.

[0117] Furthermore, the training module is used for:

[0118] Among the N noise reduction results, a first noise reduction result and a second noise reduction result are determined; the number of noise reduction processes for the first noise reduction result is a; the number of noise reduction processes for the second noise reduction result is b; ba = m; the value of m is determined by a random number generator.

[0119] A first sample noise image and a second sample noise image are determined from the N sample noise images; the first sample noise image is denoised Na times; the second sample noise image is denoised Nb times.

[0120] Based on the difference between the first denoising result and the first sample noisy image, and the difference between the second denoising result and the second sample noisy image, the parameters in the visual language fine-tuning model and the denoising fine-tuning model are adjusted.

[0121] Where m, a, and b are all positive integers less than N.

[0122] Furthermore, the training module is used for:

[0123] Based on the differences in the noise reduction results of the sample noise images with the same noise content, the parameters in the visual-text alignment sub-model of the visual language fine-tuning model, the input layer convolution, the attention layer, and the compression layer of the noise reduction fine-tuning model are adjusted.

[0124] The image processing apparatus provided in this disclosure can execute the steps performed by the client or server in the image processing method provided in this disclosure, and has the execution steps and beneficial effects, which will not be described in detail here.

[0125] Figure 10 This is a schematic diagram of the structure of an electronic device according to an embodiment of this disclosure. See below for details. Figure 10 The diagram illustrates a structural schematic suitable for implementing the electronic device 1000 in the embodiments of this disclosure. The electronic device 1000 in the embodiments of this disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), wearable electronic devices, etc., as well as fixed terminals such as digital TVs, desktop computers, smart home devices, etc. Figure 10 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0126] like Figure 10As shown, the electronic device 1000 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1008 into a random access memory (RAM) 1003 to implement the image processing method as described in the embodiments of this disclosure. The RAM 1003 also stores various programs and information required for the operation of the electronic device 1000. The processing device 1001, ROM 1002, and RAM 1003 are interconnected via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0127] Typically, the following devices can be connected to the I / O interface 1005: input devices 1006 including, for example, a touchscreen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 1007 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1008 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows electronic device 1000 to exchange information with other devices wirelessly or via wired communication. Although Figure 10 An electronic device 1000 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0128] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts, thereby implementing the image processing method as described above. In such embodiments, the computer program can be downloaded and installed from a network via communication device 1009, or installed from storage device 1008, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of embodiments of this disclosure.

[0129] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include information signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated information signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0130] In some implementations, clients and servers may communicate using any known or future network protocol, such as HTTP (Hypertext Transfer Protocol), and may interconnect with any form or medium of digital information communication (e.g., a communication network). Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any known or future network.

[0131] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0132] The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to:

[0133] Acquire the original image and the corresponding image modification prompt information; the image modification prompt information includes relevant content reflecting the modification of the original image.

[0134] Based on the original image, the image modification prompt information is encoded to obtain a feature encoding result; the feature encoding result includes features of the original image, features of the image modification prompt information, and first style features;

[0135] Based on the original image and the feature encoding result, a noise reduction network corresponding to the first style is used to perform noise reduction processing on the preset noisy image to obtain the target image; the target image is the image obtained after modifying the original image according to the image modification prompt information; the modified area in the target image presents the first style.

[0136] Optionally, when one or more of the above-described procedures are executed by the electronic device, the electronic device may also perform other steps described in the above embodiments.

[0137] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0138] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0139] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0140] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0141] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0142] According to one or more embodiments of this disclosure, this disclosure provides an electronic device, including:

[0143] One or more processors;

[0144] Memory, used to store one or more programs;

[0145] When the one or more programs are executed by the one or more processors, the one or more processors implement any of the image processing methods provided in this disclosure.

[0146] According to one or more embodiments of the present disclosure, the present disclosure provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements an image processing method as described in any of the present disclosure.

[0147] This disclosure also provides a computer program product, which includes a computer program or instructions that, when executed by a processor, implement the image processing method described above.

[0148] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0149] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image processing method, characterized in that, include: Obtain a first image; the first image includes a first object; The first image is enlarged around its edges to obtain the second image; A second object is added to the region adjacent to the first object in the second image to obtain a third image; The target video is generated using the first image as the first frame and the third image as the last frame. The target video reflects the process of the first image evolving into the third image, and is accompanied by dynamic effects of the first object and / or the second object.

2. The method according to claim 1, characterized in that, The first image is expanded around its edges to obtain a second image, which includes: The first image is cropped to obtain a fourth image; in the fourth image, the area ratio of the first object in the fourth image is greater than or equal to a set threshold. The fourth image is expanded around its edges to obtain the second image; The step of generating a target video using the first image as the first frame and the third image as the last frame includes: A target video is generated using the fourth image as the first frame and the third image as the last frame; the target video reflects the process of the fourth image evolving into the third image.

3. The method according to claim 1, characterized in that, The step of generating a target video using the first image as the first frame and the third image as the last frame includes: The target filter is determined based on the image features of the second object in the third image; Add a target filter to the second object in the third image to obtain the fifth image; The target video is generated using the first image as the first frame and the fifth image as the last frame.

4. The method according to claim 1, characterized in that, The step of adding a second object to the region adjacent to the first object in the second image to obtain a third image includes: Obtain image modification prompt information; the image modification prompt information includes a description of the second object; Based on the image modification prompt information, the second image is modified using an image modification model to obtain the third image.

5. The method according to claim 4, characterized in that, The image modification model includes a visual language model and a noise reduction network; the visual language model is the result of fusing a visual language base model and a visual language fine-tuning model. The noise reduction network is the result of fusing the noise reduction base model and the noise reduction fine-tuning model. Both the visual language fine-tuning model and the noise reduction fine-tuning model correspond to the first style; The training method for the image modification model includes: The original sample image, sample image modification prompt information, content description information of the original sample image, and content description information of the sample image modification result are input into the visual language model to obtain the sample feature encoding result; the sample image modification prompt information includes relevant content reflecting the modification of the original sample image; the sample image modification result is the image obtained after modifying the original sample image according to the sample image modification prompt information; The modified sample image is subjected to N noise addition processes to obtain N sample noise images; the number of noise addition processes performed on different sample noise images is different; N is a positive integer; The original sample image and the sample feature encoding result are input into the denoising network. The denoising network is used to perform N denoising processes on the sample noisy image that has undergone N denoising processes, resulting in N denoising results. The number of denoising processes performed on different denoising results is different. The parameters in the visual language base model and the denoising base model are frozen. Based on the differences between the sample noise images with the same noise content and the denoising results, the parameters in the visual language fine-tuning model and the denoising fine-tuning model are adjusted.

6. The method according to claim 5, characterized in that, The adjustment of parameters in the visual language fine-tuning model and the denoising fine-tuning model based on the differences between the sample noise images with the same noise content and the denoising results includes: Among the N noise reduction results, a first noise reduction result and a second noise reduction result are determined; the number of noise reduction processes for the first noise reduction result is a; the number of noise reduction processes for the second noise reduction result is b; ba = m; the value of m is determined by a random number generator. A first sample noise image and a second sample noise image are determined from the N sample noise images; the first sample noise image is denoised Na times; the second sample noise image is denoised Nb times. Based on the difference between the first denoising result and the first sample noisy image, and the difference between the second denoising result and the second sample noisy image, the parameters in the visual language fine-tuning model and the denoising fine-tuning model are adjusted. Where m, a, and b are all positive integers less than N.

7. The method according to claim 5, characterized in that, The adjustment of parameters in the visual language fine-tuning model and the noise reduction fine-tuning model is based on the differences between the sample noise images with the same noise content and the noise reduction results; including: Based on the differences in the noise reduction results of the sample noise images with the same noise content, the parameters in the visual-text alignment sub-model of the visual language fine-tuning model, the input layer convolution, the attention layer, and the compression layer of the noise reduction fine-tuning model are adjusted.

8. An image processing apparatus, characterized in that, include: The acquisition module is used to acquire a first image; the first image includes a first object: The image expansion module is used to expand the image around the first image to obtain a second image. An adding module is used to add a second object to the region adjacent to the first object in the second image to obtain a third image; The generation module is used to generate a target video using the first image as the first frame and the third image as the last frame. The target video reflects the process of the first image evolving into the third image, and is accompanied by dynamic effects of the first object and / or the second object.

9. An electronic device, characterized in that, The electronic device includes: One or more processors; Storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-7.

Images