Image processing method, image processing apparatus, and computer program product

Abstract

The present disclosure relates to an image processing method, an image processing apparatus, and a computer program product. The image processing method comprises: extracting a target image object from a first image position; in response to the target image object being moved to a second image position in the first image, adjusting a size of the target image object according to the second image position, the second image position being different from the first image position; and fusing the target image object with the first image.

Description

Technical Field

[0001] This disclosure relates to the field of image processing, specifically to image processing methods, image processing apparatus, and computer program products. Background Technology

[0002] In recent years, large-scale image processing technology has developed rapidly. Large-scale images receive user input instructions and can generate images accordingly. However, the images generated by large-scale images often fail to fully meet user needs, requiring users to edit or modify them. Existing solutions to this problem include adjusting the input instructions to the large-scale image to regenerate it. However, this method cannot precisely edit the generated image; the regenerated image may be completely different from the original and still may not fully meet user requirements. Alternatively, various image editing software / tools can be used to manually edit the images generated by the large-scale image. However, such methods rely on complex manual operations, the quality of the edited image depends on the operator's experience, and the processing results are uncertain and costly.

[0003] Furthermore, even images not generated by large models still require further editing. Summary of the Invention

[0004] In view of the above problems, this disclosure provides an image processing method, an image processing apparatus, and a computer program product that can automatically adjust the size of the moved image object according to the user's movement operation to obtain an image with a harmonious size ratio.

[0005] According to one aspect of this disclosure, an image processing method is provided, comprising: extracting a target image object from a first image position; adjusting the size of the target image object according to the second image position, the second image position being different from the first image position, in response to the target image object being moved to a second image position in the first image; and fusing the target image object with the first image.

[0006] According to one embodiment of this disclosure, the first image position is located within the first image.

[0007] According to another embodiment of this disclosure, the first image location is located in a second image that is different from the first image.

[0008] According to one embodiment of this disclosure, in response to the target image object being moved to a second image position in a first image, adjusting the size of the target image object according to the second image position includes: determining a ratio, i.e., a first ratio, between the depth of the second image position and the depth of the first image position; and adjusting the size of the target image object according to the first ratio.

[0009] According to one embodiment of this disclosure, in response to the target image object being moved to a second image position in a first image, adjusting the size of the target image object according to the second image position includes: extracting a reference image object from a third image position in the first image; determining a ratio, i.e., a second ratio, between the depth of the second image position and the depth of the third image position; determining a ratio, i.e., a third ratio, between the size of the reference image object and the size of the target image object at the same depth; and adjusting the size of the target image object according to the second ratio and the third ratio.

[0010] According to one embodiment of this disclosure, the image processing method further includes: adjusting the style of the target image object according to the style of the second image position of the first image before the fusion.

[0011] According to one embodiment of this disclosure, the image processing method further includes: filling the first image position after the target image object is moved.

[0012] According to one embodiment of this disclosure, the image processing method further includes: generating the second image through the artificial intelligence model.

[0013] According to one embodiment of this disclosure, the image processing method further includes: searching for the second image from existing images.

[0014] According to one embodiment of this disclosure, the step of searching for the second image from existing images includes: extracting keywords from user-input instructions; searching for and sorting existing images related to the keywords; and selecting the existing image with the highest ranking for the keywords as the second image.

[0015] According to another aspect of this disclosure, an image processing apparatus is provided, comprising: a processor; and a memory storing one or more computer programs, wherein when the processor executes the one or more computer programs, the processor performs the image processing method described above.

[0016] According to another aspect of this disclosure, a computer program product is provided, wherein when the computer program is executed by a processor, the computer performs the image processing method described above.

[0017] According to the image processing method, image processing apparatus, and computer program product provided in this disclosure, the size of the moved image object can be automatically adjusted according to the user's movement operation to obtain an image with a harmonious size ratio. Attached Figure Description

[0018] Figure 1This diagram illustrates an example of editing existing images generated from large models.

[0019] Figure 2 This is a flowchart of an image processing method according to an embodiment of the present disclosure.

[0020] Figures 3A-3C This is a schematic diagram illustrating an example application scenario of an image processing method according to a first embodiment of the present disclosure.

[0021] Figure 4 This is a schematic diagram illustrating an example process of an image processing method according to a first embodiment of the present disclosure.

[0022] Figure 5 This is a schematic diagram illustrating another example process of an image processing method according to a first embodiment of the present disclosure.

[0023] Figure 6 Show Figure 5 The example process shown is a schematic block diagram of the resizing module used.

[0024] Figures 7A-7C This is a schematic diagram illustrating an example application scenario of an image processing method according to a second embodiment of the present disclosure.

[0025] Figure 8 This is a schematic diagram illustrating an example process of an image processing method according to a second embodiment of the present disclosure.

[0026] Figures 9A-9C This is a schematic diagram illustrating an example application scenario of an image processing method according to a third embodiment of the present disclosure.

[0027] Figure 10 A schematic block diagram of an image processing apparatus according to an embodiment of the present disclosure is shown. Detailed Implementation

[0028] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0029] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, method embodiments may include other steps and / or omit certain steps.

[0030] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0031] It should be understood that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules, units, models, data, etc., and are not used to limit the order of the functions performed by these devices, modules, units, models, data, or their interdependencies.

[0032] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0033] Large models receive user input instructions and can generate images accordingly. However, the images generated by large models often fail to fully meet user needs, requiring users to edit or modify them. Existing solutions include adjusting the input instructions given to the large model to allow it to regenerate the image.

[0034] Figure 1 This diagram illustrates an example scenario of editing images generated from existing large models. Figure 1 In the image on the left, for example, it could be the image generated by the large model after receiving a user input instruction (also called a prompt) such as "Generate two kittens." In this image, the black cat is to the left of the white cat. Now, the user wants to modify the image, for example, by inputting the instruction "Switch the black and white cats" into the large model again. The large model receives this instruction and generates... Figure 1 The image on the right. Based on this image, it can be seen that the user wants to... Figure 1 The image on the left was edited, but after receiving new instructions, the large model generated a new image that differed from the original image in terms of background, cat's shape, and other aspects. Furthermore, although the user wanted to change the positions of the black and white cats in the image, the large model did not fulfill this requirement; in the newly generated image, the black cat remained to the left of the white cat. In other words, this method of modifying images generated by the large model through instructions cannot accurately edit already generated images; the regenerated image may differ from the original and still fail to fully meet the user's needs.

[0035] On the other hand, users can also manually edit images generated from large models using various image editing software / tools. For example, a user might want to move an object that is near in the image generated from a large model to a distant location. In this case, according to the natural law that objects appear larger when closer and smaller when farther away, the object to be moved should be shrunk. However, the extent to which the object should be shrunk depends entirely on the user's experience and judgment, and may require repeated adjustments. That is, such a method relies on complex manual operation, and the quality of the edited image depends on the operator's experience, potentially resulting in unnatural proportions, style, and other issues.

[0036] In view of the above problems, the present invention is based on the idea that when a user moves an object in an image (hereinafter also referred to as an image object), the computer can automatically adjust the size of the moved image object according to the position it is moved to, and can further adjust the style, etc., so that the edited image object is appropriately sized and can be made more natural with less human intervention.

[0037] In this disclosure, the image can be generated from a large model, or it can be any other image not generated from a large model. The image object can be any object in the image that the user may wish to edit, such as people, animals, plants, buildings, etc. The user's movement operation on the image object can be any method that allows the image object to be moved, such as clicking and dragging the image object displayed on the screen with a mouse, clicking and dragging the image object on the touch panel with a finger, etc.

[0038] The image processing method according to embodiments of the present disclosure will be described in detail below. Figure 2 This is a flowchart of an image processing method 200 according to an embodiment of the present disclosure. (As shown...) Figure 2 As shown, the image processing method 200 may include steps S210 to S230.

[0039] In step S210, the target image object is extracted from the first image location.

[0040] Here, the target image object refers to the image object that the user wants to move. The first image position is the position of the target image object in the image before it is moved. Extracting the target image object can be achieved, for example, through an image segmentation module and a user motion capture module. The image segmentation module can identify one or more image objects from an image and segment the identified image objects and the image background into different layers, allowing the user to select and move the target image object individually in subsequent processing. The image segmentation module can be implemented using neural networks, such as existing model architectures like Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and Generative Adversarial Networks (GANs). Furthermore, a user motion capture module can capture the user's selection, movement, and other operations to obtain the image position of the target image object before and after movement (e.g., the coordinates of the target image object in the image). The user motion capture module can also be implemented using existing technologies. As an example, the user motion capture module can take the image object where the pointer is when the user presses the mouse (i.e. selects it) as the target image object, take the center pixel position of the target image object as the starting position of the target image object, i.e., the first image position, and take the user pressing the mouse and dragging the target image object. When the user releases the mouse, it is considered that the movement has ended, and the center pixel position of the target image object at this time is taken as the ending position, i.e., the second image position.

[0041] The following explanation uses the example of image processing method 200 first identifying all image objects in an image and then selecting the image object clicked (selected) by the user as the target image object. However, the process of image processing method 200 identifying image objects and selecting target image objects is not limited to this. For example, the user can first select a region in the image, and image processing method 200 can identify an image object within that region as the target image object. Alternatively, the user can first click a point in the image, and image processing method 200 can identify an image object within a certain range around that point as the target image object and select it. The above description is merely an example, and this disclosure does not specifically limit the specific implementation method for extracting target image objects.

[0042] In step S220, in response to the target image object being moved to the second image position in the first image, the size of the target image object is adjusted according to the second image position.

[0043] Here, the position of the second image differs from the position of the first image. That is, in the image processing method 200 of this disclosure, the position of the target image object before and after movement is different. The size of the target image object after movement can be automatically adjusted by the resizing module according to its moved position. Here, the first image is the image to which the target image object is moved; in other words, the first image is the image that the user wants to edit and retain in the final image. Furthermore, the target image object can be moved either once or multiple times. The resizing module of this disclosure can adjust the size of the target image object each time it is moved, thereby allowing the user to quickly know the size of the target image object after each adjustment and determine whether it is necessary to continue moving the target image object. For example, the user can continuously move the target image object, and correspondingly, this disclosure can continuously adjust the size of the target image object.

[0044] For example, if the target image object is moved to the background of the first image, according to the natural law that objects appear larger when closer and smaller when farther away, the target image object should be resized smaller by the resizing module. Conversely, if the target image object is moved to the foreground of the first image, the target image object should be resized larger by the resizing module. Furthermore, regardless of whether the target image object is in the foreground or background of the first image, its appropriate size relationship with other image objects in the first image can also be considered. The specific method for the resizing module to adjust the size of the target image object will be described later.

[0045] It should be noted that the first image position can be located within the first image or in a second image different from the first image. In the former case, the target image object moves within the same image; in the latter case, the target image object moves between different images. When the first image position and the second image position are located in different images, the second image position is different from the first image position, regardless of the specific locations of the two image positions within their respective images.

[0046] In step S230, the target image object is fused with the first image. That is, after the target image object is moved to the position desired by the user, in step S230, the image fusion module can be used to fuse the target image object with the first image to form a complete image. As mentioned above, the target image object can be moved and adjusted once or multiple times. For example, the user's mouse release operation can be set as the trigger for step S230. After the user adjusts the second image position of the target image object multiple times, the image fusion module starts step S230 by recognizing the trigger operation. The fusion module can be implemented using existing methods. For example, the fusion module can merge the layer of the target image object with the background layer of the first image into one layer. The above implementation method of the fusion module is only an example. This disclosure does not particularly limit the specific implementation method of the fusion module, as long as it can combine the target image object with the first image to generate an image. In the following description, the example of the target image object being moved only once is used for simplification.

[0047] According to the image processing method 200 described above, the user only needs to move the target image object without manually adjusting its size, thus simplifying the user's operation. Furthermore, by automatically adjusting the size of the moved target image object based on its new position, an image can be obtained where the target image object's size conforms to the natural law of perspective. Moreover, the image processing method 200 does not alter other image objects or the background of the first image, thus more accurately meeting the user's needs.

[0048] The following describes the specific method of adjusting the size of the target image object according to the position of the second image in the image processing method 200.

[0049] (First Embodiment)

[0050] In the first embodiment, the first image position is located in the first image.

[0051] That is, the target image object is an image object in the first image, and the user moves the target image object from the position in the first image to the position in the second image within the same image (the first image).

[0052] (Implementation Method 1)

[0053] In one implementation, in response to a target image object being moved to a second image position within a first image, adjusting the size of the target image object according to the second image position includes: determining a ratio (referred to as a first ratio) between the depth of the second image position and the depth of the first image position; and adjusting the size of the target image object according to the first ratio. (See reference...) Figures 3A-3C and Figure 4 This implementation method will be explained.

[0054] Figures 3A-3C This is a schematic diagram illustrating an example application scenario of the image processing method 200 according to implementation method 1. In this implementation, such as Figure 3A As shown, firstly, the user inputs the instruction "Draw a puppy running on a hillside" into the large model. The large model receives this instruction and generates an image. Then, the image processing method 200, for example, identifies the puppy and flowers as image objects from this image (in... Figure 3A (The image object is represented by a dashed box). Suppose the user wants to move the puppy's position, they can select the puppy with the mouse pointer; the puppy then becomes the target image object. Then, as... Figure 3B As shown, the user can, for example, hold down the mouse button and drag (move) the puppy as the target image object. Finally, as Figure 3C As shown, when the user moves the puppy to a location on a distant hillside, the image processing method 200 automatically shrinks the puppy to a size suitable for that location on the distant hillside, conforming to the natural law of perspective. It should be noted that, for ease of explanation, only the puppy and flowers are used as examples of image objects here, but the image processing method 200 can also identify other image objects such as the sun and hillside from this image.

[0055] Figure 4 This is a schematic diagram illustrating an example process of the image processing method 200 according to this implementation. For example... Figure 4 As shown, in this implementation, the large model 410 first receives user instructions and generates an image (i.e., the first image). The large model 410 can be any suitable artificial intelligence model. The image segmentation module 420 identifies image objects from the first image and segments the image objects and background in the first image into different layers. Then, the user motion capture module 430 captures the user's selection and movement operations on the identified target image objects.

[0056] exist Figure 4 In the example, the image object selected and moved by the user, i.e., the target image object, is a puppy located near the first image. The user capture module 430 obtains the starting position of the target image object, i.e., the first image position (x1, y1), and obtains the position to which the target image object is moved, i.e., the second image position (x2, y2). Figure 4 In the example, the second image position (x2, y2) is a position far away from the first image.

[0057] Then, the resizing module 440 calculates the depth d1 of the first image position (x1, y1) and the depth d2 of the second image position (x2, y2). In this disclosure, depth is represented as the distance of a certain position from the image photographer. The resizing module 440 can use existing depth calculation (depth estimation) models to recover the depth information of the three-dimensional scene from the two-dimensional image, thereby calculating the depth of the image position. For example, it can use various existing model architectures such as convolutional neural networks, fully convolutional neural networks, encoder-decoder (ED), autoencoder (AE), and generative adversarial networks. After calculating the depth d1 of the first image position (x1, y1) and the depth d2 of the second image position (x2, y2), the resizing module 440 scales the target image object according to the ratio of the depth d2 of the second image position (x2, y2) to the depth d1 of the first image position (x1, y1) (i.e., the first ratio R1 = d1 / d2, or R1 = d2 / d1).

[0058] More specifically, for example, in the case of a first ratio R1 = d1 / d2, the size adjustment module 440 can calculate the size of the target image object after movement by the following formula (1).

[0059] H2=H1·α·R1 (1)

[0060] Where H2 represents the size of the target image object after it has been moved, H1 represents the size of the target image object before it has been moved, and α is a constant used to adjust the first ratio R1, which can be set based on experience, for example, it can be "1".

[0061] In addition, for example, in the case of the first ratio R1 = d2 / d1, the above formula (1) can also be rewritten as H2 = H1·α / R1.

[0062] Thus, the size of the puppy, which is the target image object, is adjusted by the above formula (1) to obtain an image that conforms to the law of near objects appearing larger and far objects appearing smaller. Finally, the image fusion module 470 can merge the layer of the scaled-down target image object with the other layers of the first image into one layer to obtain the final image.

[0063] Furthermore, before the image fusion module 470 performs the aforementioned fusion processing, the image processing method 200 can also utilize the style adjustment module 460 to adjust the style of the target image object based on the style of the second image position of the first image. This is because the image styles near different image positions may differ, and it may be necessary to match the target image object with the style of the images surrounding the new position. Here, the style can be, for example, the artistic features of the image formed by lines, colors, shadows, brushstrokes, etc. The style adjustment module 460 can, for example, use images within a certain range around the second image position as the generation conditions for the style adjustment model, and use the target image object as the input to the style adjustment model, adjusting the target image object to conform to the style surrounding the second image position. Such a style adjustment model can employ existing techniques, such as Neural Style Transfer (NST) based on convolutional neural networks, representing the target image object and style (i.e., the aforementioned generation conditions) as feature vectors respectively, and forming the style-adjusted target image object by minimizing the difference between the feature vectors of the image object and the style. Alternatively, a generative adversarial network (GAN)-based approach can be employed. This involves training a generator network and a discriminator network to learn the mapping relationship between the target image object and its style. The generator network generates a style-adjusted target image object, while the discriminator network determines whether the style of the generated target image object is similar to the generation conditions. This disclosure does not impose specific limitations on the specific implementation method of the style adjustment model.

[0064] Furthermore, before the image fusion module 470 performs the aforementioned fusion processing, the image processing method 200 can also utilize the image inpainting module 450 to fill the first image position after the target image object has been moved, thereby filling the blank space left at the first image position after the target image object has been moved, thus maintaining the integrity and aesthetics of the first image. Of course, according to some embodiments of this disclosure, the original position (first image position) of the target image object can also be retained after it has been moved. The image inpainting module 450 can, for example, employ various existing methods such as diffusion-based methods, patch-based methods, and generative adversarial networks to fill blank pixel areas; this disclosure does not impose any particular limitation on these methods.

[0065] The above describes one implementation of adjusting the size of the target image object when the first image is located within the first image. In the above description, an image generated by a large model was used as an example; however, as mentioned above, the first image may not be generated by a large model, but may be any other image. Furthermore, the image filling module 450 and the style adjustment module 460 may also be omitted (in...). Figure 4 (Represented by a dashed box).

[0066] According to implementation method 1, when the user moves the target image object in the same image, the image processing method 200 adjusts the size of the target image object by a first ratio R1, so that the adjusted target image object conforms to the law of near objects being larger and far objects being smaller.

[0067] (Implementation Method 2)

[0068] The following describes another implementation method for adjusting the size of the target image object when the first image is located within the first image.

[0069] In this implementation, in response to the target image object being moved to the second image position in the first image, adjusting the size of the target image object according to the second image position includes: extracting a reference image object from the third image position in the first image; determining the ratio of the depth of the second image position to the depth of the third image position, i.e., a second ratio; determining the ratio of the size of the target image object to the size of the reference image object under the same depth, i.e., a third ratio; and adjusting the size of the target image object according to the second ratio and the third ratio.

[0070] Reference Figure 5 , Figure 6 The second implementation method will be explained.

[0071] Figure 5 This is a schematic diagram illustrating an example process of image processing method 200 implementing method 2. For example... Figure 5 As shown, the processing of the image segmentation module 520, user motion capture module 530, image filling module 550, style adjustment module 560, and image fusion module 570 in this implementation is the same as that of the image segmentation module 420, user motion capture module 430, image filling module 450, style adjustment module 460, and image fusion module 470 described in implementation 1 above, and will not be repeated here. In implementation 1, the size adjustment module 440 scales the size of the target image object according to the depth (depth ratio, i.e., the first ratio) of the first image position (x1, y1) and the second image position (x2, y2) before and after the target image object is moved. However, in this implementation, the size adjustment module 540 can utilize other image objects in the first image besides the target image object (e.g., Figure 3AUsing the flowers and plants in the image as a reference image object, when calculating the size of the target image object after its movement, the ratio (second ratio R2) of the depth of the target image object at the second image position (x2, y2) to the depth (x3, y3) of the reference image object at the third image position is considered, and the ratio (third ratio R3) of the size of the target image object (e.g., a puppy) and the reference image object (e.g., flowers and plants) at the same depth is also considered. The following details the processing performed by the size adjustment module 540 in this implementation method 2.

[0072] Figure 6 A schematic block diagram of the size adjustment module 540 according to implementation method 2 is shown. Figure 6 As shown, the size adjustment module 540 in this implementation may include a depth calculation model 541, an object and pose recognition model 542, and a scale calculation model 543.

[0073] The depth calculation model 541 can calculate the depth of a certain image position in the image as described in implementation method 1 above. However, in this implementation method 2, the depth calculation model 541 calculates the depth d2 of the image position (second image position) to which the target image object (puppy) is moved and the depth d3 of the image position (third image position (x3, y3)) of the reference image object (flowers). The size adjustment module 540 calculates the ratio of depth d2 to depth d3 (i.e., the second ratio R2 = d3 / d2, or R2 = d2 / d3).

[0074] Furthermore, in this implementation 2, the object and pose recognition model 542 of the size adjustment module 540 can identify what the target image object and the reference image object are, and recognize their respective poses. For example, in Figure 5 In the example, the object and pose recognition model 542 can identify the target image object as a puppy, with the puppy in a four-legged upright posture. Simultaneously, it identifies the reference image object as flowers and plants, also in an upright posture. This object and pose recognition model 542 can be implemented using existing models, such as neural networks of various architectures. Furthermore, the object and pose recognition model 542 can further detect the specific species of the puppy and flowers / plants to make the calculations of the proportion calculation model 543 described later more accurate. This is merely an example, and this disclosure does not specifically limit the specific implementation of the object and pose recognition model 542.

[0075] Then, the scaling calculation model 543 of the size adjustment module 540 calculates the ratio (third ratio R3) between the target image object and the reference image object at the same depth, based on their respective types and postures. Here, the third ratio R3 can be, for example, the ratio of the true height h1 of the target image object to the true height h2 of the reference image object, i.e., R3 = h1 / h2. For example, the true height h1 of the target image object (a dog) in an upright posture might be in the range of 20 cm to 60 cm, and the true height h2 of the reference image object (flowers / plants) in an upright posture might be in the range of 5 cm to 20 cm. A value can be selected from each of these two true height ranges, and the ratio of these two values ​​can be used as the aforementioned third ratio R3. The scaling calculation model 543 can, for example, pre-store the true heights of various objects in various postures, which can be obtained, for example, from network information.

[0076] The size adjustment module 540 can scale the target image object according to the second ratio R2 and the third ratio R3 calculated above.

[0077] More specifically, in the case of the second ratio R2 = d3 / d2, the size adjustment module 540 can calculate the size of the target image object after movement by the following formula (2).

[0078] H2=H3·α·R2·R3 (2)

[0079] Where H2 represents the size of the target image object in the first image after it has been moved, H3 represents the size of the reference image object in the first image, and α is a constant used to adjust the second scale R2 and the third scale R3, which can be set empirically, for example, it can be "1".

[0080] Similar to the above implementation method 1, under the second ratio R2=d2 / d3, the above equation (2) can also be rewritten as H2=H3·α / R2·R3.

[0081] According to this implementation method 2, when the user moves the target image object in the same image, the image processing method 200 adjusts the size of the target image object by using a second ratio and a third ratio, so that the adjusted target image object conforms to the law of near objects being larger and far objects being smaller.

[0082] (Second Embodiment)

[0083] In the second embodiment, the first image location is located in a second image that is different from the first image.

[0084] That is, the target image object is an image object in the second image, and the user moves the target image object from the second image across images to the first image. At this time, the image processing method 200 adjusts the size of the target image object to conform to the principle of objects appearing larger when closer and smaller when farther away in the first image. Figures 7A-7C , Figure 8 The second embodiment will be described in detail.

[0085] Figures 7A-7C This is a schematic diagram illustrating an example application scenario of the image processing method 200 according to the second embodiment. For example... Figure 7A As shown, firstly, the user inputs the instruction "Draw a puppy and a child running on a hillside" into the large model. The large model receives this instruction and generates two images. Then, the image processing method 200, for example, identifies the puppy and child as image objects from the two images respectively (in... Figure 7A (Image objects are represented by dashed boxes). Suppose a user wants to move a child from the right image (second image) located in the first image to the left image (first image). They can select the child with the mouse pointer, making the child the target image object. Then, as... Figure 7B As shown, the user can, for example, hold down the mouse button and drag (move) the target image object. Finally, as... Figure 7C As shown, when the user moves the child to a certain position in the first image (the second image position), the image processing method 200 automatically shrinks the child to a size suitable for the second image position, which conforms to the natural law that things appear larger when they are closer and smaller when they are farther away.

[0086] Figure 8 This is a schematic diagram illustrating an example process of an image processing method according to a second embodiment of the present disclosure. Unless otherwise specified, the processing of the image segmentation module 820, user motion capture module 830, size adjustment module 840, image filling module 850, style adjustment module 860, and image fusion module 870 is the same as that of the image segmentation module 520, user motion capture module 530, size adjustment module 540, image filling module 550, style adjustment module 560, and image fusion module 570 in the implementation of the first embodiment 2 described above, and will not be repeated here. Specifically, as... Figure 8As shown, in the second embodiment, the target image object is the little girl in the second image, and the user moves the target image object to the second image position (x2, y2) in the first image. In this case, the image processing method 200 can select the puppy in the first image as the reference image object, which is located at the third image position (x3, y3) of the first image. The process of calculating the second ratio R2 by the depth calculation model 841 (not shown) of the size adjustment module 840 can be the same as the implementation method 2 of the first embodiment described above. The object and pose recognition model 842 (not shown) of the size adjustment module 840 can identify that the object of the target image object is a child, and the child is in an upright posture. At the same time, it can identify that the reference image object is a puppy, and the puppy is in a posture with its limbs upright. In addition, the ratio calculation model 843 (not shown) of the size adjustment module 840 can, for example, select a value from the range of the child's actual height in an upright posture (80 cm to 120 cm) and the puppy's actual height in a posture with its limbs upright (20 cm to 60 cm), and use the ratio of these two values ​​as the third ratio R3. Therefore, the size adjustment module 840 can calculate the size of the target image object (child) in the first image according to the second ratio R2 and the third ratio R3 using the above formula (2). Then, the image filling module 850 can fill the second image to ensure that no blank pixel areas are displayed, thus ensuring aesthetics. Furthermore, the style adjustment module 860 can adjust the style of the target image object to suit the style of the first image. For example, the style adjustment module 860 can adjust the color (e.g., blue) of the little girl as the target image object to a color (e.g., green) suitable for the area near the second image position in the first image. Finally, the image fusion module 870 fuses the target image object with the first image. According to the second embodiment, the image processing method 200 supports the user moving the target image object between different images, enabling the user to select a wider variety of target image objects. Furthermore, the image processing method 200 automatically adjusts the size of the target image object, ensuring that the adjusted target image object conforms to the principle of near objects appearing larger and far objects smaller.

[0087] (Third Embodiment)

[0088] In the second embodiment described above, both the first and second images are generated by a large model according to user instructions. In the third embodiment, the second image may not be generated by a large model, but may be obtained by searching for existing images. Figures 9A-9C This is a schematic diagram illustrating an example application scenario of an image processing method according to a third embodiment of the present disclosure.

[0089] like Figure 9AAs shown, the user inputs the instruction "Draw a puppy running on a hillside" into the large model. The large model receives this instruction and generates an image (first image). Furthermore, the image processing method 200 can search for one or more reference images (second images) from existing images. The user can select a target image object from the reference images and move it to the position in the second image within the first image. Figure 9B In the example, the user selects the flowers and plants in the third reference image and moves them into the image generated by the larger model (the first image). Image processing method 200 automatically adjusts the size and / or style of the target image object in the first image. Figure 9C In the example, image processing method 200 adjusts the target image object to a size and style suitable for the first image.

[0090] In the third embodiment, searching for a second image from existing images includes: extracting keywords from user-input instructions; searching for and sorting existing images related to the keywords; and selecting the existing image with the highest ranking for the keywords as the second image.

[0091] For example, image processing method 200 can extract keywords such as "puppy running" and "hillside" from the instruction "draw a puppy running on a hillside." For each keyword, it searches for multiple reference images related to that keyword from existing images. These existing images can be retrieved from existing image libraries, the internet, etc., and can include real photographs, paintings, etc. Additionally, existing images can also include images generated using large models. Furthermore, image processing method 200 can sort the images retrieved for each keyword based on predetermined rules and select the top-ranked image as a reference image for display. For example, in... Figure 9A In the example, image processing method 200 provides the two top-ranked reference images for the keyword "puppy running" and one reference image for the keyword "hillside". The number of reference images provided for each keyword can be user-specified or flexibly changed, for example, based on the size of the display interface; no limitation is imposed here.

[0092] The ranking rules for the existing images retrieved for each keyword can be, for example, at least one of the following: the relevance of each existing image to the keyword; the richness of the image objects in each existing image (e.g., the more types and / or the greater the number of identifiable image objects, the higher the ranking); the similarity of each existing image to the first image generated by the large model (e.g., similar to some extent, but not exactly the same); and user history information (e.g., the user's preference for realistic styles in past image editing). This ranking rule is merely an example; any other suitable rules can be used.

[0093] Other processes of the image processing method in the third embodiment can be implemented by the various modules described in the second embodiment, and repeated descriptions are omitted here.

[0094] According to the third embodiment, the image processing method 200 searches for more reference images from existing images and supports moving target image objects between different images, enabling users to select a wider variety of target image objects. Furthermore, the image processing method 200 automatically adjusts the size of the target image objects, ensuring that the adjusted target image objects conform to the principle of objects appearing larger when closer and smaller when farther away. In addition, the image processing method 200 extracts keywords based on user input instructions for the large model and searches for reference images according to these keywords, providing reference images that better meet the user's needs.

[0095] The image processing method 200 of this disclosure has been described in detail above.

[0096] According to one aspect of this disclosure, an image processing apparatus 1000 is also provided. Figure 10 A schematic block diagram of an image processing apparatus 1000 according to an embodiment of the present disclosure is shown.

[0097] like Figure 10 As shown, the image processing apparatus 1000 of this embodiment includes a processor 1010 and a memory 1020. The memory 1020 stores a computer program.

[0098] Processor 1010 is any processing device such as a microprocessor, which operates according to a program stored in memory 1020. Memory 1020 is any volatile or non-volatile storage element, such as a hard disk, solid-state drive, ROM, or RAM. The program executed by processor 1010, etc., is stored in memory 1020. Figure 10 The image processing apparatus 1000 shown can be used to implement an image processing method according to an embodiment of the present disclosure.

[0099] According to another aspect of this disclosure, a computer program product is also provided, comprising a computer program that, when executed by a processor, can implement the image processing method according to embodiments of this disclosure.

[0100] The hardware computing device or its components described in this disclosure can be implemented by various suitable hardware means, including but not limited to FPGA, ASIC, SoC, discrete gate or transistor logic, discrete hardware components, or any combination thereof. The devices, apparatuses, methods, and systems involved in this disclosure are not limited to any particular hardware architecture or configuration. Components in the disclosed devices, apparatuses, and systems can be discrete or integrated, and can be combined in different ways and / or replaced or supplemented by other components. It should be understood that the teachings of this disclosure can be implemented in various forms, including hardware, software, firmware, dedicated processors, or combinations thereof.

[0101] The block diagrams of apparatuses, devices, methods, systems, etc., involved in this disclosure are merely exemplary and are not intended to require or imply that connections, arrangements, or configurations must be made in the manner shown in the block diagrams. As those skilled in the art will recognize, these circuits, devices, apparatuses, and systems can be connected, arranged, and configured in any manner that achieves the desired purpose.

[0102] In the foregoing description, the present invention has been described based on embodiments. These embodiments are merely illustrative, and those skilled in the art should understand that the combination of constituent elements and processes of these embodiments can be modified in various ways, and such modifications are also within the scope of the present invention.

Claims

1. An image processing method, comprising: Extract the target image object from the first image location; In response to the target image object being moved to a second image position in the first image, the size of the target image object is adjusted according to the second image position, which is different from the first image position; and The target image object is fused with the first image.

2. The image processing method as described in claim 1, wherein, The first image position is located within the first image.

3. The image processing method as described in claim 1, wherein, The first image is located in a second image that is different from the first image.

4. The image processing method as described in claim 2, wherein, The step of adjusting the size of the target image object according to the second image position in response to the target image object being moved to a second image position in the first image includes: Determine the ratio of the depth of the second image position to the depth of the first image position, i.e., the first ratio; and The size of the target image object is adjusted according to the first ratio.

5. The image processing method according to any one of claims 1-3, wherein, The step of adjusting the size of the target image object according to the second image position in response to the target image object being moved to a second image position in the first image includes: Extract the reference image object from the third image location in the first image; The ratio of the depth of the second image position to the depth of the third image position is determined, i.e., the second ratio; Determine the ratio, i.e., the size of the target image object to the size of the reference image object at the same depth; and The size of the target image object is adjusted according to the second ratio and the third ratio.

6. The image processing method according to any one of claims 1-3, further comprising: Prior to the fusion, the style of the target image object is adjusted according to the style of the second image position of the first image.

7. The image processing method according to any one of claims 1-3, further comprising: After the target image object is moved, the position of the first image is filled.

8. The image processing method as described in claim 3, further comprising: The first image is generated using an artificial intelligence model; and The second image is generated using the artificial intelligence model, or the second image is searched for from existing images.

9. An image processing apparatus, comprising: processor; as well as Memory, which stores computer programs When the computer program is executed by the processor, the processor performs the image processing method as described in any one of claims 1 to 8.

10. A computer program product comprising a computer program that, when executed by a processor, performs the image processing method as described in any one of claims 1 to 8.

Images