Image processing method, image processing apparatus, and computer program product

Abstract

The present invention provides an image processing method, an image processing apparatus, and a computer program product that automatically adjust the size of a moved image object based on the user's movement operation, thereby obtaining an image with a balanced size ratio. [Solution] The image processing method includes extracting a target image object from a first image position, adjusting the size of the target image object according to the second image position in response to the movement of the target image object to a second image position in the first image, provided that the second image position is different from the first image position, and fusing the target image object with the first image.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing, and specifically, to an image processing method, an image processing apparatus, and a computer program product.

Background Art

[0002] In recent years, large-scale model technology has been developing rapidly. A large-scale model can receive an instruction input by a user and generate an image based on the instruction. However, an image generated by a large-scale model usually hardly satisfies the user's needs completely, and the user needs to edit or modify the generated image. For this problem, as a conventional countermeasure, for example, the instruction input to the large-scale model may be adjusted to cause the large-scale model to generate the image again. However, such a method cannot accurately edit the generated image, and the image generated again may be completely different from the original image and still may not fully satisfy the user's needs. Or, for example, various image editing software / tools may be further used to manually edit the image generated by the large-scale model.

Summary of the Invention

Problems to be Solved by the Invention

[0003] However, such a method depends on complex operations by humans, the quality of the edited image depends on the experience of the operator, the processing result is uncertain, and the cost is high. Also, there is a problem that even if it is not an image generated by a large-scale model, it still needs to be further edited.

Means for Solving the Problems

[0004] In view of the above problems, the present disclosure provides an image processing method, an image processing apparatus, and a computer program product that can automatically adjust the size of a moved image object based on a user's movement operation and obtain an image with a balanced size ratio.

[0005] According to one aspect of the present disclosure, an image processing method is provided which includes extracting a target image object from a first image position, adjusting the size of the target image object according to the second image position in response to the movement of the target image object to a second image position in the first image, wherein the second image position is different from the first image position, and merging the target image object with the first image.

[0006] According to one embodiment of the present disclosure, the first image position is located within the first image.

[0007] According to another embodiment of the present disclosure, the first image location is located within a second image that is different from the first image.

[0008] According to one embodiment of the present disclosure, adjusting the size of the target image object in response to the movement of the target image object to a second image position in a first image includes identifying a first ratio which is the ratio of the depth of the second image position to the depth of the first image position, and adjusting the size of the target image object based on the first ratio.

[0009] According to one embodiment of the present disclosure, adjusting the size of the target image object in response to the movement of the target image object to a second image position in a first image includes extracting a reference image object from a third image position in the first image, identifying a second ratio which is the ratio of the depth of the second image position to the depth of the third image position, identifying a third ratio which is the ratio of the size of the target image object to the size of the reference image object when the depths are the same, and adjusting the size of the target image object based on the second ratio and the third ratio.

[0010] According to one embodiment of the present disclosure, the image processing method further includes adjusting the style of the target image object according to the style at a second image position in the first image, prior to the fusion.

[0011] According to one embodiment of the present disclosure, the image processing method further includes interpolating the first image position after the movement of the target image object.

[0012] According to one embodiment of the present disclosure, the image processing method further includes generating the second image using an artificial intelligence model.

[0013] According to one embodiment of the present disclosure, the image processing method further includes searching for the second image from an existing image.

[0014] According to one embodiment of the present disclosure, searching for the second image from the existing images includes extracting keywords from a command entered by the user, searching for and ranking existing images related to those keywords, and selecting the existing image with the highest ranking for the keywords as the second image.

[0015] According to another aspect of the present disclosure, an image processing apparatus is provided, comprising a processor and memory storing one or more computer programs, wherein when the one or more computer programs are executed by the processor, the processor performs the image processing method described above.

[0016] According to yet another aspect of this disclosure, a computer program product is provided which, when executed by a processor, causes the computer to perform the image processing method described above. [Effects of the Invention]

[0017] According to the image processing method, image processing apparatus, and computer program product provided by the present disclosure, in response to a user's movement operation, the size of the moved image object can be automatically adjusted to obtain an image with a balanced size ratio.

Brief Description of the Drawings

[0018] [Figure 1] FIG. 1 is a diagram showing an exemplary case of editing an image generated by a conventional large-scale model. [Figure 2] FIG. 2 is a flowchart of an image processing method according to an embodiment of the present disclosure. [Figure 3] FIGS. 3A to 3C are schematic diagrams showing exemplary application scenarios of an image processing method according to a first embodiment of the present disclosure. [Figure 4] FIG. 4 is a schematic diagram showing an exemplary procedure of an image processing method according to a first embodiment of the present disclosure. [Figure 5] FIG. 5 is a schematic diagram showing another exemplary procedure of an image processing method according to a first embodiment of the present disclosure. [Figure 6] FIG. 6 is a schematic block diagram showing a size adjustment module used in the exemplary procedure shown in FIG. 5. [Figure 7] FIGS. 7A to 7C are schematic diagrams showing exemplary application scenarios of an image processing method according to a second embodiment of the present disclosure. [Figure 8] FIG. 8 is a schematic diagram showing an exemplary procedure of an image processing method according to a second embodiment of the present disclosure. [Figure 9] FIGS. 9A to 9C are schematic diagrams showing exemplary application scenarios of an image processing method according to a third embodiment of the present disclosure. [Figure 10] FIG. 10 is a schematic block diagram showing an image processing apparatus according to an embodiment of the present disclosure.

Modes for Carrying Out the Invention

[0019] Hereinafter, embodiments of the present disclosure will be described in more detail with reference to the drawings. Although several embodiments of the present disclosure are shown in the drawings, the present disclosure can be realized in various aspects and should not be construed as being limited to the embodiments described below. Rather, these embodiments should be understood to be provided for a more thorough and complete understanding of the present disclosure. The drawings and embodiments of the present disclosure are not intended to limit the protection scope of the present disclosure, but should be understood as being exemplary.

[0020] It should be understood that each step described in the embodiment of the method of the present disclosure may be executed in a different order and / or may be executed in parallel. Further, the embodiment of the method may include other steps and / or may omit some steps.

[0021] As used herein, the term "comprising" and its variations are open-ended inclusion, that is, it means "including... but not limited to this". The term "based on" means "at least partially based on...". The term "one embodiment" represents "at least one embodiment". The term "another embodiment" represents "at least one another embodiment". The term "some embodiments" represents "at least some embodiments". Definitions of other terms will be described later.

[0022] It should be understood that the concepts such as "first", "second", etc. referred to in the present disclosure are merely for distinguishing different devices, modules, units, models, data, etc., and are not for limiting the order of functions, generation procedures or mutual dependencies executed by these devices, modules, units, models, data.

[0023] Note that the modifiers "one" and "plural" referred to in the present disclosure are not limiting but general. As can be understood by those skilled in the art, unless otherwise specified in the context, it should be understood as "one or more".

[0024] <Large-scale models can receive commands entered by the user and generate images based on those commands. However, images generated by large-scale models usually do not fully meet the user's needs, and the user needs to edit or modify the generated images. A conventional approach would be, for example, to adjust the commands entered into the large-scale model and have the model regenerate the image.

[0025] Figure 1 illustrates an example of editing an image generated by a conventional large-scale model. In Figure 1, the image on the left is generated after the large-scale model receives a command (also called a prompt) "Generate two kittens" from the user. In this image, the black cat is positioned to the left of the white cat. At this point, the user attempts to modify the image and, for example, inputs another command to the large-scale model: "Swap the positions of the black cat and the white cat." The large-scale model receives this command and generates the image on the right in Figure 1. As shown in this image, the user attempts to edit the image on the left in Figure 1, but the large-scale model generates a new image after receiving the new command, and the background, the cats' postures, etc., are all different between the new image and the original image. Furthermore, the user attempts to change the positions of the black cat and the white cat in the image, but the large-scale model does not fulfill this need, and in the newly generated image, the black cat is still positioned to the left of the white cat. In other words, this method of modifying images generated by large-scale models through commands cannot accurately edit the generated images, the newly generated images may differ from the original images, and still does not fully meet the user's needs.

[0026] On the other hand, users may manually edit images generated by large-scale models using various image editing software / tools. For example, a user might try to move a nearby object to a more distant location in an image generated by a large-scale model. In this case, the object being moved should be reduced in size according to the natural law that objects become smaller the further away they are. However, the extent to which the object being moved should be reduced depends entirely on the user's experience and may require repeated adjustments by the user. In other words, such a method relies on complex human operation, the quality of the edited image depends on the operator's experience, and there is a risk of unnatural proportions and styles.

[0027] In light of the above issues, the concept of this disclosure is that a user may perform a move operation on an object in an image (hereinafter also referred to as an image object), and the computer may automatically adjust the size of the image object after the move based on the destination position of the image object, and further adjust its style, etc. This makes it possible to appropriately size the edited image object even with little human intervention, and to make the style look more natural.

[0028] In this disclosure, the image may be an image generated by a large-scale model, or it may be any other image not generated by a large-scale model. The image object may be any object in the image that the user intends to edit, such as a person, animal, plant, or building. The user's movement operation on the image object may be any method that allows the image object to be moved, such as clicking and dragging the image object displayed on the screen with a mouse, or clicking and dragging the image object with a finger on a touch panel.

[0029] The image processing method according to the embodiment of this disclosure will be described in detail below. Figure 2 is a flowchart of the image processing method 200 according to the embodiment of this disclosure. As shown in Figure 2, the image processing method 200 may include steps S210 to S230.

[0030] In step S210, the target image object is extracted from the first image position.

[0031] Here, the target image object refers to the image object that the user intends to move. The first image position is the position of the target image object in the image before it is moved. Extraction of the target image object may be achieved, for example, by an image segmentation module and a user motion capture module. The image segmentation module can recognize one or more image objects from an image and divide the recognized image objects and the image background into different layers so that the user can select and move the target image object independently in subsequent processing. The image segmentation module can be implemented using a neural network, and may employ any of the conventional model architectures, such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), and generative adversarial networks (GANs). In addition, user operations such as selection and movement may be captured by a user motion capture module to obtain the image positions of the target image object before and after movement (e.g., the coordinates of the target image object in the image). The user motion capture module may be implemented using prior art. For example, a user motion capture module may define the image object where the pointer is located when the user presses the mouse (i.e., makes a selection) as the target image object, and the pixel position at the center of the target image object may be defined as the starting position of the target image object, i.e., the first image position. The user may then drag the target image object while holding down the mouse button, and the movement may be considered complete when the user's finger is released from the mouse, with the pixel position at the center of the target image object at this point being defined as the ending position, i.e., the second image position.

[0032] The following explanation will describe an example in which the image processing method 200 first recognizes all image objects from an image, and the image object clicked (selected) by the user is designated as the target image object. However, the procedure for recognizing image objects from an image and selecting the target image object using the image processing method 200 is not limited to this. For example, the user may first select a region in the image using a box selection, and the image processing method 200 may recognize one image object within that region as the target image object. Alternatively, the user may first click a point in the image, and the image processing method 200 may recognize one image object within a predetermined range around that point as the target image object and select it. The above explanation is merely illustrative, and this disclosure is not particularly limited to specific implementation methods for extracting target image objects.

[0033] In step S220, in response to the movement of the target image object to the second image position in the first image, the size of the target image object is adjusted according to the second image position.

[0034] Here, the second image position is different from the first image position. That is, in the image processing method 200 of this disclosure, the target image object has a different position before and after movement. The size of the target image object after movement can be automatically adjusted by the size adjustment module according to the destination position. In other words, the first image here is the image to which the target image object is moved. In other words, the first image is the image that the user edits and intends to keep within the final image. The target image object may be moved only once or multiple times. The size adjustment module of this disclosure can adjust the size of the target image object each time it is moved, so that the user can quickly grasp the size of the target image object each time an adjustment is made and decide whether or not it is necessary to continue moving the target image object. For example, the user may move the target image object continuously, and accordingly, the size of the target image object may be continuously adjusted in this disclosure.

[0035] For example, if the target image object is moved to the background of the first image, it should be reduced in size by the resizing module, according to the natural law that objects become smaller the further away they are. Conversely, if the target image object is moved to the foreground of the first image, it should be increased in size by the resizing module. Furthermore, it may be considered that the relationship between the target image object and other image objects in the first image satisfies appropriate size relationships, regardless of whether the target image object is located in the foreground or background of the first image. A specific method for adjusting the size of the target image object using the resizing module will be described later.

[0036] The first image position may be located within the first image, or it may be located within a second image that is different from the first image. In the former case, the target image object is moved within the same image, and in the latter case, the target image object is moved between different images. If the first and second image positions are located within different images, the second image position is different from the first image position, regardless of the specific location of each of the two image positions within those images.

[0037] In step S230, the target image object and the first image are merged. That is, after the target image object is moved to the position desired by the user, in step S230, the image fusion module may be used to fuse the target image object and the first image to form a complete final image. As described above, the target image object may be moved and adjusted once or multiple times. For example, the user may set an operation such as releasing their finger from the mouse as the trigger for step S230, and after the user has adjusted the second image position of the target image object multiple times, the image fusion module may recognize the trigger operation and start step S230. The fusion module may be implemented in a conventional way. For example, the fusion module may merge the layer of the target image object and the background layer of the first image into a single layer. The above-described implementation method of the fusion module is merely illustrative, and this disclosure is not particularly limited to the specific implementation method of the fusion module; it is sufficient that the target image object and the first image can be merged to generate a final image. In the following description, the explanation will be simplified by using an example in which the target image object is moved only once.

[0038] According to the image processing method 200 described above, the user only performs a move operation on the target image object without manually adjusting its size, thus simplifying the user's operation. Furthermore, by automatically adjusting the size of the target image object after it has been moved according to its destination position, an image can be obtained that follows the natural law that the size of the target image object decreases as it moves further away. Moreover, since the image processing method 200 does not change other image objects, background, etc., in the first image, it can more accurately meet the user's needs.

[0039] The following describes a specific method for adjusting the size of a target image object according to the second image position using image processing method 200.

[0040] (First example) In the first embodiment, the first image position is located within the first image.

[0041] In other words, the target image object is the image object in the first image, and the user moves the target image object from the position in the first image to the position in the second image within the same image (the first image).

[0042] (Implementation Method 1) In one implementation, adjusting the size of a target image object in response to its movement to a second image position in a first image includes determining the ratio (referred to as the first ratio) between the depth of the second image position and the depth of the first image position, and adjusting the size of the target image object according to the first ratio. This implementation will be explained with reference to Figures 3A to 3C and Figure 4.

[0043] Figures 3A to 3C are schematic diagrams illustrating exemplary application scenarios of the image processing method 200 according to Implementation Embodiment 1. In this implementation embodiment, as shown in Figure 3A, first, the user inputs a command to the large-scale model to "draw a puppy running up a mountain slope." The large-scale model receives this command and generates an image. The image processing method 200 then recognizes, for example, the puppy and the plants as image objects (in Figure 3A, image objects are represented by dotted lines) from this image. Assuming the user wants to move the puppy's position, the user selects the puppy with the mouse pointer, and at this time, the puppy becomes the target image object. Subsequently, as shown in Figure 3B, the user can, for example, drag (move) the puppy as the target image object by holding down the mouse button. Finally, as shown in Figure 3C, the user moves the puppy to a location on a distant mountain slope, and at this time, the image processing method 200 automatically reduces the size of the puppy to a size appropriate for that location on the distant mountain slope, following the natural law that the further away it is, the smaller it becomes. For the sake of explanation, only puppies and flowers are used as examples of image objects here, but the image processing method 200 can recognize other image objects from the image, such as the sun or a mountain slope.

[0044] Figure 4 is a schematic diagram illustrating an exemplary procedure of the image processing method 200 according to this embodiment. As shown in Figure 4, in this embodiment, the large-scale model 410 first receives a command from the user and generates an image (i.e., a first image). The large-scale model 410 may be any suitable artificial intelligence model. The image segmentation module 420 recognizes image objects from the first image and divides the image objects and background in the first image into different layers. Subsequently, the user motion capture module 430 captures the user's selection and movement operations on the recognized target image object.

[0045] In the example in Figure 4, the image object selected and moved by the user, i.e., the target image object, is a puppy located nearby in the first image. The user motion capture module 430 obtains the starting position of the target image object, i.e., the first image position (x1, y1), and obtains the destination position of the target image object, i.e., the second image position (x2, y2). In the example in Figure 4, the second image position (x2, y2) is a distant position in the first image.

[0046] Subsequently, the size adjustment module 440 calculates the depth d1 of the first image position (x1, y1) and the depth d2 of the second image position (x2, y2), respectively. In this disclosure, depth represents the distance from a certain position to the image photographer. The size adjustment module 440 can calculate the depth of an image position by reconstructing depth information of a 3D scene from a 2D image using conventional depth calculation (depth estimation) models, and can calculate the depth of an image position using multiple conventional model architectures such as convolutional neural networks, all-layer convolutional neural networks, encoder-decoders (EDs), autoencoders (AEs), and generative adversarial networks. After calculating the depth d1 of the first image position (x1, y1) and the depth d2 of the second image position (x2, y2), the size adjustment module 440 scales the target image object according to the ratio of the depth d2 of the second image position (x2, y2) to the depth d1 of the first image position (x1, y1) (i.e., the first ratio R1 = d1 / d2, or R1 = d2 / d1).

[0047] More specifically, for example, in the case of the first ratio R1 = d1 / d2, the size adjustment module 440 may calculate the size of the moved target image object by the following formula (1). H2 = H1·α·R1 (1)

[0048] Here, H2 represents the size of the target image object after movement, H1 represents the size of the target image object before movement, and α is a constant for adjusting the first ratio R1, which may be set according to experience, for example, "1".

[0049] For example, if the first ratio R1 = d2 / d1, then equation (1) above may be rewritten as H2 = H1·α / R1.

[0050] In this way, the size of the target image object, the puppy, is adjusted according to equation (1) above, and an image is obtained that follows the rule that the further away it is, the smaller it becomes. Finally, the image fusion module 470 may fuse the layer of the reduced target image object with the other layers in the first image into a single layer to obtain the final image.

[0051] Furthermore, the image processing method 200 may further utilize the style adjustment module 460 before the image fusion module 470 performs the fusion processing described above, to adjust the style of the target image object according to the style at the second image position in the first image. This is because the surrounding image style may differ at different image positions, and therefore, it is necessary to match the target image object to the surrounding image style of the new position. Here, the style may be, for example, the artistic features of the image formed by drawing lines, colors, shadows, brushstrokes, etc. The style adjustment module 460 may, for example, use an image in a predetermined range around the second image position as the generation conditions for a style adjustment model, the target image object as the input to the style adjustment model, and the style adjustment model may adjust the target image object to have a style suitable for the surrounding style of the second image position. Such a style adjustment model may employ conventional techniques. For example, the target image object and style (i.e., the generation conditions described above) may be represented by feature vectors using Neural Style Transfer (NST) based on a convolutional neural network, and the target image object after style adjustment may be formed by minimizing the difference between the feature vector of the image object and the feature vector of the style. Alternatively, for example, a method based on generative adversarial networks may be used to train one generator network and one discriminator network to learn the mapping relationship between target image objects and styles. The generative network may then generate a style-adjusted target image object, and the discriminator network may determine whether the style of the generated target image object is similar to the generation conditions. The specific implementation method of the style adjustment model is not particularly limited in this disclosure.

[0052] Furthermore, the image processing method 200 may further utilize the image interpolation module 450 before the image fusion module 470 performs the fusion processing described above to interpolate the first image position after the target image object has moved, thereby preserving the integrity and aesthetics of the first image. Of course, according to some embodiments of this disclosure, the target image object may still remain at its original position (first image position) after it has been moved. The image interpolation module 450 may interpolate the blank pixel regions using, for example, several conventional methods such as diffusion-based methods, patch-based methods, and generative adversarial networks, and is not particularly limited in this disclosure.

[0053] The above describes one implementation method for adjusting the size of a target image object when the first image position is located within the first image. In the above description, the case in which the first image is an image generated by a large-scale model was used as an example, but as mentioned above, the first image may be any other image that is not generated by a large-scale model. Note that the image interpolation module 450 and the style adjustment module 460 may be omitted (indicated by a dashed frame in Figure 4).

[0054] According to the first embodiment, when a user moves a target image object within the same image, the image processing method 200 adjusts the size of the target image object based on a first ratio R1, thereby making the adjusted target image object obey the rule that it becomes smaller the further away it is.

[0055] (Implementation Method 2) The following describes another implementation method for adjusting the size of a target image object when the first image position is within the first image.

[0056] In this embodiment, adjusting the size of a target image object in response to its movement to a second image position in a first image includes extracting a reference image object from a third image position in the first image, identifying a second ratio which is the ratio of the depth of the second image position to the depth of the third image position, identifying a third ratio which is the ratio of the size of the target image object to the size of the reference image object when the depths are the same, and adjusting the size of the target image object in accordance with the second and third ratios.

[0057] The implementation embodiment 2 will be explained with reference to Figures 5 and 6.

[0058] Figure 5 is a schematic diagram illustrating an exemplary procedure of the image processing method 200 of Implementation Embodiment 2. As shown in Figure 5, the processing performed by the image segmentation module 520, user motion capture module 530, image interpolation module 550, style adjustment module 560, and image fusion module 570 in this Implementation Embodiment is the same as the processing performed by the image segmentation module 420, user motion capture module 430, image interpolation module 450, style adjustment module 460, and image fusion module 470 described in Implementation Embodiment 1 above, so redundant explanations are omitted here. In Implementation Embodiment 1, the size adjustment module 440 scales the size of the target image object based on the depth (depth ratio, i.e., the first ratio) of the first image position (x1, y1) and the second image position (x2, y2) before and after the target image object is moved. However, in this Implementation Embodiment, the size adjustment module 540 uses other image objects in the first image besides the target image object (for example, the flowers and plants in Figure 3A) as reference image objects, and when calculating the size of the target image object after movement, it may consider the ratio (second ratio R2) between the depth of the second image position (x2, y2), which is the destination of the target image object, and the depth (x3, y3) of the third image position where the reference image object is located. It may also consider the size ratio (third ratio R3) when the depths of the target image object (for example, the puppy) and the reference image object (for example, the flowers and plants) are the same. The processing performed by the size adjustment module 540 in Implementation Embodiment 2 will be described in detail below.

[0059] Figure 6 shows a schematic block diagram of the size adjustment module 540 according to implementation embodiment 2. As shown in Figure 6, the size adjustment module 540 in this implementation embodiment may include a depth calculation model 541, an object and attitude recognition model 542, and a ratio calculation model 543.

[0060] The depth calculation model 541 may calculate the depth of an image at a certain image position according to the method described in the implementation embodiment 1 above, but in this implementation embodiment 2, the depth calculation model 541 calculates the depth d2 of the image position (second image position) to which the target image object (puppy) moves, and the depth d3 of the image position (third image position (x3, y3)) of the reference image object (flowers and plants). The size adjustment module 540 calculates the ratio of depth d2 to depth d3 (i.e., the second ratio R2 = d3 / d2, or R2 = d2 / d3).

[0061] Furthermore, in this embodiment 2, the object and posture recognition model 542 of the size adjustment module 540 may recognize what kind of objects the target image object and the reference image object are, and recognize their respective postures. For example, in the example of Figure 5, the object and posture recognition model 542 may recognize that the type of object in the target image object is a puppy and that the puppy is standing on all four legs, and may also recognize that the reference image object is a flower and that the flower is standing upright. Here, the object and posture recognition model 542 may be implemented by a conventional model, for example, a neural network of various architectures. The object and posture recognition model 542 may further detect the specific breeds of the puppy and the flower in order to make the calculation of the ratio calculation model 543 described later more accurate. This is merely an example, and this disclosure is not particularly limited to specific embodiments of the object and posture recognition model 542.

[0062] Subsequently, the ratio calculation model 543 of the size adjustment module 540 may calculate the ratio (third ratio R3) between the size of the target image object and the size of the reference image object, assuming the same depth, based on the respective types and orientations of the target image object and the reference image object. Here, the third ratio R3 may be, for example, the ratio of the actual height h1 of the target image object to the actual height h2 of the reference image object, i.e., R3 = h1 / h2. For example, the actual height h1 of a puppy as the target image object in a standing position on all four legs may be in the range of 20cm to 60cm, and the actual height h2 of a flowering plant as the reference image object in an upright position may be in the range of 5cm to 20cm. One value may be selected from each of these two ranges of actual heights, and the ratio of these two values ​​may be the third ratio R3 described above. The ratio calculation model 543 may, for example, pre-store the actual heights of multiple different types of objects in different orientations, and these actual heights may be obtained, for example, from network information.

[0063] The size adjustment module 540 may scale the target image object based on the second ratio R2 and the third ratio R3 calculated above.

[0064] More specifically, in the case of the second ratio R2 = d3 / d2, the size adjustment module 540 may calculate the size of the moved target image object using the following formula (2). H2 = H3·α·R2·R3 (2)

[0065] Here, H2 represents the size of the target image object in the first image after movement, H3 represents the size of the reference image object in the first image, and α is a constant for adjusting the second ratio R2 and the third ratio R3, which may be set empirically, for example, "1".

[0066] Similar to the implementation embodiment 1 described above, if the second ratio R2 = d2 / d3, the above equation (2) may be rewritten as H2 = H3·α / R2·R3.

[0067] According to this embodiment 2, when a user moves a target image object within the same image, the image processing method 200 can adjust the size of the target image object according to the second ratio and the third ratio, thereby making the adjusted target image object follow the rule that it becomes smaller the further away it is.

[0068] (Second example) In the second embodiment, the position of the first image is located within a second image that is different from the first image.

[0069] In other words, the target image object is the image object in the second image, and the user moves the target image object across images, from the second image to the first image. At this time, the image processing method 200 adjusts the size of the target image object according to the rule that it becomes smaller the further away it is in the first image. The second embodiment will be specifically explained using Figures 7A to 7C and Figure 8.

[0070] Figures 7A to 7C are schematic diagrams illustrating exemplary application scenarios of the image processing method 200 of the second embodiment. As shown in Figure 7A, first, the user inputs a command to the large-scale model, "Draw a puppy and a child running up a mountain slope." The large-scale model receives this command and generates two images. The image processing method 200 then recognizes the puppy and the child from these two images as image objects (in Figure 7A, the image objects are indicated by dotted lines). If the user intends to move the child, located at the position in the first image in the right-hand image (second image), to the left-hand image (first image), the user may select the child with the mouse pointer, at which point the child becomes the target image object. Subsequently, as shown in Figure 7B, the user may, for example, drag (move) the target image object while holding down the mouse button. Finally, as shown in Figure 7C, the user moves the child to a position in the first image (the second image position), and at this time, the image processing method 200 automatically reduces the child to a size suitable for the second image position, following the natural law that objects become smaller the further away they are.

[0071] Figure 8 is a schematic diagram illustrating an exemplary procedure of an image processing method according to a second embodiment of the present disclosure. Unless otherwise specified, the processing by the image segmentation module 820, user motion capture module 830, size adjustment module 840, image interpolation module 850, style adjustment module 860, and image fusion module 870 is the same as that of the image segmentation module 520, user motion capture module 530, size adjustment module 540, image interpolation module 550, style adjustment module 560, and image fusion module 570 in the implementation 2 of the first embodiment described above, so redundant explanations are omitted here. In particular, as shown in Figure 8, in the second embodiment, the target image object is a girl in the second image, and the user moves the target image object to the second image position (x2, y2) in the first image. In such a case, the image processing method 200 may select a puppy in the first image as a reference image object, and the reference image object is located at the third image position (x3, y3) in the first image. The procedure by which the depth calculation model of the size adjustment module 840 calculates the second ratio R2 may be the same as that of implementation embodiment 2 of the first embodiment described above. The object and posture recognition model of the size adjustment module 840 may recognize that the type of object of the target image object is a child and that the child is in a standing posture, or it may recognize that the reference image object is a puppy and that the puppy is in a standing posture on all four legs. Furthermore, the ratio calculation model of the size adjustment module 840 may, for example, select one value from each of the two actual height ranges, based on the actual height range of 80cm to 120cm for a child in a standing posture and the actual height range of 20cm to 60cm for a puppy in a standing posture on all four legs, and the ratio of these two values ​​may be taken as the third ratio R3. As a result, the size adjustment module 840 can calculate the size of the target image object (child) in the first image using equation (2) described above, based on the second ratio R2 and the third ratio R3. Subsequently, the image interpolation module 850 may perform image interpolation on the second image to ensure aesthetic appeal by preventing blank pixel areas from being displayed in the second image.The style adjustment module 860 may then adjust the style of the target image object to a style suitable for the first image. For example, the style adjustment module 860 may adjust the color of the target image object, which is the girl (e.g., blue), to a color suitable for the vicinity of the second image position in the first image (e.g., green). Finally, the image fusion module 870 fuses the target image object with the first image. According to the second embodiment, the image processing method 200 can support the user moving the target image object between different images and diversify the target image objects that can be selected by the user. The image processing method 200 can also automatically adjust the size of the target image object, making it smaller the further it is from the adjusted target image object.

[0072] (Third example) In the second embodiment described above, both the first and second images are generated by a large-scale model based on user commands. In the third embodiment, the second image may not be generated by a large-scale model, but may be obtained by searching from existing images. Figures 9A to 9C are schematic diagrams illustrating exemplary application scenarios of the image processing method according to the third embodiment of this disclosure.

[0073] As shown in Figure 9A, the user inputs a command to the large-scale model: "Draw a puppy running up a mountain slope." The large-scale model receives this command and generates one image (the first image). The image processing method 200 may then search through existing images to obtain one or more reference images (the second image), and the user may select a target image object from the reference images and move it to the position of the second image in the first image. In the example in Figure 9B, the user selects flowers and plants from the third reference image and moves them to the image (the first image) generated by the large-scale model. The image processing method 200 automatically adjusts the size and / or style of the target image object in the first image. In the example in Figure 9C, the image processing method 200 adjusts the size and style of the target image object to an extent suitable for the first image.

[0074] In the third embodiment, searching for a second image from existing images includes extracting keywords from commands entered by the user, searching for and ranking existing images related to those keywords, and selecting the existing image with the highest keyword ranking as the second image.

[0075] For example, the image processing method 200 may extract keywords such as "puppy running" and "mountain slope" from the command "draw a puppy running up a mountain slope," and for each keyword, search for multiple reference images related to that keyword from existing images. Here, existing images may be searched from, for example, conventional image libraries, the internet, etc., and may include actual photographs, paintings, etc. Existing images may also include images generated by large-scale models. The image processing method 200 may then rank the searched images for each keyword according to predetermined rules, and select and display the image with the highest ranking as a reference image. For example, in the example in Figure 9A, the image processing method 200 provides the two highest-ranking reference images for the keyword "puppy running" and provides one reference image for the keyword "mountain slope." Here, the number of reference images provided for each keyword may be specified by the user, and may be flexibly changed, for example, according to the size of the display interface, and is not limited thereto.

[0076] The rules for ranking existing images found through keyword searches may include, for example, the degree of relevance between each existing image and the keyword, the richness of image objects in each existing image (e.g., the more types and / or numbers of recognizable image objects, the higher the ranking), the similarity between each existing image and a first image generated by a large-scale model (e.g., they are somewhat similar but not exactly the same), or user history information (e.g., the user prefers a realistic style in past image editing). Here, the ranking rules are merely examples, and any other appropriate rules may be adopted.

[0077] Other steps in the image processing method according to the third embodiment can be implemented using the modules described in the second embodiment, so redundant explanations are omitted here.

[0078] According to the third embodiment, the image processing method 200 can diversify the target image objects that can be selected by the user by searching for more reference images from existing images and supporting the movement of target image objects between different images. Furthermore, the image processing method 200 can automatically adjust the size of the target image object so that it follows the rule that the further away the adjusted target image object is, the smaller it becomes. In addition, the image processing method 200 can extract keywords based on commands entered by the user into a large-scale model and search for reference images according to those keywords, thereby providing reference images that better suit the user's needs.

[0079] The image processing method 200 of this disclosure has been described in detail above.

[0080] According to one aspect of the present disclosure, an image processing apparatus 1000 is further provided. Figure 10 shows a schematic block diagram of the image processing apparatus 1000 according to an embodiment of the present disclosure.

[0081] As shown in Figure 10, the image processing device 1000 of this embodiment comprises a processor 1010 and a memory 1020. A computer program is stored in the memory 1020.

[0082] The processor 1010 is any processing unit, such as a microprocessor, and operates based on a program implemented in the memory 1020. The memory 1020 is any volatile or non-volatile memory element, such as a hard disk, solid-state disk, ROM, or RAM. The program executed by the processor 1010, etc., is stored in the memory 1020. The image processing device 1000 shown in Figure 10 may be used to implement the image processing method according to the embodiment of this disclosure.

[0083] According to another aspect of the present disclosure, a computer program product is further provided which includes a computer program that, when executed by a processor, implements an image processing method according to an embodiment of the present disclosure.

[0084] The hardware computing devices described herein, in whole or in their components, may be implemented by a variety of suitable hardware means, including, but not limited to, FPGAs, ASICs, SoCs, discrete gates or transistor logic, discrete hardware components, or any combination thereof. The devices, equipment, methods, and systems relating to this disclosure are not limited to any particular hardware architecture or configuration. The components of the disclosed devices, equipment, and systems may be individual, integrated, combined in different ways, and / or replaced or supplemented by other components. It should be understood that this disclosure may be implemented in various forms, such as hardware, software, firmware, a dedicated processor, or a combination thereof.

[0085] The block diagrams of the apparatus, equipment, methods, and systems relating to this disclosure are illustrative only and do not require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will understand, these circuits, devices, apparatus, equipment, and systems may be connected, arranged, and configured in any manner that achieves the desired purpose.

[0086] In the above description, the present invention has been explained based on examples. These examples are for illustrative purposes only, and it should be understood by those skilled in the art that the combinations of components and processes in these examples can be modified in various ways, and such modifications are also within the scope of the present invention.

Claims

1. Extracting the target image object from the first image position, In response to the movement of the target image object to a second image position in the first image that is different from the first image position, the size of the target image object is adjusted according to the second image position, provided that the second image position is different from the first image position, and This includes fusing the target image object with the first image, Image processing methods.

2. The first image position is located within the first image. The image processing method according to claim 1.

3. The aforementioned first image position is located within a second image that is different from the aforementioned first image. The image processing method according to claim 1.

4. Adjusting the size of the target image object according to the second image position in response to the movement of the target image object to the second image position in the first image is: To identify a first ratio which is the ratio of the depth of the second image position to the depth of the first image position, and This includes adjusting the size of the target image object based on the first ratio, The image processing method according to claim 2.

5. Adjusting the size of the target image object according to the second image position in response to the movement of the target image object to the second image position in the first image is: Extracting a reference image object from the third image position in the first image, To identify a second ratio, which is the ratio of the depth of the second image position to the depth of the third image position. To identify a third ratio, which is the ratio between the size of the target image object and the size of the reference image object when the depths are the same, and This includes adjusting the size of the target image object based on the second and third ratios, The image processing method according to any one of claims 1 to 3.

6. The process further includes adjusting the style of the target image object according to the style at the second image position in the first image before the fusion, The image processing method according to any one of claims 1 to 3.

7. The process further includes interpolating the first image position after the movement of the target image object. The image processing method according to any one of claims 1 to 3.

8. To generate the first image using an artificial intelligence model, and The process further includes generating the second image using the artificial intelligence model, or searching for the second image from existing images. The image processing method according to claim 3.

9. Processor and A computer comprising memory in which a computer program is stored, When the computer program is executed by the processor, the processor executes the image processing method described in any one of claims 1 to 4. Image processing device.

10. When executed by a processor, the computer program includes a computer program that performs the image processing method described in any one of claims 1 to 4. Computer program products.

Images