Image processing device, image processing method and program

Abstract

To provide a technique for obtaining a mask that makes it easy to change an object. [Solution] An image processing device according to one aspect of the present disclosure is configured to perform the following: extracting the range of an object appearing in a target image through image analysis; enlarging the extracted range to create a padding area around the extracted range; generating a mask that designates the area within the enlarged range as a change area and the area outside the enlarged range as a no-change area; and outputting the generated mask.

Description

[Technical Field]

[0001] The present disclosure relates to an image processing device, an image processing method, and a program. [Background technology]

[0002] In recent years, technological developments have been progressing in image generation using trained machine learning models (image generation models) generated by machine learning. For example, Non-Patent Document 1 proposes a technology for changing objects in a target image using a trained machine learning model. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Gaurav Parmar, et al. “Zero-shot Image-to-Image Translation”, [online], [Retrieved September 5, 2025], Internet<URL:https: / / arxiv.org / pdf / 2302.03027> [Non-patent document 2] "How to use Stable Diffusion inpaint! How to edit images and erase unwanted parts", [online], [searched September 5, 2025], Internet<URL:https: / / highreso.jp / edgehub / stablediffusion / inpaint.html> [Non-patent document 3] “Extreme Prediction AI Official Website”, [online], [Retrieved September 5, 2025], Internet<URL:https: / / www.ca-kiwami.ai / > Summary of the Invention [Problem to be solved by the invention]

[0004] If a change to an object is executed without specifying a change area in the target image, there is a possibility that areas other than the object will also be changed. Therefore, the present inventors have considered a method of providing a mask along with the object to an image generation model (Non-Patent Document 2, etc.). By using a mask to specify the area of ​​the object as a change area and the other area as a change-prohibited area, it is possible to restrict changes to areas other than the object.

[0005] By automatically generating this mask from the target image, the effort required for generating a new image can be reduced. By extracting the range of the object from the target image using image processing, it is possible to automatically generate a mask that designates the extracted range as a change area and the remaining area as a prohibited change area. However, the inventors of the present invention have discovered that the following problem occurs with masks generated in this manner. That is, if a mask is generated using the extracted range of the object as is, the shape of the object is restricted. This can lead to problems with modifying the object, such as not being able to change a person's hairstyle, the shape of a face, or the shape of an object.

[0006] In one aspect, the present disclosure has been made in view of the above circumstances, and one of the objects of the present disclosure is to provide a technique for obtaining a mask that makes it easy to change an object. [Means for solving the problem]

[0007] In order to solve the above-mentioned problems, the present disclosure employs the following configurations. Note that the following configurations can be combined as appropriate.

[0008] An image processing device according to an aspect of the present disclosure includes a control unit that performs image analysis to extract an area of ​​an object appearing in a target image, expands the extracted area to provide a padding area around the extracted area, and generates a mask that designates an area within the expanded area as a change area and an area outside the expanded area as a no-change area; The method is configured to perform the step of outputting the generated mask.

[0009] In this configuration, by expanding the range of the object, a padding area is provided around the range of the object. This padding area can reduce constraints on the shape of the object. In other words, it can absorb shape changes caused by changes to the object. Therefore, this configuration can provide a mask that makes it easier to change the object.

[0010] In the image processing device according to the above aspect, outputting the generated mask may be configured by inputting the target image and the mask into an image generation model, and causing the image generation model to generate a new image in which the target object in the modification area specified by the mask has been modified. The control unit may further be configured to acquire the generated new image from the image generation model and output the acquired new image. With this configuration, it is possible to automate at least a portion of the process leading up to obtaining the new image. This is expected to reduce the effort required to generate the new image.

[0011] In the image processing device according to the above aspect, the control unit may further be configured to determine whether the acquired new image satisfies a predetermined quality condition. If the control unit determines that the acquired new image does not satisfy the predetermined quality condition, the control unit may be configured to reduce the magnification rate of the extracted range from the original magnification rate, and then re-execute the steps of providing a padding area, generating a mask, and outputting the mask. Expanding the padding area makes it easier to modify the object. However, it also increases the likelihood of quality degradation, such as unwanted modifications. This configuration allows the generated new image to be evaluated for its satisfaction of the predetermined quality condition, and if the new image does not satisfy the predetermined quality condition, the padding area can be narrowed and a new image can be generated again. This makes it easier to obtain a new image with good quality by dynamically adjusting the size of the padding area (the magnification rate of the extracted range).

[0012] In the image processing device according to the above aspect, the control unit may further be configured to receive designation of a prohibited area in the target image where modification is not permitted. Generating the mask may be configured by generating a mask so that the modified area does not include the designated prohibited area. With this configuration, even within the scope of the target object, a portion where modification is not permitted can be designated as a prohibited area. This makes it possible to obtain a mask for modifying the target object while avoiding modification of that portion.

[0013] In the image processing device according to the above aspect, the object may be a target person. With this configuration, when the target person is to be changed, a mask that makes it easier to change the object (target person) can be obtained.

[0014] In the image processing device according to the above aspect, the target image may be an advertisement image. With this configuration, when changing an object appearing in the advertisement image, a mask that makes it easier to change the object can be obtained.

[0015] Note that the embodiments of the present disclosure may not be limited to the image processing device described above. As another aspect of the image processing device according to each of the above aspects, one aspect of the present disclosure may be an image processing method that realizes all or part of each of the above configurations, a program, or a machine-readable storage medium storing such a program, such as a computer. Here, the machine-readable storage medium may be a non-transitory medium that stores information such as a program by electrical, magnetic, optical, mechanical, or chemical action. The non-transitory storage medium may include a storage medium (e.g., a CD, a DVD, a semiconductor memory), an auxiliary storage device of a computer, an external storage device connected to a computer, etc.

[0016] For example, an image processing method according to an aspect of the present disclosure may be executed by a computer and may include extracting an area of ​​an object appearing in a target image through image analysis, enlarging the extracted area to provide a padding area around the extracted area, generating a mask that designates an area within the enlarged area as a change area and an area outside the enlarged area as a no-change area, and outputting the generated mask.

[0017] Furthermore, for example, a program according to an aspect of the present disclosure may be a program for causing a computer to execute an image processing method, which may include extracting an area of ​​an object appearing in a target image through image analysis, enlarging the extracted area to provide a padding area around the extracted area, generating a mask that designates an area within the enlarged area as a change area and designates an area outside the enlarged area as a no-change area, and outputting the generated mask. [Effects of the Invention]

[0018] According to one aspect of the present disclosure, a technique can be provided for obtaining a mask that facilitates modification of an object. [Brief explanation of the drawings]

[0019] [Figure 1] FIG. 1 schematically illustrates an example of a situation to which the present disclosure is applied. [Figure 2] FIG. 2 shows a schematic example of a scene in which a mask is output. [Figure 3] FIG. 3 shows a schematic diagram of an example of an image generation model. [Figure 4] FIG. 4 shows an example of a scene in which a disallowed area is set. [Figure 5] FIG. 5 is a diagram illustrating an example of a hardware configuration of an image processing device. [Figure 6] FIG. 6 is a diagram illustrating an example of the software configuration of the image processing device. [Figure 7]FIG. 7 is a flowchart illustrating an example of a processing procedure of the image processing device. [Figure 8] FIG. 8 shows a schematic diagram of an example of a scene in which quality conditions are evaluated. [Figure 9] FIG. 9 is a flowchart illustrating an example of a processing procedure of the image processing device. [Figure 10] FIG. 10 shows the target image (image before modification) in the first experimental example. [Figure 11] FIG. 11 shows the mask generated in the first experimental example. [Figure 12] FIG. 12 shows the new image generated in the first experimental example. [Figure 13] FIG. 13 shows the target image (image before modification) in the second experimental example. [Figure 14] FIG. 14 shows the initial mask generated in the second experimental example. [Figure 15] FIG. 15 shows the new image generated using the first mask in the second example experiment. [Figure 16] FIG. 16 shows a second mask generated by reducing the magnification of the first mask in the second experimental example. [Figure 17] FIG. 17 shows the new image generated using the second mask in the second example experiment. DETAILED DESCRIPTION OF THE INVENTION

[0020] An embodiment according to one aspect of the present disclosure will be described below with reference to the drawings. However, the embodiment described below is merely an example of the present disclosure in all respects. Various improvements or modifications may be made without departing from the scope of the present disclosure. In implementing the present disclosure, specific configurations according to the embodiment may be appropriately adopted. Note that while data appearing in the present embodiment is described in natural language, more specifically, it may be specified using pseudo-language, commands, parameters, machine language, electrical signals, etc. that can be recognized by a machine such as a computer.

[0021] §1 Application Examples 1 is a schematic diagram showing an example of a scene to which the present disclosure is applied. The apparatus 1 is one or more computers configured to generate a mask 3 for use in modifying an object TO appearing in an image 2 of the object.

[0022] In this embodiment, the image processing device 1 extracts a range 20 of the object TO appearing in the target image 2 through image analysis. The image processing device 1 enlarges the extracted range 20 to provide a padding area 27 around the extracted range 20. As a result, the image processing device 1 obtains an enlarged range 25. The image processing device 1 generates a mask 3 that designates the area within the enlarged range 25 as a change area 30 and designates the area outside the enlarged range 25 as a change-prohibited area 35. The image processing device 1 outputs the generated mask 3.

[0023] In this embodiment, by expanding the range 20 of the object TO, a padding region 27 is provided around the range 20 of the object TO. This padding region 27 can reduce restrictions on the shape of the object TO. In other words, by providing the padding region 27, it is possible to absorb changes in shape due to changes to the object TO. Therefore, according to this embodiment, a mask 3 that makes it easy to change the object TO can be obtained.

[0024] [Image analysis] As long as the range (range 20) in which the object TO appears can be extracted, the image analysis method is not particularly limited and may be appropriately selected depending on the embodiment. For example, the image processing device 1 may perform image analysis using a known method, such as a general image analysis method or a method using a trained machine learning model. General image analysis methods may include edge extraction, pattern matching, etc. When using a method using a trained machine learning model, the type of trained machine learning model is not particularly limited and may be appropriately selected depending on the embodiment, as long as it has the ability to accept an input image and extract the contour of an object appearing in the input image. A trained machine learning model capable of extracting the contour of an object appearing in an image may be generated by any machine learning method. For example, multiple datasets may be collected, each consisting of a combination of training images and true values ​​of the contours of objects appearing in the training images. The trained machine learning model may be generated by machine learning using the training images of each collected dataset as input and the corresponding true values ​​as a training signal. The trained machine learning model may include a large-scale generative model, such as a large-scale visual language model (VLM).

[0025] The calculation processing of the image analysis may be executed on the image processing device 1, or may be executed on another computer other than the image processing device 1. Extracting the range 20 of the object TO by image analysis may be configured by the image processing device 1 extracting the range 20 of the object TO by executing image analysis. Extracting the range 20 of the object TO by image analysis may be configured by having another computer extract the range 20 and directly or indirectly acquiring the results of the extracted range 20 from the other computer. Indirect acquisition may also mean acquiring via yet another computer, etc.

[0026] [Target image] The target image 2 is any image that is a target for changing an object (object TO) using an image generation model. The type of the target image 2 may be selected arbitrarily.

[0027] In one example, the target image 2 may be an advertising image 2Z. The advertising image 2Z may include any image used in advertising media. The advertising media may include, for example, digital media such as the web, digital signage, and television. The advertising image 2Z may include an image printed on paper media. That is, the advertising media may include paper media such as newspapers, magazines, hanging advertisements, and posters. The advertising image 2Z may include, for example, banner advertisements (such as direct banners), pop-up images, thumbnails, landing page images, images used in direct mail, posters, and the like. The advertising image 2Z is ​​an image that is actually used as an advertisement. The image may be an image that has been used as an advertisement in the past, or an image that has not yet been used as an advertisement. According to one example of this embodiment, a mask 3 can be obtained that makes it easier to change the object TO in a scene where the object TO shown in the advertisement image 2Z is ​​to be changed.

[0028] The type of target image 2 does not have to be limited to an advertising image 2Z. The target image 2 may include any image that can be output in any medium (digital medium, paper medium, etc.). In another example, the target image 2 may be any image used in other media such as games, manga, video distribution, etc. For example, the target image 2 may be a thumbnail used in other media.

[0029] The image processing device 1 may acquire the target image 2 in which the target object TO appears by any method. In one example, the image processing device 1 may acquire the target image 2 by specification (input) by an operator. The operator may input the target image 2 by any method. For example, the operator may input the target image 2 by operating the image processing device 1, transmitting it from a user terminal, using a storage medium, or other known method.

[0030] In one example, the image processing device 1 may acquire an image placed in a specified storage area as the target image 2. For example, an advertising image (advertising image 2Z) may be placed in a predetermined storage area and distributed as an advertisement. The predetermined storage area may be selected arbitrarily as long as it is accessible by the image processing device 1 and the user's terminal. The predetermined storage area may be composed of memory resources of the image processing device 1, memory resources of another computer, an external storage device (including a storage medium), etc. The external storage device may include a data server such as a network-attached storage (NAS). The image processing device 1 may acquire an advertising image placed in a predetermined storage area as the target image 2 and generate a mask 3 for the advertising image. A new advertising image may be generated from the advertising image using the generated mask 3. The generated new advertising image may be distributed as an advertisement by being placed in a predetermined storage area in place of the original advertising image. This allows the advertising image to be distributed to be changed.

[0031] In one example, the image processing device 1 may acquire the target image 2 from a plurality of images (candidates). The plurality of images may be stored in any storage area. The storage area may be, for example, the memory resources of the image processing device 1, the memory resources of another computer, an external storage device (including a storage medium), or the like. The target image 2 may be selected from the plurality of images in any manner. For example, the image processing device 1 may randomly select the target image 2 from the plurality of images. For example, the image processing device 1 may select an image that satisfies a predetermined condition as the target image 2. The predetermined condition may be defined arbitrarily. If each image candidate for the target image 2 is an advertising image, the predetermined condition may include, for example, the delivery period exceeding a threshold, the advertising effectiveness falling below a threshold, etc. The advertising effectiveness may be measured using known methods, such as estimated click-through rate (estimated CTR), relevance, impression share (IS), impression share loss rate (IS loss rate), conversion rate (CVR), return on advertising spend (ROAS), bounce rate, keyword relevance, etc. The threshold may be arbitrarily defined.

[0032] As long as the computer can interpret it as an image, the data format of the target image 2 is not particularly limited and may be appropriately selected depending on the embodiment. The target image 2 may be obtained in a data format such as JPEG, PNG, GIF, TIFF, BMP, HEIF, HEIC, MPEG, SVG, AI, PSD, PDF, PPT, PPTX, RAW, etc. The target image 2 may also be divided into layers (AI, PSD, etc.). The target image 2 may be a composite image containing images. The data may be available as at least part of the format (pdf, ppt, pptx, etc.).

[0033] [Object] The object TO may include any object (person, object, etc.) that can be detected by image analysis. The object TO may be a single object or multiple objects (a group of objects). The object TO may also be a component of an object, such as symbolic information, graphic information, etc., attached to the object. Symbolic information may include letters, numbers, symbols, etc. Graphic information may include drawings such as characters.

[0034] In one example, the target object TO may be a target person TZ. The target person TZ may be a real person or a fictional person (including a character). The target person TZ may include a virtual character. According to one example of this embodiment, in a scene where the target person TZ is changed, a mask 3 can be obtained that makes it easier to change the target object TO (target person TZ). In addition, the target object TO may include any kind of subject, such as a commodity, an animal, etc.

[0035] The target object TO may be an object to be changed among the objects extracted by image analysis. In one example, all objects extracted by image analysis may be treated as the target object TO. That is, the image analysis may be configured to extract the range of objects limited to the type that will become the target object TO. Also, in one example, only at least some objects selected from the objects extracted by image analysis may be treated as the target object TO. That is, the image analysis may be configured to extract the range of objects without being limited to the type that will become the target object TO. The target object TO may be selected by any method from the objects whose ranges have been extracted. The target object TO may be selected by, for example, accepting designation by an operator, selecting an object that satisfies a predetermined condition, or the like. The predetermined condition may include, for example, that the object falls into a predetermined type (such as being a person), etc.

[0036] [Padding area] The padding region 27 is a region provided to expand the extracted range 20 and may be a region added around the original range 20 of the object TO. In other words, the padding region 27 may be a difference region between the original range 20 of the object TO and the expanded range 25. The padding region 27 may be provided around the entire range 20, or only a portion of the range 20. In the example of FIG. 1, the padding region 27 is provided around the entire range 20. However, the range in which the padding region 27 is provided is not limited to this example. There may be directions around the range 20 in which the padding region 27 is not provided. For example, if the object TO is a target person TZ, it is unlikely that the target person TZ will be deformed downward in a change using an image generation model. Therefore, the padding region 27 may not be provided below the object TO (target person TZ), but may be provided in other directions.

[0037] The method of enlarging the range 20 (i.e., providing the padding region 27) is not particularly limited and may be defined appropriately depending on the embodiment. In one example, the enlarged range 25 may be obtained by enlarging the extracted range 20 while maintaining its aspect ratio. In another example, the enlarged range 25 may be obtained by enlarging the range 20 in the direction in which the padding region 27 is provided. The degree of enlargement (enlargement ratio) may be specified by an arbitrary specified value. The specified value may be specified by an operator or may be provided in advance in the program. The specified value may indicate the degree of enlargement as an absolute value or a relative value. The specified value may be determined from the dimensions of the target object TO, for example, N% of the vertical / horizontal lengths. A known method may be used to enlarge the range 20.

[0038] It should be noted that either the process of expanding the range 20 or the process of generating the mask 3 may be performed first. In one example, expanding the range 20 and generating the mask 3 may be performed by expanding the extracted range 20, obtaining the expanded range 25, and then generating the mask 3. In another example, expanding the range 20 and generating the mask 3 may be performed by The mask 3 may be provisionally generated in the extracted range 20, and the extracted range 20 (change area 30) may be expanded to generate a mask 3 in which the expanded range 25 is specified in the change area 30.

[0039] [mask] The mask 3 may be data that is input to the image generation model together with the target image 2 and instructs the image generation model which areas in the target image 2 are to be changed and which areas are not allowed to be changed. The change area 30 may be an area that instructs the image generation model to change. The change-prohibited area 35 may be an area that instructs the image generation model not to allow changes. As long as the change area 30 and the change-prohibited area 35 can be instructed, the data format of the mask 3 is not particularly limited and may be determined appropriately depending on the embodiment. In one example, the mask 3 may be composed of an image of the same size as the target image 2. The mask 3 may be configured so that the value of a pixel in the change area 30 indicates that the change is permitted, and the value of a pixel in the change-prohibited area 35 indicates that the change is not permitted.

[0040] [Mask Output] The content of the output process of the mask 3 may be determined appropriately depending on the embodiment. In one example, the image processing device 1 may output the mask 3 as is. In another example, the image processing device 1 may perform predetermined information processing on the mask 3. The image processing device 1 may output the result of performing the predetermined information processing. The predetermined information processing may include, for example, a process of generating a new image from the target image 2 using the mask 3.

[0041] FIG. 2 schematically illustrates an example of a scene in which a mask 3 according to this embodiment is output. In one example, outputting the generated mask 3 may be configured by inputting the target image 2 and the mask 3 into an image generation model M0, and causing the image generation model M0 to generate a new image 5 in which the object TO in the modification area 30 specified by the mask 3 has been modified. This allows a new image 5 in which the object TO has been modified into a new object TO2 to be obtained. The image processing device 1 may acquire the generated new image 5 from the image generation model M0. The image processing device 1 may output the acquired new image 5. If the target image 2 is an advertising image 2Z, outputting the new image 5 may include replacing the original advertising image 2Z with a new image 5 (new advertising image). The data format of the new image 5 may be selected as appropriate depending on the embodiment of the image generation model M0 to be used, etc. The new image 5 may be obtained in a data format such as JPEG, PNG, GIF, TIFF, BMP, HEIF, HEIC, MPEG, SVG, AI, PSD, PDF, PPT, PPTX, or RAW. In one example, the data format of the new image 5 may be the same as that of the target image 2. According to one example of the present embodiment, at least a part of the process for obtaining the new image 5 can be automated. This is expected to reduce the effort required to generate the new image 5.

[0042] (change) Changing the object TO may involve changing at least some of the attributes of the object TO. Attributes to be changed may include personality, type, appearance, state (posture, etc.), etc. In one example, if the object TO is a target person TZ, changing the object TO (target person TZ) may include changing the personality of the target person TZ (i.e., changing the person), as well as changing the appearance (appearance) of the same person, changing the posture, etc. The appearance to be changed may include, for example, hairstyle, hair color, clothing (clothing, accessories, etc.), ornaments (accessories, glasses, hats, etc.), age, facial expression, body type, etc. Accessories may include glasses, hats, rings, combs, hairpins, necklaces, cufflinks, tie clips, etc. Changing the posture may include changing at least one of the position and orientation of at least some body parts of the target person TZ, such as changing the direction of the face or the position of the hands, etc. Note that changing the appearance may include adding any new object, such as adding glasses to a naked-eye person or adding a new decoration to a product. This may involve excluding the object TO, in which case the new object TO2 in the new image 5 may be the background.

[0043] (Image generation model) The configuration of the image generation model M0 is not particularly limited as long as it has the ability to generate a new image in which the attributes of the object in the modification region have been modified from the image and the mask, and may be determined appropriately depending on the embodiment. In one example, the image generation model M0 may be configured by a trained machine learning model. The configuration (type, structure, etc.) of the machine learning model may be determined arbitrarily. The trained machine learning model may include a large-scale generative model such as a large-scale visual language model (VLM).

[0044] FIG. 3 schematically illustrates an example of an image generation model M0 (large-scale visual language model M1) according to this embodiment. The image generation model M0 may be configured using a large-scale visual language model M1. Known models such as stable diffusion, flux, and ideogram may be used for the large-scale visual language model M1. The large-scale visual language model M1 may be a multimodal model that can also accept data other than images, such as audio data. When the large-scale visual language model M1 is used, inputting the target image 2 and mask 3 to the image generation model M0 may be configured by inputting a prompt including the target image 2, mask 3, and instruction 4 to the large-scale visual language model M1.

[0045] The instruction 4 may be appropriately configured to instruct the large-scale visual language model M1 to change the object TO within the modification area 30 indicated by the mask 3. As long as it is possible to instruct the modification of the object TO, the configuration of the instruction 4 is not particularly limited and may be appropriately determined depending on the embodiment. In one example, the instruction 4 may be configured as text in a natural language, such as "Please modify the object within the mask." The instruction 4 may also be configured as instruction data (tokens, etc.) other than text that can be interpreted by the large-scale visual language model M1.

[0046] In addition, in one example, the instruction 4 may include a specification of the changes (the attributes of the object TO2 after the change). This makes it possible to obtain a new image 5 in which the object TO2 is drawn according to the specification of the instruction 4. The changes may be indicated directly, for example, by saying "Please change to a different person" or "Please add glasses." The changes may also be indicated indirectly, for example, by saying "Please change the person to match the current trend."

[0047] The computational processing of the image generation model M0 (including the large-scale visual language model M1) may be executed on the image processing device 1, or may be executed on a computer other than the image processing device 1. Having the image generation model M0 generate a new image 5 may include executing computational processing of the image generation model M0 by the image processing device 1, or may include transmitting the target image 2 and mask 3 to another computer and having the other computer execute computational processing of the image generation model M0. The image processing device 1 may obtain the generated new image 5 directly or indirectly from the other computer.

[0048] [Specify prohibited areas] 4 schematically shows an example of a scene in which a disallowed area 29 according to this embodiment is set. In one example, the image processing device 1 may accept designation of a disallowed area 29 in which no changes are permitted in the target image 2. In response to this, generating the mask 3 may be configured to generate the mask 3 so that the change area 30 does not include the designated disallowed area 29. The disallowed area 29 may be designated by any method. For example, the disallowed area 29 may be designated by a method such as designation by an operator or extraction by image analysis. In one example, the image processing device 1 may extract a range of text written therein (for example, a rectangular range that includes the range of text written therein) by image analysis, and designate the extracted range of text written therein as the disallowed area 29.

[0049] The disallowed region 29 may be specified in any range within the target image 2. In one example, the disallowed region 29 may be specified so as to at least partially overlap with the range 20 of the object TO or the enlarged range 25. The disallowed region 29 may be specified outside the range 20 of the object TO and the enlarged range 25. The change region 30 not including the disallowed region 29 may mean that the change prohibited region 35 includes the disallowed region 29. The image processing device 1 may generate the mask 3 so that the disallowed region 29 becomes the change prohibited region 35. If the enlarged range 25 overlaps with the disallowed region 29, the image processing device 1 may set the overlapping portion with the disallowed region 29 as the change prohibited region 35, and set the change region 30 excluding this overlapping portion.

[0050] According to one example of this embodiment, even within the range of the object TO (range 20, range 25), a portion where modification is not permitted can be designated as the disallowed region 29. This makes it possible to obtain a mask 3 for modifying the object TO while avoiding modification of that portion (disallowed region 29).

[0051] §2 Configuration example [Hardware configuration] 5 is a diagram illustrating an example of a hardware configuration of the image processing device 1 according to this embodiment. In one example, the image processing device 1 may be configured as a computer to which a control unit 11, a storage unit 12, a communication module 13, an input device 14, and an output device 15 are electrically connected.

[0052] The control unit 11 is configured to execute image processing based on a program and various data. For example, the control unit 11 includes a hardware processor such as a CPU (Central Processing Unit), a RAM (Random Access Memory), and a ROM (Read Only Memory). The control unit 11 (CPU) is an example of a processor resource.

[0053] The storage unit 12 is configured to hold any data. For example, the storage unit 12 may include a hard disk drive, a solid state drive, a semiconductor memory, etc. The storage unit 12, RAM, and ROM are examples of memory resources. In one example of this embodiment, the storage unit 12 may store various information such as a program 81. The program 81 is a program for causing the image processing device 1 to execute information processing related to the generation of the mask 3 (see FIG. 7, which will be described later). The program 81 includes a series of instructions for the image processing.

[0054] In one example, the program 81 may be stored in a storage medium 91 instead of or together with the storage unit 12. The storage medium 91 is configured to store various information (such as the stored program) by electrical, magnetic, optical, mechanical, or chemical action so that a machine such as a computer can read the information. The storage unit 12 and the storage medium 91 are examples of non-transitory storage media. The image processing device 1 may acquire the program 81 from the storage medium 91. The storage medium 91 may be a disk-type storage medium (such as a CD or DVD) or a non-disk-type storage medium such as a semiconductor memory (such as a flash memory). Any drive device may be used to read information stored in the storage medium 91. The type of drive device may be selected depending on the storage medium 91. The drive device may be connected to the image processing device 1 in any manner. The storage medium 91 may include an external storage device.

[0055] In one example, when a trained machine learning model is used for image analysis and the trained machine learning model is deployed in the image processing device 1, model data representing the trained machine learning model may be stored in at least one of the storage unit 12 and the storage medium 91. As long as the calculation of the trained machine learning model can be reproduced, the configuration of the model data is not particularly limited and may be determined appropriately depending on the embodiment. For example, the model data may include values ​​of calculation parameters adjusted by machine learning, the structure of the model (e.g., the structure of a neural network), etc. The model data may be incorporated into the program 81. Similarly, in one example, when an image generation model M0 (including a large-scale visual language model M1) for generating a new image 5 is deployed in the image processing device 1, model data representing the image generation model M0 may be stored in at least one of the storage unit 12 and the storage medium 91. The model data of the image generation model M0 may be incorporated into the program 81. In addition, in one example, a plurality of images that are candidates for the target image 2 may be stored in at least one of the storage unit 12 and the storage medium 91.

[0056] The communication module 13 is configured to perform wired or wireless communication via a network. The communication module 13 may be configured, for example, by a wired LAN (Local Area Network) module, a wireless LAN module, or the like. The network standard may be selected appropriately depending on the embodiment. For example, the type of network may be selected appropriately from the Internet, a wireless communication network, a mobile communication network, a telephone network, a dedicated network, or the like. The image processing device 1 may perform data communication with another computer via the communication module 13.

[0057] The input device 14 is configured to accept input of information. The input device 14 may be configured, for example, by an imaging device, a microphone, a mouse, a keyboard, a touch panel, an operator, etc. The output device 15 is configured to output information. The output device 15 may be configured, for example, by a display, a speaker, etc. The image processing device 1 may be operated using the input device 14 and the output device 15. The input device 14 and the output device 15 may be directly connected to the image processing device 1, or may be indirectly connected via at least one of the communication module 13 and an external interface. The external interface may be appropriately configured to connect to an external device via a wired or wireless connection, for example, a USB (Universal Serial Bus) port, a dedicated port, etc. The input device 14 and the output device 15 may be at least partially integrated into a touch panel display, etc.

[0058] It should be noted that, with regard to the specific hardware configuration of the image processing device 1, components can be omitted, replaced, or added as appropriate depending on the embodiment. For example, the control unit 11 may include multiple hardware processors. The hardware processors may be a microprocessor, an FPGA (field-programmable gate array), a DSP (digital signal processor), a GP The communication module 13, the input device 14, and the output device 15 may be configured by a graphics processing unit (U), an application specific integrated circuit (ASIC), etc. Either one of them may be omitted. The program 81 may be stored in an external storage device such as a NAS. An external storage device is also an example of a non-transitory storage medium. The image processing device 1 may be configured with multiple computers. In this case, the hardware configuration of each computer may or may not be the same. The image processing device 1 may be a computer designed specifically for the service provided, as well as a general-purpose server device, a general-purpose PC (Personal Computer), a notebook PC, a terminal device, etc. The terminal device may be a smartphone, This may include a tablet device, etc.

[0059] [Software configuration] 6 schematically shows an example of the software configuration of the image processing device 1 according to this embodiment. The control unit 11 executes instructions included in the program 81 stored in the storage unit 12 using the CPU. As a result, the image processing device 1 operates as a computer including an acquisition unit 111, an extraction unit 112, an area designation unit 113, an enlargement processing unit 114, a mask generation unit 115, and an output processing unit 116 as software modules. That is, in one example, each software module of the image processing device 1 may be realized by the control unit 11 (CPU).

[0060] The acquisition unit 111 is configured to acquire a target image 2 depicting the target object TO. The extraction unit 112 is configured to extract a range 20 of the target object TO depicted in the target image 2 by image analysis. The region designation unit 113 is configured to accept designation of a disallowed region 29 in the target image 2 where modification is not permitted. The enlargement processing unit 114 is configured to enlarge the extracted range 20 to provide a padding region 27 around the extracted range 20. The mask generation unit 115 is configured to generate a mask 3 that designates a region within the enlarged range 25 as a modification region 30 and designates a region outside the enlarged range 25 as a modification-prohibited region 35. When the disallowed region 29 is designated, generating the mask 3 may be configured to generate the mask 3 so that the specified disallowed region 29 is not included in the modification region 30. The output processing unit 116 is configured to output the generated mask 3. In one example, outputting the generated mask 3 may be configured to input the target image 2 and the mask 3 to an image generation model M0, and have the image generation model M0 generate a new image 5 in which the object TO in the modification region 30 designated by the mask 3 has been modified. In this case, the output processing unit 116 may be further configured to obtain the generated new image 5 from the image generation model M0, and output the obtained new image 5.

[0061] In one example of this embodiment, each software module of the image processing device 1 is implemented by a general-purpose CPU. However, the method of implementing each of the above modules is not limited to this example and may be changed as appropriate depending on the embodiment. Some or all of the above software modules may be implemented by one or more dedicated processors or chipsets. Each of the above modules may be implemented as a hardware module. With regard to the software configuration of the image processing device 1, modules may be omitted, replaced, or added as appropriate depending on the embodiment.

[0062] §3 Example of operation FIG. 7 is a flowchart showing an example of a processing procedure for generating a mask 3 by the image processing device 1 according to this embodiment. The image processing device 1 (controller 11) is configured to execute the processing of each of the following steps in accordance with instructions included in the program 81. The following processing procedure is an example of an image processing method executed by a computer. The following processing procedure is merely an example, and each step may be changed as much as possible. Furthermore, steps in the following processing procedure may be omitted, replaced, or added as appropriate depending on the embodiment.

[0063] (Step S101) In step S101, the control unit 11 operates as the acquisition unit 111. That is, the control unit 11 acquires a target image 2 in which the target object TO is drawn.

[0064] The target image 2 may be acquired by any method. In one example, the control unit 11 may accept input of the target image 2 from an operator. The control unit 11 may acquire an image placed in a specified storage area as the target image 2. The control unit 11 may acquire the target image 2 from a plurality of images (candidates). The type of the target image 2 may also be selected arbitrarily. In one example, the target image 2 may be an advertising image 2Z. Once the target image 2 is acquired, the control unit 11 proceeds to the next step S102.

[0065] (Step S102) In step S102, the control unit 11 operates as the extraction unit 112. That is, the control unit 11 extracts the range 20 of the object TO appearing in the target image 2 by image analysis.

[0066] The target object TO may include any object that can be detected by image analysis. In one example, the target object TO may be a target person TZ. After extracting the range 20 of the target object TO, the control unit 11 proceeds to the next step S103.

[0067] (Steps S103 to S105) In step S103, the control unit 11 operates as the area designation unit 113. That is, the control unit 11 accepts designation of a disallowed area 29 in which changes are not permitted in the target image 2. If a disallowed area 29 is not designated, step S103 may be omitted.

[0068] In step S104, the control unit 11 operates as the enlargement processing unit 114. That is, the control unit 11 enlarges the extracted range 20 to provide a padding area 27 around the extracted range 20.

[0069] In step S105, the control unit 11 operates as a mask generation unit 115. That is, the control unit 11 generates a mask 3 that designates the area within the enlarged range 25 in the target image 2 as a change area 30 and designates the area outside the enlarged range 25 as a change-prohibited area 35. In one example, when a disallowed area 29 is designated, the control unit 11 may generate the mask 3 so that the change area 30 does not include the designated disallowed area 29.

[0070] The processing order of steps S103 to S105 is not limited to the example in Fig. 7 and may be changed as appropriate depending on the embodiment. For example, the processing of step S103 may be executed before step S102, may be executed at least partially in parallel with the processing of step S102, may be executed during the processing of step S102, or may be executed after step S104.

[0071] In one example, the control unit 11 may enlarge the extracted range 20 and generate the mask 3 after obtaining the enlarged range 25. In another example, the control unit 11 may temporarily generate the mask 3 using the extracted range 20, and then enlarge the extracted range 20 (change area 30) to generate the mask 3 in which the enlarged range 25 is specified as the change area 30.

[0072] For example, if the target object TO is a target person TZ, in the image analysis of step S102, the control unit 11 may extract a range of text along with the range 20 in which the target person TZ appears. In step S103, the control unit 11 may designate the extracted range of text as a not-permitted area 29. In steps S104 and S105, the control unit 11 may form an enlarged range 25 by enlarging the range 20 in which the target person TZ appears. If the not-permitted area 29 (range of text) and the enlarged range 25 overlap, the control unit 11 may generate the mask 3 by excluding the overlapping range and setting the enlarged range 25 as a change area 30, and setting the remaining range as a change-prohibited area 35.

[0073] After generating mask 3, the control unit 11 advances the process to the next step S106.

[0074] (Step S106) In step S106, the control unit 11 operates as the output processing unit 116. That is, the control unit 11 outputs the generated mask 3.

[0075] The output contents and output destination may be determined appropriately depending on the embodiment. In one example, the control unit 11 may output the mask 3 as is. In another example, the control unit 11 may perform predetermined information processing on the mask 3 and output the result of the predetermined information processing. For example, the control unit 11 may input the target image 2 and the mask 3 to the image generation model M0, causing the image generation model M0 to generate a new image 5 in which the target object TO in the modification area 30 specified by the mask 3 has been modified. In this case, the control unit 11 may acquire the generated new image 5 from the image generation model M0. The control unit 11 may output the acquired new image 5. The configuration of the image generation model M0 may be determined appropriately depending on the embodiment. In one example, the image generation model M0 may be configured using a large-scale visual language model M1. In another example, the mask 3 and the new image 5 may be output to a RAM, a memory unit 12, an output device 15, a storage medium, or the like. The host 91 may be another computer, an external storage device, etc.

[0076] After outputting the mask 3, the control unit 11 ends the processing procedure according to this operation example. The control unit 11 may repeatedly execute the processes of steps S101 to S106 at any timing. In one example, the control unit 11 may generate the mask 3 for the given target image 2 by executing the processes of steps S102 to S106 at any timing when the target image 2 is given.

[0077] (Features) In this embodiment, by expanding the range 20 of the object TO through the processing of step S104, a padding region 27 is provided around the range 20 of the object TO. When a new image (new image 5) is obtained from the target image 2 using the image generation model (image generation model M0), this padding region 27 can reduce restrictions on the shape of the object TO. In other words, providing the padding region 27 can absorb changes in shape due to changes to the object TO. Therefore, according to this embodiment, a mask 3 can be obtained that makes it easier to change the object TO.

[0078] §4 Variations Although the embodiments of the present disclosure have been described in detail above, the above description is merely an example of the present disclosure in every respect. The processes and means described in the present disclosure can be freely combined and implemented as long as no technical contradiction occurs. Furthermore, various improvements or modifications may be made to the above embodiments as appropriate. For example, the following modifications are possible. Note that, in the following, the same reference numerals are used for components similar to those in the above embodiments, and descriptions of the same points as those in the above embodiments are omitted as appropriate. The following modifications can be combined as appropriate.

[0079] <4.1> In the processing procedure of the image processing device 1 according to the above embodiment, steps may be omitted, replaced, or added. For example, the processing of step S103 may be omitted. When the processing of step S103 is omitted, the area designation unit 113 may be omitted from the software configuration of the image processing device 1.

[0080] <4.2> Expanding the padding area 27 makes it easier to change the object TO. On the other hand, it may lead to deterioration in quality, such as unwanted modifications. Therefore, in the above embodiment, when generating a new image 5, the image processing device 1 may be configured to evaluate whether the generated new image 5 satisfies a predetermined quality condition.

[0081] FIG. 8 schematically illustrates an example of a scene in which quality conditions are evaluated in the modified example. In one example, the control unit 11 (output processing unit 116) may be configured to further determine whether the acquired new image 5 satisfies a predetermined quality condition. If it is determined that the new image 5 satisfies the predetermined quality condition, the control unit 11 may accept the new image 5. On the other hand, if it is determined that the new image 5 does not satisfy the predetermined quality condition, the control unit 11 may not accept the new image 5. In this case, the control unit 11 (enlargement processing unit 114, mask generation unit 115, output processing unit 116) may be configured to reduce the enlargement rate of the extracted range 20 from the original enlargement rate, and then provide a padding region 27, generate a mask 3, and output the mask 3 again. A new image 5 may be generated again by outputting the mask 3 (inputting the target image 2 and the mask 3 to the image generation model M0).

[0082] For convenience of explanation, the original expanded range 25 is also referred to as range 251, and the original modified range 3 0 is also referred to as modified area 301, the original mask 3 is also referred to as mask 31, the new image 5 generated using mask 31 is also referred to as new image 51, and the new object TO2 drawn in new image 51 is also referred to as object TO21. Furthermore, if it is determined that new image 51 does not satisfy the quality conditions, the enlarged range 25 obtained by lowering the original enlargement rate is also referred to as range 252, the modified area 30 defined by range 252 is also referred to as modified area 302, the obtained mask 3 is also referred to as mask 32, the new image 5 generated using mask 32 is also referred to as new image 52, and the new object TO2 drawn in new image 52 is also referred to as object TO22.

[0083] The predetermined quality condition may be defined as appropriate depending on the embodiment. For example, if the target image 2 includes text, the predetermined quality condition may include the absence of text corruption. Text corruption may include poor text reproduction, such as the appearance of illegible characters, breaks in words, characters appearing in unnatural positions, or unnatural character strings. Text corruption may be evaluated as appropriate using image analysis. For example, text may be extracted from each of the original target image 2 and the new image 5 using image analysis, and whether or not the text has been corrupted may be evaluated based on the relevance rate of the extracted text. As a specific example, if the relevance rate of the text extracted from each of the original target image 2 and the new image 5 is equal to or exceeds a threshold, it may be evaluated that the text in the new image 5 is not corrupted. If not, it may be evaluated that the text in the new image 5 has been corrupted. The threshold may be defined as appropriate.

[0084] In one example, the predetermined quality condition may include the absence of defects other than text. Defects other than text may include, for example, abnormalities in the object TO2, violation of physical laws, incongruity with the background, unnatural texture, the occurrence of artifacts, etc. Abnormalities in the object TO2 may include, for example, an unnatural shape of the object TO2, such as the number of fingers not being five if the object TO2 is a human subject. Violation of physical laws may include, for example, a mismatch between light sources and shadows, unnatural light reflections, the placement of objects that defies gravity, etc. Incongruity with the background may include unnatural blurring of edges, unclear boundaries between the object TO2 and the background, inconsistencies in perspective, etc. Unnatural texture may include, for example, uneven texture of the object TO2, the occurrence of unnatural patterns (repeated patterns, etc.), etc. The occurrence of artifacts may include, for example, the occurrence of noise, distortion, the appearance of unintended lines or shapes, the occurrence of mosaic-like patterns, etc. Defects other than text may be evaluated appropriately using image analysis.

[0085] In one example, if the target image 2 is an advertisement image 2Z, the predetermined quality condition may include whether the object TO2 depicted in the new image 5 is consistent with the advertisement content. Consistency with the advertisement content may be evaluated appropriately by image analysis. For example, if the target image 2 is a job advertisement, it may be evaluated whether the object TO2 is suitable for the job advertisement scene.

[0086] Image analysis for each quality condition may be performed using a known method, such as a general image analysis method or a method using a trained machine learning model. Each quality condition may be evaluated rule-based or using a trained machine learning model. The trained machine learning model may include a large-scale generative model such as a large-scale visual language model (VLM). Known models such as ChatGPT, Claude, and Qwen may be used as the large-scale generative model.

[0087] In addition, in one example, when the target image 2 is an advertising image 2Z, the predetermined quality condition may include that the advertising effect predicted for the generated new image 5 exceeds a threshold. The advertising effect of the new image 5 may be predicted by any method. In one example, the advertising effect of the new image 5 may be predicted by a known method such as Non-Patent Document 3.

[0088] In addition, in one example, when restricting significant alterations, the predetermined quality condition is the difference between the original target image 2 and the new target image. The similarity may include whether the similarity between the target image 2 and the new image 5 exceeds a threshold or is equal to or greater than a threshold. The similarity may be calculated as appropriate by comparing the target image 2 with the new image 5. Whether the similarity satisfies the conditions may be evaluated using a trained machine learning model. As above, the trained machine learning model that evaluates the similarity may include a large-scale generative model such as a large-scale visual language model (VLM).

[0089] The arithmetic processing for determining whether the new image 5 satisfies the predetermined quality condition may be executed on the image processing device 1, or may be executed on another computer other than the image processing device 1. Determining whether the new image 5 satisfies the predetermined quality condition may be configured by executing a determination processing by the image processing device 1. Determining whether the new image 5 satisfies the predetermined quality condition may be configured by causing another computer to execute a determination processing and directly or indirectly acquiring the determination result by the other computer.

[0090] Note that the amount by which the magnification rate of the range 20 is reduced when the new image 5 does not satisfy the predetermined quality conditions may be appropriately defined depending on the embodiment. The amount by which the magnification rate is reduced may be specified by an operator or may be set in advance in the program. As with the degree of enlargement (magnification rate) in the above embodiment, the amount by which the magnification rate is reduced may be defined as an absolute value or a relative value. In one example, the magnification rate of the range 20 may be defined according to the number of executions, such as N% the first time, M% the second time, etc. Furthermore, the image processing device 1 (control unit 11) may repeatedly execute a series of processes from generating the mask 3 to generating the new image 5 until the generated new image 5 satisfies the predetermined quality conditions.

[0091] (Processing Procedure) FIG. 9 is a flowchart showing an example of a processing procedure for generating a mask 3 by the image processing device 1 according to this modified example. The image processing device 1 (controller 11) is configured to execute the processing of each of the following steps in accordance with instructions included in the program 81. The following processing procedure is also an example of an image processing method executed by a computer. The following processing procedure is merely an example, and each step may be modified as much as possible. Furthermore, steps in the following processing procedure may be omitted, replaced, or added as appropriate depending on the embodiment.

[0092] The processing of steps S101 to S105 may be performed in the same manner as in the above embodiment. As a result, a mask 3 (31) can be obtained in which the enlarged range 25 (251) is set in the modification area 30 (301). In step S1061, the control unit 11 operates as the output processing unit 116. As a result, the control unit 11 inputs the target image 2 and the mask 3 to the image generation model M0, causing the image generation model M0 to generate a new image 5 in which the object TO in the modification area 30 specified by the mask 3 has been modified. As a result, a new image 5 (51) in which a new object TO2 (TO21) is drawn can be generated.

[0093] In step S107, the control unit 11 operates as the output processing unit 116 to acquire the generated new image 5 from the image generation model M0. In step S108, the control unit 11 operates as the output processing unit 116 to determine whether the acquired new image 5 satisfies a predetermined quality condition.

[0094] If it is determined that the new image 5 satisfies the predetermined quality condition, the control unit 11 proceeds to the next step S109. On the other hand, if it is determined that the new image 5 does not satisfy the predetermined quality condition, the control unit 11 reduces the magnification rate of the extracted range 20 from the original magnification rate, returns the process to step S104, and executes the process again from step S104. This allows for obtaining a mask 3 (32) in which a range 25 (252) wider than the extracted range 20 and narrower than the original range 251 is set in the modified region 30 (302). Then, using this mask 3 (32), a new image 5 (52) in which a new object TO2 (TO22) is drawn is generated. ) can be generated. The processes of steps S104 to S107 may be repeated until the new image 5 satisfies a predetermined quality condition.

[0095] In step S109, the control unit 11 operates as the output processing unit 116 to output the acquired new image 5 (51, 52). The output destination may be RAM, the storage unit 12, the output device 15, the storage medium 91, another computer, an external storage device, etc. After outputting the new image 5 (51, 52), the control unit 11 ends the processing procedure according to this modified example.

[0096] According to this modification, the process of step S108 evaluates whether the generated new image 5 satisfies predetermined quality conditions, and if the new image 5 does not satisfy the predetermined quality conditions, it is possible to narrow the padding area 27 and try again to generate the new image 5. This makes it possible to expect that by dynamically adjusting the size of the padding area 27 (the magnification rate of the extracted range 20), it will be easier to obtain a new image 5 with good quality.

[0097] 9, steps may be omitted, replaced, or added. For example, the number of times steps S104 to S107 are repeated may be limited to a predetermined number. That is, if a new image 5 satisfying a predetermined quality condition is not obtained even after repeating the process a predetermined number of times, the control unit 11 may interrupt the generation of the mask 3 and the new image 5 and terminate the process according to this modification. Furthermore, the control unit 11 may generate a new image 5 multiple times using the same mask 3 without changing the magnification ratio. If a new image 5 satisfying a predetermined quality condition is not obtained even after generating a new image 5 a predetermined number of times, the control unit 11 may reduce the magnification ratio of the extracted range 20 and then perform the process of steps S104 to S107 again.

[0098] §5 Experimental Examples The following experiments were conducted to verify the effects of the present disclosure, but the present disclosure is not limited to the following experimental examples.

[0099] (First Experimental Example) In the first experimental example, we verified that providing a padding area makes it easier to change the target object. First, a fictitious advertisement image was prepared as the target image. The fictitious advertisement image depicted a target person as the target object. A general-purpose object detection model was used to extract candidates (bounding boxes) for the range of the target person. The range of the target person was identified from the extracted candidates based on the area of ​​the extracted candidates. The identified range (bounding box) was given to a segmentation model (SAM: Segment Anything Model), and the segmentation model was allowed to segment the range of the target person, thereby extracting the range of the target person. The magnification ratio was set to 25% of the maximum length of the extracted range, and the extracted range was enlarged to obtain the enlarged range of the target person. In addition, a text region was extracted using an appropriate OCR (Optical Character Recognition). After excluding the extracted text region, A mask was generated by setting the enlarged area of ​​the target person as the change area and the other area as the no-change area. A new advertising image was generated by inputting the generated mask and a fictitious advertising image into an image generation model. Flux and ideogram were used in combination for the image generation model.

[0100] FIG. 10 shows the fictitious advertising image (image before modification) used in the first experimental example. FIG. 11 shows the mask generated in the first experimental example. FIG. 12 shows the new advertising image generated in the first experimental example. As shown in FIGS. 10 to 12, the object could be appropriately modified by enlarging the range of the object and using a mask that specifies the enlarged range as the modification area. This verified that providing a padding area makes it easier to modify the object.

[0101] (Second Experimental Example) In the second experiment, we verified that expanding the padding area too much can result in a deterioration in image quality, while narrowing the padding area can sometimes produce a more appropriate image. First, as in the first experiment, a fictitious advertising image was prepared as the target image. The fictitious advertising image contained a target person as the object. The target person's area was extracted using the same method as in the first experiment. The extracted area was enlarged by setting the magnification ratio to 25% of the maximum length of the extracted area, thereby obtaining an enlarged area of ​​the target person. Additionally, text areas were extracted using an appropriate OCR. The extracted text area was removed, and the enlarged area of ​​the target person was set as the change area, while the remaining area was set as the no-change area, thereby generating an initial mask. A new advertising image was generated by inputting the generated initial mask and the fictitious advertising image into an image generation model. Flux and ideogram were used in combination for the image generation model.

[0102] Figure 13 shows the fictitious advertising image (image before modification) used in the second experimental example. Figure 14 shows the first mask generated in the second experimental example. Figure 15 shows a new advertising image generated using the first mask in the second experimental example. As shown in Figure 15, the first mask had a problem where the box shape for "2-3 days a week" was inverted upside down and the "HTML" and "CSS" speech bubbles were crushed. Therefore, the magnification ratio was reset to 3% of the maximum length of the extracted range to enlarge the extracted range, resulting in an enlarged range. The obtained range was set as the change area, and the remaining area was set as the prohibited area, thereby generating a second mask. A new advertising image was generated again by inputting the generated second mask and the fictitious advertising image into the image generation model.

[0103] Figure 16 shows the second mask generated by reducing the magnification rate from the first mask in the second experimental example. Figure 17 shows a new advertising image generated using the second mask in the second experimental example. As shown in Figure 17, by reducing the magnification rate from 25% of the maximum length to 3%, the image distortion that occurred in Figure 15 was resolved. This enabled us to verify that if the new image generated does not satisfy the quality conditions, it is possible to obtain a mask that can generate a new image that satisfies the quality conditions by reducing the magnification rate of the extracted range (narrowing the padding area). [Explanation of symbols]

[0104] 1... image processing device, 11... control unit, 12... storage unit, 2...target image, TO...target, 20...range, 25...(expanded) range, 27...padding area, 30...Change area, 35...Prohibited area, 3...Mask

Claims

1. Extracting the range of the object in the target image through image analysis; Expanding the extracted region to provide a padding region around the extracted region; generating a mask that designates areas within the expanded range as modified areas and areas outside the expanded range as prohibited areas; and outputting the generated mask; a control unit configured to perform Image processing device.

2. outputting the generated mask comprises inputting the target image and the mask into an image generation model, and causing the image generation model to generate a new image in which the target object in the modification area specified by the mask has been modified; The control unit obtaining the generated new image from the image generation model; and outputting the new image obtained; and further configured to perform The image processing device according to claim 1 .

3. The control unit is further configured to determine whether the acquired new image satisfies a predetermined quality condition; when it is determined that the predetermined quality condition is not satisfied, the control unit is configured to reduce the enlargement rate of the extracted range from the original enlargement rate, and then perform the steps of providing the padding area, generating the mask, and outputting the mask again. The image processing device according to claim 2 .

4. the control unit is further configured to receive designation of a non-permission area in the target image where no change is permitted; generating the mask includes generating the mask so that the change area does not include the specified disallowed area; The image processing device according to claim 1 .

5. The object is a person. The image processing device according to claim 1 .

6. The target image is an advertising image. The image processing device according to claim 1 .

7. 1. A computer-implemented image processing method, comprising: Extracting the range of the object in the target image through image analysis; Expanding the extracted region to provide a padding region around the extracted region; generating a mask that designates areas within the expanded range as modified areas and areas outside the expanded range as prohibited areas; and outputting the generated mask; Including, Image processing methods.

8. A program for causing a computer to execute an image processing method, The image processing method includes: Extracting the range of the object in the target image through image analysis; Expanding the extracted region to provide a padding region around the extracted region; generating a mask that designates areas within the expanded range as modified areas and areas outside the expanded range as prohibited areas; and outputting the generated mask; Including, program.

Images