Methods, computer devices, and computer programs for selective background removal

Abstract

The present invention provides a method, computer apparatus, and computer program for selective background removal. The method for selective background removal includes the steps of: providing a given image with words representing the salient object in the image as text prompt candidates; generating a mask based on the salient object of the text prompt selected from the text prompt candidates; and performing selective background removal on the image using the mask.

Description

Technical Field

[0001] The following description relates to a technique for removing a background obtained by excluding a main object from an image.

Background Art

[0002] Background removal (BGR) is a technique for removing a background obtained by excluding a main object from an image, and is used, for example, in the field of generating advertisements by compositing another background with the main object in an image.

[0003] As an example, Patent Document 1 (registration date: April 11, 2019) discloses a technique for compositing a specific object image onto another background image.

[0004] A main technique for removing a background from an image can be implemented by applying a salient object detection method.

[0005] Generally, saliency in an image is determined based on image characteristics (e.g., brightness, contrast, sharpness, texture, etc.) or based on the presence or absence of a semantic object, so it is not easy to select a main object from an image.

[0006] For example, when a dog and a cat are simultaneously captured in one image, most background removal techniques segment both the dog and the cat at the same time.

Prior Art Documents

Patent Documents

[0007]

Patent Document 1

Summary of Invention

Problem to be Solved by the Invention

[0008] For background removal, it is possible to provide an intuitive interface that allows the user to easily select the main object detected from the image.

[0009] It is possible to generate a mask for the main object selected by the user and perform selective background removal based on that mask. [Means for solving the problem]

[0010] A background removal method performed on a computer device, wherein the computer device includes at least one processor configured to execute computer-readable instructions contained in memory, the background removal method comprising the steps of: providing a given image with a list of words representing salient objects as text prompt candidates using at least one processor; generating a mask based on the salient object of a text prompt selected from the text prompt candidates using at least one processor; and performing selective background removal on the image using the mask using at least one processor.

[0011] In one respect, the steps provided may include extracting words from an image through I2T (image to text) or image captioning technology.

[0012] In other words, the steps provided may include generating text related to the main object from an image, and extracting words of a predetermined pattern using a parts-of-speech system defined in an open-source library for natural language processing.

[0013] Another aspect of the process may include assigning a priority to each word based on its similarity to at least one other word that has been used recently, and then sorting the words according to that priority.

[0014] From another perspective, the other words may be a list of words used by all users who have used the service within a certain period in recent times, or a list of words used by users who have requested background removal for images.

[0015] Another aspect of the process is that the generation step may involve segmenting the image using a zero-shot learning method based on selected text prompts to generate a mask.

[0016] From another perspective, the generation step may involve generating a mask corresponding to the selected text prompt using open-set semantic segmentation.

[0017] From another perspective, the steps to be performed may include compositing the mask and the image, and generating a resulting image by removing the remaining area from the image, excluding the main object of the selected text prompt.

[0018] From another perspective, the steps to be performed may further include the step of combining a thresholded mask with the resulting image to generate a final resulting image.

[0019] From another perspective, the step of generating the resulting image may include a step of applying a degradation process to the image, and then compositing the degraded image with a mask.

[0020] From another perspective, the steps to be performed may further include the step of using a mask to filter the resulting image and generate a final resulting image.

[0021] According to still another aspect, the step of generating a result image may generate the result image through a foreground segmentation model trained to use a mask as a conditional condition.

[0022] There is provided a computer program recorded in a non-transitory computer-readable recording medium for causing a computer device to execute the background removal method described above.

[0023] There is provided a computer device, comprising at least one processor configured to execute computer-readable instructions stored in a memory, wherein the at least one processor executes, for a given image: a process of providing words representing a main object in the image as text prompt candidates; a process of generating a mask based on the main object of a text prompt selected from among the text prompt candidates; and a process of performing selective background removal on the image using the mask. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] [Figure 1] FIG. 1 is a diagram illustrating an example of a network environment according to an embodiment of the present invention. [Figure 2] FIG. 2 is a block diagram illustrating an example of a computer device according to an embodiment of the present invention. [Figure 3] FIG. 3 is a diagram illustrating an example of a method for selecting a main object. [Figure 4] FIG. 4 is a flowchart illustrating an example of a method that can be executed by a computer device according to an embodiment of the present invention. [Figure 5] FIG. 5 is a diagram illustrating an example of a part-of-speech (POS) system according to an embodiment of the present invention. [Figure 6] FIG. 6 is a diagram for explaining a word providing process for main object selection according to an embodiment of the present invention. [Figure 7] This figure shows an example of a mask-conditional background removal model in one embodiment of the present invention. [Figure 8] This figure shows an example of a user interface for a background removal process in one embodiment of the present invention. [Figure 9] This figure shows an example of a user interface for a background removal process in one embodiment of the present invention. [Figure 10] This figure shows an example of a user interface for a background removal process in one embodiment of the present invention. [Figure 11] This figure shows an example of a user interface for a background removal process in one embodiment of the present invention. Modes for carrying out the invention

[0025] Embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0026] Embodiments of the present invention relate to a technique for removing the background from an image while excluding the main object.

[0027] Embodiments including those specifically disclosed herein can detect a group of candidates for a main object from an image and perform selective background removal through user selection of the detected group of candidates.

[0028] A selective background removal system according to an embodiment of the present invention may be implemented by at least one computer device, and a selective background removal method according to an embodiment of the present invention may be executed by at least one computer device included in the selective background removal system. In this case, a computer program according to one embodiment of the present invention may be installed and executed in the computer device, and the computer device may execute the selective background removal method according to an embodiment of the present invention in accordance with the control of the executed computer program. The above-described computer program may be combined with the computer device and recorded on a computer-readable recording medium in order to have the computer execute the selective background removal method.

[0029] Figure 1 is a diagram showing an example of a network environment in one embodiment of the present invention. The network environment in Figure 1 shows an example that includes a plurality of electronic devices 110, 120, 130, 140, a plurality of servers 150, 160, and a network 170. Figure 1 is merely an example for the purpose of explaining the invention, and the number of electronic devices and servers is not limited to that shown in Figure 1. Furthermore, the network environment in Figure 1 is merely an example of an environment applicable to this embodiment, and the environment applicable to this embodiment is not limited to the network environment in Figure 1.

[0030] The multiple electronic devices 110, 120, 130, and 140 may be fixed terminals or mobile terminals implemented by computer devices. Examples of the multiple electronic devices 110, 120, 130, and 140 include smartphones, mobile phones, navigation systems, personal computers (PCs), notebook PCs, digital broadcasting terminals, PDAs (Personal Digital Assistants), PMPs (Portable Multimedia Players), and tablets. As an example, Figure 1 shows a smartphone as an example of electronic device 110, but in embodiments of the present invention, electronic device 110 may mean one of a variety of physical computer devices that can communicate with other electronic devices 120, 130, 140, and / or servers 150, 160 via the network 170 using substantially wireless or wired communication methods.

[0031] The communication method is not limited, and may include not only communication methods that utilize communication networks that can be included in network 170 (for example, mobile communication networks, wired internet, wireless internet, broadcasting networks), but also short-range wireless communication between devices. For example, network 170 may include one or more arbitrary networks such as PAN (personal area network), LAN (local area network), CAN (campus area network), MAN (metropolitan area network), WAN (wide area network), BBN (broadband network), and the Internet. Furthermore, network 170 may include, but is not limited to, one or more network topologies, including bus networks, star networks, ring networks, mesh networks, star-bus networks, tree or hierarchical networks.

[0032] Servers 150 and 160 may each be implemented by one or more computer devices that communicate with multiple electronic devices 110, 120, 130, and 140 via a network 170 to provide commands, code, files, content, services, etc. For example, server 150 may be a system that provides services (for example, an image editing service) to multiple electronic devices 110, 120, 130, and 140 connected via a network 170.

[0033] Figure 2 is a block diagram showing an example of a computer device in one embodiment of the present invention. Each of the aforementioned electronic devices 110, 120, 130, and 140, as well as each of the servers 150 and 160, may be implemented by the computer device 200 shown in Figure 2.

[0034] Such a computer device 200 may include a memory 210, a processor 220, a communication interface 230, and an input / output interface 240, as shown in Figure 2. The memory 210 is a computer-readable recording medium and may include RAM (random access memory), ROM (read-only memory), and a permanent mass storage device such as a disk drive. Here, the permanent mass storage device such as ROM or a disk drive may be included in the computer device 200 as a separate permanent storage device distinct from the memory 210. The memory 210 may also contain an operating system and at least one program code. Such software components may be loaded into the memory 210 from a computer-readable recording medium separate from the memory 210. Such a separate computer-readable recording medium may include a floppy disk drive, disk, tape, DVD / CD-ROM drive, memory card, and other computer-readable recording media. In other embodiments, the software components may be loaded into the memory 210 through a communication interface 230, which is not a computer-readable recording medium. For example, software components may be loaded into the memory 210 of the computer device 200 based on a computer program installed by a file received via the network 170.

[0035] The processor 220 may be configured to process computer program instructions by performing basic arithmetic, logic, and input / output operations. Instructions may be provided to the processor 220 by memory 210 or a communication interface 230. For example, the processor 220 may be configured to execute instructions received according to program code stored in a recording device such as memory 210.

[0036] The communication interface 230 may provide a function for the computer device 200 to communicate with other devices (for example, the recording device described above) via the network 170. For example, requests, instructions, data, files, etc., generated by the processor 220 of the computer device 200 according to program code recorded in a recording device such as memory 210 may be transmitted to other devices via the network 170 under the control of the communication interface 230. Conversely, signals, instructions, data, files, etc., from other devices may be received by the computer device 200 via the network 170 through the communication interface 230 of the computer device 200. Signals, instructions, data, etc., received via the communication interface 230 may be transmitted to the processor 220 or memory 210, and files, etc., may be recorded on a recording medium (the persistent recording device described above) that the computer device 200 may further include.

[0037] The input / output interface 240 may be a means for interface with the input / output device 250. For example, the input device may include a microphone, keyboard, or mouse, and the output device may include a display or speaker. In another example, the input / output interface 240 may be a means for interface with a device that integrates input and output functions into one, such as a touchscreen. The input / output device 250 may consist of the computer device 200 and one other device.

[0038] In other embodiments, the computer device 200 may include fewer or more components than those shown in Figure 2. However, it is not necessary to explicitly show most of the conventional components in the figure. For example, the computer device 200 may be implemented to include at least some of the input / output devices 250 described above, and may further include other components such as transceivers and databases.

[0039] The following describes specific embodiments of methods and apparatus for selective background removal.

[0040] In primary object detection, the importance of an object within an image is determined either by its image-related characteristics (e.g., brightness, contrast, texture) or by its semantic presence or absence, making it difficult to select the primary object from an image.

[0041] Background removal requires the user to directly select the main object from the image. For example, selecting the main object can be done by accurately drawing the boundary of the main object in the image, by filling in the area with a box, or by using the scribble function.

[0042] Figure 3 shows an example of the scribble function as a method for selecting the main object. As shown in Figure 3, the user can use the scribble function to select the main object by sketching directly onto image 30 as if doodling.

[0043] However, in reality, users found it difficult and burdensome to select the main object from an image, and the accuracy of the selection process decreased depending on the method used. Furthermore, if the user made a mistake in the selection process, it could necessitate restarting the selection process or affect the results of background removal.

[0044] Therefore, there is a need for technology that allows users to easily select the main object detected from an image and perform selective background removal.

[0045] The computer device 200 according to this embodiment may provide image editing services to clients through a dedicated application installed on the client or by connecting to a web / mobile site associated with the computer device 200. The computer device 200 may be configured with a computer-implemented selective background removal system. For example, the selective background removal system may be implemented as an independently operating program, or it may be implemented in an in-app form for a specific application so that it can operate on the specific application.

[0046] The processor 220 of the computer device 200 may be implemented by components for performing the following selective background removal method. Depending on the embodiment, the components of the processor 220 may be selectively included in or excluded from the processor 220. Also, depending on the embodiment, the components of the processor 220 may be separated or merged for the expression of the functions of the processor 220.

[0047] Such a processor 220 and its components may control the computer device 200 to perform steps included in the following selective background removal method. For example, the processor 220 and its components may be implemented to execute instructions from the operating system code contained in the memory 210 and the code of at least one program.

[0048] Here, the components of the processor 220 may be representations of different functions that are executed by the processor 220 in accordance with instructions provided by the program code recorded in the computer device 200.

[0049] The processor 220 may read necessary instructions from the memory 210, which is loaded with instructions related to the control of the computer device 200. In this case, the instructions read may include instructions for controlling the processor 220 to perform the steps described below.

[0050] The steps included in the selective background removal method described below may be performed in a different order than shown in the diagram, some steps may be omitted, or additional processes may be included.

[0051] The steps included in the selective background removal method may be performed on the server 150, and in some embodiments, some of the steps may be performed on the client.

[0052] Figure 4 is a flowchart showing an example of a method that a computer device can perform according to one embodiment of the present invention.

[0053] Referring to Figure 4, in step 410, if an image for background removal is provided, the processor 220 may detect major objects from the image and provide the user with a word that refers to each detected major object. The processor 220 may generate words from a given image using I2T (image to text) technology, image captioning technology, etc. I2T or image captioning technology can determine the major objects contained in an image when an image is given and create a word that represents that object. At this time, the processor 220 may assign appropriate priority and sort the words generated from the image in the process of providing them to the user. As an example, the processor 220 may use a word database and arrange the words higher in the word database. The word database may consist of words related to the overall service history, for example, the names of items that have been frequently used in advertisements targeting all users within a certain period recently. Depending on the embodiment, the word database may be composed of words based on the history of the target user (for example, the names of items that the user advertised frequently) rather than the overall service history, and individualized priorities may be assigned to the words generated from images for each user.

[0054] In step 420, the processor 220 may generate a background removal mask based on the main object corresponding to the word selected by the user from the words generated in the image. The processor 220 may generate a mask the same size as the image, but with the main object region of the user-selected word set to white and the remaining region other than the selected region set to black. In other words, the processor 220 may generate a background removal mask by setting the region to be emphasized to white and the region to be removed to black. At this time, the processor 220 may apply a technique to segment the image using a zero-shot learning method with a prompt given by the user selection when generating the mask. The processor 220 may generate a mask for the user-selected region through open-set semantic segmentation. The mask may be generated using an open-set semantic segmentation method, one of the well-known segmentation methods, which detects and segments only unknown regions without additional processing. For example, the processor 220 may generate a mask based on user-selected regions by applying a CLIP (contrastive language-image pre-training model)-based segmentation technique. In addition to applying an open semantic segmentation method on a CLIP basis such as CLIPseq, known segmentation techniques may also be used for mask generation. Depending on the embodiment, it is also possible to first determine whether an ID class (in-distribution class) was used for training, and if it is an OOD class (out-of-distribution class) that was not used for training, apply an open-world semantic segmentation method that segments the image through few-shot learning on an existing knowledge basis.

[0055] In step 430, the processor 220 may use the mask to perform background removal on the image. In this embodiment, by applying the mask to the background removal on the image, the given image may be combined with the mask, user-selected objects may be excluded through the mask, and the remaining area may be processed and removed as the background area.

[0056] Figure 5 shows an example of a part-of-speech (POS) system in one embodiment of the present invention.

[0057] When the processor 220 generates words from a given image through technologies such as I2T or image captioning, and generates a sentence rather than just a list of words, it may extract key words from the generated sentence.

[0058] For example, processor 220 may use a part-of-speech system defined in open-source libraries for natural language processing in Python (see Figure 5) to determine the part of speech of each word in a text. For instance, processor 220 may extract words that follow a predetermined pattern, as shown in Table 1, from text generated from an image, to be used as the main words for text prompts. For example, in the case of pattern 1, combinations of words in the order of adjective -> proper noun may be detected.

[0059] [Table 1]

[0060] Figure 6 is a diagram illustrating the word provisioning process for selecting a primary object in one embodiment of the present invention.

[0061] The processor 220 provides the user with words extracted from the corresponding image (i.e., words referring to the main object) for user selection of the main object detected from the image, and may provide these words in order of priority.

[0062] For example, the processor 220 may learn words and patterns that are frequently used in a service (e.g., advertising image generation) and, based on this, determine the priority of words extracted from images.

[0063] Referring to Figure 6, the word DB60 may include a list of words frequently used by all users who have used the service in a recent period, or a list of words frequently used by users who have requested background removal.

[0064] The processor 220 may calculate the similarity between the words extracted from the image and the words in the word DB 60 (S601). In other words, the processor 220 may calculate the similarity relationship between the words extracted from the image and words that have been used recently.

[0065] The processor 220 may arrange and provide the words extracted from the image based on the similarity calculated in step 601 (S602).

[0066] Therefore, when the processor 220 provides the user with words extracted from an image, it can assign higher priority to words that are similar to words in the word DB 60 and arrange them higher up. This allows the user to be provided with a group of prompt candidates for images to be removed from, arranged according to a priority based on recent cumulative history.

[0067] In this embodiment, a mask is generated by zero-shot learning using user-selected prompts, and the mask may be used as a preprocessor for background removal.

[0068] To improve image quality and performance, a technique is needed that filters background removal techniques based on a mask. In this embodiment, a mask is used as a preprocessor, and the precision of the target area selected by the user can be improved through background removal using a mask-based preprocessing base.

[0069] For example, the processor 220 may precisely reveal user-selected objects in an image by intentionally degrading the input image for background removal. Basically, the main object detection methods primarily used in background removal techniques are greatly influenced by image characteristics (brightness, contrast, sharpness, texture, etc.). Therefore, intentionally blurring areas other than the mask or lowering the brightness of pixels in a given image increases the probability that those areas will be extracted as foreground. The resulting image after background removal may be defined as shown in equation (1).

[0070]

number

[0071] Here, I(x,y) and M(x,y) represent the pixel value and mask value at position (x,y), respectively. I'(x,y) is the degraded input image, which can be defined, for example, as shown in equation (2).

[0072]

number

[0073] Here, beta can represent a constant for reducing the brightness of the image (0 ≤ Beta < 1), and Blur can represent a function for blurring the image (e.g., a Gaussian filter).

[0074] As another example, the processor 220 may filter the resulting image after background removal using a mask. The mask result will have a value between 0 and 1 for each pixel position, in which case the mask value may be interpreted as the probability that each pixel is the target object. Similarly, the background removal result will also have a value between 0 and 1 for each pixel, and this value may be interpreted as the probability that each pixel is the foreground. Therefore, the result of equation (3) can be generated for the resulting image after background removal.

[0075]

number

[0076] Here, BGR(x,y) is the resulting image after background removal, and represents the BGR value at position (x,y).

[0077] In some embodiments, M(x,y) in equation (3) may be replaced with a mask value to which a filter that enhances the characteristics of the image has been applied (for example, (M'(x,y)=Laplacian(M(x,y))). In this case, Laplacian is the Laplacian function, which corresponds to a filter used to improve the sharpness of the image.

[0078] As another example, the processor 220 may perform background removal using thresholding, one of the segmentation techniques. The processor 220 may threshold the mask for a cleaner result image and then generate the resulting image. For the resulting image after background removal, the result of equation (4) can be generated.

[0079]

number

[0080] Here, M'(x,y) may represent the mask value to which thresholding has been applied (i.e., M'(x,y)=0 if M(x,y) <Given Threshold,otherwise M(x,y))。

[0081] As another example, the processor 220 may perform background removal using a conditional background removal (BGR) model. The model may be trained to use a mask for background removal itself. Referring to Figure 7, given an input image 701, the feature extractor 710, which is configured as an image encoder, can be used to convert it into an embedding, and then the output image 703 is generated by removing the background through the foreground partitioner 720. In this embodiment, the foreground partitioner 720 may be trained using a diffusion model or the like that is effective for image partitioning, and in particular, the foreground partitioner 720 may be trained to conditionally remove the background using a mask 702 generated from the input image 701.

[0082] In addition to this method, it is also possible to select a final result obtained by performing background removal through a similar method aimed at increasing the precision of the target region.

[0083] Figures 8-11 show examples of user interfaces for background removal processes in one embodiment of the present invention.

[0084] Figures 8-10 show examples of service screens included in the background removal process.

[0085] Referring to Figure 8, the processor 220 may receive an image 701 given in accordance with the user's background removal request, and at this time, it may provide the user with words generated from the main object detection process for the image 701 as text prompt candidates 80.

[0086] In this embodiment, instead of the user directly writing the main object in the image 701 using a scribble function or the like, candidate text prompts 80 for the main object in the image 701 may be automatically extracted and provided.

[0087] This allows the user to select a specific text prompt from the text prompt candidates 80 for the main object in the image 701. As shown in Figure 9, if the user selects a specific text prompt 91 from the text prompt candidates 80, the processor 220 may use the user-selected specific text prompt 91 to generate a mask 900 for background removal.

[0088] For example, if words such as "man," "tennis racket," and "tennis ball" are provided as text prompt candidates 80 for image 701, and the user selects "tennis ball," a mask 900 may be generated in which the object region 901 corresponding to the tennis ball is set to white, and the remaining region is set to black.

[0089] The user may request background removal after confirming the main object region 901 selected by the user through the mask 900. Referring to Figure 10, the processor 220 may, in accordance with the user's request, use the mask 900 to perform background removal on the image 701 and generate and provide the user with the resulting image 1000 after background removal. In other words, the processor 220 can combine the image 701 and the mask 900 and provide the resulting image 1000 in which the remaining region, excluding the main object 1001 selected by the user, has been removed as the background region through the mask 900.

[0090] This embodiment can be linked to text-based interfaces such as conversational artificial intelligence services (e.g., chatGPT). Since the service according to the present invention can perform background removal using text prompts, it can be implemented as a user scenario using a chat room.

[0091] For example, as shown in Figure 11, suppose the processor 220 provides a chat room 1100 to a user, in which the user and the service chatbot participate. The user may upload an image 701 to the chat room 1100 and input necessary commands in the form of dialogue messages during the process. The processor 220 may provide candidate text prompts 80 for the main object in the image 701 through the chat room 1100, and may generate and provide a mask 900 with a specific text prompt 91 selected by the user. The processor 220 may, in accordance with the user's request, perform background removal on the image 701 using the mask 900, and may provide the resulting image 1000 with the background removed through the chat room 1100.

[0092] This embodiment can provide an easy-to-use interface for user selection by suggesting a group of candidate main objects in image 701, and in particular, it can provide an intuitive interface that allows users to select main objects using words.

[0093] Thus, according to embodiments of the present invention, an intuitive interface can be provided that allows the user to easily select a main object detected from an image for background removal, a mask can be generated for the main object selected by the user, and selective background removal can be performed based on that mask.

[0094] The above-described apparatus may be implemented by hardware components, software components, and / or combinations of hardware and software components. For example, the apparatus and components described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as processors, controllers, ALUs (arithmetic logic units), digital signal processors, microcomputers, FPGAs (field programmable gate arrays), PLUs (programmable logic units), microprocessors, or various devices capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and one or more software applications running on the OS. The processing unit may also respond to software execution, access data, record, manipulate, process, and generate data. For convenience of understanding, it may be described as if a single processing unit is used, but those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. Other processing configurations, such as parallel processors, are also possible.

[0095] Software may include computer programs, code, instructions, or a combination of one or more of these, which may configure a processing unit to operate as desired, or which may instruct the processing unit independently or collectively. Software and / or data may be embodied in any kind of machine, component, physical device, computer recording medium, or device for interpretation based on the processing unit or for providing instructions or data to the processing unit. Software may be distributed across a networked computer system, and may be recorded or executed in a distributed manner. Software and data may be recorded on one or more computer-readable recording media.

[0096] The method according to the embodiment may be implemented in the form of program instructions executable by various computer means and recorded on a computer-readable medium. In this case, the medium may continuously record computer-executable programs or may temporarily record them for execution or download. Furthermore, the medium may be various recording or storage means in the form of a combination of one or more hardware components, and may be a medium directly connected to a computer system or distributed on a network. Examples of media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and media configured to record program instructions such as ROM, RAM, and flash memory. Other examples of media include recording media and storage media managed by app stores that distribute applications, and sites and servers that supply and distribute various other software.

[0097] As described above, embodiments have been explained based on limited embodiments and drawings, but those skilled in the art will be able to make various modifications and variations from the above description. For example, the described technique may be performed in a different order than described, and / or the components of the described system, structure, apparatus, circuit, etc. may be combined or assembled in a different manner than described, or opposed or replaced by other components or equivalents, and still achieve suitable results.

[0098] Therefore, even if the embodiment is different, it falls within the scope of the attached claims if it is equivalent to the claims.

Claims

1. A background removal method performed on a computer device, The computer device includes at least one processor configured to execute computer-readable instructions contained in memory, The aforementioned background removal method is The steps include: providing a given image with at least one processor, and providing a word representing the main object in the image as a text prompt candidate; The steps include: generating a mask based on the main object of the text prompt selected from the text prompt candidates by at least one processor, and The step of using the mask to perform selective background removal on the image using at least one of the processors. A background removal method, including the following.

2. The steps provided above are: Steps to extract the word from the image through I2T (image to text) or image captioning technology. The background removal method according to claim 1, including the method described in claim 1.

3. The steps provided above are: The steps of generating text related to the main object in the aforementioned image, and Steps to extract words that follow a predetermined pattern using a parts-of-speech system defined in an open-source library for natural language processing. The background removal method according to claim 1, including the method described in claim 1.

4. The steps provided above are: The steps include assigning a priority order to the aforementioned words based on their similarity to at least one other word that has been used recently, and sorting the words according to the priority order. The background removal method according to claim 1, including the method described in claim 1.

5. The aforementioned other words are those that correspond to a list of words used by all users who have used the service within a certain period of time recently, or to a list of words used by users who have requested background removal for the aforementioned images. The background removal method according to claim 4, characterized by the above.

6. The above generation step is, The image is divided and the mask is generated using a zero-shot learning method that utilizes the selected text prompt. The background removal method according to claim 1, characterized by the above.

7. The above generation step is, To generate a mask corresponding to the selected text prompt through open semantic segmentation. The background removal method according to claim 1, characterized by the above.

8. The steps to be performed are: The step of combining the mask and the image to generate a result image by removing the remaining area from the image, excluding the main object of the selected text prompt. The background removal method according to claim 1, including the method described in claim 1.

9. The steps to be performed are: Step 1: Generate a final result image by combining the mask to which threshold processing has been applied with the result image. The background removal method according to claim 8, further comprising:

10. The step of generating the aforementioned result image is: The step of applying a degradation process to the aforementioned image, and then compositing the degraded image with the mask. The background removal method according to claim 8, including the method described in claim 8.

11. The steps to be performed are: The step of filtering the aforementioned result image using the mask to generate a final result image. The background removal method according to claim 8, further comprising:

12. The step of generating the aforementioned result image is: The resulting image is generated through a foreground segmentation model that has been trained to conditionally utilize the aforementioned mask. The background removal method according to claim 8, characterized by the above.

13. A computer program that causes a computer device to execute the background removal method described in any one of claims 1 to 12.

14. A computer device, At least one processor configured to execute computer-readable instructions contained in memory Includes, When the at least one processor executes the instruction, A process that provides a text prompt candidate with words representing the main objects in a given image. A process for generating a mask based on the main object of the text prompt selected from the aforementioned text prompt candidates, and A process of performing selective background removal on the image using the aforementioned mask. A computer device that processes data.

15. The aforementioned at least one processor is Assigning a priority order to the aforementioned word based on its similarity to at least one other word that has been used recently, and sorting the aforementioned words according to the priority order. The computer device according to claim 14, characterized by the above.

16. The aforementioned at least one processor is The image is divided and the mask is generated using a zero-shot learning method that utilizes the selected text prompt. The computer device according to claim 14, characterized by the above.

17. The aforementioned at least one processor is The mask and the image are combined to generate a result image from which the remaining area, excluding the main object of the selected text prompt, is removed. The mask to which thresholding has been applied and the resulting image are combined to generate the final resulting image. The computer device according to claim 14, characterized by the above.

18. The aforementioned at least one processor is The mask and the image are combined to generate a result image from which the remaining area, excluding the main object of the selected text prompt, is removed. After applying a degradation process to the aforementioned image, the degraded image is combined with the aforementioned mask. The computer device according to claim 14, characterized by the above.

19. The aforementioned at least one processor is The mask and the image are combined to generate a result image from which the remaining area, excluding the main object of the selected text prompt, is removed. The aforementioned result image is filtered using the mask to generate the final result image. The computer device according to claim 14, characterized by the above.

20. The aforementioned at least one processor is The mask and the image are combined to generate a result image from which the remaining area, excluding the main object of the selected text prompt, is removed. The resulting image is generated through a foreground segmentation model that has been trained to conditionally use the aforementioned mask. The computer device according to claim 14, characterized by the above.

Images