Method, computer device, and computer program for selective background removal

Abstract

A method, a computer device, and a computer program for selective background removal are disclosed. The method for removing a selected background may comprise the steps of: providing, as text prompt candidates, words representing salient objects in a given image; generating a mask on the basis of a salient object of a text prompt selected from among the text prompt candidates; and performing selective background removal on the image by using the mask.

Description

Method, computer device, and computer program for selective background removal

[0001] The description below is about a technique for removing the background except for the main object from an image.

[0002] Background removal technology (BGR) is a technology that removes the background except for the main object in an image. It can be used in fields such as creating advertisements by synthesizing a different background for the main object in an image.

[0003] For example, Korean Patent No. 10-1969864 (registration date: April 11, 2019) discloses a technology for synthesizing a specific object image onto another background image.

[0004] The technique of removing the background from an image can be implemented mainly by applying the salient object detection method.

[0005] In general, saliency in an image is judged by image characteristics (e.g., brightness, contrast, sharpness, texture, etc.) or by the presence or absence of semantic objects, so it is not easy to select the main object in the image.

[0006] For example, when a dog and a cat are present in an image at the same time, most background removal techniques segment the dog and the cat at the same time.

[0007] An intuitive interface can be provided for users to easily select key objects detected in an image for background removal.

[0008] You can create a mask for the main object selected by the user and perform selective background removal based on that mask.

[0009] A method for removing a background, which is executed on a computer device, comprises at least one processor configured to execute computer-readable instructions contained in a memory, wherein the method comprises the steps of: providing, by the at least one processor, a word representing a salient object in a given image as a text prompt candidate; generating, by the at least one processor, a mask based on a salient object of a text prompt selected from among the text prompt candidates; and performing, by the at least one processor, selective background removal of the image using the mask.

[0010] According to one aspect, the step of providing may include a step of extracting the word from the image through I2T (image to text) or image captioning technology.

[0011] According to another aspect, the step of providing may include a step of generating a sentence related to the main object in the image; and a step of extracting words of a predetermined pattern using a part-of-speech system defined in an open source-based library for natural language processing.

[0012] According to another aspect, the step of providing may include a step of assigning a priority based on similarity between the word and at least one other word recently used for the word and sorting the word according to the priority.

[0013] According to another aspect, the above other words may correspond to a list of words used by all users using the service over a recent period of time or a list of words used by users who requested background removal for the above image.

[0014] According to another aspect, the generating step may generate the mask by segmenting the image through a zero-shot learning method using the selected text prompt.

[0015] According to another aspect, the generating step may generate a mask corresponding to the selected text prompt through open-set semantic segmentation.

[0016] According to another aspect, the performing step may include a step of generating a result image by synthesizing the mask and the image to remove the remaining area from the image except for the main object of the selected text prompt.

[0017] According to another aspect, the performing step may further include a step of generating a final result image by synthesizing the mask to which thresholding has been applied and the result image.

[0018] According to another aspect, the step of generating the result image may include a step of performing a degradation process on the image and then synthesizing the degraded image with the mask.

[0019] According to another aspect, the performing step may further include a step of filtering the result image using the mask to generate a final result image.

[0020] According to another aspect, the step of generating the result image may generate the result image through a foreground segmentation model learned to conditionally use the mask.

[0021] A computer program stored in a non-transitory computer-readable recording medium is provided to execute the above background removal method on the computer device.

[0022] In a computer device, at least one processor configured to execute computer-readable instructions contained in a memory, the at least one processor is configured to perform the steps of: providing a word representing a main object in a given image as a text prompt candidate for a given image; generating a mask based on a main object of a text prompt selected from among the text prompt candidates; and performing selective background removal on the image using the mask.

[0023] FIG. 1 is a diagram illustrating an example of a network environment according to one embodiment of the present invention.

[0024] FIG. 2 is a block diagram illustrating an example of a computer device according to one embodiment of the present invention.

[0025] Figure 3 illustrates an example of how to select a main object.

[0026] FIG. 4 is a flowchart illustrating an example of a method that a computer device according to an embodiment of the present invention can perform.

[0027] FIG. 5 illustrates an example of a part-of-speech (POS) system in one embodiment of the present invention.

[0028] FIG. 6 is a drawing for explaining a word provision process for selecting a main object in one embodiment of the present invention.

[0029] FIG. 7 is a diagram illustrating an example of a background removal model that conditions a mask in one embodiment of the present invention.

[0030] FIGS. 8 to 11 illustrate examples of a user interface for a background removal process in one embodiment of the present invention.

[0031] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings.

[0032] Embodiments of the present invention relate to a technique for removing a background excluding a main object from an image.

[0033] Embodiments including those specifically disclosed herein can detect candidates for a primary object in an image and perform selective background removal through user selection of the detected candidates.

[0034] The selective background removal system according to embodiments of the present invention can be implemented by at least one computer device, and the selective background removal method according to embodiments of the present invention can be performed by at least one computer device included in the selective background removal system. At this time, a computer program according to an embodiment of the present invention can be installed and run on the computer device, and the computer device can perform the selective background removal method according to embodiments of the present invention under the control of the run computer program. The above-described computer program can be stored on a computer-readable recording medium so as to be combined with the computer device and cause the computer to execute the selective background removal method.

[0035] FIG. 1 is a diagram illustrating an example of a network environment according to an embodiment of the present invention. The network environment of FIG. 1 represents an example including a plurality of electronic devices (110, 120, 130, 140), a plurality of servers (150, 160), and a network (170). FIG. 1 is merely an example for explaining the invention, and the number of electronic devices or servers is not limited to that of FIG. 1. In addition, the network environment of FIG. 1 merely illustrates one example of environments applicable to the present embodiments, and the environments applicable to the present embodiments are not limited to the network environment of FIG. 1.

[0036] The plurality of electronic devices (110, 120, 130, 140) may be fixed terminals or mobile terminals implemented as computer devices. Examples of the plurality of electronic devices (110, 120, 130, 140) include smart phones, mobile phones, navigation systems, computers, laptops, digital broadcasting terminals, PDAs (Personal Digital Assistants), PMPs (Portable Multimedia Players), tablet PCs, etc. For example, in FIG. 1, the shape of a smart phone is shown as an example of the electronic device (110), but in embodiments of the present invention, the electronic device (110) may actually mean one of various physical computer devices that can communicate with other electronic devices (120, 130, 140) and / or servers (150, 160) via a network (170) using a wireless or wired communication method.

[0037] The communication method is not limited, and may include not only a communication method that utilizes a communication network (e.g., a mobile communication network, a wired Internet, a wireless Internet, a broadcasting network) that the network (170) may include, but also short-range wireless communication between devices. For example, the network (170) may include any one or more of a personal area network (PAN), a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), a wide area network (WAN), a broadband network (BBN), the Internet, and the like. In addition, the network (170) may include any one or more of a network topology including, but not limited to, a bus network, a star network, a ring network, a mesh network, a star-bus network, a tree, or a hierarchical network.

[0038] Each of the servers (150, 160) may be implemented as a computer device or multiple computer devices that communicate with multiple electronic devices (110, 120, 130, 140) via a network (170) to provide commands, codes, files, contents, services, etc. For example, the server (150) may be a system that provides services (e.g., video editing services, etc.) to multiple electronic devices (110, 120, 130, 140) connected via a network (170).

[0039] FIG. 2 is a block diagram illustrating an example of a computer device according to an embodiment of the present invention. Each of the multiple electronic devices (110, 120, 130, 140) or servers (150, 160) described above can be implemented by the computer device (200) illustrated in FIG. 2.

[0040] Such a computer device (200) may include a memory (210), a processor (220), a communication interface (230), and an input / output interface (240), as illustrated in FIG. 2. The memory (210) may be a computer-readable recording medium, and may include a random access memory (RAM), a read only memory (ROM), and a permanent mass storage device such as a disk drive. Here, the permanent mass storage devices such as the ROM and the disk drive may be included in the computer device (200) as a separate permanent storage device distinct from the memory (210). In addition, the memory (210) may store an operating system and at least one program code. These software components may be loaded into the memory (210) from a computer-readable recording medium separate from the memory (210). The separate computer-readable recording medium may include a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD / CD-ROM drive, a memory card, etc. In another embodiment, the software components may be loaded into the memory (210) via a communication interface (230) other than a computer-readable recording medium. For example, the software components may be loaded into the memory (210) of the computer device (200) based on a computer program installed by files received over a network (170).

[0041] The processor (220) may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input / output operations. Instructions may be provided to the processor (220) via the memory (210) or the communication interface (230). For example, the processor (220) may be configured to execute instructions received according to program code stored in a storage device such as the memory (210).

[0042] The communication interface (230) may provide a function for the computer device (200) to communicate with other devices (e.g., the storage devices described above) via the network (170). For example, requests, commands, data, files, etc. generated by the processor (220) of the computer device (200) according to program codes stored in a recording device such as the memory (210) may be transmitted to other devices via the network (170) under the control of the communication interface (230). Conversely, signals, commands, data, files, etc. from other devices may be received by the computer device (200) via the communication interface (230) of the computer device (200) via the network (170). Signals, commands, data, etc. received via the communication interface (230) may be transmitted to the processor (220) or the memory (210), and files, etc. may be stored in a storage medium (the permanent storage device described above) that the computer device (200) may further include.

[0043] The input / output interface (240) may be a means for interfacing with an input / output device (250). For example, the input device may include a device such as a microphone, keyboard, or mouse, and the output device may include a device such as a display or speaker. As another example, the input / output interface (240) may be a means for interfacing with a device that integrates input and output functions, such as a touchscreen. The input / output device (250) may also be configured as a single device with the computer device (200).

[0044] Additionally, in other embodiments, the computer device (200) may include fewer or more components than those illustrated in FIG. 2. However, it is not necessary to explicitly illustrate most conventional components. For example, the computer device (200) may be implemented to include at least some of the input / output devices (250) described above, or may further include other components such as a transceiver, a database, etc.

[0045] Hereinafter, specific embodiments of a method and device for selective background removal will be described.

[0046] In detecting key objects, it is not easy to select key objects in an image because the importance in the image is judged by image characteristics (e.g., brightness, contrast, texture, etc.) or by the presence or absence of semantic objects.

[0047] Background removal requires the user to manually select key objects in the image. For example, selecting key objects involves drawing precise boundaries, filling in the area with a box, or using the scribble function.

[0048] Figure 3 illustrates an example of a doodle function for selecting a key object. As illustrated in Figure 3, a user can select a key object by directly sketching it on an image (30) using the doodle function.

[0049] However, real-world users often find it difficult and burdensome to select key objects from images, and some selection methods can result in poor accuracy. Furthermore, errors in user input during the selection process can lead to the need to repeat the selection process or affect the background removal results.

[0050] Therefore, a technology is needed that allows users to easily select and perform selective background removal on key objects detected in an image.

[0051] The computer device (200) according to the present embodiment can provide a video editing service to a client through a dedicated application installed on the client or access to a web / mobile site related to the computer device (200). The computer device (200) can be configured with a computer-implemented selective background removal system. For example, the selective background removal system can be implemented in the form of an independently operating program, or can be configured in the form of an in-app of a specific application so as to be operable on the specific application.

[0052] The processor (220) of the computer device (200) may be implemented as a component for performing the following selective background removal method. Depending on the embodiment, the components of the processor (220) may be optionally included or excluded from the processor (220). Furthermore, depending on the embodiment, the components of the processor (220) may be separated or combined to express the function of the processor (220).

[0053] These processors (220) and components of the processor (220) may control the computer device (200) to perform steps included in the following selective background removal method. For example, the processor (220) and components of the processor (220) may be implemented to execute instructions according to the code of the operating system included in the memory (210) and the code of at least one program.

[0054] Here, the components of the processor (220) may be representations of different functions performed by the processor (220) according to instructions provided by the program code stored in the computer device (200).

[0055] The processor (220) can read necessary commands from the memory (210) loaded with commands related to the control of the computer device (200). In this case, the read commands may include commands for controlling the processor (220) to execute steps to be described later.

[0056] The steps included in the selective background removal method described below may be performed in a different order than the illustrated order, and some of the steps may be omitted or additional processes may be included.

[0057] The steps included in the selective background removal method may be performed on the server (150), and in some embodiments, at least some of the steps may also be performed on the client.

[0058] FIG. 4 is a flowchart illustrating an example of a method that a computer device according to an embodiment of the present invention can perform.

[0059] Referring to FIG. 4, in step S410, when an image for background removal is provided, the processor (220) can detect key objects in the image and provide the user with words referring to each detected key object. The processor (220) can generate words from the given image using I2T (image to text) technology, image captioning technology, etc. When an image is provided, the I2T or image captioning technology can determine key objects included in the image and create words that express the objects. At this time, the processor (220) can sort the words generated from the image by assigning appropriate priorities in the process of providing them to the user. For example, the processor (220) can use a word database (DB) to arrange words higher in the word DB as they exist in the DB. The word DB can be composed of words related to the entire service history, and for example, can be composed of product names that have been frequently used in advertisements targeting all users over a recent period of time. In some embodiments, the word DB may be configured for each user based on their user history (e.g., product names frequently advertised by the user), rather than the entire service history, so that each user can be given a personalized priority for words generated from the video.

[0060] In step (S420), the processor (220) may generate a mask for background removal based on a main object corresponding to a word selected by the user among words generated in the image. The processor (220) may generate a mask having the same size as the image, but setting the main object area of ​​the word selected by the user to white and setting the remaining area excluding the selected area to black. In other words, the processor (220) may generate a mask for background removal by setting the area to be revealed to white and the area to be removed to black. At this time, the processor (220) may apply a technology for segmenting the image using a zero-shot learning method using a given prompt according to the user's selection when generating the mask. The processor (220) may generate a mask for the user-selected area through open-set semantic segmentation. The mask may be generated using an open-set semantic segmentation method, which is one of the already known segmentation methods, which detects and segments only an unknown region without additional processing. For example, the processor (220) may generate a mask according to a user-selected area by applying a segmentation technique based on a contrastive language-image pre-training model (CLIP). In addition to applying an open semantic segmentation method based on CLIP, such as CLIPseq, any known segmentation technique may be utilized to generate the mask.In some embodiments, it is also possible to apply an open-world semantic segmentation method that first determines whether the image is an ID class (in-distribution class) used in learning, and then segments the image through few-shot learning based on existing knowledge if it is an OOD class (out-of-distribution class) not used in learning.

[0061] In step (S430), the processor (220) can perform background removal on an image using a mask. In the present embodiment, by applying a mask to the background removal on an image, a given image can be synthesized with the mask, and the remaining area can be processed and removed as a background area, excluding a user-selected object through the mask.

[0062] FIG. 5 illustrates an example of a part of speech (POS) system according to one embodiment of the present invention.

[0063] When generating words from a given image using a technology such as I2T or image captioning, the processor (220) can extract key words from the generated sentences when generating sentences themselves rather than a list of words.

[0064] For example, the processor (220) can identify the part of speech of each word included in a sentence by using the part-of-speech system (see FIG. 5) defined in an open source library for natural language processing in Python. For example, the processor (220) can extract words with a predetermined pattern as key words to be used as text prompts through Table 1 for sentences generated from an image. For example, in the case of pattern 1, a word combination in the order of adjective → proper noun can be detected.

[0065] pattern1 = [{"POS": "ADJ"}, {"POS": "PROPN"}]pattern2 = [{"POS": "ADJ"}, {"POS": "NOUN"}]pattern3 = [{"POS": "ADJ"}, {"POS": "NOUN"}, {"POS": "NOUN"}]pattern4 = [{"POS": "NOUN"}, {"POS": "NOUN"}]pattern5 = [{"POS": "PROPN"}] pattern6 = [{"POS": "NOUN"}]

[0066] FIG. 6 is a drawing for explaining a word provision process for selecting a main object in one embodiment of the present invention.

[0067] The processor (220) provides the user with words extracted from the image (i.e., words referring to the main object) for user selection of the main object detected in the image, and at this time, the words may be provided in a sorted order according to the priority of each word.

[0068] For example, the processor (220) can learn words and patterns frequently used in services (e.g., generating advertisement videos, etc.) and prioritize words extracted from videos based on these.

[0069] Referring to FIG. 6, the word DB (60) may include a list of words frequently used by all users using the service over a recent period of time, or a list of words frequently used by individual users who requested background removal.

[0070] The processor (220) can calculate the similarity between words extracted from the image and words in the word database (60) (S601). In other words, the processor (220) can calculate the similarity relationship between words extracted from the image and recently used words.

[0071] The processor (220) can arrange and provide words extracted from the image based on the similarity calculated in step (S601) (S602).

[0072] Accordingly, when providing words extracted from an image to a user, the processor (220) can assign a higher priority to words similar to words in the word database (60) and arrange them higher. Accordingly, words arranged according to priorities based on recently accumulated history can be provided to the user as a set of prompt candidates for an image to be subject to background removal.

[0073] In this embodiment, a mask is generated through zero-shot learning using a prompt based on user selection, and the mask can be used as a preprocessor for background removal.

[0074] To improve image quality and performance, a filtering technique for mask-based background removal is needed. The present embodiments utilize a mask as a preprocessor and, through mask-based preprocessing, enhance the precision of a user-selected target area.

[0075] For example, the processor (220) can precisely reveal a user-selected object within an input image through intentional degradation of the input image for background removal. Since the primary object detection methods commonly used in background removal technology are significantly affected by the characteristics of the image (brightness, contrast, sharpness, texture, etc.), if an area other than the mask in a given image is intentionally blurred or the pixel brightness is lowered, the probability that the area will be extracted as the foreground increases. The resulting image after background removal can be defined as in Mathematical Expression 1.

[0076] [Mathematical Formula 1]

[0077] I''(x, y) = M(x, y) × I(x, y) + (1 - M(x, y)) × I'(x, y)

[0078] Here, I(x, y) and M(x, y) represent the pixel value and mask value at the location (x, y), respectively. I'(x, y) is the degraded input image, which can be defined, for example, as in Equation 2.

[0079] [Equation 2]

[0080] I'(x, y) = beta × Blur(I(x, y))

[0081] Here, beta refers to a constant (0≤beta<1) for lowering the image brightness, and Blur may refer to a function for blurring the image (e.g., Gaussian filter, etc.).

[0082] As another example, the processor (220) may filter the resulting image after background removal using a mask. The resulting mask has a value between 0 and 1 for each pixel position, and the mask value can be interpreted as the probability that each pixel is a target object. In addition, the resulting background removal also has a value between 0 and 1 for each pixel, and the corresponding value can be interpreted as the probability that each pixel is a foreground. Accordingly, the result of mathematical expression 3 can be generated for the resulting image after background removal.

[0083] [Equation 3]

[0084] BGR'(x, y) = M(x, y) × BGR(x, y)

[0085] Here, BGR(x, y) is the resulting image after background removal, and means the BGR value at the (x, y) location.

[0086] In some embodiments, M(x, y) in mathematical expression 3 may be replaced with a mask value (e.g., (M'(x, y) = Laplacian(M(x, y), etc.)) to which a filter that emphasizes the characteristics of the image is applied. In this case, Laplacian corresponds to a filter used to increase the clarity of the image with a Laplacian function.

[0087] As another example, the processor (220) may perform background removal using thresholding, a segmentation technique. The processor (220) may threshold a mask to produce a cleaner resulting image and then generate the resulting image. The result of Equation 4 may be generated for the resulting image after background removal.

[0088] [Equation 4]

[0089] BGR'(x, y) = M'(x, y) × BGR(x, y)

[0090] Here, M'(x, y) can mean the mask value with thresholding applied (i.e., M'(x, y) = 0 if M(x, y) < Given Threshold, otherwise M(x, y)).

[0091] As another example, the processor (220) can perform background removal using a conditional BGR model. The model can be trained to utilize a mask in the background removal itself. Referring to FIG. 7, when an input image (701) is given, a feature extractor (710) configured as an image encoder can be used to convert it into an embedding, and then an output image (703) with the background removed can be created through a foreground segmentor (720). In the present embodiment, the foreground segmentor (720) can be trained using a diffusion model or the like that is effective for image segmentation, and in particular, the foreground segmentor (720) that removes the background conditionally can be trained using a mask (702) generated from the input image (701).

[0092] In addition to the above-mentioned method, the final result of background removal can be selected through similar methods aimed at improving the precision of the target area.

[0093] FIGS. 8 to 11 illustrate examples of a user interface for a background removal process in one embodiment of the present invention.

[0094] Figures 8 to 10 illustrate examples of service screens included in the background removal process.

[0095] Referring to FIG. 8, the processor (220) can receive a given image (701) in response to a user's request for background removal, and at this time, can provide the user with a word generated through a main object detection process for the image (701) as a text prompt candidate (80).

[0096] In this embodiment, rather than the user directly writing the main object in the image (701) using a doodle function or the like, a text prompt candidate (80) for the main object in the image (701) can be automatically extracted and provided.

[0097] Accordingly, the user can select a specific text prompt from the text prompt candidates (80) for the main object in the image (701). As illustrated in FIG. 9, when the user selects a specific text prompt (91) from the text prompt candidates (80), the processor (220) can generate a mask (900) for background removal using the specific text prompt (91) selected by the user.

[0098] For example, for an image (701), words such as man, tennis racket, and tennis ball may be provided as text prompt candidates (80), and if the user selects a tennis ball among them, a mask (900) may be created in which the object area (901) corresponding to the tennis ball is set to white and the remaining area is set to black.

[0099] The user can confirm the main object area (901) selected by the user through the mask (900) and then request the execution of background removal. Referring to FIG. 10, the processor (220) can execute background removal for the image (701) using the mask (900) according to the user request, and can generate a result image (1000) according to the background removal and provide it to the user. In other words, the processor (220) can synthesize the image (701) and the mask (900) and provide a result image (1000) in which the remaining area is removed as a background area except for the main object (1001) selected by the user through the mask (900).

[0100] These embodiments can be integrated into text-based interfaces, such as conversational AI services (e.g., chatGPT). The service according to the present invention can perform background removal using text prompts, making it feasible for implementation in user scenarios utilizing chat rooms.

[0101] For example, as illustrated in FIG. 11, the processor (220) may provide a chat room (1100) for the user in which the user and the service chatbot participate. The user may upload an image (701) through the chat room (1100) and input commands required during the process in the form of a chat message. The processor (220) may provide text prompt candidates (80) for a main object in the image (701) through the chat room (1100) and may generate and provide a mask (900) according to a specific text prompt (91) selected by the user. The processor (220) may perform background removal on the image (701) using the mask (900) according to a user request and may provide a result image (1000) according to the background removal through the chat room (1100).

[0102] The present embodiments can provide an interface that is easy for the user to select by proposing a candidate group of main objects in an image (701), and in particular, can provide an intuitive interface by allowing the user to select main objects using words.

[0103] According to embodiments of the present invention, an intuitive interface can be provided that allows a user to easily select a key object detected in an image for background removal, and a mask can be created for the key object selected by the user to perform selective background removal based on the mask.

[0104] The devices described above may be implemented as hardware components, software components, and / or a combination of hardware components and software components. For example, the devices and components described in the embodiments may be implemented using one or more general-purpose computers or special-purpose computers, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0105] Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may independently or collectively command the processing device. The software and / or data may be embodied in any type of machine, component, physical device, computer storage medium, or device for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on one or more computer-readable recording media.

[0106] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. In this case, the medium may be one that continuously stores a computer-executable program or one that temporarily stores it for execution or download. In addition, the medium may be various recording means or storage means in the form of a single or multiple hardware combinations, and is not limited to a medium directly connected to a computer system, but may also be distributed over a network. Examples of the medium may include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and those configured to store program commands, including ROM, RAM, and flash memory. In addition, examples of other media may include recording media or storage media managed by app stores that distribute applications, sites that supply or distribute various software, servers, etc.

[0107] Although the embodiments described above have been described by way of limited examples and drawings, those skilled in the art will appreciate that various modifications and variations can be made based on the above teachings. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

[0108] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims described below.

Claims

1. In a method for removing a background running on a computer device, The computer device comprises at least one processor configured to execute computer-readable instructions contained in a memory, The above background removal method is, A step of providing, by at least one processor, a word representing a salient object in a given image as a text prompt candidate; generating a mask based on a main object of a text prompt selected from among the text prompt candidates by at least one processor; and A step of performing selective background removal on the image using the mask by at least one processor. A method of removing background including:

2. In paragraph 1, The steps provided above are: A step of extracting the word from the image using I2T (image to text) or image captioning technology. A method of removing background including:

3. In paragraph 1, The steps provided above are: A step of generating a sentence related to the main object in the above image; and A step for extracting words with predetermined patterns using the part-of-speech system defined in an open-source library for natural language processing. A method of removing background including:

4. In paragraph 1, The steps provided above are: A step of assigning a priority based on the similarity between the word and at least one other word recently used, and sorting the word according to the priority. A method of removing background including:

5. In paragraph 4, The above other words correspond to a list of words used by all users using the service over a recent period of time or a list of words used by users who requested background removal for the above video. A background removal method characterized by .

6. In paragraph 1, The above generating steps are: Generating the mask by segmenting the image using a zero-shot learning method using the selected text prompt A background removal method characterized by .

7. In paragraph 1, The above generating steps are: Generating a mask corresponding to the selected text prompt through open-set semantic segmentation. A background removal method characterized by .

8. In paragraph 1, The steps performed above are: A step of generating a result image by synthesizing the mask and the image and removing the remaining area except for the main object of the selected text prompt from the image. A method of removing background including:

9. In paragraph 8, The steps performed above are: A step of generating a final result image by synthesizing the mask to which thresholding has been applied and the result image. A method of removing background that includes more.

10. In paragraph 8, The steps for generating the above result image are: A step of performing degradation processing on the above image and then synthesizing the degraded image with the mask. A method of removing background including:

11. In paragraph 8, The steps performed above are: A step of filtering the above result image using the above mask to generate a final result image. A method of removing background that includes more.

12. In paragraph 8, The steps for generating the above result image are: Generating the result image through a foreground segmentation model learned to conditionally use the above mask. A background removal method characterized by .

13. A computer program stored in a non-transitory computer-readable recording medium for executing the background removal method of any one of claims 1 to 12 on the computer device.

14. In computer devices, At least one processor configured to execute computer-readable instructions contained in memory Including, At least one processor, A process of providing, for a given image, words representing key objects within the image as text prompt candidates; A process of generating a mask based on the main object of the text prompt selected from the above text prompt candidates; and A process of performing selective background removal on the image using the above mask. A computer device that processes.

15. In paragraph 14, At least one processor, Sort the words according to the priority given to the words based on their similarity to at least one other word recently used. A computer device characterized by:

16. In paragraph 14, At least one processor, Generating the mask by segmenting the image using a zero-shot learning method using the selected text prompt. A computer device characterized by:

17. In paragraph 14, At least one processor, By synthesizing the above mask and the above image, an image is generated by removing the remaining area except for the main object of the selected text prompt from the above image, Generating a final result image by synthesizing the above mask with threshold holding applied and the above result image. A computer device characterized by:

18. In paragraph 14, At least one processor, By synthesizing the above mask and the above image, an image is generated by removing the remaining area except for the main object of the selected text prompt from the above image. After performing deterioration processing on the above image, synthesizing the deteriorated image with the mask. A computer device characterized by:

19. In paragraph 14, At least one processor, By synthesizing the above mask and the above image, an image is generated by removing the remaining area except for the main object of the selected text prompt from the above image, Filtering the above result image using the above mask to generate a final result image A computer device characterized by:

20. In paragraph 14, At least one processor, By synthesizing the above mask and the above image, an image is generated by removing the remaining area except for the main object of the selected text prompt from the image. Generating the result image using a foreground segmentation model learned to conditionally use the above mask. A computer device characterized by:

Images