Information processing device and information processing method for processing an overlay image

Abstract

To provide an information processing device capable of reusing images overlaid with additional information by erasing the additional information from the images more appropriately with higher accuracy, and an information processing method.SOLUTION: An overlay image processing device 1 comprises: a region detection unit which inputs a first image containing an object and additional information to a machine learning model for region detection and causes the machine learning model for region detection to detect an additional information region; a mask generation unit which generates a mask to be applied to the region of the additional information from information of the region of the additional information detected by the region detection unit; and an output unit which inputs the mask generated by the mask generation unit and the first image to the machine learning model for region detection and causes the machine learning model for region detection to output a second image different from the first image. The second image is an image in which additional information has been erased from the first image and a region in the first image from which the additional information has been erased has been restored.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing apparatus and method for processing an overlay image, and more particularly to a technique for reusing an image on which additional information such as text or a logo is overlaid. [Background technology]

[0002] Many electronic commerce (EC) platforms rely heavily on numerous images to visualize the products and services they provide to users. How attractively the products and services are presented to users through images has a major impact on the sales of EC sites.

[0003] In many cases, images of products and services provided by these EC sites are overlaid with additional information such as product names and logos, brand names and logos, advertising captions, product and service descriptions, price information, campaign information, etc. as digital elements to encourage users to purchase the products and services. By allowing users to simultaneously view the overlaid additional information along with the images of the products and services displayed on the EC site, the information required to purchase the products and services can be provided to the users correctly.

[0004] However, in order to reuse an image on which such additional information is overlaid (hereinafter also referred to as an "overlay image"), it is often necessary to erase the overlaid additional information. The purposes for reusing an overlay image are diverse, but include, for example, image editing to create a product image with a different color or design, and replacement with different additional information (a different caption, a different description or price, a different campaign, etc.). Erasing the additional information from the overlay image is also useful for improving the accuracy of image classification of products, etc. Furthermore, in an image from which the additional information has been erased, pixels are missing in the areas from which the additional information has been erased, and so the image in the areas from which the pixels are missing must be repaired to restore it to the original state before the additional information was overlaid.

[0005] US Pat. No. 6,399,433 discloses a computer-implemented method for replacing objects in an image. Specifically, the computer-implemented method disclosed in US Pat. No. 6,399,633 identifies a first object in a first image that is a target object to be replaced at a certain location, and masks the target area based on the first image and the location of the first object to generate a mask image. The computer-implemented method disclosed in Patent Document 1 further generates a second image based on the generated mask image and the inpainting machine learning model, the second image being an image that does not include the target object and in which a portion of the target object has been inpainted, further generates a third image from the mask image and the second image, and adds a new object different from the first object to the generated third image. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Patent Publication No. 2023-500203 Summary of the Invention [Problem to be solved by the invention]

[0007] According to the technology of Patent Document 1, it is possible to detect a text label overlaid on an image, video, etc., and replace the detected text label with a new label different from the text label, for example, text translated into a different language.

[0008] However, it is not easy to detect and erase only the additional information that should be erased from the overlay image. In particular, on EC sites, many product images must be displayed on a single web page in a comparative manner, which limits the display size that can be allocated to each product image. For this reason, in many cases, additional information is overlaid on the product object in the image. This makes it difficult to accurately remove only the additional information, and may reduce the accuracy of image restoration.

[0009] Furthermore, it is not easy to accurately distinguish between the overlaid additional information and information in the object or background of the image that should not be erased. For example, when a product object in an image is a product package including text information such as a product name, there is a risk of the product name, etc. on the package being mistakenly erased. On the other hand, when the additional information is a logo image other than text, or the text of the additional information is superimposed on a frame image such as a circle or ribbon, there is a risk that the additional information is not erased correctly and remains.

[0010] The present invention has been made to solve the above-mentioned problems, and its object is to provide an information processing device and an information processing method that are capable of appropriately and accurately erasing additional information from an image on which the additional information is overlaid, making the additional information reusable. [Means for solving the problem]

[0011] In order to solve the above problem, one aspect of an information processing device according to the present invention includes an area detection unit that inputs a first image including an object and additional information into a first machine learning model and causes the first machine learning model to detect an area of ​​the additional information; a mask generation unit that generates a mask for the area of ​​the additional information from information of the area of ​​the additional information detected by the area detection unit; and an image output unit that inputs the mask generated by the mask generation unit and the first image into a second machine learning model and causes the second machine learning model to output a second image different from the first image, wherein the second image is an image in which the additional information has been erased from the first image and the area of ​​the first image from which the additional information has been erased is repaired.

[0012] One aspect of the information processing method of the present invention is an information processing method executed by an information processing device, comprising the steps of: inputting a first image including an object and additional information into a first machine learning model and causing the first machine learning model to detect an area of ​​the additional information; generating a mask for the area of ​​the additional information from information of the detected area of ​​the additional information; inputting the generated mask and the first image into a second machine learning model and causing the second machine learning model to output a second image different from the first image, wherein the second image is an image in which the additional information has been erased from the first image and the area of ​​the first image from which the additional information has been erased has been repaired.

[0013] One aspect of a computer-readable recording medium according to the present invention is a computer-readable recording medium storing an information processing program for causing a computer to execute information processing, the program causing the computer to execute processes including: an area detection process for inputting a first image including an object and additional information into a first machine learning model and causing the first machine learning model to detect an area of ​​the additional information; a mask generation process for generating a mask for the area of ​​the additional information from information of the area of ​​the additional information detected by the area detection process; and an image output process for inputting the mask generated by the mask generation process and the first image into a second machine learning model and causing the second machine learning model to output a second image different from the first image, the second image being an image in which the additional information has been erased from the first image and the area of ​​the first image from which the additional information has been erased has been repaired. Effect of the Invention

[0014] According to the present invention, additional information can be appropriately and accurately erased from an image on which the additional information is overlaid, making the additional information reusable. The above-mentioned objects, aspects, and advantages of the present invention, as well as objects, aspects, and advantages of the present invention not described above, will be understood by those skilled in the art from the following detailed description of the invention by referring to the accompanying drawings and the claims. [Brief description of the drawings]

[0015] [Figure 1] FIG. 1 is a block diagram showing an example of the functional configuration of an overlay image processing system according to each embodiment of the present invention. [Figure 2A] FIG. 2A is a diagram illustrating an example of an overlay image input to the overlay image processing device according to this embodiment. [Figure 2B] FIG. 2B is a diagram illustrating an example of an image in which the additional information has been deleted from the overlay image of FIG. 2A, which is output from the overlay image processing device according to this embodiment. [Figure 3A]FIG. 3A is a diagram illustrating another example of an overlay image input to the overlay image processing device according to this embodiment. [Figure 3B] FIG. 3B is a diagram illustrating another example of an image in which the additional information has been deleted from the overlay image of FIG. 3A, which is output from the overlay image processing device according to this embodiment. [Figure 4] FIG. 4 is a conceptual diagram showing an example in which the overlay image processing device according to this embodiment is implemented in a network for area detection and a network for image restoration. [Diagram 5] FIG. 5 is a flowchart showing an example of an outline of the processing procedure of the overlay image processing executed by the overlay image processing device according to this embodiment. [Figure 6] FIG. 6 is a flowchart showing an example of detailed processing procedures of the pre-learning processing for the area detection machine learning model 2 in step S1 of FIG. 5, which is executed by the overlay image processing device. [Figure 7] FIG. 7 is a flowchart showing an example of detailed processing procedures of the pre-learning processing for the image restoration machine learning model 3 in step S1 of FIG. 5, which is executed by the overlay image processing device. [Figure 8] FIG. 8 is a flowchart showing an example of a detailed processing procedure of the image cleaning process in step S2 of FIG. 5, which is executed by the overlay image processing device. [Figure 9] FIG. 9 is a diagram illustrating an example of a mask adjustment process executed by a mask generation unit of the overlay image processing device. [Figure 10] FIG. 10 is a flowchart showing an example of detailed processing procedures of the post-processing in step S3 of FIG. 5, which is executed by the overlay image processing apparatus. [Figure 11] FIG. 11 is a block diagram showing an example of the hardware configuration of the overlay image processing device according to this embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Hereinafter, with reference to the attached drawings, an embodiment for carrying out the present invention will be described in detail. Among the components disclosed below, those having the same functions are given the same reference numerals, and their description will be omitted. Note that the embodiment disclosed below is an example of a means for realizing the present invention, and should be appropriately modified or changed depending on the configuration of the device to which the present invention is applied and various conditions, and the present invention is not limited to the following embodiment. Furthermore, not all of the combinations of features described in the present embodiment are necessarily essential to the solution of the present invention.

[0017] The overlay image processing device of this embodiment inputs a first image in which additional information is overlaid on an image including an object into a first machine learning model, causes the first machine learning model to detect areas of additional information in the first image, and generates a mask for the detected areas of additional information. The overlay image processing device of this embodiment further inputs the first image and the generated mask to a second machine learning model, and causes the second machine learning model to output a second image in which the additional information has been erased from the first image and the area from which the additional information has been erased has been repaired.

[0018] Below, we will explain a non-limiting example in which the overlay image processing device of this embodiment processes as an input image an image that includes an image showing a product or service listed on an EC site, etc. as a main object (hereinafter also referred to as the "main object"), and additional information such as product name, product logo, brand name, brand logo, advertising caption, product or service description, price information, campaign information, etc. as information to encourage the purchase of the product or service is overlaid as digital elements on the same image. However, this embodiment is not limited to this, and the input image processed by the overlay image processing device may be an image containing any object such as text, graphics, images, etc., or an image of only the background, and the additional information overlaid on the image may include any type of digital element such as text, graphics, images, etc., and may or may not be related to a primary object contained in the same image.

[0019] In the following, an example will be described in which the overlay image processing device according to this embodiment erases additional information overlaid as a digital element from an input image, and restores and outputs an image of the area from which the additional information has been erased. However, this embodiment is not limited to this, and the overlay image processing device may execute either the erasure of additional information or the restoration of an image. Hereinafter, an image in which additional information is overlaid on an image including an object to be input to the overlay image processing device is referred to as an overlay image. The additional information overlaid on an image is also referred to as a digital overlay. Meanwhile, the image before the additional information is overlaid is referred to as an original image.

[0020] <Functional configuration of the overlay image processing device> Fig. 1 is a block diagram showing an example of the functional configuration of an overlay image processing system according to this embodiment. The overlay image processing system shown in Fig. 1 includes an overlay image processing device 1, a machine learning model for area detection 2, and a machine learning model for image restoration 3. The overlay image processing device 1 includes an image input unit 11, a region detection unit 12, a mask generation unit 13, an image restoration unit 14, an output unit 15, and a post-processing unit 16. The overlay image processing device 1 may further include a pre-learning unit 17, but instead, the pre-learning unit 17 may be provided in a computing component other than the overlay image processing device 1.

[0021] Moreover, the overlay image processing device 1 may be communicably connected to a client device (not shown) configured as a PC (Personal Computer) or the like via a network. In this case, the overlay image processing device 1 may be implemented in a server, and the client device may provide a user interface when the overlay image processing device 1 executes input and output of information with the outside, and may also include some or all of the components 11 to 17 of the overlay image processing device 1.

[0022] The image input unit 11 acquires an overlay image stored in a storage device as an input image, and supplies the acquired overlay image to the area detection unit 12 and the image restoration unit 14 . The storage device that stores the overlay image may be configured as a non-volatile storage device such as a hard disk drive (HDD) or a solid state drive (SSD). The image input unit 11 may acquire the overlay image by reading out an overlay image stored in advance in such a storage device, or may receive the overlay image via a communication I / F from the same or a different counterpart device that stores the overlay image. Alternatively, the image input unit 11 may acquire the overlay image in real time using a technique such as web crawling via the communication I / F.

[0023] The image input unit 11 also accepts input of various parameters necessary for executing overlay image processing in the overlay image processing device 1. The image input unit 11 may accept input of various parameters via a user interface of a client device communicably connected to the overlay image processing device 1.

[0024] The area detection unit 12 detects an area of ​​a digital overlay, which is additional information to be erased, from the overlay image supplied from the image input unit 11, and supplies the detection result of the area to be erased in the overlay image to the mask generation unit 13. Specifically, the area detection unit 12 inputs the input image, which is the overlay image, into the area detection machine learning model 2, and causes the area detection machine learning model 2 to detect the area of ​​additional information overlaid on the overlay image, and place a bounding box around the detected area.

[0025] In this embodiment, the additional information to be erased is information added to an object such as a product, such as a product name, product logo, brand name, brand logo, advertising caption, product or service description, price information, campaign information, etc., and may include text, graphics, and images. The additional information to be erased (digital overlay) may include digital text that is diverse in language, font, size, color, and text effect. The additional information to be erased may also include color frames around text, logos, ribbons, badges, and other decorative elements. In this embodiment, the area detection unit 12 can detect not only areas of additional information such as text, but also areas of additional information such as graphics and images as areas to be erased.

[0026] The area detection machine learning model 2 is a machine learning model capable of detecting the position, type, and number of a predetermined object from an input image. In this embodiment, the area detection machine learning model 2 may be implemented in a network or algorithm suitable for object detection, such as YOLO (You Only Look Once), and is pre-trained using a training data set prepared in advance. Details of the pre-training for the area detection machine learning model 2 will be described later with reference to FIG. 6.

[0027] The mask generation unit 13 generates a mask for masking the area of ​​additional information detected in the overlay image based on the area detection result of the overlay image supplied from the area detection unit 12, i.e., the area detection result of the area of ​​additional information (digital overlay) detected in the overlay image, and supplies the generated mask to the image restoration unit 14. Specifically, the mask generation unit 13 generates a binary mask in which the pixel values ​​of the areas of additional information detected in the overlay image, i.e., the areas of the digital overlay that should be erased from the overlay image, are set to 1, and the pixel values ​​of other areas other than the areas of additional information detected in the overlay image are set to 0.

[0028] In this embodiment, the mask generation unit 13 has a function of adjusting the shape and size of the mask generated in response to the rectangular bounding box output by the area detection machine learning model 2. This allows the area to be restored in the overlay image to be made smaller or larger, thereby optimizing the target area for image restoration and improving the accuracy of image restoration. Details of the mask adjustment process performed by the mask generation unit 13 will be described later with reference to Figs. 8 and 9.

[0029] The image restoration unit 14 supplies to the output unit 15 as an image restoration result a restored image in which the additional information to be erased from the overlay image is erased from the overlay image based on the overlay image supplied from the image input unit 11 and the mask supplied from the mask generation unit 13, and the image in the area from which the additional information has been erased is restored. Specifically, the image restoration unit 14 applies a mask to the overlay image, which is an input image, to generate an intermediate image in which the area of ​​additional information detected from the overlay image is erased. The image restoration unit 14 also inputs the generated intermediate image and mask to the machine learning model 3 for image restoration, and causes the machine learning model 3 for image restoration to output a restored image in which the image of the area of ​​additional information erased from the overlay image is restored to the state before the additional information was overlaid. Alternatively, the image restoration unit 14 may input the overlay image, which is an input image, and the mask to the machine learning model 3 for image restoration, and cause the machine learning model 3 for image restoration to generate an intermediate image in which the area of ​​additional information is erased from the overlay image.

[0030] The image restoration machine learning model 3 is a machine learning model capable of restoring pixels in a predetermined area masked from an input image. In this embodiment, the image restoration machine learning model 3 may be implemented in a network or algorithm suitable for image restoration, such as LaMa (Resolution-robust Large Mask Inpainting with Fourier Convolutions), and is pre-trained using a training data set prepared in advance. Details of the pre-training for the image restoration machine learning model 3 will be described later with reference to FIG. 7.

[0031] The output unit 15 outputs the restored image supplied from the image restoration unit 14, i.e., an image in which the area of ​​additional information has been erased from the overlay image and the image of the erased area has been restored to the state before the additional information was overlaid (original image). Specifically, the output unit 15 may supply the restored image to the post-processing unit 16, store the restored image in a storage device, or display the restored image on a display device.

[0032] The post-processing unit 16 performs post-processing to arbitrarily process the restored image supplied from the output unit 15 or read from the storage device. The post-processing performed by the post-processing unit 16 may include, for example, a process of adjusting the size and position of a main object in the restored image, the details of which will be described later with reference to FIG.

[0033] The pre-learning unit 17 pre-learns the area detection machine learning model 2 and the image restoration machine learning model 3. The area detection machine learning model 2 and the image restoration machine learning model 3 are machine learning models having mutually different structures, and are pre-learned using different learning data sets, the details of which will be described later with reference to Figs. 6 and 7. However, the machine learning model that can be used for overlay image processing in this embodiment is not limited to a supervised machine learning model, and for example, an unsupervised machine learning model based on reinforcement learning or the like may be used.

[0034] A specific example of the overlay image processing executed by the overlay image processing device 1 according to this embodiment will be described with reference to FIGS. 2A to 3B. Fig. 2A is a diagram for explaining an example of an overlay image input to the overlay image processing device 1. Referring to Fig. 2A, an image of a tire and wheel is shown in the center of the image, and a description "Tire & Wheel Set" identifying the product is overlaid at the bottom of the image, and a brand logo surrounded by a circular color frame is overlaid at the top right of the image. The description and notes identifying the product located at the bottom of the overlay image are additional information 21 in the form of text, and the brand logo located at the top right of the overlay image is additional information 22 in the form of text described on an image in a circular frame.

[0035] Fig. 2B is a diagram illustrating an example of a restored image in which overlay information is erased from the overlay image of Fig. 2A, output from the overlay image processing device 1. Referring to Fig. 2B, both the additional information of the text at the bottom of the overlay image and the additional information of the image in the upper right corner of the overlay image are erased, and the image in the area of ​​the erased additional information is restored. For the additional information in the upper right corner image, not only the text portion but also the circular image portion surrounding the text is erased, and the image in the erased area is restored as an image of part of the original tire and wheel. In this way, according to this embodiment, not only areas of additional information such as text, but also areas of additional information such as graphics and images can be detected as areas to be erased.

[0036] Fig. 3A is a diagram for explaining another example of an overlay image input to the overlay image processing device 1. Referring to Fig. 3A, an image of a nutritional supplement package is shown in the center of the image, and a description of the product sale surrounded by a ribbon-shaped color frame, "FREE SHIPPING!!", and a description of the sales format surrounded by a circular color frame at the bottom right of the image are overlaid. The description of the product sale surrounded by the ribbon-shaped frame located at the top left of the overlay image and the description of the sales format surrounded by a circular frame at the bottom right of the overlay image are both additional information 31, 32 in which text is written in a frame image.

[0037] FIG. 3B is a diagram for explaining an example of an image in which overlay information has been erased from the overlay image of FIG. 3A, output from the overlay image processing device 1. Referring to FIG. 3B, both pieces of additional information at the upper left and lower right of the overlay image have been erased, and the image in the area of ​​the erased additional information has been restored. In both pieces of additional information, not only the text portion but also the ribbon-shaped and circular image portions surrounding the text have been erased, and the image in the erased area has been restored as an image of part of the background of the product package, which is the object. On the other hand, the product name "ABC" and other text written on the product package are not overlaid additional information, but are elements that constitute part of the original product image, and therefore have not been erased from the overlay image. In this way, according to this embodiment, the text overlaid as additional information can be appropriately erased, while the text that constituted the objects and background of the original image before it was overlaid, rather than the information overlaid as additional information, can be retained as the original image without being erased.

[0038] <Network configuration of machine learning model for region detection and machine learning model for image restoration> FIG. 4 is a conceptual diagram showing an example of a network configuration and a schematic processing procedure when the area detection machine learning model 2 and the image restoration machine learning model 3 according to this embodiment are implemented in a machine learning network. The network shown in FIG. 4 is configured in a pipeline of a network for area detection and a subsequent image restoration network that is configured separately from the network for area detection. Referring to Figure 4, input image 41 is an overlay image in which an object image showing a product is overlaid with additional information about the product, and additional information describing the product is overlaid as images and text on an original image obtained by capturing an image of a person carrying an umbrella, which is the product, with an imaging device such as a camera, as a post-processing step in generating the original image.

[0039] Here, a single composite image is generated by combining two image layers, an original image and a digital overlay of additional information, using alpha composite, as expressed by the following equation 1. comp is the color value at each pixel location.

[0040]

number

[0041] Here, I d represents the digital overlay layer, and I s represents the scene, i.e., the layer of the original image before overlay, and α d represents the alpha channel of a digital overlay layer. The alpha channel is an auxiliary data channel that is used to set the transparency of pixels as part of the color information of the image data. The alpha value generally represents opacity, with the minimum value being completely transparent and the maximum value being completely opaque, and is often expressed as an 8-bit value with 256 levels from 0 to 255. In this case, pixel areas with positive alpha values ​​will be detected as digital overlay in the input image.

[0042] The region detection network 42 corresponds to the machine learning model 2 for region detection shown in Fig. 1. When the region detection network 42 is configured based on YOLOv5 or the like, the neural network of YOLOv5 resizes the overlay image that is the input image, divides the resized image into finer grid cells, recursively detects objects by convolution calculation based on the features of the grid cells, and outputs the coordinates of the bounding box surrounding the predicted object and the class of the predicted object together with the reliability of the prediction.

[0043] In this embodiment, the area detection network 42 detects additional information overlaid on the overlay image as an area to be erased, by referring to an image 43 of the area detection result output from the area detection network 42. The area detection network 42 detects additional information 43a of an image in which text describing a product is written within a rectangular color frame, additional information 43b of an image of the brand logo of the product seller, additional information 43c of text describing the product seller, and additional information 43d of an image showing color variations of the product, respectively.

[0044] The image 43 resulting from the region detection is input to a subsequent network 44. The network 44 includes a component for generating a mask in the previous stage and erasing the region to be erased using the mask, and an image restoration network 47 in the subsequent stage. Specifically, the region detection result image 43 and the overlay image, which is the input image, are input to the mask generation unit 13 of the overlay image processing device 1, and the mask generation unit 13 predicts a mask 45 for the overlay image, which is the input image, and inputs it to the subsequent image restoration network 47. In the mask 45 in Fig. 4, a value of 1 is set to each of the regions 43a to 43d of the additional information (digital overlay) detected by the region detection network 42, and these operate as mask regions 45a to 45d corresponding to these detected regions. On the other hand, a value of 0 is set to the region of the original image where no additional information is detected.

[0045] In the first stage of the network 44, a mask 45 is applied to the overlay image, which is the input image, i.e., calculation is performed using the mask 45, thereby erasing the area of ​​additional information (digital overlay) from the overlay image, which is the input image. As a result, an image 46 is generated in which the area of ​​additional information (digital overlay) has been erased from the overlay image, which is the input image.

[0046] This image 46 with the areas of additional information (digital overlay) removed is input, together with the mask 45, to an image inpainting network 47. The image restoration network 47 corresponds to the machine learning model 3 for image restoration shown in Fig. 1. Alternatively, the network 44 may be configured with the machine learning model 3 for image restoration, and components for mask generation and area deletion of additional information may be incorporated into a part of the machine learning model 3 for image restoration. When the network 44 including the image restoration network 47 is constructed based on LaMa, etc., the LaMa image restoration network 47 is constructed by constructing the residual block with a Fast Fourier Convolutions (FFCs) network 48 and calculating the loss function L final 49, which uses relatively large area features to restore the image of the masked area, deletes the area of ​​additional information from the input image, the overlay image, and outputs a restored image 50 in which the deleted area is restored to the original image before overlay.

[0047] Loss function L final 49 is the predicted value of the restored image ^I during pre-training and re-training of the image restoration network 47. s and the original image before overlay I s The image restoration network 47 is trained so that the loss between the ground truth data and the original data is smaller. This loss function L final Details of 49 will be described later with reference to FIG.

[0048] <Overlay Image Processing Executed by Overlay Image Processing Device 1> FIG. 5 is a flowchart showing an example of an outline of the processing procedure of the overlay image processing executed by the overlay image processing device 1 according to the present embodiment. Each step in FIG. 5 is realized by a CPU reading and executing a program stored in a storage device such as an HDD of the overlay image processing device 1. At least a part of the flowchart shown in FIG. 5 may be realized by other hardware such as a GPU. When realized by hardware, for example, a specific compiler may be used to automatically generate a dedicated circuit on an FPGA (Field Programmable Gate Array) from a program for realizing each step. Also, a gate array circuit may be formed in the same manner as an FPGA to realize the hardware. Also, it may be realized by an ASIC (Application Specific Integrated Circuit).

[0049] In S1, the pre-learning unit 17 of the overlay image processing device 1 pre-learns the area detection machine learning model 2 and the image restoration machine learning model 3 using their respective learning data sets. Details of this pre-learning process will be described later with reference to Figs. 6 and 7. In S2, the image input unit 11, area detection unit 12, mask generation unit 13, and image restoration unit 14 of the overlay image processing device 1 execute an image cleaning process. This image cleaning process is a process of erasing additional information (digital overlay) in the overlay image, which is the input image, and restoring the image of the area from which the additional information has been deleted. Details of this image cleaning process will be described later with reference to FIG. 8. The image cleaning process executed in S2 corresponds to an inference phase in which a restored image is predicted using the pre-trained machine learning model 2 for area detection and the machine learning model 3 for image restoration.

[0050] In S3, the post-processing unit 16 of the overlay image processing device 1 executes the image cleaning process of S2, i.e., post-processing of deleting the additional region and restoring the image. The post-processing executed by the post-processing unit 16 may include, for example, a process of adjusting the restored image output by the image cleaning process, the details of which will be described later with reference to FIG.

[0051] <Pre-training of machine learning model 2 for area detection> FIG. 6 is a flowchart showing an example of detailed processing procedures of the pre-learning processing for the area detection machine learning model 2 in step S1 of FIG. 5, which is executed by the overlay image processing device 1. In S11, the pre-learning unit 17 prepares a learning data set for pre-learning the area detection machine learning model 2, and stores the data in the storage device. Specifically, the pre-learning unit 17 prepares a large number of overlay images in which additional information of an object, such as a product, is overlaid on an image including the object. In each of these overlay images, a bounding box is annotated in advance for the area of ​​the additional information to be detected.

[0052] In this embodiment, the pre-learning unit 17 further labels each area of ​​additional information to be detected with a class of the additional information (digital overlay). The class may include the text of the additional information, the shape of the area of ​​the additional information, the combination of colors of the area of ​​the additional information and the text (color palette), and a corner. The corner class is a class for classifying whether the area of ​​the additional information is located at a corner of the overlay image or not, that is, it indicates the relative positional relationship between the area of ​​the additional information and the periphery of the overlay image. In an overlay image in which a product or the like posted on an EC site is the main object, the additional information is often placed at the corner of the image under the constraint of a limited display size. Therefore, by setting the class of the corner, it is possible to improve the detection accuracy of the area of ​​the additional information and to easily refine the shape and size of the mask corresponding to the detected area.

[0053] In this manner, in this embodiment, not only are the regions of additional information in an overlay image annotated as bounding boxes as training data for pre-learning the region detection machine learning model 2, but each region of additional information is labeled with a class and its value (attribute value). That is, the ground truth data for each overlay image includes the bounding box annotated to the region of additional information, and the class and its value.

[0054] In addition to text, the classes to which the additional information area is labeled can also label shape (including the presence or absence of a frame surrounding the text), color (including color combinations), and corners (including the relative positional relationship between the original image and the additional information area) as correct answer data. This makes it possible to avoid a situation where only the additional information text is detected and the frame surrounding the text remains undetected, or to avoid erroneous detection of text that is not additional information, i.e., text that exists in the original image before overlay.

[0055] In S12, the pre-learning unit 17 trains the area detection machine learning model 2 using the learning data set prepared in S11 as an input. In S13, the pre-learning unit 17 evaluates the area detection machine learning model 2 trained in S12 using a predetermined loss function. Specifically, the pre-learning unit 17 trains the area detection machine learning model 2 so that the difference between the area of ​​the predicted additional information output by the area detection machine learning model 2 trained in S12 and the corresponding correct answer data becomes smaller. The predetermined loss function used in S13 may be, for example, a binary cross-entropy loss function, but is not limited thereto. The pre-learning unit 17 may repeat the processes of S12 and S13 until the loss output by the loss function converges to a desired threshold in S13. When the process of S13 ends, the overlay image process proceeds to S2 shown in FIG.

[0056] <Pre-training of Machine Learning Model 3 for Image Restoration> FIG. 7 is a flowchart showing an example of detailed processing procedures of the pre-learning processing for the image restoration machine learning model 3 in step S1 of FIG. 5, which is executed by the overlay image processing device 1. In S111, the pre-learning unit 17 prepares a learning data set for pre-learning the image restoration machine learning model 3, and stores it in the storage device. Specifically, the pre-learning unit 17 prepares a large number of overlay images in which additional information of an object such as a product is overlaid on an image including the object, and masks for the area of ​​the additional information of the overlay images. Furthermore, the pre-learning unit 17 prepares original images including only the object such as a product before the additional information is overlaid, as ground truth data for image restoration, corresponding to each overlay image, and stores these overlay images, masks, and ground truth data in a training dataset. Alternatively, the pre-learning unit 17 may store an image in which the additional area to be erased is erased from the overlay image, instead of the mask, in the training dataset, and train the image restoration machine learning model 3 using the training dataset.

[0057] In S112, the pre-learning unit 17 trains the image restoration machine learning model 3 using the learning data set prepared in S111 as an input. In S113, the pre-learning unit 17 evaluates the image restoration machine learning model 3 trained in S112 by using a predetermined loss function. Specifically, the pre-learning unit 17 trains the machine learning model 3 for image restoration so that the difference between the predicted restored image output by the machine learning model 3 for image restoration trained in S112 and the corresponding ground truth data becomes smaller. The predetermined loss function used in S113 may be, for example, a loss function expressed by the following Equation 2, but is not limited to this.

[0058]

number

[0059] Here, L adv represents the adversarial loss, and L pglobal represents the loss in global consistency of perceptual loss, and L plocal represents the loss in local consistency of the perceptual loss. R1 represents the gradient penalty. These perceptual losses evaluate the distance between the feature vectors of the original image and the inpainted image predicted by the machine learning model for image inpainting3. In S113, the pre-learning unit 17 may repeat the processes of S112 and S113 until the loss output by the loss function converges to a desired threshold value.

[0060] <Image cleaning process> Fig. 8 is a flowchart showing an example of a detailed processing procedure of the image cleaning process in step S2 of Fig. 5, which is executed by the overlay image processing device. The overlay information erasure process is a process of predicting the original image before overlay from the overlay image, using two trained machine learning models, namely, the area detection machine learning model 2 and the image restoration machine learning model 3.

[0061] In S21, the area detection unit 12 of the overlay image processing device 1 inputs, as an input image, an overlay image in which additional information is overlaid on an image including a main object, to the area detection machine learning model 2. In S22, the area detection machine learning model 2 detects an area of ​​additional information in the overlay image, and outputs the detection result of the area of ​​additional information as a predicted area to be erased. The detection result of the area of ​​additional information includes four coordinate values ​​of a bounding box surrounding the detected area of ​​additional information and a confidence score. The confidence score indicates the probability that the bounding box contains the object to be detected (the area of ​​additional information in this embodiment).

[0062] The region detection machine learning model 2 learns parameters of the text, color, shape, and corners (including the relative position of the region of the additional information with respect to the overlay image, in particular, whether it is located at a corner of the overlay image) of the additional information as multiple classes of the additional information. The region detection machine learning model 2 may include an output channel that outputs the class of the additional information predicted for the input overlay image.

[0063] When implementing the region detection machine learning model 2 in YOLOv5 or the like, the region detection machine learning model 2 divides the input image, which is an overlay image, into S x S grid cells, and calculates the confidence score of the bounding box of each divided grid cell. The region detection machine learning model 2 selects a highly reliable bounding box as a prediction result by suppressing areas with a large IoU (Intersection over Union) value, i.e., areas with a high degree of overlap, using a threshold value using NMS (Non-Maximum Suppression), and outputs the coordinates of the selected bounding box and the predicted object class together with the confidence score.

[0064] In this embodiment, the area detection unit 12 may perform area detection of additional information using a prediction result of the class of additional information output from the output channel of the area detection machine learning model 2. Alternatively, the size, shape, and arrangement in the overlay image of the area of ​​additional information may be optimized using the prediction result of the class of additional information in the input overlay image in the area detection machine learning model 2, and weighting according to the class may be performed in area detection. The class of additional information may include text, shape (including the presence or absence of a frame surrounding the text), color (including a combination of colors), and corners (including the relative positional relationship between the original image and the area of ​​additional information).

[0065] In S23, the mask generating unit 13 of the overlay image processing device 1 generates a binary mask for masking the area of ​​the additional information detected from the overlay image in S22. Specifically, the mask generation unit 13 generates a mask (mask 45 in FIG. 4) by setting the area of ​​the additional information detected from the overlay image to 1, and setting the remaining area (main object and background area) other than the area of ​​the additional information in the overlay image to 0. By applying the generated mask to the overlay image, it is possible to obtain an intermediate image (image 46 in FIG. 4) from which the area of ​​the detected additional information has been erased from the overlay image.

[0066] In S24, the mask generation unit 13 of the overlay image processing device 1 adjusts the size, shape, or arrangement of the mask generated in S23. Specifically, the mask generation unit 13 may optimize the mask generated in S23 by referring to the class of the additional information output from the output channel of the machine learning model 2 for area detection.

[0067] The mask generated in S23 is usually rectangular in shape since it has a shape that substantially matches the bounding box that encloses the area of ​​the additional information detected in S22. In this embodiment, the shape, size, arrangement, etc. of this rectangular mask are adjusted based on the class of the additional information. The mask generating unit 13 may change the shape of the mask generated in S22 in a rectangular shape so that it matches the shape indicated by the class of the additional information. For example, if the class of the predicted additional information is a class indicating a circular or elliptical shape, changing the rectangular mask to a circular or elliptical shape can reduce the size of the area that must be restored, and the accuracy of image restoration can be improved.

[0068] Fig. 9 is a diagram for explaining an example of such a mask adjustment process executed by the mask generation unit 13. With reference to Fig. 9, it is assumed that in S23, a rectangular mask 92 is generated as a mask for the area of ​​additional information. Meanwhile, the additional information shown in Fig. 9 is located in the upper right corner of the overlay image, and text related to the product object is written in a triangular color frame. In S24, the mask generation unit 13 adjusts the rectangular mask 92 to, for example, a triangular mask 93 that contacts two sides of a corner of the overlay image based on the "corner" class of the additional information. This makes it possible to avoid erasing background and object regions that do not actually need to be masked and subjecting them to image restoration.

[0069] Furthermore, in this embodiment, the shape, size, arrangement, etc. of the generated mask may be adjusted according to the relative positions of the main object in the overlay image, for example, a product object, and the area of ​​the additional information. For example, when the rectangular mask 92 generated in S23 overlaps with the product object of the overlay image, the rectangular mask 92 is adjusted to a triangular mask 93 located at the corner, thereby making it unnecessary to perform image restoration of the area of ​​the product object 91. This makes it possible to prevent image degradation of the product object after image restoration.

[0070] On the other hand, for example, if the detection area of ​​the additional information is too far from the product object of the overlay image, the size and shape of the mask generated in S23 may be adjusted so that it is larger. For example, if the color of the product object is bleeding onto other surrounding objects or the background, the area where the color has bleed and changed to a color different from the original color can be included in the target of image restoration and the area can be restored.

[0071] Furthermore, the boundary of the mask generated in S23 may be adjusted, for example by a few pixels, based on the color difference between the predicted class (particularly, the color class) of the additional information to which the mask corresponds and the result of a local analysis of the color of the area around the boundary of the mask. Returning to Figure 8, in S25, the image restoration unit 14 of the overlay image processing device 1 applies the mask generated in S23 and adjusted as necessary in S24 to the overlay image, which is the input image, to generate an intermediate image in which the areas of additional information to be erased from the overlay image are erased.

[0072] In S26, the image restoration unit 14 of the overlay image processing device 1 inputs to the machine learning model 3 for image restoration the mask generated in S23 and adjusted as necessary in S24, and the intermediate image in which the area of ​​additional information to be erased from the overlay image has been erased. In S27, the machine learning model 3 for image restoration restores the image by restoring the area of ​​the additional information erased in the intermediate image to an area of ​​the original image without the additional information. When the machine learning model 3 for image restoration is implemented in LaMa, the image of the masked area is restored using relatively large area features, the area of ​​additional information is erased from the input image, the overlay image, and the erased area is restored to the original image before overlay, and a restored image 50 is output. In S28, the output unit 15 of the overlay image processing device 1 outputs the restored image output in S27, and the process proceeds to S3 in FIG.

[0073] <Post-processing of image cleaning> FIG. 10 is a flowchart showing an example of a detailed processing procedure of the post-processing in step 3 of FIG. 5, which is executed by the overlay image processing apparatus 1. In FIG. In S31, the post-processing unit 16 of the overlay image processing device 1 may adjust the machine learning model 3 for image restoration. Specifically, the post-processing unit 16 uses the restored image output in S28 of FIG. 8, the corresponding mask, and the overlay image, which is the input image, as input, and calculates a new loss function L, whose parameters are adjusted so that a higher cost (loss) is given to visual artifacts such as image blur. final The machine learning model 3 for image restoration may be retrained using

[0074] In S32, the post-processing unit 16 may input the restored image output in S28 of Fig. 8 and predict the image quality score of the restored image using a network pre-trained with learning data in which high image quality samples are given high scores and low image quality samples are given low scores. In this case, the post-processing unit 16 may delete restored images with image quality scores below a predetermined threshold value from the final output. This makes it possible to ensure the image quality of the restored image that is finally output to the outside.

[0075] At S33, the post-processing unit 16 may adjust the main object in the restored image. Specifically, if the main object remaining in the restored image is displaced to one side due to the erasure of the additional information area, the post-processing unit 16 may relocate the main object to the center of the restored image. Also, if the erased additional information area is relatively large and the main object becomes excessively small compared to the background in the restored image, the post-processing unit 16 may enlarge and relocate the main object so that the main object has an appropriate margin from the image edge. The processes from S31 to S33 may be executed appropriately as necessary. However, by having the post-processing unit 16 execute the processes from S31 to S33, a more highly accurate and visually natural restored image can be obtained, and the restored image can be reused more effectively.

[0076] As described above, according to this embodiment, the overlay image processing device inputs a first image in which additional information is overlaid on an image including an object to a first machine learning model, causes the first machine learning model to detect an area of ​​additional information in the first image, and generates a mask for the detected area of ​​additional information. The overlay image processing device further inputs the first image and the generated mask to a second machine learning model, and causes the second machine learning model to output a second image in which the additional information has been erased from the first image and the area from which the additional information has been erased has been restored.

[0077] This makes it possible to erase additional information appropriately and with high accuracy from an image on which the additional information is overlaid, thereby making it possible to reuse the image, thereby contributing to improving the usability of images that include a variety of objects.

[0078] <Hardware configuration of overlay image processing device> FIG. 11 is a diagram showing a non-limiting example of the hardware configuration of the overlay image processing device 1 according to this embodiment. The overlay image processing apparatus 1 according to this embodiment can be implemented on any single or multiple computers, mobile devices, or any other processing platform. 11, an example in which the overlay image processing device 1 is implemented in a single computer is shown, but the overlay image processing device 1 according to this embodiment may be implemented in a computer system including multiple computers. The multiple computers may be connected to each other so as to be able to communicate with each other via a wired or wireless network.

[0079] 11, the overlay image processing device 1 may include a CPU 111, a ROM 112, a RAM 113, a HDD 114, an input unit 115, a display unit 116, a communication I / F 117, and a system bus 118. The overlay image processing device 1 may also include an external memory. CPU (Central Processing Unit) 111 centrally controls the operation of overlay image processing device 1, and controls each component (112-117) via system bus 118, which is a data transmission path. Note that instead of or in addition to CPU 111, overlay image processing device 1 may include a GPU (Graphics Processing Unit), which executes pre-learning and inference processing of machine learning models 2 and 3.

[0080] The ROM (Read Only Memory) 112 is a non-volatile memory that stores control programs and the like necessary for the CPU 111 to execute processing. Note that the programs may be stored in a non-volatile memory such as an HDD (Hard Disk Drive) 114 or an SSD (Solid State Drive) or an external memory such as a removable storage medium (not shown). The RAM (Random Access Memory) 113 is a volatile memory and functions as a main memory, a work area, etc. of the CPU 111. That is, when executing a process, the CPU 111 loads necessary programs, etc. from the ROM 112 into the RAM 113 and executes the programs, etc. to realize various functional operations.

[0081] The HDD 114 stores, for example, various data and various information required when the CPU 111 performs processing using a program. The HDD 114 also stores, for example, various data and various information obtained when the CPU 111 performs processing using a program. The input unit 115 is composed of a keyboard and a pointing device such as a mouse. The display unit 116 is configured with a monitor such as a liquid crystal display (LCD). The display unit 116 may provide a GUI (Graphical User Interface) that is a user interface for inputting various parameters used in machine learning processing, communication parameters used in communication with other devices, and the like, to the overlay image processing device 1.

[0082] The communication I / F 117 is an interface that controls communication between the overlay image processing device 1 and an external device. The communication I / F 117 provides an interface with a network and executes communication with an external device via the network. Various data, various parameters, etc. are transmitted and received between the external device and the communication I / F 117. In this embodiment, the communication I / F 117 may execute communication via a wired LAN (Local Area Network) or a dedicated line conforming to a communication standard such as Ethernet (registered trademark). However, the network available in this embodiment is not limited to this, and may be configured as a wireless network. This wireless network includes wireless PAN (Personal Area Network) such as Bluetooth (registered trademark), ZigBee (registered trademark), and UWB (Ultra Wide Band). It also includes wireless LAN (Local Area Network) such as Wi-Fi (Wireless Fidelity) (registered trademark), and wireless MAN (Metropolitan Area Network) such as WiMAX (registered trademark). It also includes wireless WAN (Wide Area Network) such as LTE / 3G, 4G, and 5G. Note that the network only needs to connect each device so that they can communicate with each other and be capable of communication, and the communication standard, scale, and configuration are not limited to those described above.

[0083] At least some of the functions of each element of the overlay image processing device 1 shown in Fig. 1 can be realized by the CPU 111 executing a program. However, at least some of the functions of each element of the overlay image processing device 1 shown in Fig. 1 may be operated as dedicated hardware. In this case, the dedicated hardware operates under the control of the CPU 111.

[0084] Although specific embodiments have been described above, the embodiments are merely illustrative and are not intended to limit the scope of the present invention. The apparatus and method described in this specification can be embodied in forms other than those described above. Furthermore, the above-described embodiments can be appropriately omitted, substituted, and modified without departing from the scope of the present invention. Such omitted, substituted, and modified forms are included in the scope of the claims and their equivalents, and belong to the technical scope of the present invention.

[0085] (Embodiments of the present disclosure) The present disclosure includes the following embodiments. [1] An information processing apparatus comprising: an area detection unit that inputs a first image including an object and additional information into a first machine learning model, and causes the first machine learning model to detect an area of ​​the additional information; a mask generation unit that generates a mask for the area of ​​the additional information from information of the area of ​​the additional information detected by the area detection unit; and an image output unit that inputs the mask generated by the mask generation unit and the first image into a second machine learning model, and causes the second machine learning model to output a second image different from the first image, wherein the second image is an image in which the additional information is erased from the first image and the area of ​​the first image from which the additional information was erased is restored.

[0086] [2] The information processing device described in [1], characterized in that the area detection unit determines the area of ​​the additional information based on the class of the additional information output from the first machine learning model.

[0087] [3] The information processing device described in [1] or [2], characterized in that the mask generation unit adjusts the generated mask based on the class of the additional information output from the first machine learning model.

[0088] [4] The information processing device according to [3], further characterized in that the mask generation unit adjusts the generated mask based on the relative position of the object and the additional information.

[0089] [5] An information processing device according to any one of [2] to [4], characterized in that the class of the additional information output from the first machine learning model includes one or more of the color, shape, and relative position of the additional information with respect to the first image or the object.

[0090] [6] The information processing device described in any one of [2] to [5], characterized in that the first machine learning model is trained using the class of the additional information as input.

[0091] [7] The information processing device described in any one of [2] to [6], characterized in that the first machine learning model has an output channel that outputs the class of the additional information.

[0092] [8] An information processing device described in any of [1] to [7], further comprising an image adjustment unit that adjusts the object in the second image based on the relative position of the first image or the object and the additional information.

[0093] [9] The information processing device according to any one of [1] to [8], wherein the second machine learning model is trained to reduce the loss of a loss function that evaluates at least the adversarial loss and the perceptual loss of the second image.

[0094]

[10] The information processing device according to [8], wherein the image adjustment unit adjusts one or more of the position and size of the object in the second image from which the additional information has been deleted.

[0095]

[11] An information processing method executed by an information processing device, comprising the steps of: inputting a first image including an object and additional information into a first machine learning model, and causing the first machine learning model to detect an area of ​​the additional information; generating a mask for the area of ​​the additional information from information of the detected area of ​​the additional information; inputting the generated mask and the first image into a second machine learning model, and causing the second machine learning model to output a second image different from the first image; and the second image being an image in which the additional information has been erased from the first image and the area of ​​the first image from which the additional information has been erased has been restored.

[0096]

[12] A computer-readable recording medium storing an information processing program for causing a computer to execute information processing, the program causing the computer to execute processes including: an area detection process of inputting a first image including an object and additional information into a first machine learning model, and causing the first machine learning model to detect an area of ​​the additional information; a mask generation process of generating a mask for the area of ​​the additional information from information of the area of ​​the additional information detected by the area detection process; and an image output process of inputting the mask generated by the mask generation process and the first image into a second machine learning model, and causing the second machine learning model to output a second image different from the first image; and the second image is an image in which the additional information has been erased from the first image and the area of ​​the first image from which the additional information has been erased has been restored. [Explanation of symbols]

[0097] 1...Overlay image processing device, 2...Machine learning model for area detection, 3...Machine learning model for image restoration, 11...Image input unit, 12...Area detection unit, 13...Mask generation unit, 14...Image restoration unit, 15...Output unit, 16...Post-processing unit, 17...Pre-learning unit, 111...CPU, 112...ROM, 113...RAM, 114...HDD, 115...Input unit, 116...Display unit, 117...Communication I / F, 118...System bus

Claims

1. a region detection unit that inputs a first image including an object and additional information to a first machine learning model and causes the first machine learning model to detect a region of the additional information; a mask generation unit that generates a mask for the area of ​​the additional information based on information of the area of ​​the additional information detected by the area detection unit; an image output unit that inputs the mask generated by the mask generation unit and the first image into a second machine learning model and causes the second machine learning model to output a second image different from the first image; and the second image is an image in which the additional information is erased from the first image and a region of the first image from which the additional information has been erased is restored. An information processing device comprising:

2. The area detection unit determines an area of ​​the additional information based on a class of the additional information output from the first machine learning model.

2. The information processing apparatus according to claim 1,

3. The mask generation unit adjusts the generated mask based on a class of the additional information output from the first machine learning model.

2. The information processing apparatus according to claim 1,

4. The mask generation unit further adjusts the generated mask based on a relative position between the object and the additional information.

4. The information processing apparatus according to claim 3.

5. The class of the additional information output from the first machine learning model includes at least one of a color, a shape, and a relative position between the area of ​​the additional information and an outer periphery of the first image of the additional information.

3. The information processing apparatus according to claim 2.

6. The first machine learning model is trained using the class of the additional information as an input.

3. The information processing apparatus according to claim 2.

7. The first machine learning model has an output channel that outputs the class of the additional information.

3. The information processing apparatus according to claim 2.

8. The apparatus further includes an image adjustment unit that adjusts the object in the second image based on a relative position between the object and the additional information.

2. The information processing apparatus according to claim 1,

9. The image adjustment unit adjusts one or more of a position and a size of the object in the second image from which the additional information has been deleted.

9. The information processing apparatus according to claim 8,

10. The second machine learning model is trained to have a smaller loss of a loss function that evaluates the adversarial loss and the perceptual loss of the second image.

2. The information processing apparatus according to claim 1,

11. An information processing method executed by an information processing device, inputting a first image including an object and additional information into a first machine learning model to cause the first machine learning model to detect an area of ​​the additional information; generating a mask for the area of ​​the additional information from information of the area of ​​the detected additional information; inputting the generated mask and the first image into a second machine learning model to cause the second machine learning model to output a second image different from the first image; and the second image is an image in which the additional information is erased from the first image and a region of the first image from which the additional information has been erased is restored.

13. An information processing method comprising:

12. A computer-readable recording medium storing an information processing program for causing a computer to execute information processing, the program comprising: A region detection process of inputting a first image including an object and additional information into a first machine learning model and causing the first machine learning model to detect a region of the additional information; a mask generation process for generating a mask for the area of ​​the additional information based on information of the area of ​​the additional information detected by the area detection process; an image output process of inputting the mask generated by the mask generation process and the first image into a second machine learning model and causing the second machine learning model to output a second image different from the first image; and the second image is an image in which the additional information is erased from the first image and a region of the first image from which the additional information has been erased is restored. A computer-readable recording medium for executing a process including the steps of:

Images