Program, model generation method, image processing method, and image processing apparatus
By employing a two-stage learning model process to generate and correct mask images based on user input, the program enhances the precision of foreground extraction in image synthesis, addressing the inaccuracies of existing technologies.
Patent Information
- Application Number
- JP2023079603
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-12
- Publication Date
- 2024-11-22
AI Technical Summary
Existing image synthesis technologies inaccurately classify image regions, leading to unnatural boundaries and reduced precision in extracting foreground images, particularly when adjacent regions have differing pixel values.
A program that utilizes a first learning model to generate an initial mask image and a second learning model to infer and correct the foreground region based on user input, combining these models to generate a highly accurate mask image for precise foreground extraction.
The approach enables the generation of a highly accurate foreground image by accurately classifying and correcting foreground regions, reducing unnatural boundaries and improving precision in image synthesis.
Smart Images

Figure 2024163746000001_ABST
Abstract
Description
[Technical field]
[0001] The present disclosure relates to a program, a model generation method, an image processing method, and an image processing device. [Background technology]
[0002] Image synthesis is performed by extracting a foreground region from an image captured by a camera and synthesizing it with another image. For example, Patent Document 1 discloses a technique in which a processing target image is classified into a foreground region, a background region, a first unknown region, and a second unknown region, each pixel of the first unknown region is classified into a foreground region and a background region by segmentation, the transparency of each pixel of the second unknown region is obtained by alpha matting, a processing result for the first unknown region and a processing result for the second unknown region are integrated to generate a mask image, and a foreground image is cut out from the processing target image using the mask image. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] Patent No. 6676563 Summary of the Invention [Problem to be solved by the invention]
[0004] In the technology disclosed in Patent Document 1, the foreground region, background region, first unknown region, and second unknown region are classified based on the luminance information of the processing target image, so that there is a possibility that a region not intended by the user is classified as the foreground region or background region. In addition, in the technology disclosed in Patent Document 1, when the pixel values of the first unknown region and the second unknown region are different in the mask image in which the processing result for the first unknown region and the processing result for the second unknown region are integrated, it is considered that an unnatural boundary may occur at the integrated portion of the first unknown region and the second unknown region. It is difficult to cut out the foreground image from the processing target image with high accuracy using such a mask image.
[0005] An object of the present disclosure is to provide a program and the like capable of generating a mask image that enables extraction of a foreground region with high accuracy from a captured image. [Means for solving the problem]
[0006] A program according to an aspect of the present disclosure causes a computer to execute a process of acquiring a captured image, generating a first mask image indicating a foreground region in the acquired captured image, receiving a correction instruction for the foreground region or background region in the first mask image, and inputting the acquired captured image, the first mask image indicating the foreground region in the captured image, and information indicating the content of the correction instruction received for the first mask image into a learning model trained to output a second mask image indicating the foreground region in the captured image inferred based on the input information when the captured image, the first mask image indicating the foreground region in the captured image, and information indicating the content of the correction instruction received for the first mask image, to acquire a second mask image indicating the foreground region in the captured image inferred based on the input information, and generating a third mask image to be output based on the first mask image, the second mask image, and the information indicating the content of the correction instruction received for the first mask image. Effect of the Invention
[0007] According to one aspect of the present disclosure, it is possible to generate a mask image that allows for extraction of a foreground region with high accuracy from a captured image. [Brief description of the drawings]
[0008] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of an image processing device. [Diagram 2] FIG. 2 is an explanatory diagram showing an example of the configuration of a learning model. [Diagram 3] 13 is a flowchart illustrating an example of a procedure for generating an output mask image. [Figure 4] 13 is a flowchart illustrating an example of a procedure for generating an output mask image. [Diagram 5] FIG. 11 is an explanatory diagram of a merge process. [Figure 6] FIG. 11 is an explanatory diagram of a merge process. [Figure 7] FIG. 11 is an explanatory diagram of a merge process. [Figure 8] FIG. 13 is an explanatory diagram showing an example of a screen. [Figure 9] 13 is a flowchart showing an example of a procedure of a composite image providing process. [Figure 10] FIG. 13 is an explanatory diagram showing an example of a screen. [Figure 11] FIG. 13 is an explanatory diagram showing an example of a screen. [Figure 12] FIG. 13 is an explanatory diagram showing an example of a screen. [Figure 13] 13 is a flowchart showing an example of a merge processing procedure according to the second embodiment. [Figure 14] 13 is a flowchart showing an example of a merge processing procedure according to the third embodiment. [Figure 15] 13 is a flowchart showing a modified example of the procedure for generating an output mask image. [Figure 16] FIG. 13 is an explanatory diagram showing an example of a screen. [Figure 17] 13 is a flowchart showing a modified example of the procedure of the composite image providing process. [Figure 18] 13 is a flowchart showing an example of a synthetic image providing process procedure according to the fifth embodiment. [Figure 19] FIG. 13 is an explanatory diagram showing an example of a screen. [Figure 20] FIG. 1 is an explanatory diagram illustrating an example of the configuration of an image processing system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0009] Hereinafter, the program, model generation method, image processing method, and image processing device of the present disclosure will be described in detail with reference to the drawings showing embodiments thereof.
[0010] (Embodiment 1) FIG. 1 is a block diagram showing an example of the configuration of an image processing device. In this embodiment, an image processing device is described that extracts a photographed area of a subject from a photographed image captured by a camera and synthesizes the extracted photographed area of the subject with another image. In this embodiment, in the photographed area of the subject in the photographed image, the area of the subject that is to be synthesized with another image is set as a foreground area, and the area of the subject that is not to be synthesized with another image is set as a background area. Which subject in the photographed image is set as the foreground area or the background area is specified by an operation by a user.
[0011] The image processing device 10 is an information processing device capable of various information processing and transmitting and receiving information, such as a personal computer, a server computer, etc. The image processing device 10 has a control unit 11, a storage unit 12, a communication unit 13, an input unit 14, a display unit 15, a reading unit 16, etc., and these units are connected via a bus. The control unit 11 includes one or more processors such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), a GPU (Graphics Processing Unit), a TPU (Tensor Processing Unit), or an AI chip (AI semiconductor). The control unit 11 executes the program 12P stored in the storage unit 12 as appropriate to execute the information processing and control processing to be performed by the image processing device 10.
[0012] The storage unit 12 includes a RAM (Random Access Memory), a flash memory, a hard disk, an SSD (Solid State Drive), and the like. The storage unit 12 stores a program 12P (program product) executed by the control unit 11 and various data. The storage unit 12 also temporarily stores data and the like generated when the control unit 11 executes the program 12P. The program 12P and various data may be written to the storage unit 12 during the manufacturing stage of the image processing device 10, or the control unit 11 may download them from another device via the communication unit 13 and store them in the storage unit 12. The storage unit 12 also stores a first learning model M1 and a second learning model M2 that have learned training data by machine learning, for example. The learning models M1 and M2 are expected to be used as program modules that constitute artificial intelligence software. The learning models M1 and M2 perform a predetermined calculation on an input value and output the calculation result, and the storage unit 12 stores data such as coefficients and thresholds of functions that define this calculation as the learning models M1 and M2.
[0013] The storage unit 12 also stores a photographed image DB 12a and a composite background DB 12b. The photographed image DB 12a is a database in which images to be processed by the image processing device 10 are stored. The photographed images stored in the photographed image DB 12a may be images received from a camera or another information processing device via the communication unit 13, or may be images captured by the camera of the image processing device 10 if the image processing device 10 is equipped with a camera. The composite background DB 12b is a database in which background images for compositing (compositing images) for compositing a photographed area of a subject (hereinafter referred to as a foreground image) extracted from a photographed image are stored. The composite background images are a plurality of types of images, and may be still images or moving images. A part or all of the learning models M1, M2, the photographed image DB 12a, and the composite background DB 12b may be stored in another storage device connected to the image processing device 10, or in another storage device with which the image processing device 10 can communicate.
[0014] The communication unit 13 is a communication module for performing processes related to wired or wireless communication, and transmits and receives information to and from other devices via the network N. The network N may be the Internet or a public telephone line network, or may be a LAN (Local Area Network) constructed in a facility where the image processing device 10 is installed. The input unit 14 accepts operation input by a user and sends a control signal corresponding to the operation content to the control unit 11. The display unit 15 is a liquid crystal display or an organic EL display, etc., and displays various information according to instructions from the control unit 11. A part of the input unit 14 and the display unit 15 may be a touch panel configured as an integrated unit, or the touch panel may be configured to be externally attached to the image processing device 10.
[0015] The reading unit 16 reads information stored in the portable storage medium 10a, such as a CD (Compact Disc), a DVD (Digital Versatile Disc), a USB (Universal Serial Bus) memory, an SD card, a micro SD card, a Compact Flash (registered trademark), etc. The program 12P and various data stored in the storage unit 12 may be read by the control unit 11 from the portable storage medium 10a via the reading unit 16 and stored in the storage unit 12.
[0016] In this embodiment, the image processing device 10 may be a multi-computer including multiple computers, a virtual machine virtually constructed by software in one device, or a cloud server. The program 12P may be deployed and executed on a single computer or at one site, or may be distributed across multiple sites and deployed to be executed on multiple computers connected to each other via a network N. Furthermore, the input unit 14 and the display unit 15 are not essential for the image processing device 10, and the image processing device 10 may be configured to accept operations through a connected computer, or to output information to be displayed to an external display device.
[0017] In the image processing device 10 of this embodiment, the control unit 11 reads out and executes the program 12P stored in the storage unit 12, thereby performing a process of removing a background region from a photographed image, extracting a foreground region (subject region), and generating a foreground image. The control unit 11 also performs a process of compositing the foreground image generated from the photographed image with a background image for compositing. Therefore, the image processing device 10 of this embodiment can provide a user with a composite image in which the foreground image generated from the photographed image is composited with an arbitrary background image for compositing. Note that, in the image processing device 10 of this embodiment, the control unit 11 uses the first learning model M1 when extracting the foreground region from the photographed image to generate a first mask image, and uses the second learning model M2 when inferring the foreground region based on the photographed image and the first mask image to generate a second mask image.
[0018] 2 is an explanatory diagram showing an example of the configuration of learning models M1 and M2. The learning models M1 and M2 are models that recognize a specific object contained in an input image, and can classify objects in an image pixel by pixel, for example, by semantic segmentation. The learning models M1 and M2 can be configured using algorithms such as U-Net, FCN (Fully Convolutional Network), SegNet, etc., and may be configured by combining multiple algorithms.
[0019] The first learning model M1 is a model that takes one captured image as input, performs a calculation to recognize a background region and a foreground region (subject region) included in the captured image based on the input captured image, and outputs the recognition result. For the first learning model M1, for example, "Robust Video Matting" developed by ByteDance Inc. can be used. Specifically, the first learning model M1 classifies each pixel of the input captured image into a background region and a foreground region, and outputs a classified captured image (label image) in which a label for each region is associated with each pixel. In the example shown in FIG. 2A, the first learning model M1 outputs a multi-value label image in which pixels classified into the background region are indicated by black (for example, pixel value 0) and pixels classified into the foreground region are indicated by white (for example, pixel value 255), and each pixel has a pixel value of 0 to 255. The label image output from the first learning model M1 is a first mask image indicating the foreground region in the captured image, and the image processing device 10 of this embodiment uses the first mask image as an initial mask image (hereinafter referred to as a front mask image).
[0020] The first learning model M1 can be generated by machine learning using training data in which a photographed image for training and a correct labeled image in which data indicating an object to be discriminated (here, a background region and a foreground region) are labeled for each pixel in the photographed image are associated with each other. The first learning model M1 learns to output the correct labeled image included in the training data when a photographed image included in the training data is input. Specifically, the first learning model M1 performs a calculation based on the input photographed image, and obtains a detection result in which each object (here, a background region and a foreground region) in the photographed image is detected. More specifically, the first learning model M1 obtains, as an output, a labeled image in which a value indicating the type of classified object is labeled for each pixel in the photographed image. Then, the first learning model M1 compares the obtained detection result (label image) with the range and type of each object in the correct labeled image, and optimizes parameters such as weights (coupling coefficients) between neurons so that the two are similar. The method of optimizing the parameters is not particularly limited, but a steepest descent method, an error backpropagation method, etc. can be used. This provides a first learning model M1 that outputs a label image indicating the background region and the foreground region in a captured image when the captured image is input.
[0021] The label image generated from the captured image using the first learning model M1 may not be accurately classified into the foreground and background regions intended by the user. In addition, when the captured image includes multiple objects, which object is to be synthesized into the background image for synthesis varies depending on the user or what kind of synthesized image is to be generated, so it is difficult to generate a label image in which the area of the object to be synthesized is appropriately classified into the foreground area. Therefore, in this embodiment, a process is performed to generate an inferred mask image in which the foreground area intended by the user is inferred based on the captured image, the front mask image obtained by the first learning model M1, and a correction instruction to the background area and foreground area for the foreground area and background area classified in the front mask image, using the second learning model M2.
[0022] The second learning model M2 is a model that receives as input one captured image, a front mask image generated from the captured image using the first learning model M1, and the contents of a correction instruction to the background region or foreground region given to the foreground region or background region classified in the front mask image, performs a calculation to infer the foreground region intended by the user based on the input image and information, and outputs an inference mask image that is the calculation result. Specifically, the second learning model M2 infers (predicts) the foreground region intended by the user from the captured image and the front mask image based on the input correction instruction contents, and outputs an inference mask image showing the inferred foreground region. In the inference mask image, the excess or deficiency of the foreground region and background region in the front mask image is adjusted, and the background region that the user instructed to correct to the foreground region is corrected to the foreground region, and the foreground region that the user instructed to correct to the background region is corrected to the background region.
[0023] The second learning model M2 can be generated by machine learning using training data in which a photographed image for training, a front mask image generated from the photographed image using the first learning model M1, correction instructions to the background region or foreground region for any pixel in the foreground region or background region in the front mask image, and a correct label image in which data indicating an object to be determined (here, the background region and the foreground region) is labeled for each pixel in the photographed image are associated. The correct label image here is a label image indicating the foreground region in the photographed image in which the correction instructions are reflected. The second learning model M2 learns to output the correct label image included in the training data when the photographed image, front mask image, and correction instructions included in the training data are input. Specifically, the second learning model M2 performs calculations based on the input photographed image, front mask image, and correction instructions, and obtains an inferred mask image as a result of inferring the foreground region intended by the user in the input photographed image. The second learning model M2 then compares the acquired inference mask image with the correct labeled image indicated by the training data, and optimizes parameters using the steepest descent method, backpropagation method, etc. so that the two are close to each other. In this way, when the captured image, the front mask image, and the correction instruction are input, the second learning model M2 is obtained, which outputs an inference mask image in which the captured image is classified into the foreground region and background region intended by the user. In the example shown in FIG. 2B, a correction instruction is given to the pixels of the umbrella region classified into the background region in the front mask image to the foreground region, and in this case, in the inference mask image, the umbrella region instructed to be corrected is corrected to the foreground region.
[0024] The learning models M1 and M2 may be learned by the image processing device 10 or by another learning device. The learned learning models M1 and M2, which are generated by learning by another learning device, are downloaded from the learning device to the image processing device 10 via the network N or the portable storage medium 10a, for example, and stored in the storage unit 12. The image processing device 10 prepares the learning models M1 and M2 as described above in advance and uses them in a background removal process for removing a background area in a captured image captured by a camera and extracting a foreground area. Specifically, the image processing device 10 uses the first learning model M1 to obtain a front mask image in which the foreground area and the background area in the captured image are classified, and uses the second learning model M2 to obtain an inference mask image in which the captured image is classified into the foreground area and the background area intended by the user. Then, the image processing device 10 generates a mask image (output mask image for output) in which the background area in the captured image is masked from the front mask image and the inference mask image, and uses the generated output mask image to extract the unmasked foreground area from the captured image, thereby generating a foreground image from which the background area has been removed.
[0025] The following describes the process in which the image processing device 10 of this embodiment extracts a foreground area intended by the user from a captured image to generate a foreground image, and synthesizes the generated foreground image with an arbitrary background image for synthesis to provide it. First, the process of generating a mask image (output mask image) used when generating a foreground image from a captured image will be described. FIGS. 3 and 4 are flowcharts showing an example of the process procedure for generating an output mask image, FIGS. 5 to 7 are explanatory diagrams of merging processing, and FIG. 8 is an explanatory diagram showing an example screen. The merging processing shown in FIG. 4 is the process of step S17 in FIG. 3. The following process is performed by the control unit 11 of the image processing device 10 in accordance with a program 12P stored in the storage unit 12.
[0026] The control unit 11 of the image processing device 10 acquires a photographed image to be processed (S11). The photographed image may be any photographed image, such as an image photographed at an arbitrary photographing location, an image photographed by a camera held by a cameraman, or an image photographed by a camera with a fixed photographing position. The photographed image to be processed may be read from the photographed image DB 12a, or may be acquired from a camera or another information processing device via the communication unit 13. The control unit 11 executes a background removal process on the acquired photographed image to generate a front mask image (first mask image) showing a foreground region in the photographed image (S12). Specifically, the control unit 11 inputs the photographed image to the first learning model M1 and acquires a label image (front mask image) output from the first learning model M1. Here, the control unit 11 acquires a front mask image in which the foreground region and the background region in the photographed image are classified. Note that the control unit 11 may generate a front mask image by extracting a foreground region in the photographed image by a background removal process using a background difference method or the like. The background subtraction method is a process for extracting a subject that does not exist in a background image by calculating the difference between a background image captured in advance and a newly captured image.
[0027] Based on the photographed image and the front mask image, the control unit 11 displays on the display unit 15 a foreground / background correction screen for receiving a correction instruction to the background region or the foreground region for any pixel classified as the foreground region or the background region in the front mask image (S13). For example, the control unit 11 generates and displays a screen as shown in FIG. 8A. The screen shown in FIG. 8A is configured to display an image in which the foreground region in the front mask image is superimposed as a gray region on the photographed image, and to receive a correction instruction to the pixel in the foreground region to the background region and a correction instruction to the pixel in the background region to the foreground region via the photographed image being displayed. In the screen shown in FIG. 8A, the user selects any pixel in the foreground region that the user wants to correct to the background region, or any pixel in the background region that the user wants to correct to the foreground region, via the input unit 14, and operates a correction button to give a correction instruction to the arbitrary pixel. Note that the correction location may be specified by, for example, a click operation on an arbitrary location (pixel), as shown by a black circle in FIG. 8A, or by drawing a line on the arbitrary region by a drag operation. In the following, the operation of instructing a pixel in the foreground region in the front mask image to be modified in the background region will be referred to as a background click, and the operation of instructing a pixel in the background region in the front mask image to be modified in the foreground region will be referred to as a foreground click.
[0028] The foreground / background correction screen is configured to be able to switch between and display a superimposed image in which a foreground region in a front mask image is superimposed as a gray region on a photographed image as shown in Fig. 8A, a foreground image in which a foreground region in a front mask image is extracted from a photographed image by background removal processing based on the front mask image as shown in Fig. 8B, and a front mask image as shown in Fig. 8C. The foreground / background correction screen shown in Fig. 8A is provided with a "Photographed image + front mask image" button, a "Photographed image (background removal)" button, and a "front mask image" button for instructing switching between the images. The control unit 11 accepts the selection of an image by operating any of the buttons, displays the selected image on the foreground / background correction screen, and can accept a correction instruction for any pixel in the image via the displayed image.
[0029] The control unit 11 judges whether or not a correction instruction for any part (pixel) has been received via the foreground / background correction screen (S14). If the control unit 11 judges that a correction instruction has not been received (S14: NO), that is, if the end button is operated without a correction instruction being given to the foreground region or background region in the front mask image displayed on the screen of Fig. 8A, the control unit 11 specifies the front mask image generated in step S12 as the output mask image (S19).
[0030] When it is determined that a correction instruction has been received (S14: YES), the control unit 11 acquires the content of the received correction instruction. Specifically, the control unit 11 acquires the position (correction position) of the portion (pixel) instructed to be corrected and the correction content indicating whether a foreground click or a background click has been performed (S15). As the correction content, the control unit 11 acquires a correction instruction indicating that a background click has been performed (correction instruction from the foreground region to the background region) when the pixel at the correction position is a pixel in the foreground region of the front mask image, and acquires a correction instruction indicating that a foreground click has been performed (correction instruction from the background region to the foreground region) when the pixel at the correction position is a pixel in the background region of the front mask image. Then, the control unit 11 inputs the photographed image acquired in step S11, the front mask image generated in step S12, and the correction instruction content (correction position and correction content) acquired in step S15 to the second learning model M2, and generates an inference mask image (second mask image) by the second learning model M2 (S16). Specifically, the control unit 11 acquires an inferred mask image output from the second learning model M2 by inputting the captured image, the front mask image, and the correction instruction content.
[0031] Next, the control unit 11 executes a merging process (S17) for merging the front mask image generated in step S12 with the inference mask image generated in step S16 to generate an output mask image. Here, the front mask image and the inference mask image are multi-valued images and are generated by different learning models M1 and M2, so there is a possibility that there will be a difference in inference accuracy. When such two mask images are merged, there is a possibility that an unnatural feeling (stranger feeling) will occur in the merged part due to the difference in inference accuracy. Therefore, in this embodiment, the difference in inference accuracy between the two mask images is reduced by performing a binarization process on the two mask images. It is assumed that the inference accuracy of the second learning model M2 is lower than that of the first learning model M1. For a front mask image with high inference accuracy, when a foreground click is performed, binarization processing is performed with a pixel value (e.g., 255) of a white pixel (pixel in the foreground region) in the front mask image or a value close to the pixel value of a white pixel (e.g., 254 or 250) as the threshold T1, and when a background click is performed, binarization processing is performed with a pixel value (e.g., 0) of a black pixel (pixel in the background region) in the front mask image or a value close to the pixel value of a black pixel (e.g., 1 or 5) as the threshold T1. By switching the threshold T1 used for the binarization processing performed on the front mask image in this way when a foreground click is performed and when a background click is performed, the difference in inference accuracy with the inference mask image can be further reduced, and the sense of strangeness based on the difference in inference accuracy can be suppressed. On the other hand, for an inference mask image with low inference accuracy, binarization processing is performed using the same threshold T2 regardless of whether a foreground click or a background click is performed. In addition, the threshold value T2 for the inference mask image may be the intermediate value, median value, or average value between the pixel values of the white pixels (e.g., 255) and the pixel values of the black pixels (e.g., 0) in the inference mask image, or may be automatically determined from the pixel values of each pixel of the inference mask image using Otsu's binarization process or the like.
[0032] In the merge process shown in FIG. 4, the control unit 11 specifies a threshold value T1 in the binarization process to be performed on the front mask image according to the correction content acquired in step S15 (S21). Here, when a foreground click is performed, the control unit 11 specifies the pixel value of a white pixel in the front mask image or a value close to this pixel value as the threshold value T1, and when a background click is performed, the control unit 11 specifies the pixel value of a black pixel in the front mask image or a value close to this pixel value as the threshold value T1. Then, the control unit 11 performs binarization process on the front mask image based on the specified threshold value T1 (S22). As a result, an image such as that shown in the front mask image (binary) in FIG. 5 is obtained. The control unit 11 performs binarization process on the inference mask image based on, for example, a preset threshold value T2 (S23). As a result, an image such as that shown in the inference mask image (binary) in FIG. 5 is obtained.
[0033] Next, the control unit 11 calculates an exclusive OR (XOR) between the binarized front mask image and the binarized inference mask image (S24). Specifically, the control unit 11 calculates the exclusive OR of pixel values for each pixel position in the binary front mask image and the binary inference mask image to generate a difference image between the two mask images. As a result, an image as shown in the XOR image in FIG. 5 is obtained. In the XOR image shown in FIG. 5, the difference area (area where the exclusive OR is 1, difference area) is shown by white pixels, and the other area (area where the exclusive OR is 0) is shown by black pixels. Note that, since the binarization process is performed on the front mask image and the inference mask image using different thresholds T1 and T2 in steps S22 to S23, the difference area (area of white pixels) in the XOR image tends to be larger than when the exclusive OR is calculated after performing the binarization process using the same threshold (for example, an intermediate value between the pixel value of a white pixel and the pixel value of a black pixel). Therefore, in this embodiment, the control unit 11 executes morphology processing on the XOR image (difference image) (S25) and corrects the XOR image by deleting unnecessary difference areas from the XOR image.
[0034] The control unit 11 performs a closing process as shown in FIG. 6A as a morphology process, and then performs an opening process as shown in FIG. 6B. The closing process is a process of performing a dilation operation multiple times on a binary image, and then performing an erosion operation the same number of times, and can erase noise due to black pixels as shown on the left side of FIG. 6A. The opening process is a process of performing an erosion operation multiple times on a binary image, and then performing an dilation operation the same number of times, and can erase noise and lines due to white pixels as shown on the left side of FIG. 6B. The dilation operation is a process of replacing the pixel value of a pixel to be processed with the largest pixel value in the vicinity of the pixel, and the erosion operation is a process of replacing the pixel value of a pixel to be processed with the smallest pixel value in the vicinity of the pixel. By performing the morphology process, the difference area in the XOR image can be divided. In the example shown in FIG. 5, the difference area in the XOR image after the morphology process is divided into two areas. This prevents the size of the connected area extracted from the XOR image in the next step S26 from increasing.
[0035] Next, the control unit 11 specifies a connected region in the XOR image (difference image) corrected by the morphology process, in which adjacent pixels with the same pixel value (same label) as the pixel value of the pixel at the correction position obtained in step S15 are sequentially connected (S26). For example, in an image as shown in FIG. 6C, if the pixel shown in gray is the pixel at the correction position, the pixel at the correction position is a white pixel, so the control unit 11 extracts white pixels from among the pixels adjacent to the pixel at the correction position as pixels of the connected region. The control unit 11 then specifies the connected region by repeating the process of extracting white pixels from among the pixels adjacent to the extracted pixel as pixels of the connected region until there are no more white pixels in the adjacent pixels. As a result, an image as shown by the connected region in FIG. 5 is obtained. The connected region specified in this way becomes an update region to be merged into the previous mask image. Therefore, the control unit 11 sets the specified connected region as an update region (S27).
[0036] The control unit 11 specifies the pixel value of the pixel in the update region to be merged into the previous mask image according to the correction content acquired in step S15 (S28). Here, when a foreground click is performed, the control unit 11 specifies the pixel value of a white pixel in the previous mask image or a value close to this pixel value, and when a background click is performed, the control unit 11 specifies the pixel value of a black pixel in the previous mask image or a value close to this pixel value. Then, the control unit 11 inserts a pixel of the specified pixel value into a region in the previous mask image corresponding to the update region to generate an output mask image (third mask image) (S29). Specifically, when a foreground click is performed, the control unit 11 inserts a white pixel into a region corresponding to the update region of the previous mask image, and when a background click is performed, the control unit 11 inserts a black pixel into a region corresponding to the update region of the previous mask image, thereby merging a pixel of a pixel value according to the content of the correction instructed by the user into the update region specified based on the location instructed by the user. This allows a correction process to be performed on the previous mask image to reflect the content of the correction instructed by the user. This allows an output mask image as shown in FIG. 5 to be obtained.
[0037] The control unit 11 returns to the process of Fig. 3 and sets (updates) the output mask image generated in step S29 as the front mask image in the process of generating a new output mask image based on the content of the next correction instruction given by the user (S18).Then, the control unit 11 proceeds to step S13 and displays a foreground / background correction screen as shown in Fig. 8A on the display unit 15 (S13).Here, a superimposed image is displayed on the foreground / background correction screen, in which the foreground area in the updated front mask image, which has been updated to the output mask image generated in step S29, is superimposed as a gray area on the captured image.
[0038] Thereafter, the control unit 11 continues the process from step S14 onwards. In the output mask image shown in FIG. 5, the left half of the umbrella is classified as a background region, so that the user selects any pixel in the left half of the umbrella on the newly displayed screen shown in FIG. 8A and operates the correction button to perform a foreground click on the left half of the umbrella (a correction instruction from the background region to the foreground region). In this case, the control unit 11 determines that a correction instruction for the updated front mask image has been accepted (S14: YES), and repeats the processes of steps S15 to S17 to generate a new output mask image. Specifically, the control unit 11 acquires the correction instruction contents (correction position and correction contents) as shown in FIG. 7A (S15), and inputs the captured image, the updated front mask image, and the acquired correction instruction contents into the second learning model M2 as shown in FIG. 7A to generate a new inference mask image (S16). Then, the control unit 11 performs the merging process shown in Fig. 4 based on the updated front mask image and the newly generated inference mask image, and generates a new output mask image as shown in Fig. 7B (S17). As a result, an output mask image is generated in which the updated area based on the portion further specified by the user is updated to the foreground area.
[0039] When the control unit 11 determines that the correction instruction has not been received (S14: NO), that is, when the end button in the screen of FIG. 8A is operated, it specifies the front mask image at this time (the output mask image generated in step S29) as the output mask image (S19). By the above-mentioned process, a mask image (output mask image) for extracting the foreground area intended by the user with high accuracy from the captured image is generated. In the above-mentioned process, the foreground area and background area can be corrected based on the correction instruction to the foreground area and background area intended by the user for the front mask image in which the foreground area is automatically extracted from the captured image using the first learning model M1. Therefore, it is possible to correct the excess or deficiency area of the foreground area or background area and correct the foreground area and background area to the foreground area and background area intended by the user without any sense of incongruity. Furthermore, in the above-described processing, even if there is a difference in inference accuracy between the front mask image generated from the captured image using the first learning model M1 and the inference mask image generated using the second learning model M2 from the correction instructions given by the user based on the front mask image, the front mask image and the inference mask image (update area) can be merged without any sense of incongruity, making it possible to generate a highly accurate output mask image.
[0040] Next, a process will be described in which a foreground image is generated from a photographed image using the output mask image generated by the above-mentioned process, and the generated foreground image is synthesized with a synthesis background image and provided. Fig. 9 is a flow chart showing an example of a synthetic image providing process procedure, and Figs. 10 to 12 are explanatory diagrams showing example screens. The following process is performed by control unit 11 of image processing device 10 in accordance with program 12P stored in storage unit 12.
[0041] The control unit 11 of the image processing device 10 uses the output mask image specified in step S19 in Fig. 3 as a mask image to extract an area corresponding to a white area (foreground area) in the mask image from the captured image acquired in step S11, thereby extracting the foreground area in the captured image and generating a foreground image (S31). As a result, a foreground image in which the background area has been removed from the captured image is generated.
[0042] Control unit 11 displays a list of compositing background images to receive a selection of a compositing background image to be composited with the generated foreground image (S32). Here, control unit 11 reads out a compositing background image stored in compositing background DB 12b, displays a screen as shown in FIG. 10 on display unit 15, and presents a list of compositing background images to the user. The screen shown in FIG. 10 displays a foreground image, a still image prepared as a compositing background image, and a thumbnail image of a video (for example, the first image), and is configured to receive a selection of any of the compositing background images. In addition, the screen shown in FIG. 10 displays an indicator indicating a total playback time and a playback position relative to the total playback time for the compositing background image of the video, and the playback position of the video displayed on the screen can be changed by moving the playback position via the indicator. The user selects any of the compositing background images via input unit 14 on the screen shown in FIG. 10 and operates an OK button to select any of the compositing background images. The control unit 11 determines whether or not a selection of any of the composite background images has been accepted (S33), and if it determines that a selection has not been accepted (S33: NO), it waits until it is accepted.
[0043] When control unit 11 determines that the selection of any of the synthesis background images has been accepted (S33: YES), control unit 11 displays on display unit 15 a setting screen for accepting settings of image processing to be performed on the foreground image when synthesizing the foreground image with the selected synthesis background image (S34). For example, control unit 11 displays a setting screen having an input field for inputting the processing content of the image processing to be performed on the foreground image, as shown in FIG. 11A. The image processing to be performed on the foreground image includes a magnification process for enlarging or reducing the foreground image, and a rotation process for rotating the foreground image. Thus, the setting screen has an input field for the magnification ratio for the width and height of the foreground image and an input field for the rotation angle, and each input field may be configured to allow any numerical value to be input, or a pull-down menu for selecting any one of a plurality of options may be provided. Note that the setting screen shown in FIG. 11A may be configured to allow the magnification ratio or reduction ratio to be input for the foreground image by, for example, pinching out and pinching in. Furthermore, the setting screen is configured so that a compositing position (compositing position in the image to be composited) for compositing the foreground image with the background image to be composited can be specified by a drag operation as shown in Fig. 11B. Note that the image processing that can be performed on the foreground image is not limited to the magnification processing and the rotation processing.
[0044] 11A and 11B, the user inputs the magnification ratio for the width and height of the foreground image, the rotation angle, and the composite position of the foreground image relative to the composite background image via input unit 14, and operates the composite execution button to instruct the execution of composite processing of the foreground image relative to the composite background image. Control unit 11 determines whether or not an input of the processing content of the image processing has been received on the setting screen shown in FIGS. 11A and 11B (S35), and when it is determined that the input has been received (S35: YES), it displays the input processing content (specifically, the magnification ratio and the rotation angle) in each input field. Then, control unit 11 executes the image processing of the input processing content on the foreground image being displayed (S36). Here, when a magnification ratio is input, control unit 11 executes a magnification processing (enlargement processing or reduction processing) at the input magnification ratio on the foreground image being displayed, and when a rotation angle is input, it executes a rotation processing at the input rotation angle on the foreground image being displayed. Then, control unit 11 updates the foreground image being displayed to the foreground image after the image processing. The screen in Fig. 11B displays the foreground image after reduction processing has been performed on the foreground image in the screen in Fig. 11A. Image processing of the foreground image is not necessarily required, and when it is determined that input of the image processing content has not been accepted (S35: NO), control unit 11 skips the processing of step S36.
[0045] Next, control unit 11 judges whether input of a compositing position of the foreground image relative to the compositing background image has been received on the setting screen shown in Figures 11A and 11B (S37), and if it is judged that the input has been received (S37: YES), it moves the foreground image to the specified compositing position (S38) as shown in Figure 11B. Then, control unit 11 judges whether a compositing execution button has been operated (S39), and if it is judged that the button has not been operated (S39: NO), it returns to the process of step S35 and continues to accept input of the image processing content and the compositing position. Note that control unit 11 also returns to the process of step S35 if it is judged that input of the compositing position has not been received (S37: NO).
[0046] When it is determined that the synthesis execution button has been operated (S39: YES), control unit 11 performs synthesis processing to synthesize the selected foreground image with the selected synthesis background image (S40). Specifically, control unit 11 synthesizes the foreground image subjected to image processing in step S36 with the synthesis position designated in steps S37 to S38 in the synthesis background image selected in step S33 to generate a synthetic image. Control unit 11 outputs the generated synthetic image (S41) and ends the series of processes. For example, control unit 11 displays a screen such as that shown in Fig. 12 on display unit 15 to display the generated synthetic image and present it to the user.
[0047] The screen shown in FIG. 12 displays a composite image, and has a redo button for instructing redoing the generation of the composite image, a print button for instructing printing of the composite image, and a send button for instructing transmission of the composite image. When the redo button is operated on the screen shown in FIG. 12, the control unit 11 of the image processing device 10 returns to the process of step S34 in FIG. 9, for example, and executes the image processing and the composition processing again. Note that, when the control unit 11 redoes the selection of the background image for composition, it may return to the process of step S32 in FIG. 9. When the print button is operated on the screen shown in FIG. 12, the control unit 11 of the image processing device 10 transmits the composite image to a printer connected via the network N or directly to print the composite image. Also, when the send button is operated, the control unit 11 transmits the composite image to a specified terminal via the network N or by short-distance wireless communication or the like. Note that, the control unit 11 may print the generated composite image by a printer or transmit it to a predetermined terminal without displaying the screen shown in FIG. 12.
[0048] By the above-mentioned process, the image processing device 10 of the present embodiment can realize background removal processing that can extract the foreground area intended by the user from the photographed image with high accuracy by using a mask image (output mask image) that can extract the foreground area intended by the user from the photographed image with high accuracy. Therefore, a foreground image in which the foreground area intended by the user is extracted with high accuracy can be generated for a photographed image in which a subject is photographed with a free background and a free photographing position. In addition, since the foreground image can be subjected to magnification processing and rotation processing, and an arbitrary background image for synthesis can be selected, it is possible to realize free processing of the foreground image. Furthermore, even if a subject not intended by the user is captured, it is possible to generate a foreground image in which only the subject selected by the user is in the foreground area, and a foreground image including only the arbitrary subject can be generated.
[0049] In this embodiment, a first learning model M1 is used to generate a front mask image in which a background region is removed from a photographed image and a foreground region is extracted, and a second learning model M2 is used to generate an inference mask image showing a predicted foreground region based on the photographed image, the front mask image, and the correction instruction content by the user. Then, by merging the front mask image and the inference mask image, an output mask image can be generated in which the foreground region intended by the user can be extracted from the photographed image. Therefore, by performing background removal using such an output mask image, a highly accurate foreground image in which the foreground region intended by the user is extracted can be generated. Note that the user can give a correction instruction by clicking or dragging the part of the foreground region or background region that he / she wants to correct, so operability is good.
[0050] In this embodiment, the prediction process of the mask image (the front mask image and the inference mask image) using the learning models M1 and M2 is not limited to a configuration in which the image processing device 10 locally performs the process. For example, a server that performs the process using the learning models M1 and M2 may be provided. In this case, the image processing device 10 is configured to transmit the captured image to be processed to the server, and the front mask image generated from the captured image by the server using the first learning model M1 is transmitted to the image processing device 10. Also, the image processing device 10 is configured to transmit the captured image to be processed, the front mask image, and the correction instruction content to the server, and the output mask image generated by the server using the second learning model M2 is transmitted to the image processing device 10. Even in this case, the image processing device 10 can perform the background removal process of removing the background area from the captured image by using the output mask image generated by the server. Note that a server that performs the process using the first learning model M1 and a server that performs the process using the second learning model M2 may be provided separately. Also, a server that performs the process of synthesizing the foreground image with the background image for synthesis may be provided. In this case, the image processing device 10 is configured to transmit the foreground image to be subjected to the compositing process and the background image for compositing to the server, and transmit a composite image in which the foreground image is composited with the background image for compositing in the server to the image processing device 10. In this case, too, the image processing device 10 can provide the composite image generated by the server to the user by printing or transmitting it. In such a configuration, the image processing device 10 can be configured by a smartphone, a tablet terminal, a camera having a communication function, or the like. Even in the case of the above-mentioned configuration, the same processing as in this embodiment is possible, and the same effect can be obtained.
[0051] In this embodiment, a foreground image is generated from a photographed image and synthesized with a background image for synthesis to generate a synthetic image, but this configuration is not limited to this. For example, a foreground image generated by performing background removal processing on a photographed image may be provided to a user as is. In addition, the output mask image generated in this embodiment is not limited to a configuration used for processing to generate a foreground image by performing background removal on a photographed image. For example, the output mask image may be used to generate a segmentation mask (a label image for segmentation) used for learning a segmentation DNN (Deep Neural Network) that classifies each pixel in an input image into a region of a plurality of objects.
[0052] (Embodiment 2) In this embodiment, an image processing device is described that switches whether or not to execute morphology processing depending on the number of pixels in a difference area (area of white pixels) in a difference image (XOR image) between a front mask image and an inference mask image in a process of merging the front mask image and the inference mask image. The image processing device of this embodiment has a configuration similar to that of the image processing device 10 of the first embodiment shown in Fig. 1, so a description of the configuration will be omitted.
[0053] Fig. 13 is a flowchart showing an example of a merge processing procedure according to embodiment 2. The processing shown in Fig. 13 is obtained by adding steps S51 and S52 between steps S24 and S25 in the processing shown in Fig. 4. Explanations of the same steps as in Fig. 4 will be omitted.
[0054] In the image processing device 10 of this embodiment, the control unit 11 executes the same processes as steps S21 to S24 in FIG. 4. Then, the control unit 11 specifies a connected region (second connected region) in which adjacent pixels are sequentially connected with the same pixel value (same label) as the pixel value of the pixel at the correction position acquired in step S15 as a starting point in an exclusive OR (XOR) image (difference image, XOR image) between the front mask image after binarization processing and the inferred mask image after binarization processing calculated in step S24 (S51). Then, the control unit 11 counts the number of pixels in the specified connected region (the number of white pixels in the connected region) and judges whether the number of pixels in the connected region is greater than a threshold value T3 (S52). The threshold value T3 is, for example, preset and stored in the storage unit 12, but may be configured to be appropriately changeable.
[0055] If it is determined that the number of pixels in the connected region is greater than the threshold T3 (S52: YES), the control unit 11 proceeds to step S25. In this case, the control unit 11 executes the same process as in the first embodiment to generate an output mask image. If it is determined that the number of pixels in the connected region is equal to or less than the threshold T3 (S52: NO), the control unit 11 skips steps S25 to S26 and proceeds to step S27. In this case, the control unit 11 sets the connected region identified in the XOR image in step S51 as an update region (S27), and executes the processes from step S28 onwards.
[0056] By the above-mentioned process, in the difference image (XOR image) between the front mask image after the binarization process and the inference mask image after the binarization process, it is possible to switch whether or not to perform the morphology process depending on whether the number of pixels of the connected area based on the correction position is greater than the threshold T3. In the morphology process, since an unnecessary area is deleted from the image to be processed, by performing the morphology process, it is possible to prevent the update area to be merged into the front mask image from becoming unnecessarily large. However, when the original connected area (the connected area in the state before the morphology process) is small, the update area to be merged into the front mask image may become too small or disappear by performing the morphology process. Therefore, in this embodiment, by switching whether or not to perform the morphology process depending on the size (number of pixels) of the connected area in the XOR image before the morphology process, it is possible to prevent the update area to be merged into the front mask image from becoming unnecessarily large and to control it so that it does not become too small. Therefore, the threshold T3 may be set to the number of pixels that does not make the connected area too small when the morphology process is performed.
[0057] In this embodiment, the processes other than the above-mentioned processes are the same as those in the first embodiment, and the same effects as those in the first embodiment can be obtained. Furthermore, in this embodiment, when a connected region is identified after the morphological process, if the identified connected region becomes too small, the morphological process is not performed, and the connected region (update region) is identified from the XOR image before the morphological process, so that an update region of an appropriate size can be set as the update region to be merged into the previous mask image. The modified examples described in the first embodiment as appropriate can also be applied to this embodiment.
[0058] (Embodiment 3) In the above-mentioned second embodiment, whether or not to execute morphology processing is switched depending on the number of pixels of a connected region identified from an XOR image before morphology processing. In this embodiment, an image processing device is described that switches whether an update region to be merged into a previous mask image is a connected region in an XOR image after morphology processing or a connected region in an XOR image before morphology processing depending on whether a pixel at a correction position designated by a user in an XOR image after morphology processing is a white pixel or a black pixel. The image processing device of this embodiment has a configuration similar to that of the image processing device 10 of the first embodiment shown in FIG. 1, and therefore a description of the configuration is omitted.
[0059] Fig. 14 is a flowchart showing an example of a merge processing procedure according to the third embodiment. The processing shown in Fig. 14 is obtained by adding steps S61 and S62 between steps S25 and S26 in the processing shown in Fig. 4. Explanations of the same steps as in Fig. 4 will be omitted.
[0060] In the image processing device 10 of this embodiment, the control unit 11 executes the same processes as steps S21 to S25 in FIG. 4. Then, the control unit 11 judges whether or not the pixel at the correction position acquired in step S15 in the XOR image after the morphology processing is a white pixel (S61). If the control unit 11 judges that the pixel at the correction position is a white pixel (S61: YES), the control unit 11 proceeds to step S26. In this case, the control unit 11 executes the same processes as those in the first embodiment to generate an output mask image. On the other hand, if the control unit 11 judges that the pixel at the correction position is not a white pixel (S61: NO), that is, if the control unit 11 judges that the pixel is a black pixel, the control unit 11 specifies a connected region in the XOR image (image showing the result of the exclusive OR) before the morphology processing (S62). Step S62 is the same process as step S51 in FIG. 13. After that, the control unit 11 proceeds to step S27, sets the connected region specified in step S62 as the update region (S27), and executes the processes from step S28 onward.
[0061] By the above-mentioned process, depending on whether the pixel at the correction position in the XOR image after the morphology process is a white pixel or a black pixel, it is possible to switch between setting the connected region in the XOR image after the morphology process as the update region and setting the connected region in the XOR image before the morphology process as the update region. If the pixel at the correction position in the XOR image after the morphology process is a black pixel, it is considered that the connected region (update region) has become too small or disappeared due to the morphology process. Therefore, in this case, in this embodiment, the connected region in the XOR image before the morphology process is set as the update region, and the update region is merged into the previous mask image. On the other hand, if the pixel at the correction position in the XOR image after the morphology process is a white pixel, the connected region (update region) has not become too small due to the morphology process, so the connected region in the XOR image after the excess region is deleted by the morphology process is set as the update region to be merged into the previous mask image. This allows an update region of an appropriate size to be merged into the previous mask image.
[0062] In this embodiment, when the pixel at the correction position in the XOR image after the morphology processing is a black pixel, in addition to the configuration in which the merge processing is performed with the connected region in the XOR image before the morphology processing as the update region, the merge processing may not be performed. In this case, for example, when a foreground click is performed, the control unit 11 may display a message "The foreground region cannot be inferred at the specified location. Please click a different location" on the foreground / background correction screen shown in FIG. 8A. Also, when a background click is performed, the control unit 11 may display a message "The background region cannot be inferred at the specified location. Please click a different location." When such a message is displayed, the user performs a foreground click or background click on a different location on the foreground / background correction screen, and then operates the correction button to give a correction instruction for an arbitrary location (pixel), and operates the end button if there is no location to be corrected.
[0063] Therefore, if the merge process is not performed on the previous mask image in step S17 in FIG. 3, and it is determined in the subsequent step S14 that a correction instruction has not been accepted (S14: NO), the control unit 11 sets the most recently generated previous mask image as the output mask image in step S19 in FIG. 3.
[0064] In this embodiment, the processes other than the above-mentioned processes are the same as those in the first embodiment, and the same effects as those in the first embodiment can be obtained. Also in this embodiment, when the connected region becomes too small due to the morphological process, a suitable output mask image can be generated by merging the connected region (update region) in the XOR image before the morphological process into the previous mask image without performing the morphological process, or by not performing the merging process. The modified examples described in the first embodiment as appropriate can be applied to this embodiment as well.
[0065] (Embodiment 4) In the above-described first to third embodiments, the area designated by the user in the foreground is composited as a foreground image into a composite background image. In this embodiment, an image processing device that accepts the selection of a subject to be composited from a foreground image generated from a photographed image using an output mask image will be described. The image processing device of this embodiment has the same configuration as the image processing device 10 of the first embodiment shown in FIG. 1, and therefore a description of the configuration will be omitted.
[0066] Fig. 15 is a flowchart showing a modified example of the procedure for generating an output mask image, and Fig. 16 is an explanatory diagram showing an example screen. The process shown in Fig. 15 is obtained by adding steps S71 to S73 between steps S31 and S32 in the process shown in Fig. 9. Explanations of the same steps as in Fig. 9 will be omitted. Note that steps S34 to S41 in Fig. 9 are not shown in Fig. 15.
[0067] In the image processing device 10 of this embodiment, when the control unit 11 generates a foreground image from a captured image using the output mask image specified in step S19 in FIG. 3 as a mask image (S31), the control unit 11 displays the generated foreground image on the display unit 15 (S71). For example, the control unit 11 displays a screen as shown in FIG. 16 to present the foreground image to the user. The screen shown in FIG. 16 is configured to accept a selection of an arbitrary subject included in the foreground image via the foreground image being displayed. The user selects an arbitrary subject via the input unit 14 on the screen shown in FIG. 16 and operates the OK button to select the arbitrary subject. The control unit 11 determines whether or not the selection of the arbitrary subject has been accepted via the foreground image (S72). The example shown in FIG. 16 shows a state in which the selection of the left subject of two subjects in the foreground image has been accepted. Note that multiple subjects may be selected. Furthermore, when displaying the foreground image, the control unit 11 may be configured to detect each subject in the foreground image by performing an object detection process on the foreground image, for example, and present each detected subject as a bounding box. In this case, the user can easily specify an arbitrary subject by selecting one of the bounding boxes.
[0068] When the control unit 11 determines that the selection of the subject has not been accepted (S72: NO), it waits until it is accepted. When the control unit 11 determines that the selection of the subject has been accepted (S72: YES), it extracts the area of the selected subject from the foreground image to generate a subject image (S73). For example, the control unit 11 generates a subject image such as the foreground image shown in FIG. 10. Thereafter, the control unit 11 executes the processes from step S32 onwards. Note that the control unit 11 executes the subsequent processes with the subject image as the synthesis target. That is, on the screens of FIG. 10 and FIG. 11, the subject image is displayed instead of the foreground image, image processing is performed on the subject image in step S36, the subject image is moved in step S38, and the subject image is synthesized with the synthesis background image in step S40. As a result, in this embodiment, only the subject desired by the user can be extracted from the foreground image generated with high accuracy from the captured image and synthesized with the synthesis background image.
[0069] In the process shown in FIG. 15, a subject to be synthesized is selected from a foreground image generated from a photographed image to generate a subject image, but the present invention is not limited to this configuration. For example, after an arbitrary subject is selected from a photographed image, an output mask image may be generated for the area (subject image) of the selected subject, and a foreground image may be generated from the subject image using the output mask image. FIG. 17 is a flowchart showing a modified example of a synthetic image providing process procedure. The process shown in FIG. 17 is obtained by adding steps S81 to S83 between steps S11 and S12 in the process shown in FIG. 3. Explanation of the same steps as in FIG. 3 will be omitted. Note that steps S13 to S19 in FIG. 3 are omitted from FIG. 17.
[0070] When the control unit 11 of the image processing device 10 acquires a captured image (S11), it displays the acquired captured image on the display unit 15 (S81). Here, a screen having the same configuration as the screen in FIG. 16 and displaying the captured image is displayed on the display unit 15. The screen here is configured to accept a selection of an arbitrary subject included in the captured image being displayed, and the user selects an arbitrary subject in the captured image displayed on the screen and operates an OK button to select the arbitrary subject. The control unit 11 determines whether or not the selection of an arbitrary subject in the captured image has been accepted (S82). Here, multiple subjects may also be selected. In addition, the control unit 11 may be configured to detect each subject in the captured image by performing an object detection process on the captured image, for example, when displaying the captured image, and present each detected subject in a bounding box.
[0071] When the control unit 11 determines that the selection of the subject has not been accepted (S82: NO), it waits until it is accepted. When the control unit 11 determines that the selection of the subject has been accepted (S82: YES), it extracts the area of the selected subject from the captured image to generate a subject image (S83). Thereafter, the control unit 11 executes the processes from step S12 onward. The control unit 11 executes the subsequent processes with the subject image as the captured image to be processed. That is, in step S12, a front mask image is generated from the subject image, in step S16, an inference mask image is generated based on the subject image, the front mask image generated from the subject image, and the correction instruction content specified by the user, and in step S17, an output mask image is generated based on the front mask image generated from the subject image and the inference mask image. As a result, in the process shown in FIG. 17, a highly accurate output mask image that can extract the foreground area intended by the user from the subject image can be generated based on the subject image generated by extracting the area of the subject desired by the user from the captured image.
[0072] By the above-mentioned process, in this embodiment, the selection of a subject to be composited into any composite background image can be accepted via a foreground image or a photographed image. In addition, in the process shown in Fig. 17, when the area of the selected subject is extracted from the photographed image and an output mask image is generated for the extracted area (subject image), the data amount of the image to be processed can be reduced.
[0073] In this embodiment, the process of selecting a subject to be composited from a foreground image or a photographed image is manually performed by the user, but the present invention is not limited to this configuration. For example, a configuration may be adopted in which a subject in an image is automatically detected by performing an object detection process on a foreground image or a photographed image, and a subject to be composited is automatically identified from the detected subjects. In this embodiment, the modified examples described as appropriate in the above-mentioned embodiments 1 to 3 can also be applied. The configuration of this embodiment can be applied to the image processing device 10 of the above-mentioned embodiments 1 to 3, and the same effect can be obtained even when applied to the image processing device 10 of the above-mentioned embodiments 1 to 3.
[0074] (Embodiment 5) In the above-described first to fourth embodiments, the foreground image to be composited with the composite background image is a still image. In the present embodiment, an image processing device that composites a foreground image, which is a moving image, with a composite background image will be described. In the image processing device of this embodiment, an output mask image is generated for each still image included in the moving image, and a foreground image is generated using the generated output mask image, thereby generating a plurality of foreground images from which background regions have been removed, and a foreground image of the moving image is generated by stitching together the plurality of foreground images. The image processing device of this embodiment has the same configuration as the image processing device 10 of the first embodiment shown in FIG. 1, and therefore a description of the configuration will be omitted.
[0075] Fig. 18 is a flowchart showing an example of a synthetic image providing process procedure according to the fifth embodiment, and Fig. 19 is an explanatory diagram showing an example screen. The process shown in Fig. 18 is obtained by adding steps S91-S92 instead of step S11 and adding steps S93-S95 after step S19 in the process shown in Fig. 3. Explanations of the same steps as in Fig. 3 will be omitted.
[0076] In the image processing device 10 of this embodiment, the control unit 11 acquires a moving image (a photographed image including a plurality of still images) (S91). Here, too, the moving image to be processed may be read from the photographed image DB 12a, or may be acquired from a camera or another information processing device via the communication unit 13. The control unit 11 extracts one still image from the moving image to be processed (S92), and executes the processes of steps S12 to S19 on the extracted still image. Specifically, the control unit 11 generates a front mask image and an inference mask image based on the one still image, and merges the front mask image and the inference mask image to generate an output mask image. The control unit 11 uses the generated output mask image as a mask image to extract an area corresponding to a white area (foreground area) in the mask image from the still image (photographed image) extracted in step S92, to generate a foreground image, and stores the foreground image in the storage unit 12 (S93). As a result, one foreground image is generated from one still image included in the moving image.
[0077] Control unit 11 determines whether the foreground image generation process is completed for all still images included in the moving image acquired in step S91 (S94), and if it determines that the process is not completed (S94: NO), control unit 11 returns to the process of step S92. Then control unit 11 extracts one unprocessed still image from the acquired moving image (S92), and performs the processes of steps S12 to S19 and S93 on the extracted still image, and generates a foreground image from each still image.
[0078] When it is determined that the foreground image generation process has been completed for all still images (S94: YES), control unit 11 generates a foreground moving image by joining together foreground images generated from each still image included in the moving image (S95). As a result, a foreground moving image made up of foreground areas in still images included in the moving image is generated from the acquired moving image. Thereafter, control unit 11 executes steps S32 to S41 in FIG. 9. Note that in this embodiment, control unit 11 displays a setting screen as shown in FIG. 19 in step S34. The setting screen shown in FIG. 19 has a configuration similar to that of the screens shown in FIGS. 11A and 11B, and further displays indicator 15a indicating the total playback time and the playback position relative to the total playback time for the foreground moving image being displayed, and indicator 15b indicating the total playback time and the playback position relative to the total playback time for the compositing background image being displayed. By moving the playback position using indicators 15a and 15b, it is possible to change the playback positions of the foreground moving image and the compositing background image being displayed on the screen. 19 shows a state in which a moving image is selected as the background image for compositing, so indicator 15b is displayed on the background image for compositing, but if a still image is selected as the background image for compositing, indicator 15b is not displayed. Note that even if the foreground image to be composited is a moving image, the background image for compositing may be either a still image or a moving image.
[0079] The setting screen shown in FIG. 19 may have a configuration including an input field for the total playback time, an input field for the number of times to be repeatedly played, an input field for the playback speed, and the like, for the foreground moving image to be composited. In this case, it is possible to perform not only the scaling process and the rotation process but also the total playback time, the number of times to be played, and the playback speed for the foreground moving image, and any editing process can be performed on the foreground moving image. Therefore, in step S36, control unit 11 performs the scaling process and the rotation process, as well as the video editing process based on the input contents, on the foreground moving image being displayed on the setting screen. Furthermore, the setting screen shown in FIG. 19 may be configured to allow the user to specify an arbitrary playback position of the composite background image via indicator 15b, and to specify a composite position of the foreground moving image for the image (still image) at the specified playback position by a drag operation as shown in FIG. 11B. In this case, it is possible to composite the foreground moving image from an arbitrary playback position of the composite background image, which is a moving image, to an arbitrary composite position. Therefore, in step S40, control unit 11 can generate a moving image of a composite image by compositing the foreground moving image that has been subjected to image processing and video editing processing in step S36 from an arbitrary playback position of the compositing background image to a compositing position specified for each image (still image). Through the above-mentioned processing, a composite moving image is generated in which the foreground moving image is composited with a moving image of an arbitrary compositing background image.
[0080] In the present embodiment, the above-described process is performed to remove background from each still image included in a moving image of a subject, thereby generating a foreground image, thereby generating a foreground moving image from which the background area has been removed. Therefore, not only still images of the subject but also moving images can be used as the subject for synthesis, making it possible to generate a synthetic image with a higher degree of freedom. In the present embodiment, the modified examples described as appropriate in the above-described first to fourth embodiments can also be applied.
[0081] The configuration of this embodiment is applicable to the image processing device 10 of the above-mentioned embodiments 1 to 4, and the same effect can be obtained even when applied to the image processing device 10 of the embodiments 1 to 4. When the configuration of this embodiment is applied to the image processing device 10 of the embodiment 4, steps S71 to S73 in FIG. 15 may be performed after the process of step S93 or after the process of step S95 in the process shown in FIG. 18. In this case, the area of the subject desired by the user is extracted from the generated foreground image or from each still image included in the foreground moving image, and a subject moving image including only the desired subject can be generated. Furthermore, steps S81 to S83 in FIG. 17 may be performed between steps S92 and S12 in the process shown in FIG. 18. In this case, the area of the subject desired by the user is extracted from each still image extracted from the moving image, an output mask image is generated for the subject image including only the desired subject, and the generated output mask image is used to remove the background area of the subject image, thereby generating a highly accurate foreground moving image.
[0082] (Embodiment 6) A system configuration in the case where the image processing device 10 of the above-mentioned first to fifth embodiments is installed in various facilities will be described. FIG. 20 is an explanatory diagram showing a configuration example of an image processing system. The image processing system of this embodiment is installed in facilities such as amusement parks, theme parks, aquariums, zoos, botanical gardens, exhibitions, and tourist spots, and is used for images taken as commemorative photos. The image processing system of this embodiment has an image processing device 10, a camera 20 (imaging device), a touch panel 30, and a printer 40, and each device 10, 20, 30, and 40 may be configured to communicate via a network N, or may be configured to directly communicate via wired communication or wireless communication. The camera 20 is configured to perform a shooting process of acquiring one piece of image data (still image) in response to one shooting instruction, and a shooting process of acquiring, for example, 30 or 15 pieces of image data (video) per second. The camera 20 may be a camera installed in a facility and photographed by a photographer at the facility, or may be a camera used by a user who uses the facility. The camera used by the user may be a camera mounted on a smartphone, tablet terminal, or the like.
[0083] In the image processing system of this embodiment, the various screens that were displayed on the display unit 15 of the image processing device 10 in the above-mentioned first to fifth embodiments are displayed on the touch panel 30, and the various information that was input via the input unit 14 is input via the touch panel 30. Even with this configuration, the same processing as in the image processing systems of the above-mentioned first to fifth embodiments is possible, and the same effects can be obtained.
[0084] Regarding the above embodiments including the first to sixth embodiments, the following supplementary notes are further disclosed.
[0085] (Appendix 1) Acquire the captured image, generating a first mask image showing a foreground region in the acquired captured image; receiving a correction instruction for the foreground region or the background region in the first mask image; a learning model that has been trained to output a second mask image indicating the foreground region in the captured image inferred based on the input information when a captured image, a first mask image indicating a foreground region in the captured image, and information indicating the content of a correction instruction for the first mask image are input, the acquired captured image, the first mask image indicating the foreground region in the captured image, and information indicating the content of a correction instruction received for the first mask image are input to the learning model, and a second mask image indicating the foreground region in the captured image inferred based on the input information is obtained; generating a third mask image for output based on the first mask image, the second mask image, and information indicating the contents of a correction instruction received for the first mask image; A program that causes a computer to carry out processing.
[0086] (Appendix 2) performing a binarization process on the first mask image; performing a binarization process on the second mask image; generating a difference image between the binarized first mask image and the binarized second mask image; performing a correction process on the difference image; specifying a connected area in the difference image after the correction process, in which adjacent pixels having the same pixel value as the pixel for which the correction instruction has been received are sequentially connected to each other, starting from the pixel for which the correction instruction has been received; generating the third mask image based on the first mask image and the identified connected region; 2. The program according to claim 1, which causes the computer to execute the process.
[0087] (Appendix 3) when receiving an instruction to modify a background region in the first mask image to a foreground region, performing a binarization process on the first mask image using a threshold value based on pixel values of pixels in the foreground region in the first mask image; When an instruction to modify a foreground region in the first mask image to a background region is received, a binarization process is performed on the first mask image using a threshold based on pixel values of pixels in the background region in the first mask image. 3. The program according to claim 2, which causes the computer to execute the process.
[0088] (Appendix 4) when a correction instruction is received for a background region in the first mask image to a foreground region, the pixel values of the pixels in the connected region in the first mask image are updated to pixel values of pixels in the foreground region to generate the third mask image; When an instruction to modify the foreground region in the first mask image to a background region is received, the pixel values of the pixels in the connected region in the first mask image are updated to pixel values of the pixels in the background region to generate the third mask image. 4. The program according to claim 2 or 3, which causes the computer to execute the process.
[0089] (Appendix 5) A closing process of a morphological process is performed on the difference image, and then an opening process is performed to correct the difference image. 5. The program according to any one of claims 2 to 4, which causes the computer to execute the process.
[0090] (Appendix 6) accepting a selection of one of a superimposed image in which the first mask image is superimposed on the captured image, a foreground image in which a foreground region is extracted from the captured image based on the first mask image, and the first mask image; Display the selected image 6. The program according to any one of claims 1 to 5, for causing the computer to execute the process.
[0091] (Appendix 7) A correction instruction for the foreground region or the background region in the first mask image is received via the displayed image. 7. The program according to claim 6, which causes the computer to execute the process.
[0092] (Appendix 8) identifying a second connected region in the difference image by sequentially connecting adjacent pixels having the same pixel value as the pixel for which the correction instruction was received as a starting point; determining whether a number of pixels in the second connected region is greater than a predetermined value; When it is determined that the number of pixels in the second connected region is greater than a predetermined value, a correction process is performed on the difference image. 8. The program according to any one of claims 2 to 7, for causing the computer to execute the process.
[0093] (Appendix 9) determining whether or not the pixel for which the correction instruction has been received is a pixel within a difference region in the difference image after the correction process; If it is determined that the pixel is not within the difference region, the connected region in the difference image before the correction process is identified; The third mask image is generated based on the first mask image and the connected region identified from the difference image before the correction process. 9. The program according to any one of claims 2 to 8, which causes the computer to execute the process.
[0094] (Appendix 10) determining whether or not the pixel for which the correction instruction has been received is a pixel within a difference region in the difference image after the correction process; If it is determined that the pixel is not within the difference region, the first mask image is set as the third mask image. 10. The program according to any one of claims 2 to 9, which causes the computer to execute the process.
[0095] (Appendix 11) updating the first mask image, which indicates a foreground region in the captured image, to the third mask image; receiving a modification instruction for the foreground region or the background region in the updated first mask image; The captured image, the updated first mask image, and information indicating a correction instruction received for the updated first mask image are input to the learning model, and the second mask image inferred based on the input information is newly acquired; A new third mask image is generated based on the updated first mask image, the newly acquired second mask image, and information indicating the contents of a correction instruction received for the updated first mask image. 11. The program according to any one of claims 1 to 10, for causing a computer to execute the process.
[0096] (Appendix 12) A foreground region in the captured image is extracted based on the captured image and the third mask image to generate a foreground image. 12. The program according to any one of claims 1 to 11, for causing the computer to execute the process.
[0097] (Appendix 13) selecting at least one object included in the foreground image; Extracting a photographic area of the selected object from the foreground image. 13. The program according to claim 12, which causes the computer to execute the process.
[0098] (Appendix 14) Select one of the multiple images to be combined, The foreground image is composited with the selected composite image. 14. The program according to claim 12 or 13, which causes the computer to execute the process.
[0099] (Appendix 15) receiving input of an image processing to be performed on the foreground image when compositing the foreground image with the compositing image, and an input of a compositing position with respect to the compositing image; The foreground image is subjected to the received image processing, and the foreground image after the image processing is synthesized at the received synthesis position in the image to be synthesized. 15. The program according to claim 14, which causes the computer to execute the process.
[0100] (Appendix 16) acquiring training data in which a photographed image, a first mask image showing a foreground region in the photographed image, information showing a correction instruction for the first mask image, and a second mask image showing the foreground region in the photographed image reflecting the correction instruction are associated with each other; Using the acquired training data, a learning model is generated that outputs the second mask image reflecting the correction instruction when the captured image, the first mask image, and information indicating the content of the correction instruction are input. A model generation method in which processing is performed by a computer.
[0101] (Appendix 17) Acquire the captured image, generating a first mask image showing a foreground region in the acquired captured image; receiving a correction instruction for the foreground region or the background region in the first mask image; a learning model that has been trained to output a second mask image indicating the foreground region in the captured image inferred based on the input information when a captured image, a first mask image indicating a foreground region in the captured image, and information indicating the content of a correction instruction for the first mask image are input, the acquired captured image, the first mask image indicating the foreground region in the captured image, and information indicating the content of a correction instruction received for the first mask image are input to the learning model, and a second mask image indicating the foreground region in the captured image inferred based on the input information is obtained; generating a third mask image for output based on the first mask image, the second mask image, and information indicating the contents of a correction instruction received for the first mask image; An image processing method in which the processing is performed by a computer.
[0102] (Appendix 18) An image processing device having a control unit, The control unit is Acquire the captured image, generating a first mask image showing a foreground region in the acquired captured image; receiving a correction instruction for the foreground region or the background region in the first mask image; a learning model that has been trained to output a second mask image indicating the foreground region in the captured image inferred based on the input information when a captured image, a first mask image indicating a foreground region in the captured image, and information indicating the content of a correction instruction for the first mask image are input, the acquired captured image, the first mask image indicating the foreground region in the captured image, and information indicating the content of a correction instruction received for the first mask image are input to the learning model, and a second mask image indicating the foreground region in the captured image inferred based on the input information is obtained; generating a third mask image for output based on the first mask image, the second mask image, and information indicating the contents of a correction instruction received for the first mask image; Image processing device.
[0103] The embodiments disclosed herein are illustrative in all respects and should not be considered as limiting. The scope of the present invention is defined by the claims, not by the above meaning, and is intended to include all modifications within the scope and meaning equivalent to the claims. [Explanation of symbols]
[0104] 10 Image processing device 11 Control section 12 Storage section 13. Communications Department 14 Input section 15 Display M1 First learning model M2 Second Learning Model
Claims
1. Acquire the captured image, generating a first mask image indicating a foreground region in the acquired photographed image; receiving a correction instruction for the foreground region or the background region in the first mask image; a learning model that has been trained to output a second mask image indicating the foreground region in the captured image inferred based on the input information when a captured image, a first mask image indicating a foreground region in the captured image, and information indicating the content of a correction instruction for the first mask image are input, the acquired captured image, the first mask image indicating the foreground region in the captured image, and information indicating the content of a correction instruction received for the first mask image are input to the learning model, and a second mask image indicating the foreground region in the captured image inferred based on the input information is obtained; generating a third mask image for output based on the first mask image, the second mask image, and information indicating the contents of a correction instruction received for the first mask image; A program that causes a computer to carry out processing.
2. performing a binarization process on the first mask image; performing a binarization process on the second mask image; generating a difference image between the binarized first mask image and the binarized second mask image; performing a correction process on the difference image; specifying a connected area in the difference image after the correction process, in which adjacent pixels having the same pixel value as the pixel for which the correction instruction has been received are sequentially connected to each other, starting from the pixel for which the correction instruction has been received; generating the third mask image based on the first mask image and the identified connected region; The program according to claim 1 , which causes the computer to execute a process.
3. when receiving an instruction to modify a background region in the first mask image to a foreground region, binarizing the first mask image using a threshold based on pixel values of pixels in the foreground region in the first mask image; When an instruction to modify a foreground region in the first mask image to a background region is received, a binarization process is performed on the first mask image using a threshold based on pixel values of pixels in the background region in the first mask image. The program according to claim 2 , which causes the computer to execute a process.
4. when a correction instruction for changing a background region in the first mask image to a foreground region is received, pixel values of pixels in the connected region in the first mask image are updated to pixel values of pixels in the foreground region to generate the third mask image; When an instruction to modify the foreground region in the first mask image to a background region is received, the pixel values of the pixels in the connected region in the first mask image are updated to pixel values of the pixels in the background region to generate the third mask image.
4. The program according to claim 2 or 3, which causes the computer to execute a process.
5. A closing process of a morphological process is performed on the difference image, and then an opening process is performed to correct the difference image.
4. The program according to claim 2 or 3, which causes the computer to execute a process.
6. accepting a selection of one of a superimposed image in which the first mask image is superimposed on the captured image, a foreground image in which a foreground region is extracted from the captured image based on the first mask image, and the first mask image; Display the selected image 4. The program according to claim 1, which causes the computer to execute a process.
7. A correction instruction for the foreground region or the background region in the first mask image is received via the displayed image. The program according to claim 6, which causes the computer to execute a process.
8. specifying a second connected area in the difference image by sequentially connecting adjacent pixels having the same pixel value as the pixel for which the correction instruction has been received as a starting point; determining whether a number of pixels in the second connected region is greater than a predetermined value; When it is determined that the number of pixels in the second connected region is greater than a predetermined value, a correction process is performed on the difference image.
4. The program according to claim 2 or 3, which causes the computer to execute a process.
9. determining whether or not the pixel for which the correction instruction has been received is a pixel within a difference region in the difference image after the correction process; If it is determined that the pixel is not within the difference region, the connected region in the difference image before the correction process is identified; The third mask image is generated based on the first mask image and the connected region identified from the difference image before the correction process.
4. The program according to claim 2 or 3, which causes the computer to execute a process.
10. determining whether or not the pixel for which the correction instruction has been received is a pixel within a difference region in the difference image after the correction process; If it is determined that the pixel is not within the difference region, the first mask image is set as the third mask image.
4. The program according to claim 2 or 3, which causes the computer to execute a process.
11. updating the first mask image, which indicates a foreground region in the captured image, to the third mask image; receiving a modification instruction for the foreground region or the background region in the updated first mask image; The captured image, the updated first mask image, and information indicating a correction instruction received for the updated first mask image are input to the learning model, and the second mask image inferred based on the input information is newly acquired; A new third mask image is generated based on the updated first mask image, the newly acquired second mask image, and information indicating the contents of a correction instruction received for the updated first mask image.
4. The program according to claim 1, which causes the computer to execute a process.
12. A foreground region in the captured image is extracted based on the captured image and the third mask image to generate a foreground image.
4. The program according to claim 1, which causes the computer to execute a process.
13. Selecting at least one object included in the foreground image; Extracting a photographic area of the selected object from the foreground image. The program according to claim 12, which causes the computer to execute a process.
14. Select one of the multiple images to be combined, The foreground image is composited with the selected composite image. The program according to claim 12, which causes the computer to execute a process.
15. receiving input of an image processing to be performed on the foreground image when compositing the foreground image with the compositing image, and an input of a compositing position with respect to the compositing image; The foreground image is subjected to the received image processing, and the foreground image after the image processing is synthesized at the received synthesis position in the image to be synthesized. The program according to claim 14, which causes the computer to execute a process.
16. acquiring training data in which a photographed image, a first mask image showing a foreground region in the photographed image, information showing a correction instruction for the first mask image, and a second mask image showing the foreground region in the photographed image reflecting the correction instruction are associated with each other; Using the acquired training data, a learning model is generated that outputs the second mask image reflecting the correction instruction when the photographed image, the first mask image, and information indicating the content of the correction instruction are input. A model generation method in which processing is performed by a computer.
17. Acquire the captured image, generating a first mask image indicating a foreground region in the acquired photographed image; receiving a correction instruction for the foreground region or the background region in the first mask image; a learning model that has been trained to output a second mask image indicating the foreground region in the captured image inferred based on the input information when a captured image, a first mask image indicating a foreground region in the captured image, and information indicating the content of a correction instruction for the first mask image are input, the acquired captured image, the first mask image indicating the foreground region in the captured image, and information indicating the content of a correction instruction received for the first mask image are input to the learning model, and a second mask image indicating the foreground region in the captured image inferred based on the input information is obtained; generating a third mask image for output based on the first mask image, the second mask image, and information indicating the contents of a correction instruction received for the first mask image; An image processing method in which the processing is performed by a computer.
18. An image processing device having a control unit, The control unit is Acquire the captured image, generating a first mask image indicating a foreground region in the acquired photographed image; receiving a correction instruction for the foreground region or the background region in the first mask image; a learning model that has been trained to output a second mask image indicating the foreground region in the captured image inferred based on the input information when a captured image, a first mask image indicating a foreground region in the captured image, and information indicating the content of a correction instruction for the first mask image are input, the acquired captured image, the first mask image indicating the foreground region in the captured image, and information indicating the content of a correction instruction received for the first mask image are input to the learning model, and a second mask image indicating the foreground region in the captured image inferred based on the input information is obtained; generating a third mask image for output based on the first mask image, the second mask image, and information indicating the contents of a correction instruction received for the first mask image; Image processing device.
Citation Information
Patent Citations
Image processing device, image processing method, and image processing program
JP6676563B2
Cited By
Processing device, program, and robot system
WO2026160358A1
