Image processing device, image processing method, program, and print production system
The image processing device uses semantic segmentation to detect and extract the subject from a photographed image based on pixel classification and size, overcoming the need for a screen in chromakey composition, enabling easy installation and composite image generation.
Patent Information
- Application Number
- JP2020207719
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2020-12-15
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2040-12-15
AI Technical Summary
Existing chromakey composition methods require the installation of a green or blue screen behind the subject, which can be difficult in various shooting locations, especially indoors or outdoors.
An image processing device and method that detects and extracts the subject from a photographed image using semantic segmentation, determining the subject based on pixel classification and size, allowing for the generation of a composite image without the need for a screen.
Enables the detection of the subject from a photographed image without a screen, facilitating easy installation indoors or outdoors and generating a composite image by combining the subject with a template image.
Smart Images

Figure 0007593085000001 
Figure 0007593085000002 
Figure 0007593085000003
Abstract
Description
[Technical field]
[0001] The present invention relates to an image processing device, an image processing method, a program, and a print production system. [Background technology]
[0002] Chromakey compositing, which removes a specific color from a photographed image to make it transparent and then composites another image onto it, is used in various situations. For example, a service is known in which a person is photographed against a green or blue screen, the green or blue parts of the photographed image are made transparent, another image is composited, and a print of the composite image is provided. When using chromakey compositing, it is necessary to place a green or blue screen behind the person, which can be difficult to install depending on the shooting location. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] Patent No. 6767541 Summary of the Invention [Problem to be solved by the invention]
[0004] The present invention has been made in consideration of the above-described conventional situation, and an object of the present invention is to provide an image processing device, an image processing method, a program, and a print production system that are capable of detecting only a subject from a captured image without placing a screen behind the subject. [Means for solving the problem]
[0005] The image processing device of the present invention includes an analysis unit that extracts a person from a captured image of a user based on an analysis result of whether each pixel in the captured image belongs to a person, a determination unit that determines whether the person is a subject based on the size of the extracted person, and an image processing unit that combines an image of the person determined to be the subject with a template image to generate a composite image.
[0006] The image processing method of the present invention extracts a person from a captured image of a user based on an analysis result of whether each pixel in the captured image belongs to a person, determines whether the person is a subject based on the size of the extracted person, and generates a composite image by combining the image of the person determined to be the subject with a template image.
[0007] The program of the present invention causes a computer to execute the steps of: extracting a person from a captured image of a user based on an analysis result of whether each pixel in the image belongs to a person; determining whether the person is a subject based on the size of the extracted person; and combining the image of the person determined to be the subject with a template image to generate a composite image.
[0008] The print production system of the present invention comprises a photographing device that photographs a user and generates a photographed image, an analysis unit that extracts a person from the photographed image based on an analysis result of whether each pixel in the photographed image belongs to a person, a determination unit that determines whether the person is a subject or not based on the size of the extracted person, an image processing unit that generates a composite image by combining an image of the person determined to be the subject with a template image, and a printer that prints and outputs the composite image on paper. Effect of the Invention
[0009] According to the present invention, it is possible to detect only the subject from the captured image without placing a screen behind the subject. [Brief description of the drawings]
[0010] [Figure 1] 1 is a schematic configuration diagram of a print producing system according to an embodiment of the present invention. [Diagram 2] 1A to 1C are diagrams illustrating an example of subject detection. [Diagram 3] FIG. 2 is a hardware configuration diagram of the print production system. [Figure 4]FIG. 2 is a functional block diagram of the image processing device. [Diagram 5] 1 is a flowchart illustrating a method for producing a print. [Figure 6] 13(a) to 13(c) are diagrams showing examples of subject confirmation screens. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.
[0012] 1 is a schematic diagram of a print production system according to an embodiment of the present invention. The print production system includes a print production device 1 and a server device 2. The print production device 1 and the server device 2 are connected to each other so as to be able to communicate with each other via a network such as the Internet.
[0013] The print producing device 1 includes a housing H and legs L that support the housing H. Inside the housing H, there are provided an arithmetic and control device 10 (see FIG. 3), an image capturing device 11 (image capturing unit), a touch panel 12, a printer 13 (see FIG. 3), a communication unit 14 (see FIG. 3), and the like.
[0014] The print production device 1 photographs a user (subject P), detects subject P from the photographed image, and generates a composite image by combining the detected subject image with a template image. A method such as semantic segmentation is used to detect the subject from the photographed image. For example, in semantic segmentation, all pixels in the photographed image are classified as belonging to a person or background (other than a person). Pixels classified as belonging to the background are made transparent, and the person area is extracted. Because it is possible to classify people and background, there is no need to install a green or blue screen on the back side as in chromakey compositing. Therefore, it is easy to install in an open space indoors or outdoors.
[0015] When the print producing device 1 is installed in an open space, surrounding people (non-subjects D) are also captured in the captured image when the subject P is photographed. Therefore, for example, when a person is detected from the captured image shown in Fig. 2(a), not only the subject P but also the non-subject D is detected as shown in Fig. 2(b).
[0016] When taking a photograph, the subject is often closer to the photographing device than the non-subject, and is considered to be larger in size in the photographed image. Therefore, in this embodiment, the size (number of pixels) of each person detected from the photographed image is calculated, and a person whose size is equal to or larger than a predetermined value is determined to be a subject, and a person whose size is smaller than the predetermined value is determined to be a non-subject. A person determined to be a non-subject is regarded as the background and made transparent, and only the person determined to be a subject is detected. This makes it possible to detect only the subject P, for example, as shown in FIG. 2(c).
[0017] The print producing device 1 generates a composite image by combining the detected image of the subject with a template image, prints the composite image on paper, and outputs the print 3. Thus, according to this embodiment, even if a user (subject) is photographed in a situation where a third party is present around the user, it is possible to detect only the user from the photographed image and generate a composite image by combining the user with the template image, allowing the user to enjoy the print 3.
[0018] After capturing an image, the user can operate the touch panel 12 to input his / her own email address. The print producing device 1 transmits data of the composite image together with the email address to the server device 2. The server device 2 uses the email address to attach the composite image to an email and transmits it. The user can also obtain the composite image data by receiving the email using a user terminal such as a smartphone or PC.
[0019] 3 is a diagram showing the hardware configuration of the print producing apparatus 1. The print producing apparatus 1 includes an arithmetic and control device 10, a photographing device 11, a touch panel 12, a printer 13, and a communication unit 14.
[0020] The photographing device 11 is, for example, a digital camera, and outputs an image signal to the arithmetic and control device 10.
[0021] The touch panel 12 displays operation buttons, and when an operation button is pressed, it accepts an operation instruction corresponding to the operation button. The touch panel 12 can also perform a live view in which image signals output from the image capture device 11 are sequentially displayed. The touch panel 12 may be of any type, including a resistive film type, a capacitive type, a surface acoustic wave type, an infrared type, an electromagnetic induction type, an image recognition type, and an optical sensor type, and is not particularly limited thereto.
[0022] Instead of the touch panel 12, a display without a touch function and input means such as a keyboard or a mouse may be provided.
[0023] The printing method of the printer 13 is not particularly limited, and an inkjet method, a dye-sublimation type thermal transfer method, a melting type thermal transfer method, or the like can be used.
[0024] The communication unit 14 communicates with the server device 2 to transmit and receive data.
[0025] The arithmetic and control device 10 is a computer that controls each unit of the print producing device 1 and transfers data. As shown in Fig. 4, the CPU of the arithmetic and control device 10 executes an image processing program stored in the storage unit 100 to realize the functions of an analysis unit 101, a determination unit 102, an imaging control unit 103, a display processing unit 104, an image processing unit 105, a print control unit 106, an address information acquisition unit 107, and an image transmission unit 108, and the arithmetic and control device 10 operates as an image processing device.
[0026] The storage unit 100 is a hard disk or a semiconductor memory, and stores template images for synthesis in advance. The template images are characters, illustrations related to the installation location of the print producing apparatus 1, and the like.
[0027] 5 is a flowchart for explaining the print producing method according to the present embodiment. The processing of each part of the arithmetic and control device 10 will be explained with reference to this flowchart.
[0028] When a user operates the touch panel 12 to give a start instruction, the live view mode is executed (step S1). In the live view mode, the display processing unit 104 causes the touch panel 12 to sequentially display image signals acquired from the image capturing device 11.
[0029] The analysis unit 101 analyzes the image based on the image signal and extracts a person in the image (step S2). For example, all pixels in the image are classified as belonging to a person or background by semantic segmentation. Then, the pixels classified as belonging to the background are made transparent, so that the person area can be extracted.
[0030] The image analysis may be performed within the arithmetic and control device 10, or the image may be transmitted to an external server, the analysis may be performed on the external server, and the analysis results may be obtained from the external server.
[0031] The determination unit 102 determines whether the person is a subject or a non-subject based on the size of the person extracted in step S1 (the number of pixels in the person area) (step S3). A person whose size is equal to or larger than a predetermined value is determined to be a subject, and a person whose size is less than the predetermined value is determined to be a non-subject. Here, the size of a person may be the size of the entire body or the size of the face.
[0032] The display processing unit 104 displays an image in which the area other than the person area determined to be the subject is made transparent on the touch panel 12. The user can check whether the user has been determined to be the subject or whether another person has been determined to be the subject by looking at the display screen of the touch panel 12. When the user is not shown on the display screen (when the user has not been determined to be the subject), the print producing device 1 may display a message or output a sound urging the user to move closer to the print producing device 1.
[0033] When a predetermined time has elapsed since the start of live view, or when the user operates the touch panel 12 to give an instruction to shoot, the shooting control unit 103 controls the shooting device 11 to perform shooting processing (step S4). For example, in the shooting processing, after shooting preparation operations such as AF (autofocus) processing, AE (auto exposure) processing, and AWB (auto white balance) processing, an image captured by the shooting device 11 is written to the storage unit 10 as an image file.
[0034] The analysis unit 101 extracts a person from the captured image. The determination unit 102 detects a subject from the extracted person. The subject detected here is the user himself / herself, who has been confirmed in advance by the user. The image processing unit 105 synthesizes the detected subject (the captured image in which areas other than the subject have been made transparent) with a template image to generate a composite image (step S5).
[0035] The print control unit 105 transfers the composite image to the printer 13, controls the printer 13 to print the composite image on paper, and outputs a printout (step S6).
[0036] The print producing device 1 displays an email address input screen on the touch panel 12. The user operates the email address input screen to input an email address (step S7). The address information acquiring unit 107 acquires the email address information via the touch panel 12 and stores it in the storage unit 100. The email address may be inputted not only after the print is outputted, but also after the image is captured and before the print is outputted, or before the image is captured.
[0037] The image transmitting unit 108 transmits the composite image and the input email address information to the server device 2.
[0038] Thus, according to this embodiment, only the user who is the subject can be detected from the captured image. Therefore, even if the print production device 1 is installed in an open space, a composite image can be generated by combining the user with a template image, and a print that will interest the user can be produced.
[0039] In the above embodiment, an example was described in which a composite image is attached to an email and sent using an input email address, but a card on which is printed the URL of a website from which the composite image data can be downloaded (and a two-dimensional code indicating the URL) may also be printed out from the print production device 1.
[0040] In the above embodiment, a person who happens to pass close to the user at the timing of the photographing process may be detected as a subject. Therefore, the analysis unit 101 may perform face detection, calculate the time during which the face is detected during the live view for each person, and the determination unit 102 may determine whether or not the person is a subject based on the person's size and the time during which the face is detected. For example, a person whose size (number of pixels) in the image is equal to or greater than a predetermined value and whose face is detected for a predetermined time or longer is determined to be a subject. Even if the size of a person is equal to or greater than a predetermined value, if the time during which the face is detected is less than a predetermined value, the person is determined to be a non-subject, thereby preventing erroneous detection of a person who happens to pass close by.
[0041] After the photographing process (step S4 in FIG. 5), a screen for confirming the subject determined by the determination unit 102 may be displayed on the touch panel 12. For example, a subject confirmation screen such as that shown in FIG. 6(a) is displayed. If only the subject has been correctly detected, the approval button B1 is pressed. If a third party is captured, the subject selection button B2 is pressed to select the correct subject as shown in FIG. 6(b). The periphery of the person area of the selected subject may be colored red or the like to indicate that the subject has been selected. When the selection completion button B4 is pressed, the unselected people are erased as shown in FIG. 6(c), and an image showing only the selected subject is displayed.
[0042] If a third party is caught on the subject confirmation screen shown in FIG. 6(a), the re-shoot button B3 may be pressed to take another photograph.
[0043] The present invention is not limited to the above-described embodiment, and the components can be modified and embodied in the implementation stage without departing from the gist of the invention. In addition, various inventions can be formed by appropriately combining the multiple components disclosed in the above-described embodiment. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, components from different embodiments may be appropriately combined. [Explanation of symbols]
[0044] 1 Printing equipment 2. Server equipment 3 Printed materials 10 Arithmetic and control device 11 Imaging Device 12 Touch Panel 13 Printers 100 Storage section 101 Analysis Department 102 Judgment section 103 Shooting control section 104 Display processing unit 105 Image processing section 106 Print control unit 107 Address information acquisition unit 108 Image transmission unit
Claims
1. an analysis unit that extracts a person from a captured image of a user based on an analysis result of whether each pixel in the captured image belongs to a person; a determination unit that determines whether or not the person is a subject based on the size of the extracted person; an image processing unit that generates a composite image by combining an image of the person determined to be the subject with a template image; a display control unit that executes a live view that sequentially displays image signals acquired from an image capturing device that captures an image of the user on a display device; Equipped with the analysis unit calculates a time during which a face is detected during a live view for each person; The image processing device, wherein the determination unit determines that a person whose size is equal to or larger than a predetermined value and whose face has been detected for a period of time equal to or longer than a predetermined period of time is the subject.
2. displaying the person determined as the subject by the determination unit on the display device, and accepting a selection of a correct subject from among the displayed people; 2. The image processing apparatus according to claim 1, wherein the image processing section erases unselected persons and combines an image of a selected person with the template image.
3. extracting a person from a photographed image of a user based on an analysis result as to whether each pixel in the photographed image belongs to a person; determining whether the person is a subject based on the size of the extracted person; An image processing method for generating a composite image by combining an image of the person determined to be the subject with a template image, comprising: executing a live view in which image signals acquired from an image capturing device that captures an image of the user are sequentially displayed on a display device; For each person, the time that the face is detected during live view is calculated, An image processing method in which a person whose size is equal to or larger than a predetermined value and whose face has been detected for a period of time equal to or longer than a predetermined period of time is determined to be a subject.
4. extracting a person from a captured image of a user based on an analysis result of whether each pixel in the captured image belongs to a person; determining whether or not the person is a subject based on a size of the extracted person; A step of generating a composite image by combining an image of the person determined to be the subject with a template image; A program for causing a computer to execute the following: executing a live view in which image signals acquired from an image capturing device that captures an image of the user are sequentially displayed on a display device; For each person, the time that the face is detected during live view is calculated, The program controls the computer so as to determine, as a subject, a person whose size is equal to or larger than a predetermined value and whose face has been detected for a period of time equal to or longer than a predetermined period of time.
5. an image capturing device that captures an image of a user and generates a captured image; an analysis unit that extracts a person from the photographed image based on an analysis result of whether each pixel in the photographed image belongs to a person; a determination unit that determines whether or not the person is a subject based on the size of the extracted person; an image processing unit that generates a composite image by combining an image of the person determined to be the subject with a template image; a printer for printing and outputting the composite image on paper; a display control unit that executes a live view that sequentially displays image signals acquired from the image capturing device on a display device; Equipped with the analysis unit calculates a time during which a face is detected during a live view for each person; The determination unit determines that a person whose size is equal to or larger than a predetermined value and whose face has been detected for a period of time equal to or longer than a predetermined period of time is the subject.
Citation Information
Patent Citations
Image generation apparatus, image generation method and program
JP2005079776A
Imaging apparatus and method, and program
JP2010087599A
Image generating apparatus
JP2011172103A
Image processor, image processing method and program
JP2014168126A
Photograph seal creation apparatus, photograph seal creation method, and program
JP2014170234A
