Image processing device, image processing method, program, and print production system

The image processing device uses semantic segmentation to detect and extract the subject from a photographed image, eliminating the need for a screen and enabling composite image generation with a template, thus facilitating easy installation and image composition.

JP7593086B2Active Publication Date: 2024-12-03DAI NIPPON PRINTING CO LTD
View PDF 12 Cites 0 Cited by

Patent Information

Application Number
JP2020207721
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2020-12-15
Publication Date
2024-12-03
Estimated Expiration
2040-12-15

AI Technical Summary

Technical Problem

Existing chromakey composition methods require the installation of a green or blue screen behind the subject, which can be difficult in various shooting locations, especially indoors or outdoors.

Method used

An image processing device and method that detects the subject from a photographed image without a screen by using semantic segmentation to classify pixels as belonging to the person or background, allowing for the extraction of the human region and combining it with a template image.

Benefits of technology

Enables the detection of the subject from a photographed image without a screen, facilitating easy installation indoors or outdoors and generating a composite image by combining the subject with a template image.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007593086000001
    Figure 0007593086000001
  • Figure 0007593086000002
    Figure 0007593086000002
  • Figure 0007593086000003
    Figure 0007593086000003
Patent Text Reader

Abstract

To detect only a subject out of a photographic image without installing a screen behind the subject.SOLUTION: An image processing device comprises: a display processing unit which executes live viewing of successively displaying, on a display device, image signals acquired from a photographing device for photographing a user; a registration unit which registers as a subject a person performing a predetermined gesture, among persons displayed on the display device; an analysis unit which extracts the person registered by the registration unit, out of the photographic image generated by the photographing device; and an image processing unit which generates a synthetic image by synthesizing the image of the person extracted by the analysis unit and a template image.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to an image processing device, an image processing method, a program, and a print production system. [Background technology]

[0002] Chromakey compositing, which removes a specific color from a photographed image to make it transparent and then composites another image onto it, is used in various situations. For example, a service is known in which a person is photographed against a green or blue screen, the green or blue parts of the photographed image are made transparent, another image is composited, and a print of the composite image is provided. When using chromakey compositing, it is necessary to place a green or blue screen behind the person, which can be difficult to install depending on the shooting location. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6767541 [Patent Document 2] JP 2010-272958 A Summary of the Invention [Problem to be solved by the invention]

[0004] The present invention has been made in consideration of the above-described conventional situation, and an object of the present invention is to provide an image processing device, an image processing method, a program, and a print production system that are capable of detecting only a subject from a captured image without placing a screen behind the subject. [Means for solving the problem]

[0005] The image processing device of the present invention includes a display control unit that executes a live view in which image signals obtained from an imaging device that images a user are sequentially displayed on a display device; a registration unit that registers, as a subject, a person performing a predetermined gesture from among the people displayed on the display device; an analysis unit that extracts the person registered by the registration unit from a captured image generated by the imaging device; and an image processing unit that generates a composite image by combining the image of the person extracted by the analysis unit with a template image.

[0006] The image processing device of the present invention comprises a display control unit that executes a live view in which image signals obtained from an imaging device that images a user are sequentially displayed on a display device; a registration unit that registers, among faces displayed on the display device, faces that are equal to or larger than a predetermined size and that are detected for a predetermined period of time or longer as the face of a subject; an analysis unit that extracts a person whose face has been registered by the registration unit from a captured image generated by the imaging device; and an image processing unit that combines the image of the person extracted by the analysis unit with a template image to generate a composite image.

[0007] The image processing method of the present invention performs a live view in which image signals obtained from an imaging device that images a user are sequentially displayed on a display device, and among the people displayed on the display device, a person making a predetermined gesture is registered as a subject, the registered person is extracted from the captured image generated by the imaging device, and the image of the extracted person is combined with a template image to generate a composite image.

[0008] The program of the present invention causes a computer to execute the steps of: executing a live view in which image signals obtained from an imaging device that images a user are displayed sequentially on a display device; registering as a subject a person displayed on the display device who is making a predetermined gesture; extracting the registered person from an image captured by the imaging device; and generating a composite image by combining an image of the extracted person with a template image.

[0009] The print production system of the present invention comprises an imaging device that images a user, a display control unit that executes a live view in which image signals acquired from the imaging device are sequentially displayed on a display device, a registration unit that registers as a subject a person performing a predetermined gesture from among the people displayed on the display device, an analysis unit that extracts the person registered by the registration unit from the captured image generated by the imaging device, an image processing unit that generates a composite image by combining the image of the person extracted by the analysis unit with a template image, and a printer that prints and outputs the composite image on paper. Effect of the Invention

[0010] According to the present invention, it is possible to detect only the subject from a captured image without placing a screen behind the subject. [Brief description of the drawings]

[0011] [Figure 1] 1 is a schematic configuration diagram of a print producing system according to an embodiment of the present invention. [Diagram 2] 1A to 1C are diagrams illustrating an example of subject detection. [Diagram 3] FIG. 2 is a hardware configuration diagram of the print production system. [Figure 4] FIG. 2 is a functional block diagram of the image processing device. [Diagram 5] 1 is a flowchart illustrating a method for producing a print. [Figure 6] 13(a) to 13(c) are diagrams showing examples of subject registration screens. [Figure 7] 13(a) to 13(c) are diagrams showing examples of subject registration screens. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0013] 1 is a schematic diagram of a print production system according to an embodiment of the present invention. The print production system includes a print production device 1 and a server device 2. The print production device 1 and the server device 2 are connected to each other so as to be able to communicate with each other via a network such as the Internet.

[0014] The print producing device 1 includes a housing H and legs L that support the housing H. Inside the housing H, there are provided an arithmetic and control device 10 (see FIG. 3), an image capturing device 11 (image capturing unit), a touch panel 12, a printer 13 (see FIG. 3), a communication unit 14 (see FIG. 3), and the like.

[0015] The print production device 1 photographs a user (subject P), detects subject P from the photographed image, and generates a composite image by combining the detected subject image with a template image. A method such as semantic segmentation is used to detect the subject from the photographed image. For example, in semantic segmentation, all pixels in the photographed image are classified as belonging to a person or background (other than a person). Pixels classified as belonging to the background are made transparent, and the person area is extracted. Because it is possible to classify people and background, there is no need to install a green or blue screen on the back side as in chromakey compositing. Therefore, it is easy to install in an open space indoors or outdoors.

[0016] When the print producing device 1 is installed in an open space, surrounding people (non-subjects D) are also captured in the captured image when the subject P is photographed. Therefore, for example, when a person is detected from the captured image shown in Fig. 2(a), not only the subject P but also the non-subject D is detected as shown in Fig. 2(b).

[0017] Therefore, in this embodiment, a subject is requested to perform a predetermined gesture before shooting, and a person performing the gesture is registered as a subject. Then, by detecting only the registered people from the captured image, it becomes possible to detect only subject P, for example, as shown in FIG. 2(c).

[0018] The print producing device 1 generates a composite image by combining the detected image of the subject with a template image, prints the composite image on paper, and outputs the print 3. Thus, according to this embodiment, even if a user (subject) is photographed in a situation where a third party is present around the user, it is possible to detect only the user from the photographed image and generate a composite image by combining the user with the template image, allowing the user to enjoy the print 3.

[0019] After capturing an image, the user can operate the touch panel 12 to input his / her own email address. The print producing device 1 transmits data of the composite image together with the email address to the server device 2. The server device 2 uses the email address to attach the composite image to an email and transmits it. The user can also obtain the composite image data by receiving the email using a user terminal such as a smartphone or PC.

[0020] 3 is a diagram showing the hardware configuration of the print producing apparatus 1. The print producing apparatus 1 includes an arithmetic and control device 10, a photographing device 11, a touch panel 12, a printer 13, and a communication unit 14.

[0021] The photographing device 11 is, for example, a digital camera, and outputs an image signal to the arithmetic and control device 10.

[0022] The touch panel 12 displays operation buttons, and when an operation button is pressed, it accepts an operation instruction corresponding to the operation button. The touch panel 12 can also perform a live view in which image signals output from the image capture device 11 are sequentially displayed. The touch panel 12 may be of a resistive film type, a capacitive type, a surface acoustic wave type, an infrared type, an electromagnetic induction type, an image recognition type, an optical sensor type, or any other type, and is not particularly limited.

[0023] Instead of the touch panel 12, a display without a touch function and input means such as a keyboard or a mouse may be provided.

[0024] The printing method of the printer 13 is not particularly limited, and an inkjet method, a dye-sublimation type thermal transfer method, a melting type thermal transfer method, or the like can be used.

[0025] The communication unit 14 communicates with the server device 2 to transmit and receive data.

[0026] The arithmetic and control device 10 is a computer that controls each unit of the print producing device 1 and transfers data. As shown in Fig. 4, the CPU of the arithmetic and control device 10 executes an image processing program stored in the storage unit 100 to realize the functions of an analysis unit 101, a registration unit 102, an imaging control unit 103, a display processing unit 104, an image processing unit 105, a print control unit 106, an address information acquisition unit 107, and an image transmission unit 108, and the arithmetic and control device 10 operates as an image processing device.

[0027] The storage unit 100 is a hard disk or a semiconductor memory, and stores template images for synthesis in advance. The template images are characters, illustrations related to the installation location of the print producing apparatus 1, and the like.

[0028] 5 is a flowchart for explaining the print producing method according to the present embodiment. The processing of each part of the arithmetic and control device 10 will be explained with reference to this flowchart.

[0029] When a user operates the touch panel 12 to give a start instruction, the live view mode is executed (step S1). In the live view mode, the display processing unit 104 causes the touch panel 12 to sequentially display image signals acquired from the image capturing device 11.

[0030] The analysis unit 101 analyzes the image based on the image signal and extracts a person in the image (step S2). For example, all pixels in the image are classified as belonging to a person or background by semantic segmentation. Then, the pixels classified as belonging to the background are made transparent, so that the person area can be extracted.

[0031] The image analysis may be performed within the arithmetic and control device 10, or the image may be transmitted to an external server, the analysis may be performed on the external server, and the analysis results may be obtained from the external server.

[0032] The registration unit 102 causes the touch panel 12 to display a message instructing the user to make a predetermined gesture (step S3). For example, as shown in Fig. 6(a), the message instructs the user to make a gesture (pose) of raising both hands. In addition to displaying the message, a sound may be output.

[0033] The analysis unit 101 analyzes the gestures of people in the image and detects people who make a predetermined gesture, for example, a person who raises both hands. The gesture analysis can use a known method, and for example, the skeleton of a person can be detected by skeleton detection to perform gesture recognition.

[0034] The registration unit 102 registers the person who performed the predetermined gesture as a subject (step S4). As shown in FIG. 6(b), the color of the border of the person area of ​​the registered subject may be changed and displayed in live view. Alternatively, a semi-transparent image of a specific color, such as red, may be superimposed on the person area of ​​the registered subject. The user knows that he or she has been registered as a subject by the change in the color of the person displayed on the touch panel 12.

[0035] Once registered as a subject, even if you stop making the gesture, the registration will not be cancelled, as shown in Figure 6(c).

[0036] When a predetermined time has elapsed since the start of live view, or when the user operates the touch panel 12 to give an instruction to shoot, the shooting control unit 103 controls the shooting device 11 to perform shooting processing (step S5). For example, in the shooting processing, after shooting preparation operations such as AF (autofocus) processing, AE (auto exposure) processing, and AWB (auto white balance) processing, an image captured by the shooting device 11 is written to the storage unit 10 as an image file.

[0037] The analysis unit 101 tracks a person (person area) registered as a subject and extracts the subject from the captured image. The image processing unit 105 synthesizes the detected subject (the captured image in which areas other than the subject are made transparent) with a template image to generate a composite image (step S6).

[0038] The print control unit 105 transfers the composite image to the printer 13, controls the printer 13 to print the composite image on paper, and outputs a printout (step S7).

[0039] The print producing device 1 displays an email address input screen on the touch panel 12. The user operates the email address input screen to input an email address (step S8). The address information acquiring unit 107 acquires the email address information via the touch panel 12 and stores it in the storage unit 100. The email address may be inputted not only after the print is outputted, but also after the image is captured and before the print is outputted, or before the image is captured.

[0040] The image transmitting unit 108 transmits the composite image and the input email address information to the server device 2.

[0041] In this manner, according to this embodiment, a user is instructed to make a specific gesture in advance, and the person making the gesture is registered as a subject; and only the person registered as a subject can be detected from the captured image. Therefore, even if the print production device 1 is installed in an open space, a composite image can be generated by combining the user and a template image, and a print that will interest the user can be produced.

[0042] In the above embodiment, an example was described in which a composite image is attached to an email and sent using an input email address, but a card on which is printed the URL of a website from which the composite image data can be downloaded (and a two-dimensional code indicating the URL) may also be printed out from the print production device 1.

[0043] In the above embodiment, the process of registering a person who has made a predetermined gesture as a subject is performed before shooting, but it may be performed after shooting.

[0044] In the above embodiment, a person who makes a predetermined gesture is registered as a subject, but a facial image of the person who is the subject may be registered and the subject may be detected by face recognition. For example, as shown in Fig. 7(a), a user is instructed to approach the print producing device 1 and register a facial image. The analysis unit 101 detects a face in the image by face recognition processing.

[0045] When a face of a predetermined size or larger is detected for a predetermined period of time or longer, the registration unit 102 registers the face as a subject face. After registration, as shown in FIG. 7(b), the registration unit 102 displays on the touch panel 12 that registration is complete.

[0046] A weight sensor may be installed near the print producing apparatus 1, and face detection may be performed when the user stands on the weight sensor and the weight sensor detects a load equal to or greater than a predetermined value.

[0047] 7(c), a semi-transparent image of a specific color such as red may be superimposed on the person area in which the face is registered and displayed in live view. The user knows that his or her face has been registered because the color of his or her image displayed on the touch panel 12 changes. When registration of the faces of all users is complete, the user presses the capture button B1.

[0048] When the photographing button B1 is pressed, the photographing control unit 103 controls the photographing device 11 to perform photographing processing. The analysis unit 101 performs face authentication of people in the photographed image and detects subjects whose faces have been registered.

[0049] The present invention is not limited to the above-described embodiment, and the components can be modified and embodied in the implementation stage without departing from the gist of the invention. In addition, various inventions can be formed by appropriately combining the multiple components disclosed in the above-described embodiment. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, components from different embodiments may be appropriately combined. [Explanation of symbols]

[0050] 1 Printing equipment 2. Server equipment 3 Printed materials 10 Arithmetic and control device 11 Imaging Device 12 Touch Panel 13 Printers 100 Storage section 101 Analysis Department 102 Registration Department 103 Shooting control section 104 Display processing unit 105 Image processing section 106 Print control unit 107 Address information acquisition unit 108 Image transmission unit

Claims

1. a display control unit that executes a live view that sequentially displays image signals acquired from an image capturing device that captures an image of a user on a display device; a registration unit that registers, among the faces displayed on the display device, a face that is equal to or larger than a predetermined size and is detected for a predetermined period of time or longer as a face of a subject; an analysis unit that extracts a person having a face registered by the registration unit from a captured image generated by the photographing device; an image processing unit that generates a composite image by combining the image of the person extracted by the analysis unit with a template image; An image processing device comprising:

2. A live view is executed to sequentially display on a display device image signals acquired from an image capturing device that captures an image of a user; registering a face that is a predetermined size or larger and is detected for a predetermined time or longer among the faces displayed on the display device as a subject face; Extracting the person having the registered face from the captured image generated by the photographing device; The image processing method further comprises: synthesizing the extracted person image with a template image to generate a synthetic image.

3. A step of executing a live view in which image signals acquired from an image capturing device that captures an image of a user are sequentially displayed on a display device; registering a face that is detected for a predetermined time or longer and has a predetermined size or larger among the faces displayed on the display device as a subject face; extracting the person having the registered face from a captured image generated by the photographing device; generating a composite image by combining the extracted person image with a template image; A program that causes a computer to execute the following.

4. An imaging device for imaging a user; a display control unit that executes a live view that sequentially displays image signals acquired from the image capturing device on a display device; a registration unit that registers, among the faces displayed on the display device, a face that is equal to or larger than a predetermined size and is detected for a predetermined period of time or longer as a face of a subject; an analysis unit that extracts a person having a face registered by the registration unit from a captured image generated by the photographing device; an image processing unit that generates a composite image by combining the image of the person extracted by the analysis unit with a template image; a printer for printing and outputting the composite image on paper; A print production system comprising:

Citation Information

Patent Citations

  • Image generation apparatus, image generation method and program

    JP2005079776A

  • Imaging apparatus and method, and program

    JP2010087599A

  • Monitoring device, monitoring system, and monitoring program

    JP2010272958A

  • Image processor, image processing method and program

    JP2014168126A

  • Photograph seal creation apparatus, photograph seal creation method, and program

    JP2014170234A