Image processing device, operation method for image processing device, and operation program for image processing device

Abstract

This image processing device comprises a processor that acquires a synthesis template, acquires a captured image including a discretionary subject, performs image quality correction processing for reducing the difference between the image quality of the synthesis template and the image quality of the captured image by correcting the image quality of the captured image, and generates a synthesized image by synthesizing the synthesis template with the captured image that has been subjected to image quality correction processing.

Description

Image processing device, operation method of image processing device, and operation program of image processing device

[0001] The technology of the present disclosure relates to an image processing device, an operating method for an image processing device, and an operating program for an image processing device.

[0002] "Introduction to the FUJIFILM Japan INSTAX Biz Implementation Case [Toyota Alvark Tokyo Co., Ltd.] / Fujifilm <Internet URL: https: / / www.youtube.com / watch?v=JNZU1Jkl7lY> 2023 / 06 / 26" discloses a service that generates a composite image by combining images taken by fans visiting a basketball game venue with pre-prepared compositing templates that include portrait images of basketball players, etc., and then hands the composite image printed on instant film to the fan on the spot.

[0003] International Publication No. 2004 / 057531 describes an image synthesis device that generates a composite image from a first image and a second image. The first image and the second image have an overlapping common image. The image synthesis device includes a comparison unit that compares the brightness of a first common image, which is the common image of the first image, with the brightness of a second common image, which is the common image of the second image. The image synthesis device includes an adjustment unit that adjusts the brightness of at least a portion of the second image to the brightness of the first image based on the comparison result of the comparison unit. A composite image generation unit generates a composite image including the first image and the second image, the brightnesses of which have been adjusted by the adjustment unit.

[0004] One embodiment of the technology disclosed herein provides an image processing device, an operating method for the image processing device, and an operating program for the image processing device that can reduce the sense of incongruity in a composite image of a synthesis template and a captured image including an arbitrary subject.

[0005] The image processing device of the present disclosure includes a processor, which acquires a synthesis template, acquires a captured image including an arbitrary subject, and performs image quality correction processing to reduce the difference between the image quality of the synthesis template and the image quality of the captured image by correcting the image quality of the captured image, and generates a composite image by combining the captured image that has undergone the image quality correction processing with the synthesis template.

[0006] It is preferable that the processor identify a first region relating to an arbitrary subject from the photographed image, derive a first index value for the first region, and perform image quality correction processing according to the first index value.

[0007] The first region is preferably a region of interest.

[0008] The region of interest is preferably a human face region.

[0009] The first index value is preferably derived by a statistical method.

[0010] The first index value is preferably derived after a process of removing outliers is performed.

[0011] It is preferable that the image processing device has correction reference information in which a correction amount corresponding to the first index value is registered, and the processor obtains the correction amount corresponding to the derived first index value from the correction reference information and performs image quality correction processing using the obtained correction amount.

[0012] It is preferable that the processor accepts an instruction to correct the correction amount of the correction reference information, and corrects the correction amount of the correction reference information in accordance with the instruction to correct.

[0013] It is preferable that the first index value of the correction reference information and the correction amount have a nonlinear relationship.

[0014] It is preferable that the correction reference information registers correction amounts whose absolute values ​​gradually increase as the first index value approaches the maximum or minimum value.

[0015] It is preferable that the processor extracts a first region relating to an arbitrary subject from the captured image, derives a first index value for the first region, and performs image quality correction processing according to the first index value and a second index value for a second region relating to a specific subject in the synthesis template.

[0016] The second region is preferably a region related to the first region.

[0017] The first and second regions are preferably human face regions.

[0018] The first index value and the second index value are preferably derived by a statistical method.

[0019] The first index value and the second index value are preferably derived after a process of removing outliers is performed.

[0020] As an image quality correction process, the processor performs a lighting correction process that brings the lighting of an arbitrary subject closer to the lighting of a specific subject, and in the lighting correction process, it is preferable to obtain the direction and intensity of the light source of light irradiated onto an arbitrary subject in a first region as a first index value, and the direction and intensity of the light source of light irradiated onto a specific subject in a second region as a second index value.

[0021] It is preferable that the processor performs at least one of the following image quality correction processes: brightness correction process for bringing the brightness of an arbitrary subject closer to the brightness of a specific subject in the synthesis template; saturation correction process for bringing the saturation of an arbitrary subject closer to the saturation of a specific subject; lighting correction process for bringing the lighting conditions of an arbitrary subject closer to the lighting conditions of a specific subject; noise amount correction process for bringing the amount of noise on an arbitrary subject closer to the amount of noise on a specific subject; and resolution correction process for bringing the resolution of an arbitrary subject closer to the resolution of a specific subject.

[0022] The processor performs brightness correction processing, saturation correction processing, and lighting correction processing, and it is preferable to perform lighting correction processing after performing brightness correction processing and saturation correction processing.

[0023] The processor performs brightness correction processing, saturation correction processing, and resolution correction processing, and it is preferable that the processor performs brightness correction processing and saturation correction processing after performing resolution correction processing.

[0024] It is preferable that a limit value be set for the amount of correction in the image quality correction process.

[0025] It is preferable that an upper limit and a lower limit are set for the correction amount.

[0026] Preferably, the processor acquires depth information indicating the front-to-back relationship between an arbitrary subject and a specific subject in the synthesis template, and generates a synthetic image in which the arbitrary subject and the specific subject have a front-to-back relationship in accordance with the depth information.

[0027] The operating method of the image processing device of the present disclosure includes obtaining a synthesis template, obtaining a captured image including an arbitrary subject, performing an image quality correction process to reduce the difference between the image quality of the synthesis template and the image quality of the captured image by correcting the image quality of the captured image, and synthesizing the captured image that has undergone the image quality correction process with the synthesis template to generate a composite image.

[0028] The operating program of the image processing device disclosed herein causes a computer to perform processes including acquiring a synthesis template, acquiring a captured image including an arbitrary subject, performing image quality correction processing to reduce the difference between the image quality of the synthesis template and the image quality of the captured image by correcting the image quality of the captured image, and synthesizing the captured image that has undergone the image quality correction processing with the synthesis template to generate a composite image.

[0029] 1 is a diagram showing a photography and printing system and a scene in which a photography and printing service using the photography and printing system is implemented. FIG. 2 is a diagram showing a compositing template. FIG. 3 is a diagram showing an overview of processing by an image processing server. FIG. 4 is a block diagram showing a computer constituting the image processing server. FIG. 5 is a block diagram showing a processing unit of a CPU of the image processing server. FIG. 6 is a block diagram showing a detailed configuration of an image quality correction processing unit. FIG. 7 is a diagram showing processing by a first index value derivation unit. FIG. 8 is a diagram showing processing by a brightness correction processing unit. FIG. 9 is a diagram showing processing by a saturation correction processing unit. FIG. 10 is a graph showing brightness correction reference information. FIG. 11 is a flowchart showing a processing procedure of the image processing server. FIG. 12 is a flowchart showing a processing procedure of the image processing server. FIG. 13 is a flowchart showing a processing procedure of the image processing server. FIG. 14 is a diagram showing a mode in which an instruction to modify the brightness correction amount of brightness correction reference information is received and the brightness correction amount is modified in accordance with the modification instruction. FIG. 15 is a diagram showing a second brightness index value and a second saturation index value stored in association with a compositing template. FIG. 16 is a diagram showing processing when a second brightness index value is derived. FIG. 17 is a diagram showing a second embodiment in which brightness correction processing is performed according to a first brightness index value and a second brightness index value. FIG. 18 is a diagram showing a second embodiment in which saturation correction processing is performed according to a first saturation index value and a second saturation index value. FIG. 19 is a diagram showing a third embodiment in which lighting correction processing is performed according to a first lighting index value and a second lighting index value. FIG. 10 is a diagram showing processing by a lighting correction processing unit; FIG. 11 is a diagram showing the order in which brightness correction processing, saturation correction processing, and lighting correction processing are performed; FIG. 12 is a diagram showing a fourth embodiment in which noise amount correction processing is performed; FIG. 13 is a diagram showing a fifth embodiment in which resolution correction processing is performed; FIG. 14 is a diagram showing the order in which brightness correction processing, saturation correction processing, and resolution correction processing are performed; and FIG. 15 is a diagram showing a sixth embodiment in which a composite image is generated with a posterior relationship according to depth information.

[0030] 1 , a photographing and printing system 2 includes a photographing device 10, an image processing server 11, and an instant printer 12. The photographing and printing system 2 is a system for providing a service in which the image processing server 11 generates a composite image 15 by compositing a photographed image 14 obtained by photographing a customer C with the photographing device 10 onto a compositing template 13, prints the composite image 15 on an instant film 16 with the instant printer 12, and hands the instant film 16 to the customer C on the spot.

[0031] Customer C is a basketball team fan visiting an event venue, in this case a basketball game venue. A photo booth 20 is set up at the game venue. A flag 21 with the name of the basketball team written on it is attached to the wall of the photo booth 20. Customer C stands in a designated spot in front of the flag 21 and strikes a pose of his or her choice. Staff S stationed at the photo booth 20 photographs such customer C with the photographing device 10. Customer C is an example of an "arbitrary subject" and a "person" according to the technology of the present disclosure. In the following, an arbitrary subject will be referred to as an arbitrary subject.

[0032] The compositing template 13 is prepared in advance by the organizer of the event, in this case the organizer of a basketball game, for example, the owner company of a basketball team. As an example, as shown in FIG. 2 , the compositing template 13 includes a portrait image 30 of a basketball player P belonging to the basketball team. Here, a "portrait image" is an image taken with the subject (here, basketball player P) in mind that he or she will be photographed, and includes at least the face of the subject. Basketball player P is an example of a "specific subject" and a "person" according to the technology of the present disclosure. Note that, hereinafter, a specific subject will be referred to as a specific subject.

[0033] The compositing template 13 also includes auxiliary elements 31. The auxiliary elements 31 may be, for example, a basketball team logo, text describing the basketball player P's name, height, weight, alma mater, or two-dimensional codes (here, QR (Quick Response) Code (registered trademark)) representing links to bonus videos of the basketball team or basketball player P. The compositing template 13 is configured such that the portrait image 30 and auxiliary elements 31 are appropriately arranged, leaving a blank space in which customer C can be placed. A composite image 15 of the compositing template 13 configured in this manner and the captured image 14 appears as if the basketball player P is standing next to customer C (see also FIG. 3 ). The portrait image 30 and auxiliary elements 31, and thus the compositing template 13, are RGB (red, green, blue) color images. Similarly, the captured image 14 is also an RGB color image.

[0034] The photographing device 10 has a camera unit (not shown) configured with an imaging element such as a lens and a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor, a display 23, etc. Here, a tablet terminal is illustrated as an example of the photographing device 10. Note that the photographing device 10 may be any device that has a camera unit and a display, such as a digital single-lens reflex camera, a digital still camera, a smartphone, or a notebook personal computer.

[0035] A shooting screen 25 is displayed on the display 23. The shooting screen 25 displays the compositing template 13 and a live view image 26 of the real space, including the shooting booth 20 and customer C, currently captured by the camera unit of the photographing device 10, superimposed on each other. A release button 27 is also provided on the shooting screen 25. While viewing the shooting screen 25, the staff member S adjusts the composition, such as the relative positions of the customer C and the basketball player P. Then, when an appropriate composition is achieved, the staff member S operates the release button 27. This causes the photographed image 14 to be taken.

[0036] The image processing server 11 is, for example, a server computer, a workstation, etc. The image processing server 11 is an example of an "image processing device" according to the technology of the present disclosure.

[0037] The photographing devices 10 and the image processing server 11 are connected to each other so that they can communicate with each other. Similarly, the image processing server 11 and the instant printer 12 are also connected to each other so that they can communicate with each other. The photographing devices 10 and the image processing server 11, and the image processing server 11 and the instant printer 12, are connected via a WAN (Wide Area Network) such as the Internet or a public communication network, for example. Alternatively, they are connected via short-range wireless communication such as Bluetooth (registered trademark). Note that a plurality of photographing devices 10 and a plurality of instant printers 12 may be connected to the image processing server 11.

[0038] The photographing device 10 transmits a photographed image 14 to the image processing server 11. The image processing server 11 distributes screen data of various screens, such as a photographing screen 25, to the photographing device 10. The image processing server 11 distributes the screen data to the photographing device 10 in the form of screen data for web distribution created using a markup language such as XML (Extensible Markup Language). The photographing device 10 reproduces various screens based on the screen data and displays them on the display 23. Note that other data description languages, such as JSON (Javascript (registered trademark) Object Notation), may be used instead of XML. The image processing server 11 transmits a composite image 15 to the instant printer 12.

[0039] As an example, as shown in FIG. 3 , the image processing server 11 performs image quality correction processing on the photographed image 14. The image quality correction processing is processing for reducing the difference between the image quality of the compositing template 13 and the image quality of the photographed image 14. More specifically, the image quality correction processing is processing for reducing the difference between the image quality of the basketball player P in the compositing template 13 and the image quality of the customer C in the photographed image 14. Here, "reduction" also includes eliminating the difference between the image quality of the compositing template 13 and the image quality of the photographed image 14. The image processing server 11 generates a composite image 15 by combining the compositing template 13 with a corrected photographed image 14AC, which is the photographed image 14 after the image quality correction processing.

[0040] 4, the computer constituting the image processing server 11 includes a storage 35, a memory 36, a CPU (Central Processing Unit) 37, a communication unit 38, a display 39, and an input device 40. These are interconnected via a bus line 41.

[0041] The storage 35 is a hard disk drive built into a computer constituting the image processing server 11 or connected via a cable or network. Alternatively, the storage 35 is a disk array consisting of multiple hard disk drives. The storage 35 stores control programs such as an operating system, various application programs, and various data associated with these programs. Note that a solid state drive may be used instead of a hard disk drive.

[0042] The memory 36 is a work memory for the CPU 37 to execute processing. The CPU 37 loads programs stored in the storage 35 into the memory 36 and executes processing in accordance with the programs. In this way, the CPU 37 comprehensively controls each part of the computer. The CPU 37 is an example of a "processor" according to the technology of the present disclosure. The memory 36 may be built into the CPU 37.

[0043] The communication unit 38 is a network interface that controls the transmission of various information between external devices such as the photographing device 10 and the instant printer 12. The display 39 displays various screens. Each screen has an operation function using a GUI (Graphical User Interface). The computer that constitutes the image processing server 11 accepts operation instructions input from an input device 40 via each screen. The input device 40 is a keyboard, a mouse, a touch panel, a microphone for voice input, etc.

[0044] 5, an operating program 45 is stored in the storage 35 of the image processing server 11. The operating program 45 is an application program for causing the computer constituting the image processing server 11 to function as an "image processing device" according to the technology of the present disclosure. In other words, the operating program 45 is an example of an "operating program for an image processing device" according to the technology of the present disclosure.

[0045] The storage 35 also stores a synthesis template database (hereinafter referred to as DB (Data Base)) 46, correction reference information 47, and a synthesis image DB 48. The synthesis template DB 46 stores, for example, a plurality of synthesis templates 13 for all basketball players P belonging to a basketball team. The synthesis image DB 48 stores the generated synthesis image 15.

[0046] When the operating program 45 is started, the CPU 37 of the image processing server 11 works in cooperation with the memory 36 and the like to function as a request receiving unit 50, a read / write (hereinafter referred to as RW (Read Write)) control unit 51, a screen distribution control unit 52, an image quality correction processing unit 53, a composite image generation unit 54, and a transmission unit 55.

[0047] The request receiving unit 50 receives various requests from the photographing device 10. The various requests include a request to distribute the photographing screen 25, a request to generate a composite image 15, and a request to print the composite image 15. The distribution request includes identification information of the compositing template 13 selected by the customer C to be used for photographing from among multiple compositing templates 13. The generation request includes the photographed image 14. When a distribution request is received, the request receiving unit 50 outputs the identification information of the compositing template 13 included in the distribution request to the RW control unit 51. When a generation request is received, the request receiving unit 50 outputs the photographed image 14 included in the generation request to the RW control unit 51. When a print request is received, the request receiving unit 50 outputs a message to the transmitting unit 55 indicating that the print request has been received.

[0048] The RW control unit 51 controls the storage of various data in the storage 35 and the reading of various data from the storage 35. Specifically, when the RW control unit 51 receives identification information of a compositing template 13 included in a distribution request from the request receiving unit 50, the RW control unit 51 accesses the compositing template DB 46 and reads out the compositing template 13 corresponding to the identification information. The RW control unit 51 then outputs the read out compositing template 13 to the screen distribution control unit 52 and the composite image generation unit 54.

[0049] The RW control unit 51 stores the captured image 14 input from the request receiving unit 50 in the storage 35. The RW control unit 51 reads the captured image 14 from the storage 35 and outputs the read captured image 14 to the image quality correction processing unit 53. The RW control unit 51 also reads the correction reference information 47 from the storage 35 and outputs the read correction reference information 47 to the image quality correction processing unit 53. Furthermore, the RW control unit 51 stores the composite image 15 input from the composite image generation unit 54 in the composite image DB 48.

[0050] The screen delivery control unit 52 generates screen data for various screens. For example, the screen delivery control unit 52 generates screen data for the photographing screen 25 including the synthesis template 13 from the RW control unit 51. The screen delivery control unit 52 controls the delivery output of the generated screen data to the photographing device 10.

[0051] The various screens include the photographing screen 25, a list screen of multiple compositing templates 13, and a confirmation screen for the composite image 15. The list screen is shown to the customer C by the staff member S before the photographed image 14 is taken. The customer C then selects a compositing template 13 including a portrait image 30 of the desired basketball player P from the multiple compositing templates 13 displayed in the list. When the desired compositing template 13 is selected on this list screen, the photographing device 10 transmits a delivery request for the photographing screen 25 to the image processing server 11. The confirmation screen is shown to the customer C by the staff member S after the photographed image 14 is taken. The customer C then confirms the finished composite image 15. When the customer C agrees with the finished composite image 15 on this confirmation screen, the photographing device 10 transmits a print request for the composite image 15 to the image processing server 11.

[0052] The image quality correction processing unit 53 performs image quality correction processing on the photographed image 14 to generate a corrected photographed image 14AC. The image quality correction processing unit 53 outputs the corrected photographed image 14AC to the composite image generation unit 54.

[0053] The composite image generation unit 54 generates a composite image 15 by combining the corrected captured image 14AC with the composition template 13. The composite image generation unit 54 outputs the composite image 15 to the RW control unit 51, the screen distribution control unit 52, and the transmission unit 55. When the transmission unit 55 receives an input from the request reception unit 50 indicating that a print request has been received, the transmission unit 55 transmits the composite image 15 to the instant printer 12.

[0054] 6 , the image quality correction processing unit 53 includes a Lab conversion unit 60, a first region identification unit 61, a first index value derivation unit 62, a brightness correction processing unit 63, a saturation correction processing unit 64, and an RGB conversion unit 65. The captured image 14 from the RW control unit 51 is input to the Lab conversion unit 60. The Lab conversion unit 60 converts the captured image 14, which is an RGB color image, into a Lab image 68. The Lab conversion unit 60 outputs the Lab image 68 to the first region identification unit 61 and the brightness correction processing unit 63.

[0055] The captured image 14 from the RW control unit 51 is also input to the first area identification unit 61. The first area identification unit 61 identifies a first face area FA1 that rectangularly surrounds the face of customer C in the captured image 14, for example, using a well-known face recognition technology such as a trained model for face recognition. The first face area FA1 is an example of the "first area," "area of ​​interest," and "person's face area" according to the technology disclosed herein. Here, the area of ​​interest is an area that attracts the attention (gaze) of a person viewing an image. It is known that when an image containing a person's face is viewed, attention is drawn to the person's face. For this reason, it is appropriate to set the first face area FA1 as the area of ​​interest.

[0056] The first area identification unit 61 outputs a first identification result 69 of the first face area FA1 to the first index value derivation unit 62. The first identification result 69 is a set of pixel values ​​of the pixels constituting the first face area FA1 among the pixels of the Lab image 68. Note that the first identification result 69 may exclude areas other than skin-colored areas such as hair, eyebrows, beard, eyes, nostrils, mouth, and even areas in shadow.

[0057] The first index value derivation unit 62 derives a first brightness index value 70 and a first saturation index value 71 based on the first identification result 69. The first brightness index value 70 is an index value for the brightness of the first face area FA1. The first saturation index value 71 is an index value for the saturation of the first face area FA1. The first index value derivation unit 62 outputs the first brightness index value 70 to the brightness correction processing unit 63 and the first saturation index value 71 to the saturation correction processing unit 64. The first brightness index value 70 and the first saturation index value 71 are examples of "first index values" according to the technology of the present disclosure.

[0058] The brightness correction processing unit 63 receives brightness correction reference information 47A from the correction reference information 47. The brightness correction processing unit 63 performs brightness correction processing on the Lab image 68 in accordance with the brightness correction reference information 47A and the first brightness index value 70. The brightness correction processing is an image quality correction processing that brings the brightness of the face of customer C in the photographed image 14 closer to the brightness of the face of basketball player P in the synthesis template 13. Here, "bringing closer" also includes bringing the brightness of the face of customer C in the photographed image 14 closer to the brightness of the face of basketball player P in the synthesis template 13. The same applies to the various image quality correction processing that follows. The brightness correction processing unit 63 outputs a brightness-corrected image 68ABC, which is the Lab image 68 that has undergone brightness correction processing, to the saturation correction processing unit 64. Note that the brightness correction processing may be performed limitedly to the first face area FA1, or may be performed on the entire photographed image 14.

[0059] The saturation correction processing unit 64 receives saturation correction reference information 47B from the correction reference information 47. The saturation correction processing unit 64 performs saturation correction processing on the brightness-corrected image 68ABC in accordance with the saturation correction reference information 47B and the first saturation index value 71. The saturation correction processing is an image quality correction processing that brings the saturation of the face of customer C in the photographed image 14 closer to the saturation of the face of basketball player P in the synthesis template 13. The saturation correction processing unit 64 outputs a saturation-corrected image 68ASC, which is the brightness-corrected image 68ABC that has undergone the saturation correction processing, to the RGB conversion unit 65. Note that, like the brightness correction processing, the saturation correction processing may be performed limitedly on the first face area FA1, or may be performed on the entire photographed image 14.

[0060] The RGB conversion unit 65 converts the saturation-corrected image 68ASC, which is a Lab image, into the corrected captured image 14AC, which is an RGB color image. The RGB conversion unit 65 outputs the corrected captured image 14AC to the composite image generation unit .

[0061] As an example, as shown in FIG. 7 , the first index value derivation unit 62 generates first brightness data 75 by extracting brightness (L value) from the first identification result 69. The first index value derivation unit 62 performs an outlier removal process on the first brightness data 75. In the outlier removal process, first, a representative value such as the average, median, or mode of the multiple brightness values ​​constituting the first brightness data 75, and a standard deviation are calculated. Then, brightness values ​​outside a threshold range defined by the calculated representative value ± the standard deviation are removed as outliers. FIG. 7 shows an example in which brightness values ​​such as "250" and "2" are removed as outliers.

[0062] The first index value derivation unit 62 performs statistical processing on the processed first brightness data 75AFP, which is the first brightness data 75 after the outlier removal processing. Here, the statistical processing is processing to calculate the average value of multiple brightnesses that make up the processed first brightness data 75AFP. The first index value derivation unit 62 outputs the calculated average value as the first brightness index value 70. In this manner, the first brightness index value 70 is derived by a statistical method. Note that the statistical processing may be processing to derive a median or a mode instead of processing to calculate an average value. Furthermore, the outlier removal processing and the statistical processing may be repeated multiple times.

[0063] Although not shown in the figures, the first index value derivation unit 62 generates first saturation data by deriving saturation based on the a-value and the b-value from the first identification result 69, as in the case of the first brightness index value 70. The first index value derivation unit 62 performs outlier removal processing on the first saturation data to generate processed first saturation data. The first index value derivation unit 62 then performs statistical processing on the processed first saturation data. The first index value derivation unit 62 outputs a statistical value calculated by the statistical processing, such as the average, median, or mode, as the first saturation index value 71.

[0064] As an example, as shown in FIG. 8 , the brightness correction processing unit 63 acquires a brightness correction amount 77 corresponding to the first brightness index value 70 from the brightness correction reference information 47A. Then, using the acquired brightness correction amount 77, brightness correction processing is performed on the Lab image 68 to bring the brightness of the customer C's face closer to the brightness of the basketball player P's face. More specifically, the brightness correction processing unit 63 changes the L value of the Lab image 68 according to the brightness correction amount 77. As a result, the Lab image 68 becomes a brightness-corrected image 68ABC. The brightness correction amount 77 is an example of a "correction amount" according to the technology disclosed herein. Note that the value to which the brightness correction amount 77 is applied is not limited to the L value of the illustrated Lab image 68. For example, the brightness correction amount 77 may also be applied to a value related to brightness in another color space, such as a green pixel value in the RGB space. However, by applying the brightness correction amount 77 to the L value of the Lab image 68, the change in the overall appearance of the captured image 14 and the corrected captured image 14AC can be reduced compared to when it is applied to other values.

[0065] Similarly, as shown in FIG. 9 as an example, the saturation correction processing unit 64 acquires a saturation correction amount 78 corresponding to the first saturation index value 71 from the saturation correction reference information 47B. Then, using the acquired saturation correction amount 78, saturation correction processing is performed on the brightness-corrected image 68ABC to bring the saturation of the customer C's face closer to the saturation of the basketball player P's face. More specifically, the saturation correction processing unit 64 changes the a-value and b-value of the brightness-corrected image 68ABC according to the saturation correction amount 78. As a result, the brightness-corrected image 68ABC becomes the saturation-corrected image 68ASC. The saturation correction amount 78 is an example of a "correction amount" according to the technology disclosed herein. Note that the values ​​to which the saturation correction amount 78 is applied are not limited to the a-value and b-value of the brightness-corrected image 68ABC illustrated in the example. The saturation correction amount 78 may also be applied to values ​​related to saturation in other color spaces, such as the S-value in the HLS space.

[0066] As an example, as shown in FIG. 10 , the brightness correction reference information 47A is information in which a brightness correction amount 77 for each first brightness index value 70 is registered. The brightness correction amount 77 changes nonlinearly with respect to the first brightness index value 70. The brightness correction amount 77 takes on a positive upper limit value CAMAX when the first brightness index value 70 is at its minimum value BV1MIN. The upper limit value CAMAX is a value that, if exceeded, may result in blown-out highlights in the corrected captured image 14AC. The upper limit value CAMAX is an example of a "limit value" according to the technology of the present disclosure.

[0067] The brightness correction amount 77 gradually decreases from the upper limit value CAMAX as the first brightness index value 70 increases from the minimum value BV1MIN to BV1S, which is the same brightness as the specific subject, basketball player P. In other words, the absolute value of the brightness correction amount 77 gradually increases as the first brightness index value 70 approaches the minimum value BV1MIN.

[0068] When the first brightness index value 70 is smaller than BV1S, the face of customer C is darker than the face of basketball player P. Therefore, when the first brightness index value 70 is smaller than BV1S, the brightness correction amount 77 is a positive value. Therefore, the brightness correction processing unit 63 corrects the captured image 14 so that the face of customer C is brighter.

[0069] The brightness correction amount 77 is 0 when the first brightness index value 70 is BV1S. Furthermore, the brightness correction amount 77 takes on a negative lower limit value CAMIN when the first brightness index value 70 is the maximum value BV1MAX. The lower limit value CAMIN is a value below which black crush may occur in the corrected captured image 14AC. The lower limit value CAMIN is an example of a "limit value" according to the technology of the present disclosure.

[0070] The brightness correction amount 77 gradually decreases from 0 as the first brightness index value 70 increases from BV1S to the maximum value BV1MAX. In other words, the absolute value of the brightness correction amount 77 gradually increases as the first brightness index value 70 approaches the maximum value BV1MAX.

[0071] When the first brightness index value 70 is greater than BV1S, the face of customer C is brighter than the face of basketball player P. Therefore, when the first brightness index value 70 is greater than BV1S, the brightness correction amount 77 takes a negative value. Therefore, the brightness correction processing unit 63 corrects the captured image 14 so that the face of customer C becomes darker.

[0072] Specifically, the brightness correction reference information 47A is a function of the graph in FIG. 10 or a data table of the graph in FIG.

[0073] The saturation correction reference information 47B has the same content as the brightness correction reference information 47A, except that the first brightness index value 70 changes to the first saturation index value 71 and the brightness correction amount 77 changes to the saturation correction amount 78, and the saturation correction amount 78 changes nonlinearly with respect to the first saturation index value 71, so it will not be illustrated or explained here.

[0074] The first saturation index value 71 may be either a scalar quantity or a vector quantity. In the case of a vector quantity, the saturation correction processing unit 64 performs a saturation correction process in which the direction and magnitude of the vector quantity of the first saturation index value 71 are made to approximate the direction and magnitude of the representative value of the vector quantity of saturation in the face area of ​​the basketball player P. The order in which the direction and magnitude are made to approximate may be the direction approximated first and then the magnitude, or vice versa.

[0075] Next, the operation of the above configuration will be described with reference to the flowcharts shown in Figures 11 to 13 as an example. As shown in Figure 5, when the operating program 45 is started, the CPU 37 of the image processing server 11 functions as a request receiving unit 50, an RW control unit 51, a screen distribution control unit 52, an image quality correction processing unit 53, a composite image generation unit 54, and a transmission unit 55. As shown in Figure 6, the image quality correction processing unit 53 functions as a Lab conversion unit 60, a first area identification unit 61, a first index value derivation unit 62, a brightness correction processing unit 63, a saturation correction processing unit 64, and an RGB conversion unit 65.

[0076] First, prior to capturing the photographed image 14, a plurality of compositing templates 13 are read from the compositing template DB 46 under the control of the RW control unit 51. Then, the read plurality of compositing templates 13 are output from the RW control unit 51 to the screen distribution control unit 52. The screen distribution control unit 52 generates screen data for a list screen of the compositing templates 13, and distributes the screen data to the photographing device 10.

[0077] In the photographing device 10, a list screen is displayed on the display 23. The staff member S shows the list screen to the customer C, and the customer C selects a synthesis template 13 including a portrait image 30 of the desired basketball player P. This causes the photographing device 10 to transmit a delivery request for the photographing screen 25 to the image processing server 11.

[0078] 11 , when the request receiving unit 50 receives a request to deliver the image capture screen 25 (YES in step ST100), the request receiving unit 50 outputs identification information of the compositing template 13 selected by the customer C to be used for image capture to the RW control unit 51. Then, under the control of the RW control unit 51, the compositing template 13 corresponding to the identification information is read from the compositing template DB 46. In this way, the compositing template 13 selected by the customer C is acquired (step ST110). The compositing template 13 is output from the RW control unit 51 to the screen delivery control unit 52 and the composite image generation unit 54.

[0079] Based on the synthesis template 13 from the RW control unit 51, the screen distribution control unit 52 generates screen data for the photographing screen 25, and distributes the screen data to the photographing device 10 (step ST120).

[0080] In the photographing device 10, a photographing screen 25 is displayed on the display 23. The staff member S guides the customer C to a designated location in front of the flag 21 in the photographing booth 20. The staff member S gives various instructions to the customer C, such as moving slightly away from the basketball player P so as not to overlap with him, or asking him to strike a pose. Then, when the appropriate composition is achieved, the staff member S operates the release button 27, and the photographed image 14 is taken. This causes the photographing device 10 to send a request to the image processing server 11 to generate a composite image 15.

[0081] 12 , when the request receiving unit 50 receives a request to generate a composite image 15 (YES in step ST200), the captured image 14 included in the generation request is output from the request receiving unit 50 to the RW control unit 51. Then, under the control of the RW control unit 51, the captured image 14 is stored in the storage 35 (step ST210).

[0082] The RW control unit 51 reads out the photographed image 14 and the correction reference information 47 from the storage 35. In this way, the photographed image 14 is acquired (step ST220). The read photographed image 14 and correction reference information 47 are output from the RW control unit 51 to the image quality correction processing unit 53.

[0083] 6 , the captured image 14 is input to the Lab conversion unit 60 and the first region identification unit 61. The Lab conversion unit 60 converts the captured image 14 into a Lab image 68 (step ST230). The Lab conversion unit 60 outputs the Lab image 68 to the first region identification unit 61 and the brightness correction processing unit 63.

[0084] The first area identification unit 61 identifies a first facial area FA1 of customer C appearing in the photographed image 14 (step ST240). A first identification result 69 of the first facial area FA1 is output from the first area identification unit 61 to the first index value derivation unit 62. Note that in this example, the Lab conversion unit 60 converts the photographed image 14 into a Lab image 68 (step ST230) before the first area identification unit 61 identifies the first facial area FA1 (step ST240). However, this processing order may be reversed. In other words, the Lab conversion unit 60 may convert the photographed image 14 into a Lab image 68 after the first area identification unit 61 identifies the first facial area FA1.

[0085] As shown in FIG. 7 , the first index value derivation unit 62 generates first brightness data 75 from the first identification result 69. Next, outlier removal processing is performed on the first brightness data 75. Then, statistical processing is performed on processed first brightness data 75AFP, which is the first brightness data 75 after the outlier removal processing. As a result, a first brightness index value 70 is derived. A first saturation index value 71 is also derived in a similar manner (step ST250). The first brightness index value 70 is output from the first index value derivation unit 62 to the brightness correction processing unit 63. The first saturation index value 71 is output from the first index value derivation unit 62 to the saturation correction processing unit 64.

[0086] The brightness correction processing unit 63 receives the brightness correction reference information 47A from the correction reference information 47. The saturation correction processing unit 64 receives the saturation correction reference information 47B from the correction reference information 47.

[0087] 8 , the brightness correction processing unit 63 obtains the brightness correction amount 77 corresponding to the first brightness index value 70 from the brightness correction reference information 47A. Then, using the brightness correction amount 77, brightness correction processing is performed on the Lab image 68 to bring the brightness of the customer C's face closer to the brightness of the basketball player P's face (step ST260). A brightness-corrected image 68ABC, which is the Lab image 68 after the brightness correction processing, is output from the brightness correction processing unit 63 to the saturation correction processing unit 64.

[0088] 9 , the saturation correction processing unit 64 obtains the saturation correction amount 78 corresponding to the first saturation index value 71 from the saturation correction reference information 47B. Then, using the saturation correction amount 78, saturation correction processing is performed on the brightness-corrected image 68ABC to bring the saturation of the face of customer C closer to the saturation of the face of basketball player P (step ST270). The saturation-corrected image 68ASC, which is the brightness-corrected image 68ABC after the saturation correction processing, is output from the saturation correction processing unit 64 to the RGB conversion unit 65.

[0089] The RGB converter 65 converts the saturation-corrected image 68ASC, which is a Lab image, into the corrected captured image 14AC, which is an RGB color image (step ST280). The corrected captured image 14AC is output from the RGB converter 65 (image quality correction processor 53) to the composite image generator 54.

[0090] The composite image generating unit 54 generates the composite image 15 by combining the corrected photographed image 14AC with the composition template 13 (step ST290). The composite image 15 is output from the composite image generating unit 54 to the screen distribution control unit 52.

[0091] Based on the composite image 15 from the composite image generating unit 54, the screen distribution control unit 52 generates screen data for a confirmation screen of the composite image 15, and distributes the screen data to the photographing equipment 10 (step ST300).

[0092] In the photographing device 10, a confirmation screen for the composite image 15 is displayed on the display 23. The staff member S shows the confirmation screen to the customer C, who evaluates the quality of the composite image 15. When the customer C approves of the quality of the composite image 15, the photographing device 10 transmits a print request for the composite image 15 to the image processing server 11.

[0093] 13, when the request receiving unit 50 receives a request to print the composite image 15 (YES in step ST350), the request receiving unit 50 outputs a signal indicating that the print request has been received to the transmitting unit 55. As a result, the transmitting unit 55 transmits the composite image 15 to the instant printer 12 (step ST360).

[0094] In the instant printer 12, the composite image 15 is printed on an instant film 16. The instant film 16 on which the composite image 15 has been printed is handed over to the customer C by the staff member S.

[0095] As described above, the CPU 37 of the image processing server 11 functions as the RW control unit 51, the image quality correction processing unit 53, and the composite image generation unit 54. The RW control unit 51 acquires the compositing template 13 by reading it from the storage 35. The RW control unit 51 also acquires the photographed image 14 including the customer C by reading it from the storage 35. The image quality correction processing unit 53 corrects the image quality of the photographed image 14, thereby performing image quality correction processing to reduce the difference in image quality between the compositing template 13 and the photographed image 14. The composite image generation unit 54 generates the composite image 15 by combining the corrected photographed image 14AC with the compositing template 13. This makes it possible to reduce the sense of incongruity in the composite image 15 that results from the difference in image quality between the compositing template 13 and the photographed image 14.

[0096] The portrait image 30 placed in the compositing template 13 is, for example, an image taken by a professional photographer in a photography studio, and is an image that was taken with at least more effort and time than the photographed image 14. For this reason, the portrait image 30 is considered to have image quality, such as brightness and saturation, at a certain level or above. Therefore, the technology disclosed herein, which corrects the image quality of the photographed image 14 rather than the compositing template 13 to reduce the difference between the image quality of the compositing template 13 and the image quality of the photographed image 14, is preferable because it can bring the image quality of the photographed image 14 closer to the good image quality of the compositing template 13.

[0097] As shown in FIG. 6 , the first area identification unit 61 identifies the first face area FA1 of customer C from the captured image 14. As shown in FIG. 7 , the first index value derivation unit 62 derives a first brightness index value 70 for the first face area FA1. Although not shown, the first index value derivation unit 62 also derives a first saturation index value 71 for the first face area FA1. As shown in FIGS. 8 and 9 , the brightness correction processing unit 63 and the saturation correction processing unit 64 perform brightness correction processing and saturation correction processing according to the first brightness index value 70 and the first saturation index value 71. Therefore, compared to when the entire area of ​​the captured image 14 is processed, the brightness correction processing and saturation correction processing can be performed based on a clear criterion, namely, the first face area FA1. This improves the correction accuracy of the brightness correction processing and the saturation correction processing. It should be noted that the first brightness index value 70 and the first saturation index value 71 may be derived from the entire area of ​​the photographed image 14, without being limited to the first face area FA1.

[0098] In an image that includes a person's face, attention is drawn to the person's face, and therefore the image quality of the person's face is the most noticeable. Therefore, according to the technology of the present disclosure, in which the first region is the first face region FA1 that is the attention region, it is possible to generate an ideal composite image 15 in which the difference in image quality between the most noticeable face of basketball player P in composite template 13 and the image quality of customer C in captured image 14 is reduced.

[0099] 7, the first brightness index value 70 is derived by a statistical method. Although not shown, the first saturation index value 71 is also derived by a statistical method. Therefore, it is possible to derive a plausible first brightness index value 70 and a plausible first saturation index value 71.

[0100] 7 , the first brightness index value 70 is derived after an outlier removal process has been performed. Although not shown, the first saturation index value 71 is also derived after an outlier removal process has been performed. Therefore, it is possible to derive more likely first brightness index value 70 and first saturation index value 71, with the influence of outliers removed.

[0101] As shown in FIGS. 6 , 8 , and 10 , the image processing server 11 has brightness correction reference information 47A in which a brightness correction amount 77 corresponding to a first brightness index value 70 is registered. Also, as shown in FIGS. 6 and 9 , the image processing server 11 has saturation correction reference information 47B in which a saturation correction amount 78 corresponding to a first saturation index value 71 is registered. As shown in FIG. 8 , the brightness correction processing unit 63 acquires the brightness correction amount 77 corresponding to the derived first brightness index value 70 from the brightness correction reference information 47A. Then, the brightness correction processing is performed using the acquired brightness correction amount 77. Also, as shown in FIG. 9 , the saturation correction processing unit 64 acquires the saturation correction amount 78 corresponding to the derived first saturation index value 71 from the saturation correction reference information 47B. Then, the saturation correction processing is performed using the acquired saturation correction amount 78. Therefore, by simply deriving the first brightness index value 70 and the first saturation index value 71, it is possible to easily obtain the corresponding brightness correction amount 77 and saturation correction amount 78. As a result, it is possible to reduce the time required for the brightness correction process and the saturation correction process.

[0102] 10 , the first brightness index value 70 and the brightness correction amount 77 in the brightness correction reference information 47A have a nonlinear relationship. Although not shown, the first saturation index value 71 and the saturation correction amount 78 in the saturation correction reference information 47B also have a nonlinear relationship. Therefore, it is possible to obtain a more appropriate brightness correction amount 77 and a more appropriate saturation correction amount 78 than in the linear case. As a result, it is possible to perform more appropriate brightness correction processing and saturation correction processing on the captured image 14.

[0103] 10 , the brightness correction reference information 47A stores brightness correction amounts 77 whose absolute values ​​gradually increase as the first brightness index value 70 approaches the maximum value BV1MAX or the minimum value BV1MIN. Although not shown, the saturation correction reference information 47B also stores saturation correction amounts 78 whose absolute values ​​gradually increase as the first saturation index value 71 approaches the maximum or minimum value. This makes it possible to acquire appropriate brightness correction amounts 77 and saturation correction amounts 78 according to the magnitudes of the first brightness index value 70 and the first saturation index value 71. As a result, it is possible to perform more appropriate brightness correction processing and saturation correction processing on the captured image 14.

[0104] As shown in FIG. 10 , a limit value is set for the brightness correction amount 77. More specifically, an upper limit value CAMAX and a lower limit value CAMIN are set for the brightness correction amount 77. This prevents the occurrence of blown-out highlights in the captured image 14 due to brightness correction processing using an excessively positive brightness correction amount 77, and the occurrence of crushed blacks in the captured image 14 due to brightness correction processing using an excessively negative brightness correction amount 77. Simply put, it prevents degradation of the image quality of the captured image 14 due to image quality correction processing using an excessive correction amount. Note that if the captured image 14 becomes too dark, customer C's satisfaction may decrease compared to when the captured image 14 is bright. Therefore, the lower limit value CAMIN may be set to a value that is relatively stricter than the upper limit value CAMAX. Similarly, if the saturation of the captured image 14 becomes too low, customer C's satisfaction may decrease. Therefore, the lower limit value of the saturation correction amount 78 may be set to a value that is relatively stricter than the upper limit value.

[0105] As an example, the correction reference information 47 may be configured to be modifiable, as shown in Fig. 14 . In Fig. 14 , the CPU 37 of the image processing server 11 displays a correction instruction screen 85 on the display 39 in response to an instruction from an administrator of the image processing server 11. The correction instruction screen 85 is a screen for issuing an instruction to correct the brightness correction reference information 47A. A graph of the brightness correction reference information 47A is displayed on the correction instruction screen 85. The graph of the brightness correction reference information 47A can be selected and transformed with a cursor 86. The dashed curve represents the brightness correction reference information 47A before correction, and the solid curve represents the brightness correction reference information 47A after correction.

[0106] An OK button 87 is provided at the bottom of the correction instruction screen 85. The CPU 37 also functions as an instruction accepting unit 88 in addition to the above-mentioned processing units 50 to 55 (only the RW control unit 51 is shown in FIG. 14 ). The instruction accepting unit 88 accepts various operational instructions from the administrator via the input device 40.

[0107] The administrator changes the graph of the brightness correction reference information 47A on the correction instruction screen 85 to the desired shape, and then selects the OK button 87. As a result, a correction instruction 89 for the brightness correction reference information 47A is received by the instruction receiving unit 88. The correction instruction 89 includes the corrected brightness correction reference information 47A. The instruction receiving unit 88 outputs the correction instruction 89 to the RW control unit 51. The RW control unit 51 rewrites the brightness correction reference information 47A in the storage 35 with the corrected brightness correction reference information 47A.

[0108] In this manner, the instruction receiving unit 88 receives a correction instruction 89 for the brightness correction amount 77 in the brightness correction reference information 47A. The RW control unit 51 corrects the brightness correction amount 77 in the brightness correction reference information 47A in accordance with the correction instruction 89. This allows for flexible changes to the brightness correction amount 77. This allows for a more realistic setting of the brightness correction amount 77 than when the brightness correction amount 77 is fixed, such as changing the brightness correction amount 77 in response to feedback from the field, such as customer C and staff S. This also improves responsiveness, allowing knowledgeable staff S to correct the brightness correction amount 77 on-site. While FIG. 14 illustrates an example of correcting the brightness correction amount 77 in the brightness correction reference information 47A, the saturation correction amount 78 in the saturation correction reference information 47B may be corrected instead of or in addition to this.

[0109] Second Embodiment As shown in FIG. 15 as an example, in the second embodiment, a second brightness index value 95 and a second saturation index value 96 are associated and stored in a synthesis template 13 in a synthesis template DB 46. The second brightness index value 95 is a brightness index value of a second face area FA2 that rectangularly surrounds the face of a basketball player P. The second saturation index value 96 is a saturation index value of the second face area FA2. Like the first face area FA1, the second face area FA2 is identified using well-known face recognition technology. The second brightness index value 95 and the second saturation index value 96 are examples of a "second index value" according to the technology of the present disclosure. The second face area FA2 is an example of a "second area," "area related to a first area," and "person's face area" according to the technology of the present disclosure. The term "area related to the first area" means an area of ​​the same type as the first area, such as when the first area is the first face area FA1 and the second face area FA2 is the second area.

[0110] As shown in FIG. 16 as an example, the second brightness index value 95 is derived as follows when creating the synthesis template 13. First, brightness (L value) is extracted from the second identification result 100 of the second face area FA2 to generate second brightness data 101. Then, an outlier removal process is performed on the second brightness data 101. Similar to the outlier removal process for the first brightness data 75 in the first embodiment, the outlier removal process first calculates representative values, such as the average, median, and mode, and standard deviation of the multiple brightness values ​​constituting the second brightness data 101. Then, brightness values ​​outside the threshold range defined by the calculated representative value ± the standard deviation are removed as outliers. FIG. 16 shows an example in which brightness values ​​such as "425" and "492" are removed as outliers. Note that, like the first identification result 69, the second identification result 100 may also remove areas other than skin-colored areas, such as hair, eyebrows, beard, eyes, nostrils, mouth, and even shadow areas.

[0111] Next, statistical processing is performed on the processed second brightness data 101AFP, which is the second brightness data 101 after the outlier removal processing. Here, the statistical processing is processing to calculate the average value of multiple brightness values ​​that make up the processed second brightness data 101AFP. The average value calculated in this manner is set as the second brightness index value 95. In this manner, the second brightness index value 95 is also derived using a statistical method. Note that the statistical processing may be processing to derive a median or a mode instead of processing to calculate an average value. Furthermore, the outlier removal processing and the statistical processing may be repeated multiple times.

[0112] Although not shown in the figures, the second saturation index value 96, like the second brightness index value 95, is generated by first deriving saturation based on the a-value and the b-value from the second identification result 100. Then, the second saturation data is subjected to an outlier removal process to obtain processed second saturation data. Next, statistical processing is performed on the processed second saturation data. The statistical value calculated by this statistical processing, such as the mean, median, or mode, is used as the second saturation index value 96.

[0113] 17 as an example, a Lab image 68, a first brightness index value 70, and a second brightness index value 95 are input to the brightness correction processing unit 105 of this embodiment. The brightness correction processing unit 105 calculates the difference between the first brightness index value 70 and the second brightness index value 95. The brightness correction processing unit 105 performs brightness correction processing on the Lab image 68 to reduce the calculated difference, thereby bringing the brightness of the customer C's face closer to the brightness of the basketball player P's face. As a result, the Lab image 68 is converted into a brightness-corrected image 68ABC.

[0114] 18 , the brightness-corrected image 68ABC, the first saturation index value 71, and the second saturation index value 96 are input to the saturation correction processor 106 of this embodiment. The saturation correction processor 106 calculates the difference between the first saturation index value 71 and the second saturation index value 96. The saturation correction processor 106 performs saturation correction processing on the brightness-corrected image 68ABC to reduce the calculated difference, thereby bringing the saturation of customer C closer to the saturation of basketball player P. As a result, the brightness-corrected image 68ABC becomes the saturation-corrected image 68ASC.

[0115] As described above, in the second embodiment, the brightness correction processing unit 105 performs brightness correction processing in accordance with the first brightness index value 71 of the first facial area FA1 of customer C in the photographed image 14 and the second brightness index value 95 of the second facial area FA2 of basketball player P in the compositing template 13. Furthermore, the saturation correction processing unit 106 performs saturation correction processing in accordance with the first saturation index value 71 of the first facial area FA1 and the second saturation index value 96 of the second facial area FA2. Therefore, compared to when the entire areas of the compositing template 13 and the photographed image 14 are processed, it is possible to perform brightness correction processing and saturation correction processing in accordance with clear criteria, namely, the first facial area FA1 and the second facial area FA2.

[0116] According to the technology of the present disclosure, in which the second area is an area related to the first area, more specifically, the second face area FA2, an ideal composite image 15 can be generated in which the difference in image quality between the most popular face of basketball player P in the composite template 13 and the face of customer C in the photographed image 14 is reduced.

[0117] 16 , the second brightness index value 95 is derived by a statistical method. Although not shown, the second saturation index value 96 is also derived by a statistical method. Therefore, it is possible to derive a plausible second brightness index value 95 and a plausible second saturation index value 96.

[0118] 16 , the second brightness index value 95 is derived after an outlier removal process has been performed. Although not shown, the second saturation index value 96 is also derived after an outlier removal process has been performed. Therefore, it is possible to derive more likely second brightness index value 95 and second saturation index value 96, with the influence of outliers removed.

[0119] The second brightness index value 95 and the second saturation index value 96 may be derived by the image processing server 11, or may be derived by a device other than the image processing server 11. In the latter case, the second brightness index value 95 and the second saturation index value 96 are transmitted to the image processing server 11 from the other device.

[0120] An upper limit and a lower limit may be set for the second brightness index value 95. In this case, if the derived second brightness index value 95 exceeds the upper limit or falls below the lower limit, the second brightness index value 95 is clipped to the upper limit or lower limit. Then, the brightness correction process is performed using the second brightness index value 95 clipped to the upper limit or lower limit. This makes it possible to prevent excessive brightness correction from being performed due to the influence of the second brightness index value 95 exceeding the upper limit or falling below the lower limit. Similarly, an upper limit and a lower limit may be set for the second saturation index value 96.

[0121] Third Embodiment In the third embodiment, as shown in FIG. 19 as an example, a lighting correction processing unit 110 is provided in the image quality correction processing unit. A Lab image 68, a first lighting index value 111, and a second lighting index value 112 are input to the lighting correction processing unit 110. The first lighting index value 111 is derived by a first index value derivation unit 113 based on a first identification result 69. The first lighting index value 111 includes the light source direction and intensity of light irradiated onto the face of customer C in the first face area FA1. The second lighting index value 112 is stored in association with the synthesis template 13 in the synthesis template DB 46. The second lighting index value 112 includes the light source direction and intensity of light irradiated onto the face of basketball player P in the second face area FA2. The first lighting index value 111 is an example of a “first index value” according to the technology of the present disclosure. The second lighting index value 112 is an example of a “second index value” according to the technology of the present disclosure.

[0122] The light source direction is the vertical and horizontal angles of the position where the light source is estimated to exist. In the vertical direction, the front is defined as 0°, directly above is defined as +90°, and directly below is defined as -90°. In the horizontal direction, the front is defined as 0°, the right is defined as +90°, and the left is defined as -90°. The light source direction can be derived by dividing the first face area FA1 or the second face area FA2 into multiple areas, calculating a representative value of the brightness of each divided area, and analyzing the difference between these representative values. The light intensity can be derived from the representative value of the brightness of each divided area. Alternatively, a three-dimensional face model can be applied to the face of customer C in the first face area FA1 or the face of basketball player P in the second face area FA2, determining the normal vector of the light hitting each point on the face, and estimating the light source direction from the normal vector using Phong's reflection model. Alternatively, a trained model may be used that outputs the direction and intensity of the light source in response to an input image of the first face area FA1 or the second face area FA2.

[0123] The lighting correction processing unit 110 performs lighting correction processing on the Lab image 68 based on the first lighting index value 111 and the second lighting index value 112, so as to make the way light hits the face of the customer C closer to the way light hits the face of the basketball player P. The lighting correction processing unit 110 outputs a lighting-corrected image 68ALC, which is the Lab image 68 after the lighting correction processing.

[0124] 20 as an example, the lighting correction processing unit 110 calculates a lighting correction amount 115 from a first lighting index value 111 and a second lighting index value 112. The lighting correction processing unit 110 performs lighting correction processing by multiplying the L value of the first face area FA1 in the Lab image 68 by the lighting correction amount 115. The lighting correction amount 115 is an example of a "correction amount" according to the technology of the present disclosure.

[0125] The lighting correction amount 115 is set for each pixel of the image of the first face area FA1. The lighting correction amount 115 is calculated by calculating the cosine similarity between the direction of a vector from a representative point on the face of customer C in the first face area FA1, such as the tip of the nose, to each pixel and the direction of a vector from the representative point toward the light source (second lighting index value 112). The lighting correction amount 115 can be calculated by multiplying the calculated cosine similarity by the light intensity of the first lighting index value 111. As is well known, cosine similarity takes a value between 1 and -1, with 1 indicating a match and -1 indicating a mismatch. Therefore, if the direction of the vector from the representative point and the direction of the vector from the representative point toward the light source (second lighting index value 112) are similar, the first lighting index value 111 will take a positive value. Therefore, the lighting correction processing unit 110 corrects that portion as if it were illuminated. On the other hand, if the direction of the vector from the representative point is not similar to the direction of the vector from the representative point toward the light source of the second lighting index value 112, the first lighting index value 111 will be a negative value. Therefore, the lighting correction processing unit 110 corrects that part so that it does not receive light.

[0126] To prevent the lighting-corrected image 68ALC from appearing unnatural due to a step-like change in the lighting correction amount 115 around the representative point, it is preferable to perform a smoothing process on the lighting correction amount 115 around the representative point. The smoothing process is achieved, for example, by setting the lighting correction amount 115 at the representative point to 0 and smoothly changing the lighting correction amount 115 up to a certain distance from the representative point.

[0127] As described above, in the third embodiment, the lighting correction processor 110 performs lighting correction as image quality correction processing to make the lighting on the face of customer C closer to the lighting on the face of basketball player P. The lighting correction processor 110 acquires the direction and intensity of the light source irradiating the face of customer C in the first face area FA1 as the first lighting index value 111. The lighting correction processor 110 also acquires the direction and intensity of the light source irradiating the face of basketball player P in the second face area FA2 as the second lighting index value 112. This makes it possible to acquire a composite image 15 in which the difference in the lighting on the faces of customer C and basketball player P is reduced. This eliminates the need for the troublesome adjustment of the position of customer C and / or lighting equipment in the photo booth 20 to make the lighting on the face of basketball player P closer to that of basketball player P.

[0128] As with the brightness correction amount 77, the value to which the lighting correction amount 115 is applied is not limited to the L value of the illustrated Lab image 68. For example, the lighting correction amount 115 may be applied to a value related to brightness in another color space, such as a green pixel value in the RGB color space. Furthermore, if excessive light is shining on the face of customer C in the first face area FA1 and the light intensity of the first lighting index value 111 is equal to or greater than a threshold intensity, it is preferable to perform a process to reduce the light intensity (reduce the brightness) of the captured image 14 before performing the lighting correction process. In this case, the lighting correction processing unit 110 performs the lighting correction process using the lighting correction amount 115 on the Lab image 68 of the captured image 14 that has been subjected to the light intensity reduction process.

[0129] If the difference between the light source direction and intensity of the first lighting index value 111 and the light source direction and intensity of the second lighting index value 112 is not greater than the threshold, the lighting correction process may be postponed.

[0130] When the brightness correction process and saturation correction process of the first embodiment and the lighting correction process of the third embodiment are performed as image quality correction processes, it is preferable to perform them in the order shown in FIG. 21 , for example. That is, the brightness correction process and saturation correction process are performed first, followed by the lighting correction process. Conversely, if the brightness correction process and saturation correction process are performed after the lighting correction process, the lighting correction process may be distorted by the brightness correction process and saturation correction process, which have been carefully corrected. For this reason, it is preferable to perform the lighting correction process after the brightness correction process and saturation correction process.

[0131] 22 as an example, in the fourth embodiment, a noise amount correction processing unit 120 is provided in the image quality correction processing unit. The captured image 14 and a first noise amount index value 121 are input to the noise amount correction processing unit 120. The first noise amount index value 121 is derived by a first index value derivation unit 122 based on a first identification result 69. The first index value derivation unit 122 calculates, as the noise amount, the difference in pixel values ​​between the image of the first face area FA1 and an image obtained by performing a smoothing process on the image of the first face area FA1. The first index value derivation unit 122 then performs an outlier removal process on the noise amount. The first index value derivation unit 122 performs statistical processing, such as calculating an average value, on the noise amount after the outlier removal process. The first index value derivation unit 122 outputs the calculated average value as the first noise amount index value 121. The first noise amount index value 121 is an example of a "first index value" according to the technology of the present disclosure.

[0132] The noise amount correction processing unit 120 acquires a noise amount correction amount 123 corresponding to the first noise amount index value 121 from the noise amount correction reference information 47C. The noise amount correction amount 123 is an example of a "correction amount" according to the technology of the present disclosure. The noise amount correction reference information 47C is a type of correction reference information 47, and is information in which the noise amount correction amount 123 corresponding to the first noise amount index value 121 is registered. Like the brightness correction amount 77 and the saturation correction amount 78, the noise amount correction amount 123 also has a nonlinear relationship with the first noise amount index value 121. Furthermore, the absolute value of the noise amount correction amount 123 also gradually increases as the first noise amount index value 121 approaches the maximum or minimum value. Furthermore, an upper limit and a lower limit are set for the noise amount correction amount 123.

[0133] The noise amount correction processing unit 120 uses the acquired noise amount correction amount 123 to perform noise amount correction processing on the photographed image 14 to bring the amount of noise on the face of the customer C closer to the amount of noise on the face of the basketball player P. This turns the photographed image 14 into a noise amount-corrected image 14ANC. Note that the noise amount correction processing is performed on the entire photographed image 14.

[0134] If the amount of noise on the face of customer C is less than the amount of noise on the face of basketball player P, the noise amount correction processing unit 120 performs a process to increase the amount of noise on the face of customer C as the noise amount correction process. On the other hand, if the amount of noise on the face of customer C is greater than the amount of noise on the face of basketball player P, the noise amount correction processing unit 120 performs a process to decrease the amount of noise on the face of customer C as the noise amount correction process. In this case, the noise amount correction processing unit 120 reduces the amount of noise by inputting the captured image 14 to a filter such as a smoothing filter or a bilateral filter. Alternatively, the amount of noise may be reduced by performing a wavelet transform on the captured image 14, reducing components corresponding to noise according to the noise amount correction amount 123, and then performing an inverse wavelet transform.

[0135] As described above, in the fourth embodiment, the noise amount correction processing unit 120 performs, as image quality correction processing, noise amount correction processing that brings the amount of noise on the face of the customer C closer to the amount of noise on the face of the basketball player P. Therefore, it is possible to obtain a composite image 15 in which the difference in the amount of noise on the faces of the customer C and the basketball player P is reduced.

[0136] 22 illustrates an example of noise amount correction processing using the noise amount correction amount 123 from the noise amount correction reference information 47C, but this is not limiting. Similar to the second brightness index value 95 and the second saturation index value 96 in the second embodiment, an index value for the amount of noise on the face of the basketball player P in the second face area FA2 (hereinafter referred to as the second noise amount index value) may be stored in association with the synthesis template 13. In this case, the noise amount correction processor performs noise amount correction processing on the captured image 14 to reduce the difference between the first noise amount index value 121 and the second noise amount index value. The second noise amount index value is an example of a "second index value" according to the technology of the present disclosure.

[0137] Fifth Embodiment As shown in FIG. 23 as an example, in the fifth embodiment, a resolution correction processing unit 125 is provided in the image quality correction processing unit. A captured image 14 and a first resolution index value 126 are input to the resolution correction processing unit 125. The first resolution index value 126 is the resolution of the captured image 14. Here, resolution is the number of pixels present in a unit area, i.e., pixel density. The unit area is, for example, an area of ​​one inch square. The first resolution index value 126 is an example of a "first index value" according to the technology of the present disclosure.

[0138] The resolution correction processing unit 125 obtains a resolution correction amount 127 corresponding to the first resolution index value 126 from the resolution correction reference information 47D. The resolution correction amount 127 is an example of a "correction amount" according to the technology of the present disclosure. The resolution correction reference information 47D is a type of correction reference information 47, and is information in which the resolution correction amount 127 corresponding to the first resolution index value 126 is registered. Like the brightness correction amount 77 and the saturation correction amount 78, the resolution correction amount 127 also has a nonlinear relationship with the first resolution index value 126. Furthermore, the absolute value of the resolution correction amount 127 also gradually increases as the first resolution index value 126 approaches its maximum or minimum value. Furthermore, an upper limit and a lower limit are set for the resolution correction amount 127.

[0139] The resolution correction processing unit 125 uses the acquired resolution correction amount 127 to perform resolution correction processing on the photographed image 14, bringing the resolution of the face of customer C closer to the resolution of the face of basketball player P. As a result, the photographed image 14 becomes a resolution-corrected image 14ARC. Note that the resolution correction processing is performed on the entire photographed image 14.

[0140] If the resolution of the photographed image 14 is lower than the resolution of the portrait image 30 of the basketball player P in the synthesis template 13, the resolution correction processor 125 performs a process to increase the resolution of the photographed image 14 as a resolution correction process. The resolution increase process may be, for example, pixel interpolation using bicubic interpolation or super-resolution processing using a fractal interpolation filter. A process to increase the sharpness of the photographed image 14 may also be employed. On the other hand, if the resolution of the photographed image 14 is higher than the resolution of the portrait image 30, the resolution correction processor 125 performs a process to reduce the resolution of the photographed image 14 as a resolution correction process. Examples of resolution reduction processes include a process of averaging the pixel values ​​of multiple adjacent pixels in the photographed image 14 and combining them into a single pixel, and a so-called blurring process that reduces the sharpness of the photographed image 14.

[0141] As described above, in the fifth embodiment, the resolution correction processing unit 125 performs image quality correction processing to bring the resolution of the face of the customer C closer to the resolution of the face of the basketball player P. Therefore, it is possible to obtain a composite image 15 in which the difference in resolution between the customer C and the basketball player P is reduced.

[0142] 23 illustrates an example of resolution correction processing using the resolution correction amount 127 from the resolution correction reference information 47D, but this is not limiting. Similar to the second brightness index value 95 and the second saturation index value 96 in the second embodiment, the resolution of the portrait image 30 of the basketball player P (hereinafter referred to as the second resolution index value) may be stored in association with the synthesis template 13. In this case, the resolution correction processing unit performs resolution correction processing on the captured image 14 to reduce the difference between the first resolution index value 126 and the second resolution index value. The second resolution index value is an example of a "second index value" according to the technology of the present disclosure.

[0143] When the brightness correction process and saturation correction process of the first embodiment and the resolution correction process of the fifth embodiment are performed as image quality correction processes, it is preferable to perform them in the order shown in Fig. 24, for example. That is, the resolution correction process is performed first, followed by the brightness correction process and the saturation correction process. By performing the order in this way, it is possible to perform the brightness correction process and the saturation correction process according to the resolution at which the composite image 15 is displayed.

[0144] The first index value, such as the first brightness index value 70, and the second index value, such as the second brightness index value 95, may be derived using a trained model.

[0145] As image quality correction processes, the following processes may be performed in addition to or instead of the exemplified brightness correction process, saturation correction process, lighting correction process, noise amount correction process, and resolution correction process. For example, if a black-and-white, sepia, or other effect is applied to the portrait image 30, the same effect may be applied to the captured image 14. Furthermore, a process may be performed to reduce the difference in distortion between the captured image 14 and the portrait image 30 due to lens distortion of the photographing device. Although a somewhat extreme example, if the camera unit of the photographing device 10 includes a fisheye lens, distortion of the captured image 14 due to the fisheye lens is corrected.

[0146] A process of aligning the height of the face of the specific subject with the height of the face of the arbitrary subject and / or a process of aligning the size of the faces of the arbitrary subject and the specific subject (a process of enlarging or reducing the portrait image 30) may be performed. Furthermore, a process of aligning the angle of the arbitrary subject with the angle of the specific subject may be performed. For example, if the face of the specific subject in the portrait image 30 is photographed from the front and the face of the arbitrary subject in the photographed image 14 is photographed from below looking up, a photographed image 14 in which the face of the arbitrary subject is photographed from the front is generated from the original photographed image 14 using the trained model.

[0147] Sixth Embodiment As an example, as shown in FIG. 25 , a synthesis template 13 according to a sixth embodiment includes a portrait image 30 of actor AC. Actor AC stands on the left side with his right cheek facing forward and holds out his right hand to shake hands. Therefore, the back of actor AC's right hand faces forward. Customer C in the captured image 14 stands on the right side of actor AC with his left cheek facing forward and also holds out his right hand to shake hands with actor AC. Therefore, the palm of customer C's right hand faces forward. Actor AC, like basketball player P, is an example of a "specific subject" according to the technology of the present disclosure.

[0148] First depth information 130 is stored in association with the captured image 14. Similarly, second depth information 131 is stored in association with the synthesis template 13. The first depth information 130 is information indicating the front-to-back relationship of customer C with respect to actor AC. Because customer C has the palm of his right hand facing forward, "back" is registered in the first depth information 130. The second depth information 131 is information indicating the front-to-back relationship of actor AC with respect to customer C. Because actor AC has the back of his right hand facing forward, "front" is registered in the second depth information 131.

[0149] The composite image generation unit 132 of the sixth embodiment acquires the first depth information 130 and the second depth information 131 along with the photographed image 14 and the composition template 13. The composite image generation unit 132 generates a composite image 15 in which the customer C and the actor AC are in a front-to-back relationship according to the first depth information 130 and the second depth information 131. Specifically, the composite image generation unit 132 generates a composite image 15 in which the right hand of the customer C is in the back, the right hand of the actor AC is in the front, and the right hand of the actor AC covers the right hand of the customer C, as if the customer C and the actor AC are shaking hands with their right hands.

[0150] As described above, in the sixth embodiment, the composite image generation unit 132 acquires the first depth information 130 and the second depth information 131 indicating the front-to-back relationship between the customer C and the actor AC, and generates a composite image 15 in which the customer C and the actor AC have a front-to-back relationship according to the first depth information 130 and the second depth information 131. Therefore, it is possible to acquire a composite image 15 in which the front-to-back relationship between the arbitrary subject and the specific subject is natural, such as the illustrated composite image 15 that looks as if the customer C and the actor AC are shaking hands. This makes it possible to hold an event such as a virtual handshake event with the actor AC.

[0151] The depth information is not limited to simple information such as "back" and "front" as shown in the example. The distance from the photographic device 10 to an arbitrary subject when the photographic image 14 is captured may be measured, and the measured distance may be used as the first depth information 130. Similarly, the distance from the photographic device to a specific subject when the portrait image 30 is captured may be measured, and the measured distance may be used as the second depth information 131. The distance may be measured using a distance measuring sensor such as a LiDAR (Light Detection and Ranging) sensor. Alternatively, the distance may be estimated based on the size of the first face area FA1 and the second face area FA2. Furthermore, a trained model that outputs the distance in response to an input image may be used.

[0152] In addition, the second depth information 131 may be configured to be able to change its content (for example, from "front" to "back") in response to a specific gesture by the customer C or staff member S, such as snapping their fingers or clapping their hands.

[0153] The clothing, belongings, etc. of the specific subject appearing in the portrait image 30 may be changed depending on the shooting environment of the photographed image 14. For example, if the walls of the photo booth 20 are white, the color of the jacket of the specific subject appearing in the portrait image 30 may be changed to black so that the specific subject is not obscured by the walls of the photo booth 20. Alternatively, if customer C, a basketball fan, is wearing an away uniform, basketball player P in the portrait image 30 may also be changed to an away uniform. Furthermore, for example, if it rains on the day the photographed image 14 is taken, the specific subject may be shown holding an umbrella. To achieve this, multiple patterns of portrait images 30 may be actually prepared, each depicting a specific subject wearing different colors or types of jackets and / or carrying different belongings. Alternatively, only one pattern of portrait image 30 may be prepared. Then, a semantic segmentation model may be used to identify the clothing area, and a color conversion process may be applied to the identified clothing area. Alternatively, a trained model may be used that, in response to an input image, outputs an image in which the specific subject is holding a specified item.

[0154] A video in which a specific subject appears from outside the screen and remains stationary at a predetermined position may be played on the shooting screen 25. If there is a wall or the like at the position where the specific subject appears and it would be unnatural for the specific subject to appear from that position, the position where the specific subject appears may be changed to a more natural position where there is no wall or the like.

[0155] The specific subject is not limited to a person. It may be a character, an animal, a plant, a building such as a temple, a shrine, or a tower, or even a vehicle such as a train, a car, or a motorcycle. It may also be a work of art such as a painting or a sculpture. Multiple types of specific subjects, such as a character and a person, or an animal and a plant, may be mixed in one synthesis template 13. In the case of a character, the second region is the character's face region. In the case of an animal, the second region is the animal's face region or the animal's entire body region. In the case of a plant, the second region is the flower region. In the case of a building, a vehicle, or a work of art, the second region is the entire region of the building, vehicle, or work of art.

[0156] If the specific subject is an animal, plant, building, vehicle, or work of art, the position and / or size of the specific subject may be changed depending on the position and / or size of the arbitrary subject. For example, if the specific subject is a dog, it may be placed at the feet of the arbitrary subject, smaller than the arbitrary subject.

[0157] The first area and the area of ​​interest are not limited to the first facial area FA1 of customer C. The entire body of customer C may be the first area and the area of ​​interest. Also, partial areas of customer C's face, such as the forehead or cheeks, may be the first area and the area of ​​interest. Similarly, the second area is not limited to the second facial area FA2 of basketball player P or actor AC. The entire body of basketball player P or actor AC, or even a partial area of ​​the face, may be the second area.

[0158] When multiple types of specific subjects, such as characters and people, are mixed in one synthesis template 13, a main subject is identified from the multiple types of specific subjects, and a second region of the main subject is identified. A semantic segmentation model is used to identify the main subject, and specific subjects, such as people, characters, and animals, appearing in the synthesis template 13 are classified. Priorities are set for each type of specific subject, and the specific subject with the highest priority among the classified specific subjects is designated as the main subject. Alternatively, the main subject may be identified based on the size and position of each specific subject, such as by designating the largest specific subject as the main subject or the specific subject closest to the center. An evaluation value using the size and position of the specific subject as parameters may be calculated for each specific subject, and the specific subject with the highest evaluation value may be designated as the main subject.

[0159] Alternatively, without identifying a main subject, second index values ​​may be derived for multiple types of specific subjects, and a representative value such as a weighted average may be used as the final second index value. The weighting coefficient in the weighted average is a value based on the priority set for each type of specific subject. The weighting coefficients are normalized so that the sum of all weighting coefficients equals 1. If multiple specific subjects of the same type, such as two basketball players P, are mixed in the synthesis template 13, second index values ​​may be derived for each of the multiple specific subjects, and a representative value such as the average may be used as the final second index value. If there are differences in brightness and / or saturation among multiple specific subjects of the same type, processing may be performed to reduce the differences in brightness and / or saturation among the multiple specific subjects of the same type before deriving the second index value.

[0160] Similarly, when multiple arbitrary subjects such as a couple, a parent and child, or friends appear in the captured image 14, a main subject is identified from the multiple arbitrary subjects, and a first region of the main subject is identified to derive a first index value. Alternatively, a first index value may be derived for each of the multiple arbitrary subjects, and a representative value such as the average value of these values ​​may be used as the final first index value.

[0161] For example, if the image quality of the specific subjects appearing in the multiple compositing templates 13 is considered to be roughly the same, such as when the specific subjects appearing in the multiple compositing templates 13 are the same actor AC but in different poses, a second index value is derived for each of the specific subjects appearing in the multiple compositing templates 13. Then, a representative value such as the average value of these values ​​may be used as the second index value common to all of the compositing templates 13.

[0162] The hardware configuration of the computer constituting the image processing server 11 can be modified in various ways. For example, the image processing server 11 can be configured with multiple computers separated as hardware in order to improve processing power and reliability. For example, the functions of the request receiving unit 50, RW control unit 51, and screen distribution control unit 52, and the functions of the image quality correction processing unit 53, composite image generation unit 54, and transmission unit 55 can be distributed and performed by two computers. In this case, the image processing server 11 is configured with two computers. Also, all or part of the functions of the image processing server 11 may be performed by the photographing device 10.

[0163] In this way, the hardware configuration of the computer of the image processing server 11 can be changed as appropriate depending on the required performance such as processing power, safety, reliability, etc. Furthermore, in addition to the hardware, application programs such as the operating program 45 can also be duplicated or stored in a distributed manner across multiple storage devices in order to ensure safety and reliability.

[0164] In each of the above embodiments, the hardware structure of processing units that perform various processes, such as the request receiving unit 50, the RW control unit 51, the screen distribution control unit 52, the image quality correction processing unit 53, the composite image generation units 54 and 132, the transmission unit 55, the Lab conversion unit 60, the first area identification unit 61, the first index value derivation units 62, 113, and 122, the brightness correction processing units 63 and 105, the saturation correction processing units 64 and 106, the RGB conversion unit 65, the instruction receiving unit 88, the lighting correction processing unit 110, the noise amount correction processing unit 120, and the resolution correction processing unit 125, can be formed using the various processors shown below. The various processors include the CPU 37, which is a general-purpose processor that executes software (operating program 45) and functions as various processing units, as well as programmable logic devices (PLDs), which are processors whose circuit configuration can be changed after manufacture, such as an FPGA (Field Programmable Gate Array), and / or dedicated electrical circuits, such as an ASIC (Application Specific Integrated Circuit), which are processors having a circuit configuration designed exclusively for executing specific processing.

[0165] A single processing unit may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (e.g., a combination of multiple FPGAs and / or a combination of a CPU and an FPGA).Furthermore, multiple processing units may be configured with a single processor.

[0166] Examples of configuring multiple processing units with a single processor include, first, a form in which one processor is configured with a combination of one or more CPUs and software, as typified by computers such as client and server, and this processor functions as multiple processing units. Second, a form in which a processor is used to realize the functions of the entire system including multiple processing units with a single IC (Integrated Circuit) chip, as typified by systems on chips (SoCs). In this way, various processing units are configured using one or more of the above-mentioned various processors as a hardware structure.

[0167] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit (circuitry) that combines circuit elements such as semiconductor elements.

[0168] From the above description, the technology described in the following supplementary paragraphs can be understood.

[0169] [Supplementary Item 1] An image processing device including a processor, wherein the processor acquires a compositing template, acquires a captured image including an arbitrary subject, performs image quality correction processing to reduce a difference between the image quality of the compositing template and the image quality of the captured image by correcting the image quality of the captured image, and generates a composite image by combining the captured image that has undergone the image quality correction processing with the compositing template. [Supplementary Item 2] The image processing device according to Supplementary Item 1, wherein the processor identifies a first region related to the arbitrary subject from the captured image, derives a first index value for the first region, and performs the image quality correction processing according to the first index value. [Supplementary Item 3] The image processing device according to Supplementary Item 2, wherein the first region is a region of interest. [Supplementary Item 4] The image processing device according to Supplementary Item 3, wherein the region of interest is a human face region. [Supplementary Item 5] The image processing device according to any one of Supplementary Items 2 to 4, wherein the first index value is derived using a statistical method. [Supplementary Item 6] The image processing device according to any one of Supplementary Items 2 to 5, wherein the first index value is derived after outlier removal processing has been performed. [Supplementary Item 7] The image processing device according to any one of Supplementary Items 2 to 6, further comprising correction reference information having registered therein a correction amount corresponding to the first index value, wherein the processor acquires the correction amount corresponding to the derived first index value from the correction reference information, and performs the image quality correction processing using the acquired correction amount. [Supplementary Item 8] The image processing device according to Supplementary Item 7, wherein the processor receives an instruction to modify the correction amount in the correction reference information, and modifies the correction amount in the correction reference information in accordance with the modification instruction. [Supplementary Item 9] The image processing device according to Supplementary Item 7 or Supplementary Item 8, wherein the first index value in the correction reference information and the correction amount have a nonlinear relationship. [Supplementary Item 10] The image processing device according to any one of Supplementary Items 7 to 9, wherein the correction reference information is registered with the correction amount whose absolute value gradually increases as the first index value approaches a maximum or minimum value.[Supplementary Item 11] The image processing device according to Supplementary Item 1, wherein the processor extracts a first region relating to the arbitrary subject from the captured image, derives a first index value for the first region, and performs the image quality correction processing according to the first index value and a second index value for a second region relating to the specific subject in the synthesis template. [Supplementary Item 12] The image processing device according to Supplementary Item 11, wherein the second region is a region related to the first region. [Supplementary Item 13] The image processing device according to Supplementary Item 11 or Supplementary Item 12, wherein the first region and the second region are human face regions. [Supplementary Item 14] The image processing device according to any one of Supplementary Items 11 to 13, wherein the first index value and the second index value are derived by a statistical method. [Supplementary Item 15] The image processing device according to any one of Supplementary Items 11 to 14, wherein the first index value and the second index value are derived after performing an outlier removal process. [Supplementary Item 16] The image processing device described in any one of Supplementary Items 11 to 15, wherein the processor performs a lighting correction process as the image quality correction process to bring the lighting of the arbitrary subject closer to the lighting of the specific subject, and in the lighting correction process, the direction and intensity of the light source of light irradiated onto the arbitrary subject in the first region are obtained as the first index value, and the direction and intensity of the light source of light irradiated onto the specific subject in the second region are obtained as the second index value. [Supplementary Item 17] The image processing device according to any one of Supplementary Items 1 to 16, wherein the processor performs, as the image quality correction processing, at least one of a brightness correction processing for making the brightness of the arbitrary subject closer to the brightness of a specific subject in the synthesis template, a saturation correction processing for making the saturation of the arbitrary subject closer to the saturation of the specific subject, a lighting correction processing for making the lighting of the arbitrary subject closer to the lighting of the specific subject, a noise amount correction processing for making the amount of noise on the arbitrary subject closer to the amount of noise on the specific subject, and a resolution correction processing for making the resolution of the arbitrary subject closer to the resolution of the specific subject.[Supplementary Item 18] The image processing device according to Supplementary Item 17, wherein the processor performs the brightness correction process, the saturation correction process, and the lighting correction process, and performs the lighting correction process after performing the brightness correction process and the saturation correction process. [Supplementary Item 19] The image processing device according to Supplementary Item 17 or Supplementary Item 18, wherein the processor performs the brightness correction process, the saturation correction process, and the resolution correction process, and performs the brightness correction process and the saturation correction process after performing the resolution correction process. [Supplementary Item 20] The image processing device according to any one of Supplementary Items 1 to 19, wherein a limit value is set for the amount of correction in the image quality correction process. [Supplementary Item 21] The image processing device according to Supplementary Item 20, wherein an upper limit value and a lower limit value are set for the amount of correction. [Supplementary Item 22] The image processing device described in any one of Supplementary Items 1 to 21, wherein the processor acquires depth information indicating a front-to-back relationship between the arbitrary subject and a specific subject of the synthesis template, and generates the synthetic image in which the arbitrary subject and the specific subject have a front-to-back relationship according to the depth information.

[0170] The technology of the present disclosure can be appropriately combined with the various embodiments and / or various modified examples described above. Furthermore, it is not limited to the above-described embodiments, and various configurations can be adopted without departing from the spirit of the present disclosure. Furthermore, the technology of the present disclosure extends not only to programs, but also to storage media that non-temporarily store programs, and computer program products that include programs.

[0171] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[0172] In this specification, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0173] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

Claims

1. An image processing device comprising a processor, which acquires a synthesis template, acquires a captured image including an arbitrary subject, performs image quality correction processing to reduce the difference between the image quality of the synthesis template and the image quality of the captured image by correcting the image quality of the captured image, and synthesizes the captured image that has been subjected to the image quality correction processing with the synthesis template to generate a composite image.

2. The image processing device described in claim 1, wherein the processor identifies a first area relating to the arbitrary subject from the captured image, derives a first index value for the first area, and performs the image quality correction processing according to the first index value.

3. The image processing device according to claim 2, wherein the first region is a region of interest.

4. The image processing device according to claim 3, wherein the region of interest is a human face region.

5. The image processing device according to claim 2, wherein the first index value is derived by a statistical method.

6. The image processing device according to claim 2, wherein the first index value is derived after performing processing to remove outliers.

7. An image processing device as described in claim 2, which has correction reference information in which a correction amount corresponding to the first index value is registered, and the processor obtains the correction amount corresponding to the derived first index value from the correction reference information, and performs the image quality correction process using the obtained correction amount.

8. An image processing device according to claim 7, wherein the processor receives an instruction to modify the correction amount of the correction reference information, and modifies the correction amount of the correction reference information in accordance with the modification instruction.

9. The image processing device according to claim 7, wherein the first index value of the correction reference information and the correction amount have a nonlinear relationship.

10. An image processing device according to claim 7, wherein the correction reference information is registered with the correction amount whose absolute value gradually increases as the first index value approaches the maximum or minimum value.

11. The image processing device described in claim 1, wherein the processor extracts a first region relating to the arbitrary subject from the captured image, derives a first index value for the first region, and performs the image quality correction process according to the first index value and a second index value for a second region relating to the specific subject in the synthesis template.

12. The image processing device according to claim 11, wherein the second region is a region related to the first region.

13. The image processing device according to claim 11, wherein the first area and the second area are human face areas.

14. The image processing device according to claim 11, wherein the first index value and the second index value are derived by a statistical method.

15. The image processing device according to claim 11, wherein the first index value and the second index value are derived after performing a process of excluding outliers.

16. The image processing device described in claim 11, wherein the processor performs a lighting correction process as the image quality correction process to bring the lighting of the arbitrary subject closer to the lighting of the specific subject, and in the lighting correction process, the direction and intensity of the light source of light irradiated onto the arbitrary subject in the first area are obtained as the first index value, and the direction and intensity of the light source of light irradiated onto the specific subject in the second area are obtained as the second index value.

17. The image processing device of claim 1, wherein the processor performs, as the image quality correction process, at least one of the following: brightness correction process for bringing the brightness of the arbitrary subject closer to the brightness of a specific subject in the synthesis template; saturation correction process for bringing the saturation of the arbitrary subject closer to the saturation of the specific subject; lighting correction process for bringing the lighting intensity of the arbitrary subject closer to the lighting intensity of the specific subject; noise amount correction process for bringing the amount of noise on the arbitrary subject closer to the amount of noise on the specific subject; and resolution correction process for bringing the resolution of the arbitrary subject closer to the resolution of the specific subject.

18. The image processing device according to claim 17, wherein the processor performs the brightness correction process, the saturation correction process, and the lighting correction process, and performs the lighting correction process after performing the brightness correction process and the saturation correction process.

19. The image processing device described in claim 17, wherein the processor performs the brightness correction process, the saturation correction process, and the resolution correction process, and performs the brightness correction process and the saturation correction process after performing the resolution correction process.

20. The image processing device according to claim 1, wherein a limit value is set for the amount of correction in the image quality correction process.

21. The image processing device according to claim 20, wherein an upper limit and a lower limit are set for the correction amount.

22. The image processing device described in claim 1, wherein the processor acquires depth information indicating the front-to-back relationship between the arbitrary subject and a specific subject in the synthesis template, and generates the synthetic image in which the arbitrary subject and the specific subject have a front-to-back relationship in accordance with the depth information.

23. A method for operating an image processing device, comprising: acquiring a synthesis template; acquiring a captured image including an arbitrary subject; performing image quality correction processing to reduce the difference between the image quality of the synthesis template and the image quality of the captured image by correcting the image quality of the captured image; and synthesizing the captured image that has been subjected to the image quality correction processing with the synthesis template to generate a composite image.

24. An operating program for an image processing device that causes a computer to execute processes including: acquiring a synthesis template; acquiring a captured image including an arbitrary subject; performing image quality correction processing to reduce the difference between the image quality of the synthesis template and the image quality of the captured image by correcting the image quality of the captured image; and synthesizing the captured image that has been subjected to the image quality correction processing with the synthesis template to generate a composite image.

Images