Image overlay system and image overlay method
By generating and recognizing trimmed face and superimposed object regions in the image superimposition system, the system improves facial image recognition accuracy and ensures accurate superimposition of images, addressing the challenge of positional changes in users.
Patent Information
- Application Number
- JP2023531173
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2021-06-28
- Publication Date
- 2024-12-04
- Estimated Expiration
- 2041-06-28
AI Technical Summary
Conventional image superimposition systems face challenges in maintaining recognition accuracy of facial images and types of shading when the user changes their position, leading to issues with superimposing images accurately on the face.
The system generates face area and superimposed object region images by trimming specific areas in the captured image, uses face and superimposed object recognition devices to identify these regions, and synthesizes a composite image that can track the user's face over a larger area, improving recognition accuracy and ensuring the superimposed image aligns correctly.
This approach enhances the recognition accuracy of facial images and types of shading, allowing the superimposed image to be accurately superimposed on the face even when the user changes position, particularly useful for applications like Kabuki makeup simulation.
Smart Images

Figure 0007594218000001 
Figure 0007594218000002 
Figure 0007594218000003
Abstract
Description
[Technical field]
[0001] The present disclosure relates to an image overlay system and an image overlay method. [Background technology]
[0002] 2. Description of the Related Art Conventionally, there is known a technique for separating an image of an object (foreground in which an object is captured) from a background in an image captured by a camera (Patent Document 1).
[0003] In addition, a technology called Augmented Reality (AR) overlay is known, which displays an image of a subject captured by a camera by superimposing another image on it, and a service called "Henshin Kabuki" using AR overlay technology has been proposed (Non-Patent Documents 1 and 2).
[0004] The synthesis device used in the transformation kabuki service acquires a face image from an image captured of the user, and recognizes the type of kumadori from an image captured of the kumadori mask. The image synthesis device then displays a kumadori image showing the makeup with the kumadori recognized in the preparation stage superimposed on the face image in the image captured of the user. At this time, the image synthesis device makes the kumadori image track the change in the position of the face image in the image.
[0005] Furthermore, when the facial expression of the person captured by the camera changes, the image of the face changes, and the image synthesis device can deform the shading image to be superimposed based on the change. For example, when the person opens his / her mouth, the synthesis device deforms the shading image so that the part corresponding to the mouth opens. Also, for example, when the person changes from facing the camera to facing the left and the person's right cheek faces the camera, the synthesis device superimposes the shading image as observed from the right cheek side.
[0006] In this way, the image synthesis device displays the face of the service user as if the user had applied kumadori makeup to the face of the user, so that the user can easily experience kumadori makeup without actually applying kumadori makeup to his or her own face. [Prior art documents] [Patent documents]
[0007] [Patent Document 1] JP 2020-17136 A [Non-patent literature]
[0008] [Non-Patent Document 1] Usui, et al., "Kabuki x ICT Collaboration," NTT Technical Journal, November 2018 issue, p. 26, [online], [Retrieved June 10, 2021], Internet<URL:https: / / journal.ntt.co.jp / article / 3992> [Non-Patent Document 2] “The interactive exhibit “Henshin Kabuki,” which was well received in Las Vegas, USA, will be unveiled in Japan for the first time at Tokyo Edo Week | News Release | NTT,” [online], [searched June 10, 2021], Internet<URL:https: / / group.ntt / jp / newsrelease / 2016 / 09 / 20 / 160920a.html> Summary of the Invention [Problem to be solved by the invention]
[0009] In the above-mentioned conventional technology, the area in the captured image where the synthesis device executes processing to recognize the face image, the area where the device executes processing to recognize the type of shading, and the area where the face image is recognized to superimpose the superimposed image are the same. Therefore, if these areas are enlarged, the recognition accuracy of the face image and the type of shading decreases, and if these areas are reduced, when the user changes the position of the face, the face image is positioned outside the area, and therefore the superimposed image cannot be superimposed on the face image.
[0010] An object of the present disclosure, made in consideration of such circumstances, is to provide an image overlay system and an image overlay method that can improve the accuracy of recognizing facial images and types of shading, while overlaying a superimposed image on a facial image even if the user changes the position of their face. [Means for solving the problem]
[0011] In order to solve the above problem, the image overlay system of the present disclosure includes a control device that generates face area image information indicating a face area image obtained by cropping a face area in an image captured by an imaging device of a user in a first stage, and generates superimposition area image information indicating a superimposition area image obtained by cropping a superimposition area in an image captured by the imaging device of a superimposition object in the first stage; a face recognition device that recognizes a face image of the user included in the face area image and generates face image information indicating the face image; a superimposition recognition device that recognizes a superimposition image equivalent to the superimposition area image; and a synthesis device that detects the face image of the user from a synthesis area image, which is an image of the face area and a synthesis area larger than the superimposition area in an image captured by the imaging device of the user in a second stage after the first stage, based on the face image information, and generates synthetic image information indicating a synthetic image obtained by superimposing the superimposition image on the face image.
[0012] In addition, in order to solve the above problem, the image superimposition method according to the present disclosure includes the steps of generating face area image information indicating a face area image obtained by cropping a face area in an image captured by an imaging device of a user in a first stage, generating superimposition area image information indicating a superimposition area image obtained by cropping a superimposition area in an image captured by the imaging device of a superimposition object in the first stage, recognizing a face image of the user contained in the face area image and generating face image information indicating the face image, recognizing a superimposition image equivalent to the superimposition area image, detecting a face image of the user based on the face image information from a composite area image, which is an image of the face area and a composite area larger than the superimposition area in an image captured by the imaging device of the user in a second stage after the first stage, and generating composite image information indicating a composite image obtained by superimposing the superimposition image on the face image. Effect of the Invention
[0013] According to the image overlay system and image overlay method disclosed herein, it is possible to improve the accuracy of recognizing face images and types of shading, while overlaying a superimposed image on a face image even if the user changes the position of the face. [Brief description of the drawings]
[0014] [Figure 1] 1 is a schematic diagram of an image overlay system according to a first embodiment. [Figure 2A] FIG. 2 is a diagram showing a face area in a captured image. [Figure 2B] FIG. 13 is a diagram showing an overlapping object region in a captured image. [Figure 3A] FIG. 2C is a diagram showing another example of the mask shown in FIG. 2B. [Figure 3B] FIG. 2C is a diagram showing yet another example of the mask shown in FIG. 2B. [Figure 4] 2 is a diagram showing an example of information stored in a region information storage unit shown in FIG. 1; FIG. [Diagram 5] FIG. 2 is a diagram showing a synthesis area in a captured image. [Figure 6A] 2 is a sequence diagram showing a part of an example of an operation in the image overlay system shown in FIG. [Figure 6B] 1. FIG. 4 is a sequence diagram showing the remaining part of the example of the operation in the image overlay system shown in FIG. [Figure 7] FIG. 11 is a schematic diagram of an image overlay system according to a second embodiment. [Figure 8A] 8 is a sequence diagram showing a part of an example of the operation of the image overlay system shown in FIG. 7. [Figure 8B] 8 is a sequence diagram showing the remaining part of the example of the operation in the image overlay system shown in FIG. 7. FIG. [Figure 9] FIG. 2 is a hardware block diagram of a control device and a synthesis device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0015] <<First embodiment>> The overall configuration of the first embodiment will be described with reference to Fig. 1. Fig. 1 is a schematic diagram of an image overlay system 100 according to this embodiment.
[0016] 1, an image superimposing system 100 according to the first embodiment includes an image capturing device 1, a face recognition device 2, a superimposed object recognition device 3, a control device 4, a synthesis device 5, and a display device 6. The control device 4 communicates with each of the image capturing device 1, the face recognition device 2, the superimposed object recognition device 3, and the synthesis device 5 via a communication network. The synthesis device 5 communicates with the display device 6 via the communication network.
[0017] <Configuration of imaging device> The imaging device 1 is configured with a camera having an optical element, an imaging element, a communication interface, etc. For the communication interface, standards such as Ethernet (registered trademark), FDDI (Fiber Distributed Data Interface), and Wi-Fi (registered trademark) may be used.
[0018] In a preparation stage (first stage), the imaging device 1 generates captured image information indicating a captured image of a user of the image superimposing system 100 as shown in FIG. 2A. The captured image here may be a still image. At this time, a mechanism may be provided for adjusting the position of the user's face in real space so that the face image is included in a face area R1, which will be described in detail later. Specifically, the imaging device 1 captures an image of at least a part of the user including the user's face. The at least a part of the user may be the upper half of the user's body including the user's face. Furthermore, the at least a part of the user may be the user's entire body including the user's face.
[0019] Moreover, the imaging device 1 generates imaging image information indicating an imaging image obtained by imaging an overlapping object arranged in a predetermined area in a preparation stage. The predetermined area may be, for example, an area between a face and the imaging device. Moreover, the imaging image here may be a still image. For example, the imaging device 1 may generate imaging image information indicating an imaging image obtained by imaging an overlapping object together with a user in a state in which the overlapping object is arranged between the user's face and the imaging device 1 as shown in FIG. 2B. At this time, a mechanism may be provided for adjusting the position of the overlapping object in real space so that the image of the overlapping object is included in an overlapping object area R2, which will be described in detail later. The overlapping object is an object in real space corresponding to an image that the synthesis device 5 overlaps on the image of the user's face imaged by the imaging device 1. In the example shown in FIG. 2B, the overlapping object is a mask MK applied with makeup by kumadori used in kabuki. The mask MK is not limited to the example shown in FIG. 2B, and may be a mask MK as shown in FIG. 3A and FIG. 3B, or any other mask MK.
[0020] Furthermore, the imaging device 1 generates captured image information indicating a captured image of the user in a service provision stage (second stage) after the preparation stage. The captured image here may be a moving image. In the service provision stage, the user captured by the imaging device 1 is the user captured by the imaging device 1 in the preparation stage.
[0021] Furthermore, the imaging device 1 transmits captured image information to the control device 4 via the communication network.
[0022] <Configuration of face recognition device> 1 is configured by a computer having a control unit (controller), a communication interface, etc. The control unit may be configured by dedicated hardware such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array), or may be configured by a processor, or may be configured to include both.
[0023] The face recognition device 2 receives face area image information, which will be described in detail later, transmitted from the control device 4. Then, the face recognition device 2 recognizes the face image of the user included in the face area image indicated by the face area image information, and generates face image information indicating the face image. The method by which the face recognition device 2 recognizes the face image may be any method. Furthermore, the face recognition device 2 transmits the face image information to the control device 4.
[0024] <Configuration of the overlapping object recognition device> As shown in Fig. 1, the superimposed object recognition device 3 includes a communication unit 31, a superimposed image information storage unit 32, and a recognition unit 33. The communication unit 31 is configured with a communication interface that transmits and receives information to and from the outside. The superimposed image information storage unit 32 is configured with memories such as a hard disk drive (HDD), a solid state drive (SSD), an electrically erasable programmable read-only memory (EEPROM), a read-only memory (ROM), and a random access memory (RAM). The recognition unit 33 constitutes a control unit.
[0025] The communication unit 31 receives superimposed object area image information, which will be described in detail later, transmitted from the control device 4. In addition, the communication unit 13 transmits a superimposed image ID (Identification) (superimposed image identifier), which will be described in detail later, to the control device 4.
[0026] The superimposed image information storage unit 32 stores the superimposed image in association with the superimposed image ID. The superimposed image is an image that is superimposed on the captured image of the user by the imaging device 1 during the service provision stage. The superimposed image ID is an identifier for identifying the superimposed image. As described above, in the example where the superimposed object is a mask MK made up with kumadori makeup used in Kabuki, the superimposed image is an image of the makeup part of the mask MK made up with kumadori makeup.
[0027] The recognition unit 33 recognizes a superimposed image corresponding to the superimposed object area image indicated by the superimposed object area image information. At this time, the recognition unit 33 may recognize a superimposed image corresponding to the superimposed object area image indicated by the superimposed object area image information from among the superimposed images stored in the superimposed image information storage unit 32. Then, the recognition unit 33 extracts a superimposed image ID stored in the superimposed image information storage unit 32 corresponding to the recognized superimposed image. Furthermore, the recognition unit 33 controls the communication unit 31 to transmit the superimposed image ID to the control device 4.
[0028] <Control device configuration> The control device 4 includes a communication unit 41, an area information storage unit 42, a superimposed image information storage unit 43, a trimming unit 44, and a scenario control unit 45. The communication unit 41 is configured by a communication interface. The area information storage unit 42 and the superimposed image information storage unit 43 are configured by memories. The trimming unit 44 and the scenario control unit 45 constitute a control unit.
[0029] The communication unit 41 receives captured image information from the imaging device 1. The communication unit 41 also receives face image information from the face recognition device 2. The communication unit 41 also receives a superimposed image ID from the superimposed object recognition device 3. The communication unit 41 also transmits information to the synthesis device 5 under the control of the scenario control unit 45.
[0030] In the area information storage unit 42, as shown in FIG. 4, area IDs (area identifiers) for identifying areas in a captured image and information indicating the range of the areas are stored in association with each other. The areas include a face area R1, a superimposed object area R2, and a composite area R3, and each area is a predetermined area. The face area R1 is an area in a captured image in which a user is captured in a preparation stage, as shown in FIG. 2A, and is an area expected to include a face image of the user. The superimposed object area R2 is an area in a captured image in which a superimposed object is captured in a preparation stage, as shown in FIG. 2B, and is an area expected to include an image of the superimposed object. The composite area R3 is an area in a captured image in which a user is captured in a service provision stage, as shown in FIG. 5, and is an area expected to include a face image even if the user changes the position of his or her face. The composite area R3 is larger than the face area R1 and the superimposed object area R2. The composite area R3 can be, for example, an area including a whole-body image showing the whole body of the user.
[0031] In the example shown in FIG. 4, the information indicating the range of the region is the starting point coordinate, and the X size and the Y size. Specifically, the face region R1 is identified by the region ID "1" and is a region surrounded by a rectangle having a length LX1 in the X-axis direction and a length LY1 in the Y-axis direction from the starting point coordinate (X1, Y1). The superimposed object region R2 is identified by the region ID "2" and is a region surrounded by a rectangle having a length LX2 in the X-axis direction and a length LY2 in the Y-axis direction from the starting point coordinate (X2, Y2). The superimposed region is identified by the region ID "3" and is a region surrounded by a rectangle having a length LX3 in the X-axis direction and a length LY3 in the Y-axis direction from the starting point coordinate (X3, Y3).
[0032] The superimposed image information storage unit 43, like the superimposed image information storage unit 32 of the superimposed object recognition device 3, stores a superimposed image and a superimposed image ID in association with each other.
[0033] The trimming unit 44 generates face area image information indicating a face area image obtained by trimming the face area R1 in the captured image captured by the imaging device 1 in the preparation stage. Specifically, the trimming unit 44 may acquire face area information indicating the face area R1 in the captured image from the area information storage unit 42. In the example shown in FIG. 4, the trimming unit 44 acquires face area information indicated by the start point coordinates corresponding to the area ID "1" stored in the area information storage unit 42, as well as the X size and the Y size. Then, the trimming unit 44 extracts the face area R1 indicated by the face area information in the captured image, and generates face area image information indicating an image of the face area R1.
[0034] The trimming unit 44 also generates superimposed object region image information indicating a superimposed object region image obtained by trimming the superimposed object region R2 in the captured image in which the imaging device 1 captured the superimposed object in the preparation stage. Specifically, the trimming unit 44 may acquire superimposed object region information indicating the superimposed object region R2 in the captured image from the region information storage unit 42. In the example shown in FIG. 4, the trimming unit 44 acquires superimposed object region information indicated by the start point coordinates corresponding to the region ID "2" stored in the region information storage unit 42, as well as the X size and the Y size. Then, the trimming unit 44 extracts the superimposed object region R2 indicated by the superimposed object region information in the captured image, and generates superimposed object region image information indicating an image of the superimposed object region.
[0035] The trimming unit 44 controls the communication unit 41 to transmit the face region image information to the face recognition device 2. In addition, the trimming unit 44 controls the communication unit 41 to transmit the overlapping object region image information to the overlapping object recognition device 3.
[0036] The scenario control unit 45 extracts superimposed image information stored in the superimposed image information storage unit 43 corresponding to the superimposed image ID recognized by the superimposed object recognition device 3. The scenario control unit 45 also extracts compositing area information indicating compositing area R3 corresponding to area ID "3" stored in the area information storage unit 42. In the example shown in Fig. 5, the compositing area information is the starting point coordinates (X3, Y3), as well as the X size "LX3" and the Y size "LY3".
[0037] Then, the scenario control unit 45 controls the communication unit 41 to transmit to the synthesis device 5 the face image information, the superimposition image information, the synthesis area information, and the captured image information generated by the imaging device 1 in the service provision stage.
[0038] <Configuration of synthesis equipment> 1, the synthesis device 5 includes a communication unit 51, a face image detection unit 52, and a superimposition unit 53. The communication unit 51 is configured with a communication interface. The face image detection unit 52 and the superimposition unit 53 configure a control unit.
[0039] The communication unit 51 receives face image information, superimposed image information, synthesis area information, and captured image information generated by the imaging device 1 at the service provision stage from the control device 4. The communication unit 51 also transmits synthetic image information, which will be described in detail later, to the display device 6.
[0040] The facial image detection unit 52 detects the facial image of the user from a composite area image, which is an image of a composite area R3 larger than the facial area R1 and the overlap area R2 in an image captured by the imaging device 1 during the service provision stage, based on facial image information.
[0041] The superimposing unit 53 generates composite image information indicating a composite image obtained by superimposing the superimposing image on the face image. Specifically, the superimposing unit 53 superimposes the superimposing image indicated by the superimposing image information on the face image detected by the face image detecting unit 52 in the captured image indicated by the captured image information. Then, the superimposing unit 53 generates composite image information indicating a composite image obtained by superimposing the superimposing image on the face image.
[0042] The superimposing unit 53 may also extract feature parameters from the face image detected from the composite area image. The feature parameters are parameters indicating features of the face image, such as the direction in which the face is facing, the facial expression, and the like. In such a configuration, the superimposing unit 53 may generate composite image information indicating a composite image in which the superimposed images are superimposed, according to the feature parameters. For example, when the feature parameters indicate that the left side of the face faces the imaging surface of the imaging device 1, the superimposing unit 53 may superimpose a superimposed image corresponding to the left side of the face. Furthermore, in a configuration in which the captured image is a moving image, the superimposing unit 53 may change the superimposed image to be superimposed on the face image, according to changes in the feature parameters of the face images in each of the multiple still images constituting the moving image.
[0043] Moreover, the superimposing unit 53 controls the communication unit 51 to transmit the composite image information to the display device 6 .
[0044] <Display device configuration> The display device 6 is configured by a device having a display interface, a communication interface, etc. The display interface may be configured by an organic EL (Electro Luminescence) panel, a liquid crystal panel, or the like.
[0045] The display device 6 receives the composite image information transmitted from the synthesis device 5. Then, the display device 6 displays the composite image indicated by the composite image information.
[0046] <Image overlay system operation> Here, the operation of the image overlay system 100 according to the first embodiment will be described with reference to Figures 6A and 6B. Figures 6A and 6B are sequence diagrams showing an example of the operation of the image overlay system 100 according to this embodiment. The operation of the image overlay system 100 described with reference to Figures 6A and 6B corresponds to an example of an image overlay method of the image overlay system 100 according to the first embodiment.
[0047] As shown in FIG. 6A, first, the image superimposing system 100 executes preparation processing in steps S11 to S19, which is a preparation stage.
[0048] In step S11, the imaging device 1 generates captured image information indicating a captured image of the user.
[0049] In step S12, the imaging device 1 transmits captured image information to the control device 4.
[0050] In step S13, the trimming unit 44 of the control device 4 acquires, from the region information storage unit 42, face region information indicating the face region R1 in the captured image.
[0051] In step S14, the trimming unit 44 of the control device 4 uses the captured image information transmitted in step S12 to generate face area image information indicating a face area image obtained by trimming the face area R1 in the captured image of the user captured by the imaging device 1.
[0052] In step S15, the trimming unit 44 of the control device 4 controls the communication unit 41 to transmit the facial area image information to the face recognition device 2. As a result, the communication unit 41 transmits the facial area image information to the face recognition device 2.
[0053] In step S16, the face recognition device 2 recognizes a face image included in the image indicated by the face area image information, and generates face image information indicating the face image.
[0054] In step S17, the face recognition device 2 transmits the face image information to the control device 4.
[0055] In step S18, the imaging device 1 generates captured image information indicating a captured image of the superimposed object. At this time, the imaging device 1 may generate captured image information indicating a captured image of the superimposed object together with the user in a state in which the superimposed object is disposed between the user's face and the imaging device 1.
[0056] In step S19, the imaging device 1 transmits the captured image information to the control device 4.
[0057] In step S20, the trimming unit 44 of the control device 4 acquires, from the region information storage unit 42, overlapping object region information indicating the overlapping object region R2 in the captured image.
[0058] In step S21, the trimming unit 44 of the control device 4 uses the captured image information transmitted in step S19 to generate overlap area image information indicating an overlap area image obtained by trimming the overlap area R2 in the captured image in which the imaging device 1 captured the overlap area.
[0059] In step S22, the trimming unit 44 of the control device 4 controls the communication unit 41 to transmit the overlapping object region image information to the overlapping object recognition device 3. As a result, the communication unit 41 transmits the overlapping object region image information to the overlapping object recognition device 3.
[0060] In step S23, the recognition unit 33 of the superimposed object recognition device 3 recognizes a superimposed image corresponding to the superimposed object area image indicated by the superimposed object area image information. The recognition unit 33 may recognize a superimposed image corresponding to the superimposed object area image indicated by the superimposed object area image information from among the superimposed images stored in the superimposed image information storage unit 32.
[0061] In step S24, the recognition unit 33 of the superimposed object recognition device 3 extracts the superimposed image ID stored in the superimposed image information storage unit 32 in correspondence with the recognized superimposed image.
[0062] In step S25, the recognition unit 33 of the superimposed object recognition device 3 controls the communication unit 31 to transmit the superimposed image ID to the control device 4. As a result, the communication unit 31 transmits the foldable image ID to the control device 4.
[0063] After the preparation stage from steps S11 to S25 as described above, the image superimposing system 100 executes a service providing process in steps S26 to S31 which is a service providing stage.
[0064] In step S26, the imaging device 1 generates captured image information indicating a captured image of the user.
[0065] In step S27, the imaging device 1 transmits the captured image information to the control device 4.
[0066] As shown in FIG. 6B, in step S28, the scenario control unit 45 of the control device 4 extracts the superimposed image information stored in the superimposed image information memory unit 43 corresponding to the superimposed image ID recognized by the superimposed object recognition device 3.
[0067] In step S29, the scenario control unit 45 of the control device 4 acquires, from the area information storage unit 42, synthesis area information indicating the synthesis area R3 within the captured image.
[0068] In step S30, the scenario control unit 45 of the control device 4 controls the communication unit 41 to transmit the face image information, the superimposed image information, the synthesis area information, and the captured image information generated by the imaging device 1 in the service provision stage to the synthesis device 5. As a result, the communication unit 41 transmits the face image information, the superimposed image information, the synthesis area information, and the captured image information generated by the imaging device 1 in the service provision stage to the synthesis device 5.
[0069] In step S31, the facial image detection unit 52 of the synthesis device 5 uses the captured image information to detect the facial image of the user from the synthesis area image, which is an image of a synthesis area R3 larger than the facial area R1 in the captured image of the user captured by the imaging device 1 during the service provision phase, based on the facial image information.
[0070] In step S32, the superimposing unit 53 of the synthesizing device 5 generates synthesized image information indicating an image in which the superimposing image is superimposed on the face image.
[0071] In step S33, the superimposing unit 53 of the synthesizing device 5 controls the communication unit 51 to transmit the synthetic image information to the display device 6. As a result, the communication unit 51 transmits the synthetic image information to the display device 6.
[0072] In step S34, the display device 6 displays the composite image indicated by the composite image information.
[0073] As described above, according to the first embodiment, the image superimposing system 100 generates face area image information indicating a face area image obtained by trimming a face area R1 in a captured image of a user captured by the imaging device 1 in the preparation stage, and generates superimposed object area image information indicating a superimposed object area image obtained by trimming a superimposed object area R2 in a captured image of a superimposed object captured by the imaging device 1 in the preparation stage. Then, the image superimposing system 100 recognizes a face image of the user included in the face area image, generates face image information indicating the face image, recognizes a superimposed image corresponding to the superimposed object area image, and extracts a superimposed image ID for identifying the superimposed image. Furthermore, the image superimposing system 100 recognizes a face image of the user based on face image information from a superimposed area image, which is an image of a superimposed area R3 larger than the face area R1 and the superimposed object area R2 in a captured image of a user captured by the imaging device 1 in a service provision stage after the preparation stage, and generates superimposed image information indicating a superimposed image obtained by superimposing a superimposed image identified by a superimposed image ID on the face image. Therefore, the image overlay system 100 can track the user's face image over a wide range in the captured image and overlay the overlay image while improving the recognition accuracy of the face image and the type of kumadori. In particular, when the overlay is makeup with kumadori in kabuki, there are many types of kumadori and the differences between them are subtle, in addition to some examples shown in Figures 2B, 3A, and 3B, so it is useful to improve the recognition accuracy.
[0074] In the first embodiment described above, the image superimposing system 100 includes the imaging device 1, but is not limited to this. For example, the image superimposing system 100 may not include the imaging device 1 and may acquire captured images from an external imaging device.
[0075] In the first embodiment described above, the communication unit 41 of the control device 4 transmits the superimposed image information to the synthesis device 5, but this is not limited to the above. For example, the communication unit 41 may transmit a superimposed image ID to the synthesis device 5, and the synthesis device 5 may extract a superimposed image corresponding to the superimposed image ID from a storage unit included in the synthesis device 5.
[0076] In the first embodiment described above, the image superimposing system 100 includes the display device 6, but is not limited to this. For example, the image superimposing system 100 may not include the display device 6 and may transmit captured images to an external display device, or may transmit captured images to another external device.
[0077] In the first embodiment described above, the superimposing unit 53 of the synthesis device 5 extracts the feature parameters from the face image, but this is not limited to the above. For example, the recognition unit 33 of the face recognition device 2 may recognize the face image and extract the feature parameters from the face image. In such a configuration, the feature parameters are transmitted to the synthesis device 5.
[0078] In the first embodiment described above, any two or more devices included in the image superimposition system 100 may be integrated. In such a configuration, information may be exchanged between functional units included in the two or more devices integrated without going through a communication unit.
[0079] <<Second embodiment>> The overall configuration of the second embodiment will be described with reference to Fig. 7. Fig. 7 is a schematic diagram of an image superimposing system 101 according to the second embodiment. In the second embodiment, the same functional units as those in the first embodiment are denoted by the same reference numerals, and the description thereof will be omitted.
[0080] 7, an image superimposing system 101 according to the second embodiment includes an image capturing device 1, a face recognition device 2, a superimposed object recognition device 3, a control device 4-1, a synthesis device 5, and a display device 6. The control device 4-1 communicates with each of the image capturing device 1, the face recognition device 2, the superimposed object recognition device 3, and the synthesis device 5 via a communication network.
[0081] <Control device configuration> The control device 4-1 includes a communication unit 41, a region information storage unit 42-1, a superimposed image information storage unit 43, a trimming unit 44, a scenario control unit 45, and a region determination unit 46. The region information storage unit 42-1 is configured by a memory. The region determination unit 46 constitutes a control unit.
[0082] The region determination unit 46 determines the superimposed region R2 in the captured image based on the position of the face image recognized by the face recognition device 2 in the captured image. For example, the region determination unit 46 may determine an area within a first predetermined length from the center of the face image recognized by the face recognition device 2 as the superimposed region R2. For example, the region determination unit 46 may determine an area obtained by adding an area within a second predetermined length from the periphery of the face image to the area of the face image recognized by the face recognition device 2 as the superimposed region R2. Also, for example, the region determination unit 46 may determine an area within a rectangle of a predetermined size having the center of gravity at the center of the face image recognized by the face recognition device 2 as the superimposed region R2.
[0083] In the region information storage unit 42-1, a region ID for identifying a region in a captured image and information indicating the range of the region are stored in association with each other, similar to the region information storage unit 42 in the first embodiment. As in the first embodiment, the region includes a face region R1, a superimposed region R2, and a composite region R3. The face region R1 and the composite region R3 in the second embodiment are predetermined regions, similar to the face region R1 and the composite region R3 in the first embodiment. The superimposed region R2 in the second embodiment is different from the superimposed region R2 in the first embodiment, and is not a predetermined region, but is a superimposed region R2 determined by the region determination unit 46. Note that the determined superimposed region R2 may be indicated by a start coordinate, an X size, and a Y size, similar to the first embodiment.
[0084] In the second embodiment, as described in the first embodiment, the trimming unit 44 generates superimposed object area image information indicating a superimposed object area image obtained by trimming the superimposed object area R2 in the captured image in which the imaging device captured the superimposed object in the preparation stage. In the second embodiment, since the area determination unit 46 and the area information storage unit 42 are configured as described above, the trimming unit 44 generates superimposed object area image information indicating a superimposed object area image obtained by trimming the superimposed object area R2 determined by the area determination unit 46 in the captured image.
[0085] <Image overlay system operation> Here, the operation of the image superimposing system 101 according to the second embodiment will be described with reference to Fig. 8A and Fig. 8B. Fig. 8A and Fig. 8B are flowcharts showing an example of the operation of the image superimposing system 101 according to this embodiment. The operation of the image superimposing system 101 described with reference to Fig. 8A and Fig. 8B corresponds to an example of an image superimposing method of the image superimposing system 101 according to the second embodiment.
[0086] 8A, the image superimposing system 101 executes the processes from step S41 to step S47. The processes from step S41 to step S47 are the same as the processes from step S11 to step S17 in the first embodiment.
[0087] In step S48, the region determining unit 46 determines the overlapping object region R2 based on the face image.
[0088] In step S49, the region information storage unit 42-1 stores the overlapping object region R2 determined in step S48. Specifically, the region information storage unit 42-1 stores the start coordinates and the X and Y sizes indicating the overlapping object region R2 determined in step S48 in association with an overlapping object ID for identifying the overlapping object region R2.
[0089] 8A and 8B, the image superimposing system 101 executes the processes from step S50 to step S66. The processes from step S50 to step S66 are the same as the processes from step S18 to step S34 in the first embodiment. As a result, in step S52, the trimming unit 44 generates superimposed object area image information indicating a superimposed object area image obtained by trimming the superimposed object area R2 determined by the area determination unit 46 in the captured image.
[0090] As described above, according to the second embodiment, the image superimposing system 101 determines the superimposed object region R2 based on the face image included in the face region image, and generates superimposed object region image information indicating the superimposed object region image obtained by trimming the superimposed object region R2 in the captured image. Therefore, the image superimposing system 101 can determine the region in the captured image where the image of the superimposed object is likely to be located as the superimposed object region R2. Therefore, the image superimposing system 101 can appropriately recognize the superimposed image, and accordingly, can superimpose the appropriate superimposed image on the face image. For example, when the user is an adult, a child, a wheelchair user, etc., the face region R1 has to be set widely, but the image superimposing system 101 can limit the position and size of the superimposed object region R2 by using the recognition result of the face image, and therefore can appropriately recognize the superimposed image and appropriately superimpose the superimposed image on the face image.
[0091] <Program> The above-mentioned control devices 4, 4-1, and synthesizing device 5 can be realized by a computer 102. A program for causing the control devices 4, 4-1, and synthesizing device 5 to function may be provided. The program may be stored in a storage medium or provided through a network. FIG. 9 is a block diagram showing a schematic configuration of a computer 102 functioning as each of the control devices 4, 4-1, and synthesizing device 5. Here, the computer 102 may be a general-purpose computer, a dedicated computer, a workstation, a PC (Personal Computer), an electronic notepad, or the like. The program instructions may be program code, code segments, or the like for performing necessary tasks.
[0092] 9, the computer 102 includes a processor 110, a read only memory (ROM) 120, a random access memory (RAM) 130, a storage 140, an input unit 150, a display unit 160, and a communication interface (I / F) 170. Each component is connected to each other via a bus 180 so as to be able to communicate with each other. The processor 110 is specifically a central processing unit (CPU), a micro processing unit (MPU), a graphics processing unit (GPU), a digital signal processor (DSP), a system on a chip (SoC), or the like, and may be composed of multiple processors of the same type or different types.
[0093] The processor 110 controls each component and executes various arithmetic processing. That is, the processor 110 reads a program from the ROM 120 or the storage 140, and executes the program using the RAM 130 as a working area. The processor 110 controls each component and executes various arithmetic processing according to the program stored in the ROM 120 or the storage 140. In this embodiment, the program according to the present disclosure is stored in the ROM 120 or the storage 140.
[0094] The program may be stored in a storage medium readable by the computer 102. By using such a storage medium, the program can be installed in the computer 102. Here, the storage medium in which the program is stored may be a non-transitory storage medium. The non-transitory storage medium is not particularly limited, and may be, for example, a CD-ROM, a DVD-ROM, or a USB (Universal Serial Bus) memory. In addition, the program may be in a form that is downloaded from an external device via a network.
[0095] The ROM 120 stores various programs and various data. The RAM 130 temporarily stores programs or data as a working area. The storage 140 is configured with an HDD (Hard Disk Drive) or an SSD (Solid State Drive) and stores various programs including an operating system and various data.
[0096] The input unit 150 includes one or more input interfaces that receive an input operation from a user and acquire information based on the user's operation. For example, the input unit 150 is a pointing device, a keyboard, a mouse, etc., but is not limited to these.
[0097] The display unit 160 includes one or more output interfaces that output information. For example, the display unit 160 is a display that outputs information as a video or a speaker that outputs information as a sound, but is not limited to these. Note that, if the display unit 160 is a touch panel type display, it also functions as the input unit 150.
[0098] The communication interface (I / F) 170 is an interface for communicating with an external device.
[0099] The following supplementary notes are further disclosed regarding the above embodiment.
[0100] (Additional note 1) A control unit is provided, The control unit is generating face area image information showing a face area image obtained by trimming a face area in a captured image of a user captured by an imaging device in a first stage; and generating superimposed object area image information showing a superimposed object area image obtained by trimming a superimposed object area in a captured image of a superimposed object captured by the imaging device in the first stage; Recognizing a face image of the user included in the face area image, and generating face image information indicating the face image; Recognizing a superimposed image corresponding to the superimposed object region image; An image overlay system that recognizes a facial image of the user from a composite area image, which is an image of a composite area larger than the facial area and the superimposition area in an image captured by the imaging device in a second stage after the first stage, based on the facial image information, and generates composite image information indicating a composite image in which the superimposition image identified by the superimposition image identifier is superimposed on the facial image. (Additional note 2) The image superimposition system described in Appendix 1, wherein the control unit determines the superimposition area based on the face image, and generates the superimposition area image information indicating the superimposition area image obtained by cropping the superimposition area in the captured image. (Additional note 3) A step of generating face area image information indicating a face area image obtained by trimming a face area in a captured image of a user captured by an imaging device in a first stage; generating overlapping object area image information indicating an overlapping object area image obtained by trimming an overlapping object area in a captured image in which the imaging device captures the overlapping object in the first stage; A step of recognizing a face image of the user included in the face area image and generating face image information indicating the face image; a step of recognizing a superimposed image corresponding to the superimposed object region image and extracting a superimposed image identifier for identifying the superimposed image; a step of recognizing a face image of the user based on the face image information from a composite area image, which is an image of a composite area larger than the face area and the overlapping object area in a captured image captured by the imaging device in a second step after the first step; generating composite image information indicating a composite image obtained by superimposing the superimposed image on the face image; An image overlay method comprising: (Additional note 4) The method further includes a step of determining the overlapping object region based on the face image included in the face region image, The image superimposition method according to claim 3, wherein the step of generating the overlapping object area image information includes a step of generating the overlapping object area image information indicating the overlapping object area image obtained by cropping the overlapping object area in the captured image.
[0101] All publications, patent applications, and standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, and standard was specifically and individually indicated to be incorporated by reference.
[0102] Although the above-mentioned embodiment has been described as a representative example, it is obvious to those skilled in the art that many modifications and substitutions can be made within the spirit and scope of the present disclosure. Therefore, the present invention should not be interpreted as being limited by the above-mentioned embodiment, and various modifications or changes can be made without departing from the scope of the claims. For example, it is possible to combine multiple building blocks shown in the configuration diagram of the embodiment into one, or to divide one building block. [Explanation of symbols]
[0103] 1. Imaging device 2. Facial Recognition Device 3. Overlapped object recognition device 4, 4-1 Control device 5 Synthesizer 6 Display device 31 Communications Department 32 Superimposed image information storage unit 33 Recognition part 41 Communications Department 42, 42-1 Area information storage unit 43 Superimposed image information storage unit 44 Trimming Department 45 Scenario control section 46 Area determination part 51 Communications Department 52 Face image detection unit 53 Overlapped section 100, 101 Image overlay system 102 Computer 110 Processor 120 ROM 130 RAM 140 Storage 150 Input section 160 Output section 170 Communication Interface 180 Bus
Claims
1. a control device that generates face area image information indicating a face area image obtained by trimming a face area in a captured image of a user captured by an imaging device in a first step, and generates superimposed object area image information indicating a superimposed object area image obtained by trimming a superimposed object area in a captured image of an overlapping object captured by the imaging device in the first step; a face recognition device that recognizes a face image of the user included in the face area image and generates face image information indicating the face image; an overlapping object recognition device that recognizes an overlapping image corresponding to the overlapping object region image; a synthesis device that detects a face image of the user from a synthesis area image, which is an image of a synthesis area larger than the face area and the superimposed object area in a captured image captured by the imaging device in a second step after the first step, based on the face image information, and generates synthesis image information indicating a synthesis image in which the superimposed image is superimposed on the face image; An image overlay system comprising:
2. The image superimposition system according to claim 1 , wherein the control device determines the superimposition area based on the face image, and generates the superimposition area image information indicating the superimposition area image obtained by cropping the superimposition area in the captured image.
3. A step of generating face area image information indicating a face area image obtained by trimming a face area in a captured image of a user captured by an imaging device in a first stage; generating overlapping object area image information indicating an overlapping object area image obtained by trimming an overlapping object area in a captured image in which the imaging device captures the overlapping object in the first stage; A step of recognizing a face image of the user included in the face area image and generating face image information indicating the face image; Recognizing an overlapping image corresponding to the overlapping object region image; a step of detecting a face image of the user based on the face image information from a composite area image, which is an image of a composite area larger than the face area and the overlapping object area, in a captured image captured by the imaging device in a second step after the first step; generating composite image information indicating a composite image obtained by superimposing the superimposed image on the face image; An image overlay method comprising:
4. The method further includes a step of determining the overlapping object region based on the face image included in the face region image, The image superimposing method according to claim 3 , wherein the step of generating the superimposed object area image information includes a step of generating the superimposed object area image information indicating the superimposed object area image obtained by cropping the superimposed object area in the captured image.
Citation Information
Patent Citations
Information processing apparatus, information processing method, and information processing program
JP2020016961A
Object detection and recognition apparatus, method, and program
JP2020017136A
Skin analysis device, skin analysis method, and computer program
JP2020075080A
Information processing device, information processing method, and program
JP2020113061A
