Information processing device, method, program, system, and beauty method

Abstract

This invention provides an information processing device, method, program, system, and beauty method that enhances the sense of presence of others, even when performing treatments on oneself. [Solution] The information processing device 10 includes a determination unit 103 that determines the position and orientation of an object based on a person in an image, a superposition unit 105 that superimposes the image and a CG image of the object with the determined position and orientation, and a display unit 106 that displays the superimposed superposition image.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, method, program, system, and beauty method.

Background Art

[0002] Conventionally, in beauty salons and the like, there is known a beauty technique in which a practitioner such as an esthetician performs an act related to beauty on a customer. The customer can obtain a sense of satisfaction by receiving an act related to beauty from others.

[0003] On the other hand, there is also known a beauty technique called "self-esthetic" in which the customer, rather than the practitioner, performs an act related to beauty by oneself. In addition, the customer may perform an act related to beauty by oneself at home or the like, imitating the act of a professional (Patent Document 1).

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, even when performing a treatment on oneself, it has been difficult to feel as if receiving a treatment from others.

[0006] Therefore, an object of the present invention is to improve the sense of presence of others even when performing a treatment on oneself.

Means for Solving the Problems

[0007] An information processing device according to one embodiment of the present invention comprises: a determination unit that determines the position and orientation of an object based on a person in an image; a superposition unit that superimposes the image and a CG image of the object that has the determined position and orientation; and a display unit that displays the superimposed superposition image. [Effects of the Invention]

[0008] In this invention, even when performing the procedure on oneself, it is possible to enhance the sense of presence of others. [Brief explanation of the drawing]

[0009] [Figure 1] This diagram shows the overall configuration of one embodiment of the present invention. [Figure 2] This figure shows the functional block of an information processing device according to one embodiment of the present invention. [Figure 3] This is a flowchart showing the flow of the superimposed display process (first embodiment) according to one embodiment of the present invention. [Figure 4] This is a flowchart showing the flow of the superimposed display process (second embodiment) according to one embodiment of the present invention. [Figure 5] This is an example of a screen displayed on an information processing device according to one embodiment of the present invention (first embodiment). [Figure 6] This is an example of a screen displayed on an information processing device according to one embodiment of the present invention (second embodiment). [Figure 7] This figure shows the hardware configuration of an information processing device according to one embodiment of the present invention. [Modes for carrying out the invention]

[0010] Each embodiment will be described below with reference to the attached drawings. In this specification and the drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant descriptions will be omitted.

[0011] <Explanation of Terms> "Real-time image" refers to the image captured by the camera function of the information processing device 10 (which will be explained in detail later). • "CG (computer graphics) image" refers to an image generated by the information processing device 10. Note that the CG image may also be a processed video taken in advance. "Acts related to beauty" refers to any act related to beauty performed using hands, cosmetic tools, etc. (for example, the application of cosmetics).

[0012] <Overall Structure> Figure 1 shows the overall configuration according to one embodiment of the present invention. Person 20 can feel as if they are receiving a treatment (action) from another person (for example, receiving a beauty treatment) by performing an arbitrary treatment (action) on themselves (for example, performing a beauty treatment) while looking at the screen displayed on the information processing device 10.

[0013] Specifically, in one embodiment of the present invention, person 20 performs a beauty treatment on themselves while their upper body is projected onto the lens of a camera mounted on the side of the information processing device 10 that has a screen. The information processing device 10 displays a video in which a CG image of an object (for example, a practitioner such as an esthetician) is superimposed on a real-time image of person 20. The CG image of the object is a CG image of the object performing a beauty treatment on person 20. By performing a beauty treatment on themselves while watching such a video, person 20 can feel as if they are receiving a beauty treatment from another person (i.e., an object).

[0014] The information processing device 10 captures a video of a person 20 and displays a video in which a CG image of an object (for example, a practitioner such as an esthetician) is superimposed on the real-time image of the person 20. For example, the information processing device 10 is a smartphone, tablet, personal computer, etc., that has a camera function (it may also be a 3D camera). The information processing device 10 will be described in detail later with reference to Figure 2.

[0015] In this specification, a case where the information processing apparatus 10 is one device (for example, a smartphone having a camera function or the like) will be described. However, the information processing apparatus 10 may be composed of a plurality of devices (for example, a device without a camera function and a digital camera). Further, a device other than the information processing apparatus 10 (such as a server) may execute some of the processes executed by the information processing apparatus 10 described in this specification.

[0016] <Functional blocks of the information processing apparatus 10> FIG. 2 is a diagram showing functional blocks of an information processing apparatus 10 according to an embodiment of the present invention. The information processing apparatus 10 can include a photographing unit 101, a specifying unit 102, a determining unit 103, a selecting unit 104, an overlapping unit 105, and a display unit 106. Further, the information processing apparatus 10 can function as the photographing unit 101, the specifying unit 102, the determining unit 103, the selecting unit 104, the overlapping unit 105, and the display unit 106 by executing a program. Each will be described below.

[0017] The photographing unit 101 acquires a real-time image. Specifically, the photographing unit 101 photographs a video of the person 20. For example, the photographing unit 101 photographs a video of the upper body of the person 20 performing an act related to beauty.

[0018] The specifying unit 102 specifies a part of the body of the person 20 in the real-time image acquired by the photographing unit 101. Hereinafter, <<specification for erasing (for example, transparency) of the area of a part of the body of the person 20>> and <<specification for determining the position and orientation of an object>> will be described.

[0019] <<Specification for erasing (for example, transparency) of the area of a part of the body of the person 20>> The identification unit 102 identifies a portion of the person 20's body within the real-time image (for example, a portion including at least the hand, such as from the elbow to the fingertips or from the wrist to the fingertips). Person 20 may be holding objects such as makeup tools. The identification unit 102 then removes (e.g., makes transparent) the identified portion of the person 20's body (for example, a portion including at least the hand) and the portion of any objects the person 20 is holding (if any). The process of removing the portion of the person 20's body is optional.

[0020] <<Method for identifying the area>> For example, the identification unit 102 can identify a region of the person 20 in a real-time image that includes at least the hands by referencing information about the human skeleton (such as the position of feature points) stored in the information processing device 10 beforehand to estimate the skeleton of the person 20, extracting areas of the same color, or extracting parts of the person 20 in the video that are moving at a speed of a certain speed or faster.

[0021] <<Identification for determining the position and orientation of an object>> The following will describe the first and second embodiments separately.

[0022] In the first embodiment, the identification unit 102 identifies the position and posture of at least the hands (for example, from the elbow to the fingertips, from the wrist to the fingertips, etc.) of the person 20 in the real-time image. The person 20 may be holding objects such as makeup tools.

[0023] "Position" indicates the location of person 20 within the image (in this case, the real-time image). "Posture" indicates the way person 20 is positioned (posture, stance, etc.).

[0024] <<Method for determining position and orientation>> For example, the identification unit 102 can determine at least the position and posture of the person 20 in a real-time image by referencing information about the human skeleton (such as the position of feature points) stored in the information processing device 10 in advance to estimate the skeleton of the person 20, extracting areas of the same color, or extracting parts of the person 20 in the video that are moving at a speed of a certain speed or faster.

[0025] In the second embodiment, the identification unit 102 identifies the point of contact between the hand of person 20 or an object held by person 20 (e.g., cosmetic tools) and person 20's face in the real-time image.

[0026] <<Method for identifying contact points>> For example, the identification unit 102 can analyze real-time images to identify contact points between the person's hand (e.g., fingers) or an object held by the person 20 and the person's face in the real-time images. For example, the identification unit 102 can identify contact points from real-time images using a trained model that has been machine-learned on multiple images in which a hand (e.g., fingers) is in contact with a face (e.g., cheek). Alternatively, the identification unit 102 may identify contact points between the person's hand (e.g., fingers) or an object held by the person 20 and the person's face based on detection results from sensors attached to the person's fingers, etc.

[0027] The determination unit 103 determines the position and orientation of an object based on the person 20 in the real-time image. "Position" refers to the location in the virtual space where the object is assumed to exist. "Orientation" refers to the posture (position, stance, etc.) of the object's body.

[0028] Specifically, the determination unit 103 determines the position and orientation of an object based on the correspondence between a part of a person's body and the position and orientation of the object.

[0029] The correspondence may be a predetermined database or a pre-trained machine learning model. The database links information about a part of a person's body (specifically, at least the position and posture of a person's hands, or the point of contact between a person's hands or an object they are holding and their face) with information about the position and posture of an object. The pre-trained model is a predictive model that, when information about a part of a person's body (specifically, at least the position and posture of a person's hands, or the point of contact between a person's hands or an object they are holding and their face) is input, outputs information about the position and posture of an object.

[0030] The following will describe the first and second embodiments separately.

[0031] <<First Embodiment>> In the first embodiment, the determination unit 103 determines the position and posture of an object based on the position and posture of at least the hands (e.g., from the elbow to the fingertips, from the wrist to the fingertips, etc.) of the person 20 identified by the identification unit 102. For example, the determination unit 103 can make the position and posture of at least the hands of the person 20 the same as the position and posture of at least the hands of the object. Furthermore, the determination unit 103 can determine the position and posture of other parts of the object (i.e., at least parts other than the hands) based on the condition that the position and posture of at least the hands of the person 20 and the object are the same, and that the object is standing diagonally behind the person 20, etc., which is a possible standing position of the practitioner during treatment.

[0032] <<Second Embodiment>> In the second embodiment, the determination unit 103 determines the position and orientation of an object based on the point of contact between the person 20's hand or an object held by the person 20 (e.g., cosmetic tools) identified by the identification unit 102 and the person 20's face. For example, the determination unit 103 can make the point of contact between the person 20's hand or an object held by the person 20 (e.g., cosmetic tools) and the person 20's face the same as the point of contact between the object's hand or an object held by the object (e.g., cosmetic tools) and the person 20's face. Furthermore, the determination unit 103 can determine the position and orientation of other parts of the object (at least parts other than the hand) based on the condition that the point of contact between the person 20's hand, etc. and the point of contact between the object's hand, etc. are the same, and that the object is standing behind the person 20, etc., and the practitioner's standing position during the procedure.

[0033] The selection unit 104 selects an object. For example, the selection unit 104 can select an object specified by a person 20 from among multiple objects. Person 20 can select an object on a screen where multiple objects (for example, images of the faces of the objects) are displayed.

[0034] Furthermore, for example, the selection unit 104 can select an object suitable for a person 20 predicted by machine learning. Alternatively, for example, the selection unit 104 can select a predetermined object (a uniform object for all people).

[0035] Here, we will explain what an object is. An object is not limited to a person; it may also be an animal, a character, etc. For example, an object is a person who has previously performed a cosmetic procedure on person 20. Persons who have previously performed cosmetic procedures on person 20 may be selected by referring to the treatment history linked to person 20's identification information (registered member ID, etc.). Furthermore, the CG image of an object is not limited to a CG image of the entire object, but may also be a CG image of a part of the object (for example, the object's hand). In addition, the CG image of an object may be a CG image of the object holding an object (for example, an object identical to one that person 20 possesses (for example, cosmetic tools)).

[0036] The superimposing unit 105 superimposes a real-time image of the person 20 (a real-time image in which at least the area including the person 20's hands has been removed) with a CG image of an object whose position and orientation have been determined by the determination unit 103. Specifically, the superimposing unit 105 overlays the CG image on top of the real-time image.

[0037] Furthermore, the superimposing unit 105 can interpolate images. Specifically, the superimposing unit 105 can overlay a CG image of a part of person 20's body (for example, a hand not performing a beauty treatment) recreated based on a interpolation image (for example, an image of person 20 before they begin a beauty treatment) onto the image in which the real-time image and the CG image of the object are superimposed. This makes it possible to generate a video in which person 20 is not performing a beauty treatment themselves.

[0038] Specifically, the superimposing unit 105 can overlay an image of a part of the body of the person 20 identified by the identification unit 102 (for example, an area including at least the hands) onto an image of that area in a complementary image (for example, an image of the person 20 before they begin the beauty treatment). In other words, after the area of ​​the person 20 including at least the hands is erased (for example, made transparent) in the real-time image, the superimposing unit 105 complements the background that should be present in that area and then overlays an object. This makes it possible to generate a video in which the erased area has been filled in.

[0039] The superimposing unit 105 may also change the position and orientation of the object's CG image (for example, by moving the position where the object is standing) so that the object moves naturally. In addition, the superimposing unit 105 may not synchronize the position and orientation of the object's CG image with that of the person 20 so that the object does not move unnaturally when the person 20 makes a sudden movement (or it may temporarily reduce the tracking speed for the person 20's movement). This makes it possible to generate a video in which the object does not freeze or move too much.

[0040] The display unit 106 displays the superimposed image from the superimposition unit 105.

[0041] <Processing method> The following describes the processing method for superimposed displays, which is the first embodiment, and the processing method for superimposed displays, which is the second embodiment.

[0042] <<First Embodiment>> Figure 3 is a flowchart showing the flow of the superimposed display process (first embodiment) according to one embodiment of the present invention.

[0043] In step 11 (S11), the camera unit 101 acquires real-time images. Specifically, the camera unit 101 captures a video of the person 20.

[0044] In step 12 (S12), the identification unit 102 identifies a region of the person 20 in the real-time image of S11 that includes at least the hands (for example, a region from the elbow to the fingertips, a region from the wrist to the fingertips, etc.). The identification unit 102 also identifies the position and posture of at least the hands of the person 20 in the real-time image of S11 (for example, a region from the elbow to the fingertips, a region from the wrist to the fingertips, etc.). The person 20 may be holding objects such as makeup tools.

[0045] In step 13 (S13), the identification unit 102 removes (e.g., makes transparent) the area including at least the hands of the person 20 identified in S12 and the area of ​​any objects the person 20 is holding (if any objects are being held) from the real-time image.

[0046] In step 14 (S14), the determination unit 103 determines the position and orientation of an object based on the person 20 in the real-time image from S11. Specifically, the determination unit 103 determines the position and orientation of an object based on the position and orientation of at least the hands (e.g., from the elbow to the fingertips, from the wrist to the fingertips, etc.) of the person 20 identified in S12.

[0047] In step 15 (S15), the superimposing unit 105 superimposes the real-time image of the person 20 (the real-time image in which at least the area including the person 20's hands has been erased in S13) with the CG image of the object at the position and orientation determined in S14. Furthermore, the superimposing unit 105 can interpolate the images.

[0048] In step 16 (S16), the display unit 106 displays the superimposed images from S15. Specifically, the display unit 106 displays the multiple images generated in the processes of S11 to S15 as a video.

[0049] <<Second Embodiment>> Figure 4 is a flowchart showing the flow of the superimposed display process (second embodiment) according to one embodiment of the present invention.

[0050] In step 21 (S21), the camera unit 101 acquires real-time images. Specifically, the camera unit 101 captures a video of the person 20.

[0051] In step 22 (S22), the identification unit 102 identifies a region of the person 20 in the real-time image from S21 that includes at least the hands (for example, a region from the elbow to the fingertips, a region from the wrist to the fingertips, etc.). The person 20 may be holding objects such as makeup tools.

[0052] In step 23 (S23), the identification unit 102 removes (e.g., makes transparent) the area including at least the hands of the person 20 identified in S22 and the area of ​​any objects the person 20 is holding (if any objects are being held) from the real-time image.

[0053] In step 24 (S24), the identification unit 102 identifies the point of contact between the hand of person 20 or an object held by person 20 (for example, cosmetic tools) and person 20's face in the real-time image from S21.

[0054] In step 25 (S25), determination unit 103 determines the position and orientation of the object based on person 20 in the real-time image of S21. Specifically, determination unit 103 determines the position and orientation of the object based on the contact point between the hand of person 20 specified in S24 or an object (e.g., a cosmetic tool) held by person 20 and the face of person 20.

[0055] In step 26 (S26), superimposition unit 105 superimposes the real-time image in which person 20 is photographed (the real-time image in which at least the area including the hand of person 20 is erased in S23) and the CG image of the object whose position and orientation are determined in S25. Further, superimposition unit 105 can complement the image.

[0056] In step 27 (S27), display unit 106 displays the image superimposed in S26. Specifically, display unit 106 displays a plurality of images generated in the processes of S21 to S26 as a video.

[0057] <<Creation of CG Image>> In one embodiment of the present invention, a person performing an act related to beauty can be cut out from a video. As a cutting method, it may be cut out from a video shot with a green screen (chroma key) or the like, or it may be cut out by designating a portion to be cut out from a normal video. In one embodiment of the present invention, the object is a person who performs an act related to beauty on a person, and the CG image of the object is created using a video in which the object is photographed performing an act related to beauty on the shape of a face of a predetermined color (e.g., green) with a background of a predetermined color (e.g., green) (also called a green screen). For example, a person performing an act related to beauty on a face covered with a cloth of a predetermined color (e.g., green) or the like with a cloth of a predetermined color (e.g., green) or the like as the background is photographed performing an act related to beauty. In this photographed video, the CG image of the object is created by cutting out the hand or the like of the person performing the act related to beauty on the person (including a part of the body including the hand or a beauty tool held by the hand, or the whole body of the person performing the act related to beauty may be cut out).

[0058] <<Other Embodiments>> In one embodiment of the present invention, the hand movements of the person 20 and the movement of the CG object are not synchronized. Specifically, the identification unit 102 removes (e.g., makes transparent) a portion of the person 20's body (e.g., a portion including at least the hands) and a portion of an object held by the person 20 (if an object is being held) from the real-time image. Then, the determination unit 103 determines the position and orientation of the object based on the person 20 in the real-time image (specifically, based on the position of the person 20 in the real-time image (e.g., the position and orientation of the person 20's face)). For example, the determination unit 103 determines the position and orientation of the object so that the object's hands touch the face of the person 20 in the real-time image. Then, the superimposition unit 105 superimposes the real-time image of the person 20 (the real-time image with at least the portion including the person 20's hands removed) and the CG image of the object. The person 20 performs a beauty-related action in accordance with the movement of the CG object.

[0059] Although this specification describes an embodiment in which a CG image is superimposed on a real-time image, the present invention is not limited thereto and can also be applied to an embodiment in which a CG image is superimposed on a pre-captured image.

[0060] <Display screen> The following describes the display screen of the first embodiment and the display screen of the second embodiment. In one embodiment of the present invention, the information processing device 10 displays a video of a person 20 receiving a beauty treatment from an object (for example, a practitioner such as an esthetician).

[0061] <<First Embodiment>> Figure 5 shows an example of a screen displayed on an information processing device 10 according to one embodiment of the present invention (first embodiment). In the first embodiment, the position and posture of the CG image of an object (for example, a practitioner such as an esthetician) are linked to the position and posture of at least the hands of the person 20 (for example, from the elbow to the fingertips, from the wrist to the fingertips, etc.). For example, person 20 can watch a video of person 20 receiving a beauty treatment while seated (Note that person 20 in Figure 5 is facing the screen displayed on the information processing device 10 in Figure 5 while receiving the beauty treatment).

[0062] <<Second Embodiment>> Figure 6 shows an example of a screen displayed on an information processing device 10 according to one embodiment of the present invention (second embodiment). In the second embodiment, the position and orientation of the CG image of an object (for example, a practitioner such as an esthetician) are linked to the point of contact between the hands of person 20 or an object held by person 20 (for example, cosmetic tools) and person 20's face. For example, person 20 can watch a video of themselves receiving a beauty treatment while lying on their back (in Figure 6, person 20 is facing the screen displayed on the information processing device 10 in Figure 6 while receiving the beauty treatment).

[0063] <Beauty method> One embodiment of the present invention is a beauty method in which beauty-related acts are performed while viewing the image described above. That is, one embodiment of the present invention is a beauty method in which beauty-related acts are performed while viewing an image, wherein the position and orientation of an object are determined based on a person in a real-time image, and the image is an image in which the real-time image and a CG image of the object at the determined position and orientation are superimposed.

[0064] <Effects> Thus, in one embodiment of the present invention, by performing beauty-related actions on himself while looking at the screen displayed on the information processing device 10, the person 20 can feel as if he is receiving beauty-related actions from another person (that is, an object (for example, a practitioner such as an esthetician)).

[0065] <Hardware Configuration> Figure 7 shows the hardware configuration of an information processing device according to one embodiment of the present invention.

[0066] The information processing device 10 includes a CPU (Central Processing Unit) 1001, a ROM (Read Only Memory) 1002, and a RAM (Random Access Memory) 1003. The CPU 1001, ROM 1002, and RAM 1003 form what is known as a computer.

[0067] Furthermore, the information processing device 10 may include an auxiliary storage device 1004, a display device 1005, an operating device 1006, an I / F (Interface) device 1007, and a drive device 1008.

[0068] Furthermore, each piece of hardware in the information processing device 10 is interconnected via bus B.

[0069] The CPU 1001 is a computing device that executes various programs installed on the auxiliary storage device 1004.

[0070] ROM1002 is non-volatile memory. ROM1002 functions as a main memory device that stores various programs, data, etc., necessary for the CPU1001 to execute the various programs installed on the auxiliary storage device 1004. Specifically, ROM1002 functions as a main memory device that stores boot programs such as BIOS (Basic Input / Output System) and EFI (Extensible Firmware Interface).

[0071] RAM1003 is a volatile memory such as DRAM (Dynamic Random Access Memory) or SRAM (Static Random Access Memory). RAM1003 functions as a main memory device that provides a working area that is expanded when various programs installed on the auxiliary storage device 1004 are executed by the CPU 1001.

[0072] The auxiliary storage device 1004 is an auxiliary storage device that stores various programs and information used when various programs are executed.

[0073] The display device 1005 is a display device that displays the internal state of the information processing device 10, etc.

[0074] The operating device 1006 is an input device that allows the operator of the information processing device 10 to input various instructions to the information processing device 10.

[0075] The I / F device 1007 is a communication device that connects to a network and communicates with other devices.

[0076] The drive device 1008 is a device for setting the storage medium 1009. The storage medium 1009 here includes media that record information optically, electrically, or magnetically, such as CD-ROMs, flexible disks, and magneto-optical disks. The storage medium 1009 may also include semiconductor memories that record information electrically, such as EPROMs (Erasable Programmable Read Only Memory) and flash memory.

[0077] The various programs to be installed in the auxiliary storage device 1004 are installed, for example, when the distributed storage medium 1009 is set in the drive device 1008, and the various programs recorded on the storage medium 1009 are read by the drive device 1008. Alternatively, the various programs to be installed in the auxiliary storage device 1004 may be installed by downloading them from the network via the I / F device 1007.

[0078] The information processing device 10 has a photographing device 1010. The photographing device 1010 photographs a person 20.

[0079] Although embodiments of the present invention have been described in detail above, the present invention is not limited to the specific embodiments described above, and various modifications and changes are possible within the scope of the gist of the present invention as described in the claims.

[0080] This international application claims priority based on Japanese Patent Application No. 2021-039295, filed on 11 March 2021, and the entire contents of No. 2021-039295 are incorporated herein by reference. [Explanation of Symbols]

[0081] 10 Information Processing Devices 20 people 101 Photography Department 102 Specific part 103 Decision Section 104 Selection Section 105 Overlapping section 106 Display section 1001 CPU 1002 ROM 1003 RAM 1004 Auxiliary storage device 1005 Display device 1006 Operating device 1007 I / F device 1008 Drive unit 1009 Storage medium 1010 Imaging device

[0082] (Note 1) A determination unit that determines the position and orientation of an object based on the person in the image, A superimposed area where the above image and a CG image of the object at the determined position and orientation are superimposed, A display unit that displays the superimposed superimposed images Equipped with an information processing device. (Note 2) The image above is a real-time image of the information processing device described in Appendix 1. (Note 3) The image above is a pre-existing image of the information processing device described in Appendix 1. (Note 4) The above determination unit determines the above position and posture of the object based on at least the position and posture of the person, as described in any of the appendices 1 to 3. (Note 5) The above determination unit determines the position and orientation of the object based on the point of contact between the person's hand or an object held by the person and the person's face, as described in any of the appendices 1 to 3. (Note 6) The above determination unit determines the above position and posture of the object based on the position and posture of the person's face, as described in any of the information processing apparatus described in Appendix 1 to 3. (Note 7) The superimposed portion is an information processing device according to any one of the appendices 1 to 6, which further superimposes a CG image of a part of the person's body. (Note 8) The above determination unit determines the position and orientation of the object based on the correspondence between a part of a person's body and the position and orientation of the object, as described in any of Appendix 1 to 7. (Note 9) The above-mentioned person is an information processing device described in any of the appendices 1 to 8, which is engaged in activities related to beauty. (Note 10) The CG image of the above-mentioned object is a CG image of the above-mentioned object performing a beauty-related act on the above-mentioned person, as described in Appendix 9 of the information processing device. (Note 11) The CG image of the above object is created using a video of the object performing a cosmetic procedure on a face, as described in Appendix 10 of the information processing device. (Note 12) The information processing device according to Appendix 10 or 11, wherein the CG image of the above object is a CG image of at least the hands of a person or character performing a beauty-related act, and an object held by the person or character performing the beauty-related act. (Note 13) The above object is a person who has previously performed cosmetic acts on the above person, and is an information processing device as described in any of Appendix 10 to 12. (Note 14) An information processing apparatus according to any one of appendices 1 to 13, further comprising a selection unit for selecting the above-mentioned object. (Note 15) A method by which a computer performs an action. The steps include determining the position and orientation of an object based on the person in the image, The steps include superimposing the above image with a CG image of the object at the determined position and orientation, The steps include displaying the superimposed superimposed images and A method that includes this. (Note 16) Computer A determination unit that determines the position and orientation of an object based on the person in the image. The superimposed area overlays the above image and the CG image of the object at the determined position and orientation. Display unit for showing superimposed superimposed images A program designed to function as such. (Note 17) A system including an information processing device and a server, A determination unit that determines the position and orientation of an object based on the person in the image, A superimposed area where the above image and a CG image of the object at the determined position and orientation are superimposed, A display unit that displays the superimposed superimposed images A system equipped with this feature. (Note 18) This is a beauty method in which beauty-related actions are performed while viewing images. The image above is Based on the people in the image, the position and orientation of the objects are determined. A beauty method in which the above image and a CG image of the above object in the determined position and orientation are superimposed to form a superimposed image.

Claims

1. An information processing device for generating a superimposed image in which a CG image of a second hand, different from the hand of the person, is superimposed on an image that includes the hand of the person, A determination unit that determines the position and posture of the second hand so that the position and posture of the second hand are linked to the position and posture of the person's hand, A superimposing unit that generates the superimposed image based on the determined position and orientation of the second hand, A display unit that displays the superimposed image on a display device, Equipped with an information processing device.

2. The information processing apparatus according to claim 1, wherein the superimposing unit generates an image in which at least the region including the hand of the person in the image appears to have been erased, and generates the superimposed image in which the CG image of the second hand is superimposed on the image.

3. The information processing apparatus according to claim 1 or 2, wherein the aforementioned CG image is a CG image of a person performing a beauty-related act on the aforementioned person.

4. The information processing apparatus according to any one of claims 1 to 3, wherein the aforementioned image is a real-time image.

5. The information processing apparatus according to any one of claims 1 to 3, wherein the aforementioned image is an image that has been taken in advance.

6. The information processing apparatus according to any one of claims 1 to 5, wherein the superimposing section further superimposes a CG image of a part of the person's body.

7. The information processing apparatus according to any one of claims 1 to 6, wherein the determination unit determines the position and posture of the CG image based on the correspondence between a part of a person's body and the position and posture of the CG image.

8. The information processing device according to any one of claims 1 to 7, wherein the person is performing an act related to beauty.

9. The information processing apparatus according to claim 1 or 2, wherein the CG image of the second hand is a CG image of the second hand performing a beauty-related action on the person.

10. The information processing apparatus according to claim 9, wherein the CG image of the second hand is created using a video showing the second hand performing a beauty-related action on the shape of a face.

11. The information processing apparatus according to claim 9 or 10, wherein the CG image of the second hand is a CG image of at least the hand of a person or character performing a beauty-related act, and an object held by the person or character performing the beauty-related act.

12. The information processing apparatus according to any one of claims 9 to 11, wherein the second hand is the hand of a person who has previously performed a beauty-related act on the person.

13. The information processing apparatus according to claim 1 or 2, further comprising a selection unit for selecting the second hand.

14. A method for generating a superimposed image in which a second CG image of a hand different from the hand of a person is superimposed on an image including the hand of a person, The steps include determining the position and posture of the second hand so that the position and posture of the second hand are linked to the position and posture of the person's hand, The steps include generating the superimposed image based on the determined second hand position and orientation, The steps include displaying the superimposed image on a display device, A method that includes this.

15. The method according to claim 14, wherein in the step of generating the superimposed image, an image is generated in which an image is generated in which at least the area including the hand of a person in the image is erased, and the superimposed image is generated in which the CG image of the second hand is superimposed on the image.

16. An information processing device for generating a superimposed image in which a second CG image of a hand different from the hand of the person is superimposed on an image that includes the hand of the person, A determination unit that determines the position and posture of the second hand so that the position and posture of the second hand are linked to the position and posture of the person's hand, Based on the determined position and orientation of the second hand, the superimposing unit generates the superimposed image. A display unit that displays the superimposed image on a display device, A program designed to function as such.

17. The program according to claim 16, wherein the superimposing unit generates an image in which at least the area including the hand of the person in the image appears to have been erased, and generates the superimposed image in which the CG image of the second hand is superimposed on the image.

18. A system including an information processing device and a server for generating a superimposed image in which a second CG image of a hand different from the hand of the person is superimposed on an image including the hand of the person, A determination unit that determines the position and posture of the second hand so that the position and posture of the second hand are linked to the position and posture of the person's hand, A superimposing unit that generates the superimposed image based on the determined position and orientation of the second hand, A display unit that displays the superimposed image on a display device, A system equipped with this feature.

19. The system according to claim 18, wherein the superimposing unit generates an image in which at least the area including the hand of the person in the image appears to have been erased, and generates the superimposed image in which the CG image of the second hand is superimposed on the image.

20. This is a beauty method in which beauty-related actions are performed while viewing images. The aforementioned image is, A beauty method in which the position and posture of a second hand, which is different from the position and posture of the person's hand, is determined so as to be linked to the position and posture of the person's hand, and a superimposed image is created in which a CG image of the second hand is superimposed on an image including the person's hand based on the determined position and posture of the second hand.

21. The beauty method according to claim 20, wherein the superimposed image is a superimposed image in which a CG image of the second hand is superimposed on an image in which an area including at least the hand of a person in the image appears to have been erased.

Images