Simulation system and simulation method

Abstract

The invention provides a simulation system and the like which are not easy to generate glare during simulation. According to one aspect of the present invention, a simulation system for projecting an image to a head of a living being is provided. The simulation system includes a photographing unit, an information processing unit, and a projection unit. The image capturing unit is configured so as to continuously capture captured images including the head in a time series. The information processing section is configured to calculate predetermined information on the head based on the captured image, and generate an image by performing projection. The predetermined information includes the position and shape of the head, and the form of the projection. An image to be projected includes a base portion and one or more object portions. The one or more subject portions include a portion that shields an eye portion included in the head. The projection unit is configured to project the image generated by the information processing unit so as to match the position of the head on the basis of the predetermined information. The time from the time at which the frame of the camera unit is captured to the time at which the information processing corresponding to the frame is performed and the time at which the projection unit projects the image to the head is 50 ms or less.

Description

Technical Field

[0001] The invention relates to a simulation system and a simulation method. Background Art

[0002] In recent years, the technology of projecting images onto three-dimensional objects (also known as "projection mapping") has attracted widespread attention. In addition, this technology has also been developed and applied to projecting makeup images onto parts such as the face of a person. As related technologies, there are technologies disclosed in Patent Documents 1 and 2. Prior art literature Patent Literature

[0003] Patent Document 1: Japanese Patent Publication No. 2018-195996 Patent Document 2: Japanese Patent Publication No. 2022-45944 Summary of the invention Problems to be solved by the invention

[0004] However, the inventors have discovered that when using the above type of projection technology, glare may be generated due to the light of the projector used.

[0005] In view of the above circumstances, the present invention provides a simulation system and the like that are less likely to generate glare during simulation. Means of solving problems

[0006] According to one embodiment of the present invention, a simulation system for projecting an image onto a biological head is provided. The simulation system includes a photographing unit, an information processing unit, and a projection unit. The photographing unit is configured to continuously photograph images including a head in a time series. The information processing unit is configured to calculate predetermined information about the head based on the photographed images, and to generate an image by performing projection. The predetermined information includes the position, shape, and projection form of the head. The image to be projected includes a base portion and one or more object portions. The one or more object portions include a portion that obscures an eye portion included in the head. The projection unit is configured to project the image generated by the information processing unit in a manner consistent with the position of the head based on the predetermined information. The time from the moment when the photographing unit photographs a frame of the head to the moment when the information processing corresponding to the frame is performed and the moment when the projection unit projects the image onto the head is less than 50 ms.

[0007] According to the above aspect, a simulation system or the like is provided which is less likely to cause glare during use when performing simulation. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 It is a diagram showing the configuration of the simulation system 1 according to the first embodiment. Figure 2 It is a block diagram showing the hardware configuration of the information processing unit 3 . Figure 3 3 is a functional block diagram showing the functions of the control unit 33 in the information processing unit 3 . Figure 4 It is a block diagram showing the hardware configuration of the tablet terminal (makeup input unit 6 ). Figure 5 This is an activity diagram for explaining the simulation method of the first embodiment. Figure 6 This is a conceptual diagram for explaining the image generated in the information processing step. Figure 7 This is an example of a display screen displayed on the tablet terminal (makeup input unit 6 ). Figure 8 1 is a diagram showing a configuration of a first modified example of the simulation system 1 . Fig. 9 1 is a diagram showing a configuration of a second modified example of the simulation system 1 . DETAILED DESCRIPTION

[0009] The following describes embodiments of the present invention using drawings. Various features shown in the following embodiments can be combined with each other.

[0010] In addition, the program for implementing the software appearing in this embodiment can be provided as a computer-readable non-transitory storage medium (Non-Transitory Computer-Readable Medium), can be provided in a downloadable manner from an external server, and can also be provided so that the program can be launched through an external computer to realize its functions on a client terminal (so-called cloud computing).

[0011] In addition, the "unit" in this embodiment may include, for example, a combination of hardware resources implemented by a generalized circuit and software information processing that can be specifically implemented by such hardware resources. In addition, although various information is processed in this embodiment, such information is a physical value representing a signal value of a voltage or current, or can be represented as a high or low signal value of a binary bit set composed of 0 or 1, or can be represented by quantum superposition (i.e., so-called quantum bit), and communication or operation can be performed on a generalized circuit.

[0012] In addition, a circuit in a broad sense is a circuit realized by at least appropriately combining circuits, circuit types, processors, and memories, etc. That is, it includes application-specific integrated circuits (ASICs), programmable logic devices (e.g., simple programmable logic devices (SPLDs), complex programmable logic devices (CPLDs), and field programmable gate arrays (FPGAs)).

[0013] [First embodiment] First, a first embodiment of the simulation system will be described. That is, the simulation system of the first embodiment is as follows. A simulation system for projecting an image onto a biological head, comprising: Shooting unit, information processing unit and projection unit; wherein, The imaging unit is configured to continuously capture images including the head in a time series; The information processing unit is configured to calculate predetermined information about the head based on the captured image, and generate an image by performing projection, wherein: The predetermined information includes the position, shape and projection form of the head, The image to be projected includes a base portion and one or more object portions, One or more of the object portions include a portion that obscures an eye portion included in the head; The projection unit is configured to project the image generated by the information processing unit in a manner consistent with the position of the head based on the predetermined information. The time from when the imaging unit images the frame of the head to when information processing corresponding to the frame is performed and when the projection unit projects the image onto the head is 50 ms or less. The following describes the structure and purpose of the simulation system.

[0014] 1. Hardware structure In this section, the hardware configuration of the first embodiment is described.

[0015] 1.1 Simulation system 1 Figure 11 is a block diagram showing a simulation system 1 according to a first embodiment. The simulation system 1 projects an image onto the head of a living being. Figure 1 A system for projecting an image onto the head of a person P1 is shown in . Note that the simulation system 1 can also be applied to animals other than humans. In addition, the simulation system 1 can be used for various purposes, but is typically used to simulate makeup for the head. Note that in this specification, "makeup" refers to makeup applied to various parts of the face using foundation, highlighter, shadow, eye shadow, eyeliner, eyebrow pencil, mascara, blush, lipstick, etc., and also includes dyeing of hair, etc. In addition, in addition to makeup for the head, the simulation system 1 can also be used to simulate accessories (decorations) on the head, simulate hairstyles, simulate the increase or decrease of three-dimensional structures such as pores or wrinkles, simulate the increase or decrease of uneven skin color such as acne or spots, simulate cosmetic surgery, and visualize the structure of subcutaneous tissue. In addition, the projection part can also include other than the head. For example, as described above, while simulating makeup on the head, projection can also be performed on the body or hands, feet, etc. to simulate clothing that matches the makeup.

[0016] The simulation system 1 includes a photographing unit 2, an information processing unit 3, and a projection unit 4, which are connected by a circuit. Note that although this is not a necessary structure in this embodiment, Figure 1 In the simulation system 1 of FIG. 1 , in addition to the above-mentioned structure, an illumination adjustment unit 5 and a makeup input unit 6 are further provided. These components will be further described below.

[0017] 1.2 Photography Section 2 The photographing unit 2 is configured to continuously photograph images including the head in a time series. The photographing unit 2 is, for example, a camera configured to obtain external information as an image (visible light, infrared light, etc. can be used as appropriate). Such a camera preferably uses a camera with a high operating frequency (frame rate) called high-speed vision. The frame rate (operating frequency) of the camera is, for example, 300 fps or more, preferably 400 fps or more, and more preferably 500 fps or more or 1000 fps or more. In addition, the operating frequency of the camera can be, for example, 100, 125, 150, 175, 200, 225, 250, 275, 300, 325, 350, 375, 400, 425, 450, 475, 500, 525, 550, 575, 600, 625, 650, 675, 700, 725, 750, 775, 800, 825, 850, 875, 900, 925, 950, 975, 1000, 1025, 1050, 1075, 1100, 1125, 1150, 1 175, 1200, 1225, 1250, 1275, 1300, 1325, 1350, 1375, 1400, 1425, 1450, 1475, 1500, 1525, 1550, 1575, 1600, 1625, 1650, 1675, 1700, 1725, 1750, 1775, 1800, 1825, 1850, 1875, 1900, 1925, 1950, 1975, 2000 fps or may be in the range between any two of the values ​​exemplified here.

[0018] In addition, the imaging unit 2 is not limited to a camera, and a measuring sensor capable of obtaining a three-dimensional shape may be used, or a plurality of sensors with different functions may be used. Note that the viewing angle position in the projection unit 4 described later and the viewing angle position in the imaging unit 2 are matched by prior calibration. In addition, although not shown in the figure, a calibration-free system can be realized by using a coaxial optical system.

[0019] 1.3 Information Processing Unit 3 The information processing unit 3 is configured to calculate predetermined information related to the head based on the captured image and generate an image to be projected. Figure 2 It is a block diagram showing the hardware configuration of the information processing unit 3 . Figure 33 is a functional block diagram showing the functions of the control unit 33 in the information processing unit 3. The information processing unit 3 includes a communication unit 31, a storage unit 32, and a control unit 33, and these components are electrically connected inside the information processing unit 3 via a communication bus 30. In particular, regarding the control unit 33, the information processing unit 3 includes a shooting control unit 331, a projection control unit 332, a calculation unit 333, an input receiving unit 334, and an illumination control unit 335. The following further describes each component.

[0020] <Communication Department 31> The communication unit 31 preferably uses a wired communication method such as USB, IEEE1394, Thunderbolt (registered trademark), wired LAN network communication, etc., but may also include wireless LAN network communication, mobile communication such as LTE / 3G, Bluetooth (registered trademark) communication, etc. as needed. That is, it is more preferably implemented as a collection of these multiple communication units.

[0021] For example, the communication unit 31 is preferably configured to be able to communicate with the imaging unit 2, the projection unit 4, etc. according to a predetermined high-speed communication standard. Specifically, the communication unit 31 is configured so that the light emitting unit 41 of the projection unit 4 can transmit an image of the projection light to be emitted. In addition, the communication unit 31 is configured to be able to receive the captured image obtained by the imaging unit 2.

[0022] <Storage Unit 32> The storage unit 32 stores various information defined in the above description. This can be implemented as a storage device such as a solid state drive (SSD), or as a memory such as a random access memory (RAM) that stores temporary information (parameters, arrays, etc.) related to the operation of the program. In addition, it can also be a combination of these.

[0023] In particular, the storage unit 32 stores the photographing information acquired by the photographing unit 2 and received by the communication unit 31. In addition, the storage unit 32 stores a sensor control program for controlling the photographing control unit 331 in the control unit 33 to receive the photographing information from the photographing unit 2. The storage unit 32 stores a light emission control program for controlling the projection control unit 332 in the control unit 33 to control the light emitting unit 41 to emit light at a predetermined operating frequency. Furthermore, the storage unit 32 stores various programs related to the information processing unit 3 to be executed by the calculation unit 333 and the like in the control unit 33 .

[0024] <Control Unit 33> The control unit 33 processes / controls the overall actions associated with the information processing unit 3. The control unit 33 is, for example, a central processing unit (CPU) not shown. The control unit 33 implements various functions related to the information processing unit 3 by reading a predetermined program stored in the storage unit 32. Specifically, it corresponds to a shooting control function, a projection control function, a calculation function, an input acceptance function, an illumination control function, and the like. That is, the information processing based on the software (stored in the storage unit 32) is specifically implemented by the hardware (control unit 33), so that it can be executed as a shooting control unit 331, a projection control unit 332, a calculation unit 333, an input acceptance unit 334, and an illumination control unit 335. That is, the program enables the computer to function as each part of the information processing unit 3.

[0025] Note that although Figure 2 In the figure, the control unit 33 is shown as a single control unit 33, but it is not limited thereto. The control unit 33 may have a plurality of control units 33 for each function. In addition, a combination of these may be used. The various functions that the control unit 33 can realize are described in further detail below.

[0026] (Photography Control Unit 331) In the shooting control unit 331, the information processing performed by the software (stored in the storage unit 32) is specifically implemented by the hardware (control unit 33). The shooting control unit 331 controls to receive shooting information from the shooting unit 2. In addition, the operating frequency of the shooting unit 2 can be adjusted, and the shooting information obtained by the shooting unit 2 can also be used to generate a 3D projection image model.

[0027] (Projection control unit 332) In the projection control unit 332, information processing of the software (stored in the storage unit 32) is specifically implemented by the hardware (control unit 33). The projection control unit 332 is configured to be able to control the light emitting unit 41 so that the light emitting unit 41 in the projection unit 4 emits light at a predetermined operating frequency. That is, the projection control unit 332 generates a control signal for controlling the light emitting unit 41, and sends the control signal to the light emitting unit 41 of the projection unit 4 via the communication unit 31. Then, the light emitting unit 41 emits light based on the control signal to project the projection light onto the object (person P1).

[0028] (Calculation unit 333) In the operation unit 333, information processing performed by software (stored in the storage unit 32) is specifically implemented by hardware (control unit 33). The operation unit 333 is configured to be able to perform various operations such as image processing or conversion of the projected image. These are not particularly limited and can be appropriately implemented as needed.

[0029] (Input receiving unit 334) The input receiving unit 334 specifically implements information processing based on software (stored in the storage unit 32) through hardware (control unit 33). The input receiving unit 334 is configured to receive input content to the makeup input unit 6 described later, for example. These are not particularly limited and can be implemented appropriately as needed.

[0030] (Illumination Control Unit 335) The illumination control unit 335 specifically implements information processing based on software (stored in the storage unit 32) through hardware (control unit 33). The illumination control unit 335 is configured to be able to control the illumination of the location where the person P1 exists as needed. That is, it can be configured to adjust the output of the light projected from the projection unit 4, or it can be configured so that if the illumination adjustment unit 5 is electrically connected to the information processing unit 3, the illumination adjustment unit 5 is appropriately operated through the function of the illumination control unit 335.

[0031] 1.4 Projection section 4 The projection unit 4 is configured to project an image generated by the information processing unit 3 according to the position of the head. More specifically, the projection unit 4 typically includes a light emitting unit 41, and the light emitting unit 41 emits light so as to be able to project projection light containing an image onto the projection object (person P1). The appearance and internal structure of the projection unit 4 are not particularly limited, and it is preferred to use a projection unit with a high refresh rate (operating frequency) such as a so-called high-speed projector. The operating frequency of the projector is, for example, 300 Hz or more, preferably 400 Hz or more, and more preferably 500 Hz or more or 1000 Hz or more. In addition, the operating frequency of the projector can be, for example, 100, 125, 150, 175, 200, 225, 250, 275, 300, 325, 350, 375, 400, 425, 450, 475, 500, 525, 550, 575, 600, 625, 650, 675, 700, 725, 750, 775, 800, 825, 850, 875, 900, 925, 950, 975, 1000, 1025, 1050, 1075, 1100, 1125, 1150, 1 175, 1200, 1225, 1250, 1275, 1300, 1325, 1350, 1375, 1400, 1425, 1450, 1475, 1500, 1525, 1550, 1575, 1600, 1625, 1650, 1675, 1700, 1725, 1750, 1775, 1800, 1825, 1850, 1875, 1900, 1925, 1950, 1975, 2000 Hz or may be in the range between any two of the numerical values ​​exemplified here. In addition, the projection unit 4 may be configured to be able to adjust the optical axis using a reflector or the like according to the movement of the head. Therefore, projection can be performed not only from the front of the head but also from the side or upper and lower surfaces. In addition, even when the position of the person P1 moves, it is easy to perform projection that appropriately matches the position of the movement.

[0032] 1.5 Illumination adjustment unit 5 The illumination adjustment unit 5 is used to adjust the illumination of the ambient light at the location where the projection object (person P1) exists. The illumination adjustment unit 5 is typically configured to adjust the illumination of the ambient light at the location where the head exists by means of components arranged around the head of the projection object. Here, the illumination adjustment can also be achieved by blocking the light entering from the surrounding light sources. In addition, the illumination adjustment can also be achieved by emitting light from the illumination adjustment unit 5 to become the desired ambient light. Note that in Figure 1 2 shows a mode in which the illumination adjustment unit 5 exists independently, but when the illumination adjustment unit 5 includes electrical elements, the illumination adjustment unit 5 may be connected to the information processing unit 3 or the like through a circuit or the like.

[0033] 1.6 Makeup input section 6 The makeup input unit 6 is used to input the details of the simulated makeup. The makeup input unit 6 is typically a portable terminal, and as a specific example, a tablet terminal, a smart phone, etc., but a computer (Personal Computer: PC) etc. can also be used, and the details are not limited to this. In addition, the makeup input unit 6 can be the same terminal as the information processing unit 3, and predetermined information processing can be performed by inputting to the information processing unit 3.

[0034] Hereinafter, it is assumed that the makeup input unit 6 is a tablet terminal, and its hardware structure is described. Figure 4 It is a block diagram showing the hardware configuration of the tablet terminal (makeup input unit 6 ). The tablet terminal (makeup input unit 6) includes a communication unit 61, a storage unit 62, a control unit 63, a display unit 64, and an input unit 65, and these components are electrically connected inside the tablet terminal (makeup input unit 6) via a communication bus 60. The description of the communication unit 61, the storage unit 62, and the control unit 63 is substantially the same as that of the communication unit 31, the storage unit 32, and the control unit 33 in the above-mentioned information processing unit 3, and thus the description thereof is omitted here.

[0035] The display unit 64 may be included in the housing of the tablet terminal (makeup input unit 6) or may be external. The display unit 64 displays a screen of a graphical user interface (GUI) that can be operated by the user. For example, it is preferred to use a display device such as a CRT display, a liquid crystal display, an organic EL display, and a plasma display, depending on the type of terminal. Here, the case where the display unit 64 is included in the housing of the tablet terminal (makeup input unit 6) is described.

[0036] The input unit 65 can be included in the housing of the tablet terminal (makeup input unit 6) or can be external. For example, the input unit 65 can be implemented as a touch panel integrally with the display unit 64. If it is a touch panel, the user can input tapping operations, sliding operations, etc. Of course, a switch button, a mouse, a QWERTY keyboard, etc. can also be used instead of a touch panel. That is, the input unit 65 accepts the operation input performed by the user. The input is transmitted to the control unit 63 as a command signal via the communication bus 60, and the control unit 63 can perform predetermined controls and operations as needed.

[0037] Note that the image displayed on the display unit 64 of the makeup input unit 6 or the content that can be input via the image will be described later.

[0038] 2. Simulation method This section describes each step of the simulation method using the simulation system 1. That is, the simulation method of the first embodiment includes an imaging step, an information processing step, and a projection step, and this section describes each step with reference to an activity diagram or the like.

[0039] Figure 5 This is an activity diagram for explaining the simulation method of the first embodiment. First, in activity A01, images including the head are continuously captured in time series (capturing step). This capturing step is achieved by the imaging unit 2 continuously capturing the head of the person P1 in time series.

[0040] In activity A02, based on the image captured in the capturing step, predetermined information related to the head is calculated, and an image to be projected is generated (information processing step). Here, the predetermined information includes the position and shape of the head and the form of projection, and the image to be projected has a base part and one or more object parts, and one or more object parts include a part that covers the eye part included in the head.

[0041] In order to grasp the position of the head, a learned model that has previously learned the positional relationship of parts of the head that can become feature points is used. The feature points typically correspond to eyes, noses, mouths, ears, etc. When the captured image captured in the capturing step is input to the learning model, each feature point included in the captured image is calculated. In addition, a rule-based algorithm such as template matching can also be used instead of the learned model.

[0042] This information processing step is typically performed by the information processing unit 3 performing predetermined calculations or the like. Here, refer to Figure 6 Describe the images generated in this information processing step. Figure 6 This is a conceptual diagram for explaining the image generated in the information processing step. Right now, Figure 6 The image IMG1 in corresponds to the image generated in the information processing step. When generating the image IMG1, typically, a process of combining the base part BS1 with the object parts OBJ11 to OBJ14 is performed.

[0043] exist Figure 6 In the example shown, the base part BS1 is a part that constitutes the background color (base color) of the projection head, but the color of the "base part" is not necessarily limited to a single color. That is, the color constituting the base part may also be a color that constitutes a gradient or a color with a pattern on a part. However, as described later, in discussing the relationship between the illuminance of the ambient light and the illuminance of the base part, the illuminance of the base part can be defined as the color projected to the largest area among the colors projected onto the face of the head of the person P1. In addition, in discussing the relationship of the illuminance, the illuminance of the base part refers only to the light irradiated from the projection unit 4. The illuminance value of the base part can be obtained by subtracting the illuminance of the ambient light from the illuminance of the base part when the projection light projected from the projection unit 4 and the ambient light are irradiated simultaneously.

[0044] The object part OBJ11 is a part that covers the eyes included in the head, the object part OBJ12 is a part that expresses the shape of blush, the object part OBJ13 is a part that expresses the shape of eye makeup, and the object part OBJ14 is a part that expresses the shape of lips.

[0045] Note that such an image IMG1 is generated by calculating predetermined information related to the head based on the photographed image captured in the photographing step, and specifically, the predetermined information includes the position, shape, and projection form of the head. In other words, based on the parts of the face (eyes, eyelids, cheeks, lips, etc.) present in the head captured in the photographing step, the positional relationship of various objects of the above-mentioned image IMG1 is adjusted (appropriated).

[0046] Note, about Figure 6 The image IMG1 in the figure shows the concept that each object part is separated, but the method of constructing the image IMG1 is not limited to this. For example, when constructing the image IMG1, the content of each object part pre-combined can be combined with the base part BS1, or the image IMG1 composed of the base part BS1 and each object part compared with the original combination can be used.

[0047] In addition, among the various object parts described above, the part (object part OBJ11) that shields the eye part included in the head has a function of alleviating the glare of the projection light entering the eye, but the object part OBJ11 may also shield an area including the white of the eye in addition to the black of the eye (iris) (the entire eye). Note that the part (object part OBJ11) that shields the eye part included in the head is generally composed of a dark color relative to the color that constitutes the base part BS1.

[0048] In addition, such an information processing step is typically executed by the information processing unit 3 , but the information processing unit 3 may be configured to output the contents of one or more target parts based on the input to the makeup input unit 6 .

[0049] The following combination Figure 7 A method of outputting the content of the target portion based on the input to the makeup input unit 6 will be described. Figure 7 This is an example of a display screen displayed on the tablet terminal (makeup input unit 6 ).

[0050] exist Figure 7 In the display screen D of FIG. 1 , an object OBJ21 capable of selecting the shape of various makeups, an object OBJ22 capable of selecting the color of various makeups, and an object OBJ23 capable of adjusting the shade of various makeups are shown. Note that in actual operation, a shape corresponding to pressing the object OBJ21a is selected, and a different shape is selected when the object OBJ21b is pressed. For convenience, each of the object OBJ21, the object OBJ22, and the object OBJ23 is described as a collection of objects of the same type.

[0051] That is, a person (typically person P1, but may be an operator other than person P1) touching the display screen D displayed on the display unit 64 of the makeup input unit 6 selects the shape of the makeup that is desired to be applied to person P1 based on object OBJ21, and selects the color of the makeup that is desired to be applied to person P1 based on object OBJ22. Note that although a color palette capable of selecting various colors is shown in object OBJ22, objects of different shapes capable of selecting various colors may be arranged instead. In addition, in object OBJ23, the shade of each makeup can be selected based on a sliding operation of a bar.

[0052] In this way, the content input to the makeup input unit 6 is accepted by the input acceptance unit 334 of the information processing unit 3. Furthermore, the calculation unit 333 of the information processing unit 3 performs a predetermined arithmetic operation based on the received content, and generates an image to be projected. In addition, the content input to the makeup input unit 6 is not limited to Figure 7 The manner shown may also be content indicating the atmosphere or impression of makeup, such as "gorgeous" or "soft". In addition, content indicating celebrities such as actors or idols may be input content, and the information processing unit 3 may output an object portion that can be associated with the celebrity.

[0053] In performing such an information processing step, the information processing unit 3 may be configured to calculate predetermined information using a predetermined area centered on the position of the head calculated from the previous frame for the current frame of the captured image. That is, in such a structure, the position of the head is calculated in the previous frame, predetermined information is calculated based on the surrounding area of ​​the head, and an image to be projected is generated. In other words, the operation unit 333 does not perform information processing on the entire frame area in the captured image, but only performs information processing on a part of the frame area. By performing such processing, it can help to improve the information processing speed.

[0054] In addition, when performing the information processing step, the information processing unit 3 may also adjust the projection color of the image based on the reflection characteristics of a specific part of the head. In addition, the specific part corresponds to the skin, lips, hair, etc. present in the head. In addition, the reflection characteristics of the specific part may use a database obtained in advance, or may be measured on the spot. In the case of on-the-spot measurement, the imaging unit 2 may also measure the reflection characteristics of a specific part of the head. On the other hand, the measurement of the reflection characteristics may also be performed by a structure different from that of the imaging unit 2. For example, a predetermined sensor ( Figure 1 (not shown in the figure) can also detect the reflection characteristics existing in a specific part of the head, and adjust the content of the image projected by the information processing unit 3 based on the detected reflection characteristics. As a more typical example, the information processing unit 3 can also be configured to detect the error from the target color during simulation based on the reflection characteristics of a specific part of the head, and can also control the projection data or the output level of the projection unit 4 based on the detected error.

[0055] Return to Figure 5In the next activity A03, based on the predetermined information, the image generated in the information processing step is projected at the position of the head (projection step). The projection step is typically performed by the projection unit 4, but by performing such a step, an image that matches the head of the projection object can be projected, so that makeup and the like can be simulated. In addition, in the projection step, the optical axis may be adjusted using a reflective mirror or the like according to the movement of the head.

[0056] Then, similar processing is performed every other frame (activities A01, A02, A03). Thus, even when the person P1 moves, the image follows the movement and the desired image is projected onto the head ( Figure 5 In addition, when terminating the simulation, it is sufficient to stop the function of at least one of the imaging unit 2, the information processing unit 3, and the projection unit 4 ( Figure 5 'end' path in . In addition, the repetition of this action does not necessarily have to be performed every other frame, and may be configured to be performed every several frames.

[0057] Here, the simulation method of the first embodiment has the following features. That is, the time (also referred to as "waiting time") from the time when the camera unit captures a frame in the photographing step to the time when information processing corresponding to the frame is performed and an image is projected onto the head in the projecting step is 50 ms or less.

[0058] Note that the above delay is preferably less than 45ms, more preferably less than 40ms, further preferably less than 35ms, particularly preferably less than 30ms, particularly preferably less than 25ms, even more particularly preferably less than 20ms, particularly particularly preferably less than 15ms, and further particularly preferably less than 10ms. In addition, such a waiting time can be shortened by increasing the operating frequency of the imaging unit 2 and the projection unit 4 and increasing the information processing capability of the information processing unit 3. In addition, as in the example of the above-mentioned information processing step, for the current frame of the captured image, by calculating the predetermined information using a predetermined area centered on the position of the head calculated from the previous frame, it can also contribute to shortening such a delay. In addition, the lower limit value of the delay is not particularly limited, but is, for example, 0.1 ms or more.

[0059] Furthermore, the simulation method of the first embodiment may also have the following features. That is, when the illuminance of the ambient light at the position where the head is present is I1 [lux] and the illuminance of the base portion of the projected image is I2 [lux], I2 / I1 may be greater than or equal to 1.5 and less than or equal to 30. The inventors have studied and found that when the illumination of the ambient light at the position where the head is located is extremely low, the projected image is easily glaring, and when the illumination of the ambient light at the position where the head is located is extremely high, the width of the color gamut that can be expressed tends to be limited. Therefore, it is preferred to set the ratio of the illumination of the ambient light at the position where the head is located to the illumination of the base portion of the projected image within an appropriate range. By setting it within such a range, glare is less likely to be felt, and the width of the color gamut that can be expressed can be ensured.

[0060] In addition, in this specification, "ambient light at the position where the head exists" is defined as light that is excluding the portion of light directly projected onto the head by the projection unit 4 from the entire light reaching the head. Here, the ambient light typically includes natural light such as sunlight or light emitted from a flame, artificial light sources such as light emitted from a fluorescent lamp (daylight color, daylight white, white, etc.) or a light bulb, and light emitted from an optical device other than the projection unit 4 (LED, EL, laser, etc.), and light indirectly projected onto the head by the projection unit 4. In addition, the color tone of the ambient light can also be adjusted via a color balance adjustment filter, etc. Furthermore, the light indirectly projected onto the head can typically include light that reaches the head by reflecting from a part of the body different from the head, light that reaches the head by reflecting from the wall of the simulated space, etc. Therefore, it can be said that the simulation system 1 of this embodiment can provide a makeup place that assumes day and night, weather, and various events (studio, stage, watching a game, party, eating, anti-aging), etc., by setting the type of light source and illumination of the ambient light. In addition, the "light reaching the head" refers to the light irradiated to the hairline, chin, and ears of the face surrounded by the bangs connected by a curve.

[0061] Note that from the viewpoint of reducing glare, light from a light source emitting ambient light is preferably projected as indirect light reflected on a wall, etc. rather than directly projected onto the face of person P1. Further, the projection unit 4 is more preferably directly illuminated by a light source emitting ambient light.

[0062] The above I2 / I1 is preferably 3 or more, more preferably 5 or more, further preferably 7 or more, and particularly preferably 10 or more. In addition, I2 / I1 is preferably 27 or less, more preferably 25 or less, further preferably 22 or less, and particularly preferably 20 or less.

[0063] Table 1 below summarizes the use of Figure 1The results of the simulation system 1 screening the value of I2 / I1 are shown in Table 1. As shown in Table 1, by setting the value of I2 / I1 within the range, glare is less likely to be felt and the width of the color gamut that can be expressed can be ensured.

[0064] In the results shown in Table 1, a white LED was used as the ambient light. More specifically, the output of the white LED was adjusted and evaluated while I2 was fixed to 500 [lux] and I3 described later was fixed to 0 [lux].

[0065] In addition, the "non-glare degree" in Table 1 was evaluated according to the following criteria. Specifically, 10 people were projected on a panel for 10 minutes, and the number of people who responded that it was a burden on their eyes evaluated the "non-glare degree." The specific evaluation criteria are as follows. A: Less than 1 person B: 2 or more but no more than 4 people C: 5 or more people

[0066] Note that the “width of the representable color gamut” in Table 1 is evaluated according to the following criteria. That is, 30 colors were selected from the eye shadows on the market. The selected 30 colors of eye shadow were reproduced by simulation system 1, projected on the panel, and then visually evaluated by 5 professional cosmetics evaluators. Three or more professional cosmetics evaluators evaluated the "breadth of the color range that can be expressed" based on the number of colors judged to be equivalent to the actual product. The specific judgment criteria are as follows. A: More than 25 colors B: 20 ​​colors or more but less than 24 colors C: 15 colors or more but less than 19 colors D: Less than 14 colors

[0067] [Table 1] Example A1 Example A2 Example A3 Example A4 Example A5 Example A6 Example A7 Example A8 <![CDATA[I2 / I1]]> 0.7 1.5 5 10 20 30 40 50 No glare A A A A A B C C The width of the color gamut that can be expressed D B A A A A A A

[0068] In addition, I1 is preferably 5 or more, more preferably 8 or more, further preferably 10 or more, further preferably 20 or more, and particularly preferably 30 or more. On the other hand, I1 is preferably 500 or less, more preferably 400 or less, further preferably 300 or less, particularly preferably 250 or less, and particularly preferably 200 or less. By setting the illuminance of the ambient light at the position of the head within such a range, there is an advantage that it is easy to ensure the color gamut of the projection light and to achieve a natural makeup look.

[0069] Table 2 below summarizes the use of Figure 1The results of the simulation system 1 screening the value of I1 are shown in Table 2. As shown in Table 2, by setting the value of I1 within the range, there are advantages such as reducing glare, easily ensuring the color gamut of the projected light, and easily achieving a natural makeup.

[0070] In the results shown in Table 2, a white LED was used as the ambient light, and the output of the white LED was adjusted while I2 / I1 was fixed to 10 and I3 (described later) was fixed to 0 [lux] for evaluation.

[0071] In addition, the “width of the representable color gamut” in Table 2 was evaluated based on the same criteria as those in Table 1.

[0072] In addition, the "naturalness" in Table 2 was evaluated according to the following criteria. Specifically, eye shadow, blush, and lipstick were projected onto a panel and visually evaluated by five professional cosmetics evaluators. The "naturalness" was evaluated based on the number of professional cosmetics evaluators who answered that it was "natural" compared to the actual makeup. The specific judgment criteria are as follows. A: 4 or more people B: 2 or more but less than 3 people C: Less than 1 person

[0073] [Table 2] Example B1 Example B2 Example B3 Example B4 Example B5 Example B6 Example B7 Example B8 Example B9 Example B10 <![CDATA[I1]]> 2 5 15 30 50 100 200 300 500 700 The width of the color gamut that can be expressed A A A A A A A B B D Naturalness C B B A A A A A A A

[0074] In addition, when the illuminance of the area corresponding to the black part of the eye in the image projected by the projection unit 4 is I3[lux], 13 / 11 is preferably 10 or less, more preferably 5 or less, and further preferably 3 or less. That is, by setting the illuminance of the area corresponding to the black part of the eye relatively low relative to the illuminance of the ambient light, it is easier to reduce glare. The illuminance of the region corresponding to the black part of the eye is obtained by measuring the illuminance of the center of the black part of the eye (iris). In addition, the lower limit of the value of 13 / 11 is not particularly limited, but is 0.01 or more as an example, and may be 0.

[0075] Table 3 below summarizes the use of Figure 1 The results of the simulation system 1 screening the value of I3 / I1 are shown in Table 3. As shown in Table 3, by appropriately setting the value of I3 / I1, it is easier to reduce glare.

[0076] Note that in the results shown in Table 3, the evaluation was performed by adjusting the illuminance of the area corresponding to the black part of the eye while fixing I1 at 50 [lux] and I2 at 500 [lux]. As the ambient light in this case, the light emitted from the white LED was also used.

[0077] In addition, the "non-glare degree" in Table 3 was evaluated based on the same criteria as those in Table 1.

[0078] [Table 3] Example C1 Example C2 Example C3 Example C4 Example C5 <![CDATA[I3 / I1]]> 2 3 5 10 15 No glare A A B B C

[0079] In addition, when adjusting the values ​​(value ranges) of various illuminations as described above, the function of the illumination adjustment unit 5 can also be utilized. That is, when the illumination around the head is too high, the illumination adjustment unit 5 can also block the light emitted from the surrounding light source to achieve the desired illumination of the ambient light. On the other hand, when the ambient illumination is too low, the illumination adjustment unit 5 can also emit appropriate light to achieve the desired illumination of the ambient light.

[0080] In the illustrated embodiment, the imaging unit 2 detects the illuminance of the ambient light at the position where the head is present, and based on the detected illuminance, the illuminance control unit 335 of the information processing unit 3 may also control the illuminance adjustment unit 5. In addition, the detection of the illuminance of the ambient light may be performed by a structure different from that of the imaging unit 2. For example, a predetermined sensor ( Figure 1 The illumination control unit 335 of the information processing unit 3 controls the illumination adjustment unit 5 based on the detected illumination.

[0081] In addition, the imaging unit 2 or the like may detect the illuminance of the ambient light at the position where the head is present, and based on the detected illuminance, appropriately adjust the output level of the image projected by the projection unit 4. That is, when the projection unit 4 is a projector, the illuminance of the base portion may be controlled within an appropriate range by adjusting the output of the projector. In addition, for the image projected by the projection unit 4, for the shielded portion corresponding to the eye portion, the value of I3 / I1 may be controlled by adopting a color with high light shielding properties.

[0082] In addition, in order to further reduce the glare felt by person P1, dimming contact lenses may be installed on person P1.

[0083] 3. Conclusion As described above, in the simulation system 1 of the first embodiment, a predetermined image is projected onto the head, and on the other hand, a portion that shields the eye portion included in the head is included in one or more object portions. That is, the shielding portion functions to mitigate the glare of the light emitted during the projection, and by shortening the waiting time, the shielding portion can appropriately follow the movement of the head, thereby significantly achieving the effect of alleviating the glare. From such a viewpoint, it is possible to realize a simulation system in which glare is hardly felt during use during simulation.

[0084] [Second embodiment] Next, a simulation system according to a second embodiment will be described. The second embodiment will be described mainly with respect to the differences from the simulation system according to the first embodiment, and descriptions of the same matters will be omitted.

[0085] That is, the simulation system of the second embodiment is as follows. A simulation system for projecting an image onto a biological head, comprising: Shooting unit, information processing unit and projection unit; wherein, The imaging unit is configured to continuously capture images including the head in a time series; The information processing unit is configured to calculate predetermined information about the head based on the captured image, and generate an image by performing projection, wherein: The predetermined information includes the position, shape and projection form of the head, The image to be projected includes a base portion and one or more object portions; The projection unit is configured to project the image generated by the information processing unit in a manner consistent with the position of the head based on the predetermined information. When the illuminance of the ambient light at the position where the head is present is I1 [lux] and the illuminance of the base portion in the projected image is I2 [lux], I2 / I1 is greater than or equal to 1.5 and less than or equal to 30.

[0086] That is, in the simulation system of the second embodiment, like the simulation system 1 of the first embodiment, there are a shooting unit 2, an information processing unit 3 and a projection unit 4. On the other hand, for one or more object parts included in the image generated by the information processing unit 3, it is arbitrary whether to include a part that obscures the eye part included in the head, and in addition, it is arbitrary whether the delay is less than 50ms.

[0087] On the other hand, in the simulation system of the second embodiment, it is necessary to implement such that I2 / I1 is greater than or equal to 1.5 and less than or equal to 30 (the same applies to the simulation method of the second embodiment). Since an appropriate range is set for I2 / I1 in this manner, also in the second embodiment, it is possible to realize a simulation system in which glare during use is hardly felt during simulation.

[0088] [Modifications] In addition, the simulation system 1 may be implemented in the following manner.

[0089] Figure 8 1 is a block diagram showing a first modified example of the simulation system 1. In the above embodiment, the imaging unit 2, the information processing unit 3 and the projection unit 4 are shown as separate structures, but they may also be separate structures. Figure 8 As in the simulation system 1 shown in the figure, the imaging unit 2 and the information processing unit 3 are incorporated in the projection unit 4. In other words, a high-speed and high-performance projector having the functions of the imaging unit 2, the information processing unit 3, and the projection unit 4 integrated therein can be implemented.

[0090] Fig. 9 1 is a block diagram showing a second modified example of the simulation system 1. In the above embodiment, the illumination adjustment unit 5 is described as an arcade-shaped member located near the person P1, but it may also be Fig. 9 The components may be arranged in a manner to surround the person P1, the imaging unit 2, the information processing unit 3, and the projection unit 4, as in the illumination adjustment unit 5a. That is, the illumination adjustment unit 5a may constitute a space surrounding the person P1, the imaging unit 2, the information processing unit 3, and the projection unit 4, and the person P1 may perform simulation in the space.

[0091] In the above embodiment, the information processing unit 3 is described as a computer having a predetermined function. However, the information processing unit 3 may be a control circuit that converts the information captured by the imaging unit 2 into a predetermined signal and sends the signal to the projection unit 4 .

[0092] In the above embodiments, the projection part includes the head of the organism, but as a modification, such a head may not be included. For example, the projection part may be set to the body, hands, feet, etc. of the organism. In this way, clothing simulation, image simulation, nail polish or nail art simulation, etc. may be performed.

[0093] In addition, the present invention can also be provided in the following various forms.

[0094] (1) A simulation system for projecting an image onto a biological head, characterized in that it includes: a photographing unit, an information processing unit and a projection unit; wherein the photographing unit is configured to continuously photograph images containing the head in a time series; the information processing unit is configured to calculate predetermined information about the head based on the photographed images, and generate a video by performing projection, wherein the predetermined information includes the position, shape and projection form of the head, and the image to be projected includes a base part and one or more object parts, and the one or more object parts include a part that obscures the eye part contained in the head; the projection unit is configured to project the image generated by the information processing unit in a manner consistent with the position of the head based on the predetermined information, and the time from the moment when the photographing unit photographs the frame of the head to the moment when the information processing corresponding to the frame is performed and to the moment when the projection unit projects the image onto the head is less than 50 ms.

[0095] (2) A simulation system according to (1) above, wherein, when the illuminance of the ambient light at the location where the head is present is set to I1 [lux] and the illuminance of the base portion of the projected image is set to I2 [lux], I2 / I1 is greater than 1.5 and less than 30.

[0096] (3) The simulation system according to (2) above, wherein I1 is greater than or equal to 5 and less than or equal to 500.

[0097] (4) A simulation system according to (2) or (3) above, wherein, when the illuminance of the area corresponding to the black eye part in the image projected by the projection unit is set to I3 [lux], I3 / I1 is less than or equal to 10.

[0098] (5) A simulation system according to any one of (1) to (4) above, further comprising an illumination adjustment unit, wherein the illumination adjustment unit is configured to adjust the illumination of ambient light at a location where the head is located using a component disposed around the head.

[0099] (6) The simulation system according to any one of (1) to (5) above, wherein the imaging unit is a camera having an operating frequency of 300 fps or more.

[0100] (7) A simulation system according to any one of (1) to (6) above, wherein the information processing unit is configured to calculate the predetermined information for a current frame of the captured image using a predetermined area centered on the position of the head calculated based on a previous frame.

[0101] (8) The simulation system according to any one of (1) to (7) above, wherein the projection unit is a projector having an operating frequency of 300 Hz or more.

[0102] (9) The simulation system according to any one of (1) to (8) above, wherein the simulation involves applying makeup to the head.

[0103] (10) The simulation system according to (9) above, further comprising a makeup input unit, wherein the information processing unit outputs the contents of one or more target parts based on input to the makeup input unit.

[0104] (11) A simulation system for projecting an image onto a biological head, characterized in that it includes: a photographing unit, an information processing unit and a projection unit; wherein the photographing unit is configured to continuously photograph images including the head in a time series; the information processing unit is configured to calculate predetermined information about the head based on the photographed images, and generate a video by performing projection, wherein the predetermined information includes the position, shape and projection form of the head, and the image to be projected includes a base part and one or more object parts; the projection unit is configured to project the image generated by the information processing unit in a manner consistent with the position of the head based on the predetermined information, and when the illuminance of the ambient light at the position where the head exists is set to I1 [lux] and the illuminance of the base part in the projected image is set to I2 [lux], I2 / I1 is greater than 1.5 and less than 30.

[0105] (12) A simulation method using a simulation system that projects an image onto a biological head, characterized in that it includes: a shooting step, an information processing step and a projection step; wherein, in the shooting step, a captured image including the head is continuously shot in a time series; in the information processing step, predetermined information about the head is calculated based on the captured image, and a video is generated by performing projection, wherein the predetermined information includes the position, shape and projection form of the head, and the image to be projected includes a base part and one or more object parts, and the one or more object parts include a part that obscures the eye part included in the head; in the projection step, based on the predetermined information, the image generated by the information processing unit is projected in a manner consistent with the position of the head, and the time from the moment when the frame of the head is shot in the shooting step to the moment when the information processing corresponding to the frame is performed and to the moment when the image is projected onto the head in the projection step is less than 50 ms.

[0106] (13) A simulation method using a simulation system that projects an image onto a biological head, characterized in that it includes: a shooting step, an information processing step, and a projection step; wherein, in the shooting step, a captured image including the head is continuously captured in a time series; in the information processing step, predetermined information about the head is calculated based on the captured image, and a video is generated by performing projection, wherein the predetermined information includes the position, shape and projection form of the head, and the image to be projected includes a base part and one or more object parts; in the projection step, based on the predetermined information, the image generated by the information processing unit is projected in a manner consistent with the position of the head, and when the illuminance of the ambient light at the position where the head exists is set to I1 [lux] and the illuminance of the base part in the projected image is set to I2 [lux], I2 / I1 is greater than 1.5 and less than 30. Of course, it's not limited to this.

[0107] Finally, although various embodiments of the present invention have been described, these are presented only as examples and are not intended to limit the scope of the invention. Without departing from the gist of the present invention, the new embodiments of the present invention may also be implemented in various other forms and may be variously omitted, replaced, and modified. Such embodiments and their variations are included in the scope or gist of the invention and are included in the invention described in the claims and their equivalents. Description of Reference Numerals

[0108] 1: Analog system 2: Photography Department 3: Information Processing Department 4: Projection Department 5.5a: Illumination adjustment section 6: Makeup input section 30: Communication bus 31: Ministry of Communications 32: Storage 33: Control Department 41: Luminous part 60: Communication bus 61: Ministry of Communications 62: Storage 63: Control Department 64: Display unit 65: Input 331: Shooting Control Department 332: Projection Control Department 333: Operation Department 334: Input receiving unit 335: Illumination Control Department BS1: Basics D: Display screen IMG1: Image OBJ11~OBJ14: Object part OBJ21~OBJ23, OBJ21a, OBJ21b: Object P1: People

Claims

1. A simulation system for projecting an image onto a biological head, characterized in that: include: Shooting unit, information processing unit and projection unit; wherein, The imaging unit is configured to continuously capture images including the head in a time series; The information processing unit is configured to calculate predetermined information about the head based on the captured image, and generate an image by performing projection, wherein: The predetermined information includes the position, shape and projection form of the head, The image to be projected includes a base portion and one or more object portions, One or more of the object portions include a portion that obscures an eye portion included in the head; The projection unit is configured to project the image generated by the information processing unit in a manner consistent with the position of the head based on the predetermined information. The time from when the imaging unit images the frame of the head to when information processing corresponding to the frame is performed and when the projection unit projects the image onto the head is 50 ms or less.

2. The simulation system according to claim 1, wherein: When the illuminance of the ambient light at the position where the head is present is I1 [lux] and the illuminance of the base portion in the projected image is I2 [lux], I2 / I1 is greater than or equal to 1.5 and less than or equal to 30.

3. The simulation system according to claim 2, wherein: The I1 is greater than or equal to 5 and less than or equal to 500.

4. The simulation system according to claim 2 or 3, wherein: When the illuminance of a region corresponding to the black eye portion in the image projected by the projection unit is set to I3 [lux], I3 / I1 is 10 or less.

5. The simulation system according to any one of claims 1 to 4, wherein: It also has an illumination adjustment unit. The illumination adjustment unit is configured to adjust the illumination of ambient light at a position where the head is located using a member disposed around the head.

6. The simulation system according to any one of claims 1 to 5, wherein: The photographing unit is a camera with an operating frequency of 300 fps or more.

7. The simulation system according to any one of claims 1 to 6, wherein: The information processing unit is configured to calculate the predetermined information using a predetermined area centered on the position of the head calculated from a previous frame for a current frame of the captured image.

8. The simulation system according to any one of claims 1 to 7, wherein: The projection unit is a projector with an operating frequency of 300 Hz or more.

9. The simulation system according to any one of claims 1 to 8, wherein: Simulate applying makeup to the head.

10. The simulation system according to claim 9, wherein: It also has a makeup input unit. The information processing unit outputs the content of one or more of the target parts based on the input to the makeup input unit.

11. A simulation system for projecting an image onto a biological head, characterized in that: include: Shooting unit, information processing unit and projection unit; wherein, The imaging unit is configured to continuously capture images including the head in a time series; The information processing unit is configured to calculate predetermined information about the head based on the captured image, and generate an image by performing projection, wherein: The predetermined information includes the position, shape and projection form of the head, The image to be projected includes a base portion and one or more object portions; The projection unit is configured to project the image generated by the information processing unit in a manner consistent with the position of the head based on the predetermined information. When the illuminance of the ambient light at the position where the head is present is I1 [lux] and the illuminance of the base portion in the projected image is I2 [lux], I2 / I1 is greater than or equal to 1.5 and less than or equal to 30.

12. A simulation method using a simulation system for projecting an image onto a biological head, characterized in that: include: shooting step, information processing step and projection step; wherein, In the photographing step, photographic images including the head are continuously photographed in a time series; In the information processing step, predetermined information about the head is calculated based on the captured image, and an image is generated by performing projection, wherein, The predetermined information includes the position, shape and projection form of the head, The image to be projected includes a base portion and one or more object portions, One or more of the object portions include a portion that obscures an eye portion included in the head; In the projecting step, based on the predetermined information, the image generated by the information processing unit is projected in a manner consistent with the position of the head. The time from when the frame of the head is captured in the capturing step to when information processing corresponding to the frame is performed and to when the image is projected onto the head in the projecting step is 50 ms or less.

13. A simulation method using a simulation system for projecting an image onto a biological head, characterized in that: include: shooting step, information processing step and projection step; wherein, In the photographing step, photographic images including the head are continuously photographed in a time series; In the information processing step, predetermined information about the head is calculated based on the captured image, and an image is generated by performing projection, wherein, The predetermined information includes the position, shape and projection form of the head, The image to be projected includes a base portion and one or more object portions; In the projecting step, based on the predetermined information, the image generated by the information processing unit is projected in a manner consistent with the position of the head. When the illuminance of the ambient light at the position where the head is present is I1 [lux] and the illuminance of the base portion in the projected image is I2 [lux], I2 / I1 is greater than or equal to 1.5 and less than or equal to 30.

Images