Simulation system and method of simulation

Abstract

[Problem] To provide a simulation system, etc., with which glare at use time is not readily felt felt when a simulation is performed. [Solution] According an embodiment of the present invention, provided is a simulation system that projects a video to the head of a living being. This simulation system comprises an image-capturing unit, an information processing unit, and a projection unit. The image-capturing unit is configured to successively capture images in time series, the captured images including the head. The information processing unit is configured to calculate prescribed information relating to the head and generate a video to be projected, on the basis of the captured images. The prescribed information includes the position, shape, and projected form of the head. The video to be projected has a base portion and one or more object portions. The one or more object portions include a portion that masks the eyes included in the head. The projection unit is configured to project the video generated by the information processing unit, in conformance with the position of the head, on the basis of the prescribed information. A configuration is adopted such that a duration from the time of a frame in which the image-capturing unit captures an image of the head until the time at which information processing corresponding to the frame is carried out and the projection unit projects the video to the head is 50 ms or less.

Description

Simulation system and simulation method

[0001] The present invention relates to a simulation system and a simulation method.

[0002] In recent years, a technology known as "projection mapping," which projects images onto three-dimensional objects, has been attracting attention. Furthermore, developments have been made to apply this technology to project makeup images onto human faces, etc. Related technologies include those disclosed in Patent Documents 1 and 2.

[0003] JP 2018-195996 A JP 2022-45944 A

[0004] However, the inventors have found through their research that the light from the projector used in the above-mentioned projection technology can be dazzling.

[0005] In view of the above circumstances, the present invention provides a simulation system and the like that is less susceptible to glare when used to perform a simulation.

[0006] According to one aspect of the present invention, there is provided a simulation system for projecting an image onto the head of a living creature. The simulation system includes an imaging unit, an information processing unit, and a projection unit. The imaging unit is configured to continuously capture images including the head in a time series. The information processing unit is configured to calculate predetermined information about the head based on the captured images and generate an image to be projected. The predetermined information includes the position, shape, and projection manner of the head. The image to be projected has a base portion and one or more object portions. The one or more object portions include a portion that masks the eyes included in the head. The projection unit is configured to project the image generated by the information processing unit to match the position of the head based on the predetermined information. The projection unit is configured so that the time from the time of a frame in which the imaging unit captures an image of the head to the time when the projection unit projects an image onto the head after information processing corresponding to this frame is performed is 50 ms or less.

[0007] According to the above aspect, a simulation system or the like is provided that is less susceptible to glare when used to perform a simulation.

[0008] FIG. 1 is a configuration diagram showing a simulation system 1 according to a first embodiment. FIG. 2 is a block diagram showing the hardware configuration of an information processing unit 3. FIG. 3 is a functional block diagram showing the functions of a control unit 33 in the information processing unit 3. FIG. 4 is a block diagram showing the hardware configuration of a tablet terminal (makeup input unit 6). FIG. 5 is an activity diagram for explaining a simulation method according to the first embodiment. FIG. 6 is a conceptual diagram for explaining images generated in information processing steps. FIG. 7 is an example of a display screen displayed on a tablet terminal (makeup input unit 6). FIG. 8 is a configuration diagram showing a first modified example of the simulation system 1. FIG. 9 is a configuration diagram showing a second modified example of the simulation system 1.

[0009] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described below with reference to the accompanying drawings. Various features shown in the following embodiments can be combined with each other.

[0010] Incidentally, the program for realizing the software appearing in this embodiment may be provided as a non-transitory computer-readable recording medium, or may be provided so as to be downloadable from an external server, or may be provided so that the program is started on an external computer and its functions are realized on a client terminal (so-called cloud computing).

[0011] In this embodiment, the term "unit" may include, for example, a combination of hardware resources implemented by a circuit in the broad sense and software information processing that can be specifically realized by these hardware resources. In addition, various types of information are handled in this embodiment, and this information may be represented by, for example, physical values ​​of signal values ​​representing voltages and currents, high and low signal values ​​as a binary bit set consisting of 0 or 1, or quantum superposition (so-called quantum bits), and communication and calculations may be performed on a circuit in the broad sense.

[0012] Furthermore, a circuit in the broad sense is a circuit realized by at least an appropriate combination of a circuit, circuitry, a processor, a memory, etc. That is, it includes an application specific integrated circuit (ASIC), a programmable logic device (e.g., a simple programmable logic device (SPLD), a complex programmable logic device (CPLD), and a field programmable gate array (FPGA)), etc.

[0013] First Embodiment First, a simulation system according to the first embodiment will be described. That is, the simulation system according to the first embodiment is as follows: A simulation system for projecting an image onto the head of a living creature, comprising: an imaging unit, an information processing unit, and a projection unit; the imaging unit is configured to continuously capture images including the head in a time series; the information processing unit is configured to calculate predetermined information about the head based on the captured images and generate an image to be projected, wherein the predetermined information includes the position, shape, and projection aspect of the head; the image to be projected has a base portion and one or more object portions, wherein the one or more object portions include portions that mask the eyes included in the head; and the projection unit is configured to project the image generated by the information processing unit based on the predetermined information, aligned with the position of the head; and the simulation system is configured so that the time from the time of a frame in which the imaging unit captures an image of the head to the time when the projection unit projects the image onto the head after information processing corresponding to this frame is performed is 50 ms or less. The configuration and uses of this simulation system will be described below.

[0014] 1. Hardware Configuration In this section, the hardware configuration of the first embodiment will be described.

[0015] 1.1 Simulation System 1 FIG. 1 is a configuration diagram illustrating a simulation system 1 according to a first embodiment. The simulation system 1 projects an image onto the head of a living creature. FIG. 1 illustrates a system projecting an image onto the head of a human P1. The simulation system 1 can also be applied to animals other than humans. While the simulation system 1 can be used for a variety of purposes, it is typically used to simulate makeup on the head. In this specification, "makeup" encompasses makeup applied to various parts of the face, such as foundation, highlighter, shading, eyeshadow, eyeliner, eyebrows, mascara, blush, and lipstick, as well as hair coloring. In addition to makeup on the head, the simulation system 1 may also be used for simulating accessories (decorations) on the head, simulating hairstyles, simulating the increase or decrease of three-dimensional structures such as pores and wrinkles, simulating the increase or decrease of uneven skin tone such as acne and blemishes, simulating cosmetic surgery, and visualizing the location of subcutaneous tissue. Furthermore, the projection area may include areas other than the head. For example, as described above, projection may be performed onto the torso, hands, legs, etc., so that makeup can be simulated on the head while also simulating clothing to match the makeup.

[0016] The simulation system 1 includes an imaging unit 2, an information processing unit 3, and a projection unit 4, which are connected via a circuit. Although not essential components in this embodiment, the simulation system 1 in Fig. 1 also includes an illumination adjustment unit 5 and a makeup input unit 6 in addition to the above components. These components will be further described below.

[0017] 1.2 Imaging Unit 2 The imaging unit 2 is configured to capture images including the head continuously in time series. The imaging unit 2 is, for example, a camera configured to acquire information about the external world as images (visible light, infrared light, etc. can be used as appropriate). It is preferable to use such a camera with a high operating frequency (frame rate), known as a high-speed vision. The frame rate (operating frequency) of the camera is, for example, 300 fps or higher, preferably 400 fps or higher, and more preferably 500 fps or higher or 1000 fps or higher. The operating frequency of the camera can be set to, for example, 100, 125, 150, 175, 200, 225, 250, 275, 300, 325, 350, 375, 400, 425, 450, 475, 500, 525, 550, 575, 600, 625, 650, 675, 700, 725, 750, 775, 800, 825, 850, 875, 900, 925, 950, 975, 1000, 1025, 1050, 1075, 1100, 1125, 1150, 1175, 1 The value may be 200, 1225, 1250, 1275, 1300, 1325, 1350, 1375, 1400, 1425, 1450, 1475, 1500, 1525, 1550, 1575, 1600, 1625, 1650, 1675, 1700, 1725, 1750, 1775, 1800, 1825, 1850, 1875, 1900, 1925, 1950, 1975, or 2000 fps, or may be within a range between any two of the values ​​exemplified here.

[0018] Furthermore, the imaging unit 2 is not limited to a camera, and may employ a measurement sensor capable of acquiring a three-dimensional shape, or may employ a plurality of sensors with different functions. It is assumed that the angle of view position of the projection unit 4 (described later) and the angle of view position of the imaging unit 2 are matched by prior calibration. Although not shown, a calibration-free system may be implemented by employing a coaxial optical system.

[0019] 1.3 Information Processing Unit 3 The information processing unit 3 is configured to calculate predetermined information related to the head based on the captured image and generate an image to be projected. FIG. 2 is a block diagram showing the hardware configuration of the information processing unit 3. FIG. 3 is a functional block diagram showing the function of the control unit 33 in the information processing unit 3. The information processing unit 3 has a communication unit 31, a storage unit 32, and a control unit 33, and these components are electrically connected via a communication bus 30 inside the information processing unit 3. In particular, with regard to the control unit 33, the information processing unit 3 includes an imaging control unit 331, a projection control unit 332, a calculation unit 333, an input reception unit 334, and an illuminance control unit 335. Each component will be described in further detail below.

[0020] <Communication Unit 31> The communication unit 31 is preferably a wired communication means such as USB, IEEE 1394, Thunderbolt (registered trademark), wired LAN network communication, etc., but may also include wireless LAN network communication, mobile communication such as LTE / 3G, Bluetooth (registered trademark) communication, etc. as necessary. In other words, it is more preferable to implement the communication unit 31 as a collection of these multiple communication means.

[0021] For example, the communication unit 31 is preferably configured to be able to communicate with the image capture unit 2, the projection unit 4, etc., using a predetermined high-speed communication standard. Specifically, the communication unit 31 is configured to be able to transmit an image of projection light emitted by the light emitting unit 41 of the projection unit 4. The communication unit 31 is also configured to be able to receive captured images acquired by the image capture unit 2.

[0022] <Storage Unit 32> The storage unit 32 stores various pieces of information defined above. This may be implemented as a storage device such as a solid state drive (SSD), or as a memory such as a random access memory (RAM) that stores temporarily required information (arguments, arrays, etc.) related to program calculations. Alternatively, the storage unit 32 may be implemented as a combination of these.

[0023] In particular, the storage unit 32 stores imaging information acquired by the imaging unit 2 and received by the communication unit 31. The storage unit 32 also stores a sensor control program for controlling the imaging control unit 331 in the control unit 33 to receive imaging information from the imaging unit 2. The storage unit 32 also stores a light emission control program for the projection control unit 332 in the control unit 33 to control the light emitting unit 41 to emit light at a predetermined operating frequency. The storage unit 32 also stores various other programs related to the information processing unit 3 that are executed by the calculation unit 333 in the control unit 33, etc.

[0024] <Control Unit 33> The control unit 33 processes and controls the overall operation related to the information processing unit 3. The control unit 33 is, for example, a central processing unit (CPU) (not shown). The control unit 33 realizes various functions related to the information processing unit 3 by reading out predetermined programs stored in the storage unit 32. Specifically, these functions include an imaging control function, a projection control function, a calculation function, an input acceptance function, and an illuminance control function. In other words, information processing by software (stored in the storage unit 32) is specifically realized by hardware (the control unit 33), and can be executed as the imaging control unit 331, the projection control unit 332, the calculation unit 333, the input acceptance unit 334, and the illuminance control unit 335. In other words, the programs can cause a computer to function as each unit of the information processing unit 3.

[0025] 2 shows a single control unit 33, but in reality, this is not limited to this, and the system may be implemented with multiple control units 33 for each function. Also, a combination of these may be used. The various functions that the control unit 33 can realize will be described in more detail below.

[0026] (Imaging control unit 331) The imaging control unit 331 specifically realizes information processing by software (stored in the storage unit 32) using hardware (control unit 33). The imaging control unit 331 controls the imaging unit 2 to receive imaging information and the like. The imaging control unit 331 may also adjust the operating frequency of the imaging unit 2, and may also be implemented to generate a 3D projection image model using the imaging information acquired by the imaging unit 2.

[0027] (Projection control unit 332) The projection control unit 332 is a unit in which information processing by software (stored in the storage unit 32) is specifically realized by hardware (control unit 33). The projection control unit 332 is configured to be able to control the light-emitting unit 41 in the projection unit 4 so that the light-emitting unit 41 emits light at a predetermined operating frequency. In other words, the projection control unit 332 generates a control signal for controlling the light-emitting unit 41, and this control signal is transmitted to the light-emitting unit 41 in the projection unit 4 via the communication unit 31. Then, based on this control signal, the light-emitting unit 41 emits light to project projection light onto the target (person P1).

[0028] (Calculation unit 333) The calculation unit 333 is a unit in which information processing by software (stored in the storage unit 32) is specifically realized by hardware (control unit 33). The calculation unit 333 is configured to be able to perform various calculations, such as image processing and conversion of the projected image. These are not particularly limited and can be performed as needed.

[0029] (Input Receiving Unit 334) The input receiving unit 334 is a unit in which information processing by software (stored in the storage unit 32) is specifically realized by hardware (control unit 33). The input receiving unit 334 is configured to be able to receive, for example, input content to the makeup input unit 6 described below. These are not particularly limited and can be implemented appropriately as needed.

[0030] (Illuminance control unit 335) The illuminance control unit 335 is a unit in which information processing by software (stored in the storage unit 32) is specifically realized by hardware (control unit 33). The illuminance control unit 335 is configured to be able to control the illuminance at the location where the person P1 is present as necessary. In other words, it may be configured to adjust the output of light projected from the projection unit 4, and if the illuminance adjustment unit 5 is electrically connected to the information processing unit 3, it may be configured to appropriately operate the illuminance adjustment unit 5 by the function of this illuminance control unit 335.

[0031] 1.4 Projection Unit 4 The projection unit 4 is configured to project the image generated by the information processing unit 3 in accordance with the position of the head. More specifically, the projection unit 4 typically includes a light-emitting unit 41, which is configured to emit projection light containing an image so that it can be projected onto the projection target (human P1). The external appearance and internal configuration of the projection unit 4 are not particularly limited, but it is preferable to use a so-called high-speed projector with a high refresh rate (operating frequency). The operating frequency of the projector is, for example, 300 Hz or higher, preferably 400 Hz or higher, and more preferably 500 Hz or higher or 1000 Hz or higher. The operating frequency of the projector is, for example, 100, 125, 150, 175, 200, 225, 250, 275, 300, 325, 350, 375, 400, 425, 450, 475, 500, 525, 550, 575, 600, 625, 650, 675, 700, 725, 750, 775, 800, 825, 850, 875, 900, 925, 950, 975, 1000, 1025, 1050, 1075, 1100, 1125, 1150, 1175 , 1200, 1225, 1250, 1275, 1300, 1325, 1350, 1375, 1400, 1425, 1450, 1475, 1500, 1525, 1550, 1575, 1600, 1625, 1650, 1675, 1700, 1725, 1750, 1775, 1800, 1825, 1850, 1875, 1900, 1925, 1950, 1975, or 2000 Hz, or within a range between any two of the values ​​exemplified here. Furthermore, the projection unit 4 may be configured to be able to adjust the optical axis using a mirror or the like in accordance with head movement. This makes it easier to project images not only from the front of the head, but also from the side, top, and bottom. In addition, even if the position of the person P1 moves, it becomes easier to perform projection that is appropriately adjusted to the moved position.

[0032] 1.5 Illuminance Adjustment Unit 5 The illuminance adjustment unit 5 is used to adjust the illuminance of ambient light at the location of the projection target (human P1). This illuminance adjustment unit 5 is typically configured to adjust the illuminance of ambient light at the location of the head using a component disposed around the head of the projection target. Here, the adjustment of illuminance may be achieved by blocking light from surrounding light sources. Alternatively, the illuminance adjustment unit 5 may emit light to achieve the desired ambient light. While FIG. 1 shows an embodiment in which the illuminance adjustment unit 5 exists independently, if the illuminance adjustment unit 5 includes electrical elements, the illuminance adjustment unit 5 may be connected to the information processing unit 3 or the like via a circuit or the like.

[0033] 1.6 Makeup Input Unit 6 The makeup input unit 6 is used to input details of the makeup to be simulated. The makeup input unit 6 is typically a portable terminal, and specific examples include a tablet terminal, a smartphone, etc., but the makeup input unit 6 may also be a personal computer (PC), and the details are not limited thereto. Furthermore, the makeup input unit 6 may be the same terminal as the information processing unit 3, and predetermined information processing may be performed based on the input to the information processing unit 3.

[0034] The hardware configuration of the makeup input unit 6 will be described below assuming that the unit is a tablet terminal. Figure 4 is a block diagram showing the hardware configuration of the tablet terminal (makeup input unit 6). The tablet terminal (makeup input unit 6) has a communication unit 61, a storage unit 62, a control unit 63, a display unit 64, and an input unit 65, and these components are electrically connected within the tablet terminal (makeup input unit 6) via a communication bus 60. Descriptions of the communication unit 61, storage unit 62, and control unit 63 will be omitted because they are substantially the same as the communication unit 31, storage unit 32, and control unit 33 in the information processing unit 3 described above.

[0035] The display unit 64 may be, for example, included in the housing of the tablet terminal (makeup input unit 6) or may be externally attached. The display unit 64 displays a graphical user interface (GUI) screen that can be operated by the user. This is preferably implemented by using a display device such as a CRT display, a liquid crystal display, an organic EL display, or a plasma display, depending on the type of terminal. Here, the display unit 64 will be described as being included in the housing of the tablet terminal (makeup input unit 6).

[0036] The input unit 65 may be included in the housing of the tablet terminal (makeup input unit 6) or may be externally attached. For example, the input unit 65 may be implemented as a touch panel integrated with the display unit 64. A touch panel allows the user to input tapping, swiping, and the like. Of course, a switch button, a mouse, a QWERTY keyboard, or the like may be used instead of a touch panel. That is, the input unit 65 accepts an operation input made by the user. The input is transferred as a command signal to the control unit 63 via the communication bus 60, and the control unit 63 can execute predetermined control or calculation as necessary.

[0037] The images displayed on the display unit 64 of the makeup input unit 6 and the content that can be input using these images will be described later.

[0038] 2. Simulation Method This section describes each step of the simulation method using the above-described simulation system 1. That is, the simulation method of the first embodiment includes an imaging step, an information processing step, and a projection step, and this section describes each of these steps with reference to an activity diagram, etc.

[0039] 5 is an activity diagram for explaining the simulation method of the first embodiment. First, in activity A01, images including the head are captured continuously in time series (imaging step). This imaging step is achieved by the imaging unit 2 capturing images of the head of a human P1 continuously in time series.

[0040] In activity A02, predetermined information about the head is calculated based on the captured image captured in the imaging step, and an image to be projected is generated (information processing step). Here, this predetermined information includes the position, shape, and projection manner of the head, and the image to be projected has a base portion and one or more object portions, and the one or more object portions include portions that mask the eyes included in the head.

[0041] To grasp the position of the head, a trained model is used that has previously learned the positional relationships of parts of the head that can be feature points. Typical examples of these feature points include the eyes, nose, mouth, and ears. When the captured image captured in the imaging step is input to this trained model, each feature point included in the captured image is calculated. Alternatively, a rule-based algorithm such as template matching may be used instead of the trained model.

[0042] This information processing step is typically executed by the information processing unit 3 performing predetermined calculations, etc. The image generated in this information processing step will now be described with reference to Fig. 6. Fig. 6 is a conceptual diagram for explaining the image generated in the information processing step. That is, the image IMG1 in Fig. 6 corresponds to the image generated in the information processing step, and in generating this image IMG1, a process of combining a base portion BS1 with object portions OBJ11 to OBJ14 is typically performed.

[0043] In the example shown in FIG. 6 , the base portion BS1 constitutes the skin color (base color) of the head to be projected, but the color of the "base portion" is not necessarily limited to a single color. That is, the color constituting the base portion may be a color that forms a gradation or a color with a pattern applied to a portion of the base portion. However, when discussing the relationship between the illuminance of ambient light and the illuminance of the base portion, as described below, the illuminance of this base portion can be defined as the color that is projected over the widest area of ​​the colors projected onto the face of the head of human P1. Furthermore, when discussing the relationship between illuminance, the illuminance of the base portion refers only to the light irradiated from the projection unit 4. The illuminance value of this base portion can be calculated by subtracting the illuminance of ambient light from the illuminance of the base portion when the projection light projected from the projection unit 4 and the ambient light are simultaneously irradiated.

[0044] Object part OBJ11 is a part that masks the eye part included in the head, object part OBJ12 is a part that represents the shape of the cheeks, object part OBJ13 is a part that represents the shape of the eye makeup, and object part OBJ14 is a part that represents the shape of the lips.

[0045] The image IMG1 is generated by calculating predetermined information about the head based on the captured image in the imaging step, and specifically, this predetermined information includes the position, shape, and projection mode of the head. In other words, the positional relationships of various objects in the image IMG1 are adjusted (optimized) based on the facial features (eyes, eyelids, cheeks, lips, etc.) present on the head captured in the imaging step.

[0046] 6 shows the concept of each object part being separated from the other object parts, but the method of constructing the image IMG1 is not limited to this. For example, when constructing the image IMG1, the content of each object part that has been previously combined may be combined with the base part BS1, or the image IMG1 may initially be a combination of the base part BS1 and each object part.

[0047] Furthermore, among the various object parts described above, the part that masks the eye part included in the head (object part OBJ11) has the function of reducing the glare of the projection light that enters the eyes, but this object part OBJ11 may mask not only the pupil (iris) but also the area including the white of the eye (the entire eye). Note that the part that masks the eye part included in the head (object part OBJ11) is usually made up of a darker color than the color that makes up the base part BS1.

[0048] Furthermore, such information processing steps are typically performed by the information processing unit 3, but the information processing unit 3 may be configured to output the contents of one or more object parts based on input to the makeup input unit 6.

[0049] A mode in which the content of the object portion is output based on an input to the makeup input unit 6 will be described with reference to Fig. 7. Fig. 7 shows an example of a display screen displayed on the tablet terminal (makeup input unit 6).

[0050] 7 shows an object OBJ21 that allows the user to select various makeup shapes, an object OBJ22 that allows the user to select various makeup colors, and an object OBJ23 that allows the user to adjust the shade of various makeup. Note that in actual operation, when object OBJ21a is pressed, the corresponding shape is selected, and when object OBJ21b is pressed, a different shape is selected. However, for convenience, the above-mentioned objects OBJ21, OBJ22, and OBJ23 will be described as a collection of objects of the same type.

[0051] That is, a person (typically human P1, but may be an operator other than human P1) viewing the display screen D displayed on the display unit 64 of the makeup input unit 6 selects the type of makeup they wish to apply to human P1 based on object OBJ21, and selects the color of makeup they wish to apply to human P1 based on object OBJ22. Note that while object OBJ22 displays a color palette from which various colors can be selected, objects of different shapes from which various colors can be selected may be arranged instead. Furthermore, with object OBJ23, the shade of each type of makeup can be selected by sliding a bar.

[0052] The content input to the makeup input unit 6 in this manner is accepted by the input accepting unit 334 of the information processing unit 3. The calculation unit 333 of the information processing unit 3 then executes a predetermined calculation based on the accepted content to generate an image to be projected. The content input to the makeup input unit 6 is not limited to the form shown in FIG. 7 , and may be information indicating the atmosphere or image of the makeup, such as "gorgeous" or "soft." Furthermore, the input content may represent a celebrity, such as an actor or idol, and the information processing unit 3 may output an object portion that evokes the celebrity.

[0053] Furthermore, when performing such information processing steps, the information processing unit 3 may be configured to calculate the predetermined information for the current frame of the captured image using a predetermined area centered on the head position calculated from the immediately preceding frame. That is, in such a configuration, the head position is calculated in the immediately preceding frame, and the predetermined information is calculated and an image to be projected is generated based on the area around the head. In other words, the calculation unit 333 does not process information for the entire frame area of ​​the captured image, but only for a partial area of ​​the frame. Performing such processing can contribute to improving the information processing speed.

[0054] Furthermore, when the information processing step is performed, the information processing unit 3 may adjust the projected color of the image based on the reflection characteristics of a specific part of the head. The specific part corresponds to the skin, lips, hair, etc., present on the head. The reflection characteristics of the specific part may be determined using a database acquired in advance or may be measured on the spot. When measuring on the spot, the image capture unit 2 may measure the reflection characteristics of the specific part of the head. Alternatively, the measurement of the reflection characteristics may be performed by a configuration different from the image capture unit 2. For example, a predetermined sensor (not shown in FIG. 1 ) connected to the information processing unit 3 may detect the reflection characteristics of the specific part of the head, and the content of the image projected by the information processing unit 3 may be adjusted based on the detected reflection characteristics. In a more typical example, the information processing unit 3 may be configured to detect an error from the target color during simulation based on the reflection characteristics of the specific part of the head, and the projection data, the output level of the projection unit 4, etc. may be controlled based on the detected error.

[0055] Returning to the explanation of the activity diagram in Figure 5, in the following activity A03, the image generated in the information processing step is projected based on predetermined information in accordance with the position of the head (projection step). This projection step is typically performed by the projection unit 4, and by performing such a step, it is possible to project an image that fits the head of the projection target, thereby enabling simulation of makeup, etc. Note that in this projection step, the optical axis may be adjusted using a mirror or the like in accordance with the movement of the head.

[0056] Thereafter, the same processing (activities A01, A02, A03) is executed every other frame. As a result, even if the person P1 moves, the image follows this movement and the desired image is projected onto the head ([Next] route in FIG. 5). To end the simulation, the functions of at least one of the imaging unit 2, information processing unit 3, and projection unit 4 can be stopped ([End] route in FIG. 5). Note that this activity does not necessarily have to be repeated every other frame, and it can also be configured to be repeated every few frames.

[0057] The simulation method of the first embodiment has the following characteristics: The time (which may be referred to as "latency") from the time of a frame in which the head is imaged in the imaging step to the time when information processing corresponding to this frame is performed and an image is projected onto the head in the projection step is 50 ms or less.

[0058] The latency is preferably 45 ms or less, more preferably 40 ms or less, even more preferably 35 ms or less, even more preferably 30 ms or less, particularly preferably 25 ms or less, even more preferably 20 ms or less, even more preferably 15 ms or less, even more preferably 10 ms or less, and especially preferably 5 ms or less. This reduction in latency time may be achieved by improving the operating frequency of the imaging unit 2 or the projection unit 4 or by improving the information processing capability of the information processing unit 3. Furthermore, as an example of the information processing step described above, calculating predetermined information for the current frame of the captured image using a predetermined region centered on the head position calculated from the immediately preceding frame can also contribute to reducing this latency. The lower limit of the latency is not particularly limited, but an example is 0.1 ms or more.

[0059] The simulation method of the first embodiment may have the following features: １ [lux], and the illuminance of the base part of the projected image is I ２ When [lux] is used, I ２ / I １The ratio may be configured to be 1.5 or more and 30 or less. The inventors' studies have revealed that when the illuminance of the ambient light at the position of the head is extremely low, the projected image tends to be dazzling, and when the illuminance of the ambient light at the position of the head is extremely high, the range of the color gamut that can be expressed tends to be limited. For this reason, it is preferable to set the ratio between the illuminance of the ambient light at the position of the head and the illuminance of the base part of the projected image within an appropriate range. By setting the ratio within such a range, the user is less likely to feel dazzled and the range of the color gamut that can be expressed can be ensured.

[0060] In this specification, "ambient light at the location where the head is located" is defined as the total light reaching the head, excluding the portion of light directly projected onto the head by the projection unit 4. Here, this ambient light typically includes natural light such as sunlight or light emitted from a flame, artificial light sources such as fluorescent lamps (daylight, neutral white, white, etc.), light bulbs, and light emitted from optical devices other than the projection unit 4 (LED, EL, laser, etc.), as well as light indirectly projected onto the head by the projection unit 4. The color tone of this ambient light may be adjusted using a color balance adjustment filter or the like. Furthermore, the light indirectly projected onto the head typically includes light that reaches the head after reflecting off a body part other than the head, or light that reaches the head after reflecting off a wall of the simulation space. Therefore, the simulation system 1 of this embodiment can provide makeup application opportunities that simulate day and night, weather, and various events (studio, stage, watching a game, meeting, dining, anti-aging), etc., by setting the type and illuminance of the light source of the ambient light. Furthermore, "light reaching the head" refers to light that is irradiated onto the area of ​​the face surrounded by the curved lines connecting the hairline, chin, and both ears.

[0061] From the viewpoint of reducing glare, it is preferable that the light from the light source emitting ambient light is projected as indirect light reflected from a wall or the like rather than being projected directly onto the face of the person P1. It is also more preferable that the projection unit 4 is directly illuminated by the light source emitting ambient light.

[0062] The above-mentioned I ２ / I １is preferably 3 or more, more preferably 5 or more, even more preferably 7 or more, and particularly preferably 10 or more. ２ / I １ is preferably 27 or less, more preferably 25 or less, even more preferably 22 or less, and particularly preferably 20 or less.

[0063] The following Table 1 shows the results of the simulation using the simulation system 1 shown in FIG. ２ / I １ As shown in Table 1, the results of screening the values ​​of I ２ / I １ By setting the value within a range, it is possible to reduce the sense of glare and ensure a wide color gamut that can be expressed.

[0064] In the results shown in Table 1, a white LED was used as the ambient light. ２ 500 [lux], and I ３ The evaluation was performed by adjusting the output of the white LED while fixing the luminance at 0 [lux].

[0065] Furthermore, the "lack of glare" in Table 1 was evaluated according to the following criteria. That is, the image was projected onto a panel of 10 people for 10 minutes, and the "lack of glare" was evaluated according to the number of people who responded that they felt strain on their eyes. The specific criteria for evaluation are as follows: A: 1 person or less B: 2 to 4 people C: 5 people or more

[0066] The "width of the color gamut that can be expressed" in Table 1 was evaluated according to the following criteria. That is, 30 colors were selected from eye shadows on the market. The selected 30 colors of eye shadow were reproduced using the simulation system 1, projected onto a panel, and visually evaluated by five cosmetic product specialists. The "width of the color gamut that can be expressed" was evaluated according to the number of colors that three or more cosmetic product specialists judged to be the same color as the actual product. The specific evaluation criteria are as follows: A: 25 colors or more B: 20 ​​to 24 colors C: 15 to 19 colors D: 14 ​​colors or less

[0067]

[0068] Also, I １ is preferably 5 or more, more preferably 8 or more, even more preferably 10 or more, even more preferably 20 or more, and particularly preferably 30 or more. １ is preferably 500 or less, more preferably 400 or less, even more preferably 300 or less, even more preferably 250 or less, and particularly preferably 200 or less. Setting the illuminance of the ambient light at the position where the head is present within this range has the advantage of making it easier to ensure the color gamut of the projected light and to achieve natural makeup.

[0069] The following Table 2 shows the results of the simulation using the simulation system 1 shown in FIG. １ As shown in Table 2, the results of screening the values ​​of I １ By setting the value of within a range, it is possible to reduce glare, ensure the color gamut of the projected light, and achieve natural makeup.

[0070] In the results shown in Table 2, a white LED was used as the ambient light, and ２ / I １ 10, I ３ The evaluation was performed by adjusting the output of the white LED while fixing the luminance at 0 [lux].

[0071] Furthermore, the "width of the color gamut that can be expressed" in Table 2 is evaluated according to the same criteria as those in Table 1.

[0072] The "naturalness" in Table 2 was evaluated according to the following criteria. That is, eye shadow, blush, and lipstick were projected onto a panel, and five cosmetic product specialists visually evaluated the results. The "naturalness" was evaluated according to the number of cosmetic product specialists who answered that the results were "natural" compared to when the makeup was actually applied. The specific evaluation criteria are as follows: A: 4 or more B: 2 to 3 C: 1 or less

[0073]

[0074] Furthermore, the illuminance of the area corresponding to the black part of the eye in the image projected by the projection unit 4 is I ３ When [lux] is used, I ３ / I １ is preferably 10 or less, more preferably 5 or less, and even more preferably 3 or less. In other words, by setting the illuminance of the area corresponding to the iris of the eye relatively low compared to the illuminance of the ambient light, it is easier to reduce glare. The illuminance of the area corresponding to the iris of the eye can be determined by measuring the illuminance at the center of the iris. ３ / I １ The lower limit of the value is not particularly limited, but is, for example, 0.01 or more, and can also be 0.

[0075] The following Table 3 shows the results of the simulation using the simulation system 1 shown in FIG. ３ / I １ As shown in Table 3, the results of screening the values ​​of I ３ / I １ By setting the value appropriately, it is easier to reduce glare.

[0076] In the results shown in Table 3, I １ 50 [lux], I ２ The evaluation was performed by adjusting the illuminance of the area corresponding to the black part of the eye while fixing the illuminance at 500 lux. The ambient light in this case was also light emitted from a white LED.

[0077] Furthermore, the "no glare" in Table 3 is evaluated according to the same criteria as those in Table 1.

[0078]

[0079] The functions of the illuminance adjustment unit 5 may be utilized to adjust the various illuminance values ​​(numerical ranges) as described above. That is, when the ambient illuminance relative to the head is excessively high, the illuminance adjustment unit 5 may block light emitted from surrounding light sources to achieve the desired ambient illuminance. On the other hand, when the ambient illuminance is excessively low, the illuminance adjustment unit 5 may emit appropriate light to achieve the desired ambient illuminance.

[0080] In an exemplary aspect, the imaging unit 2 may detect the illuminance of ambient light at the position where the head is present, and the illuminance control unit 335 of the information processing unit 3 may control the illuminance adjustment unit 5 based on the detected illuminance. Note that the detection of the illuminance of ambient light may be performed by a configuration different from that of the imaging unit 2. For example, a predetermined sensor (not shown in FIG. 1 ) connected to the information processing unit 3 may detect the illuminance at the position where the head is present, and the illuminance control unit 335 of the information processing unit 3 may control the illuminance adjustment unit 5 based on the detected illuminance.

[0081] Furthermore, the image capturing unit 2 or the like may detect the illuminance of the ambient light at the position where the head is present, and the output level of the image projected by the projection unit 4 may be adjusted appropriately based on the detected illuminance. That is, if the projection unit 4 is a projector, the output of the projector may be adjusted to control the illuminance of the base portion within an appropriate range. Furthermore, by using a highly light-blocking color for the mask portion corresponding to the eye portion of the image projected by the projection unit 4, it is possible to prevent the image from being blurred. ３ / I １ The value of may be controlled.

[0082] Additionally, in order to further reduce the glare felt by the person P1, the person P1 may wear photochromic contact lenses.

[0083] 3. Conclusion In this way, in the simulation system 1 of the first embodiment, a predetermined image is projected onto the head, while one or more object portions include a portion that masks the eye portion of the head. That is, this masking portion functions to reduce the glare of the light emitted during this projection, and by shortening the latency time, the masking portion can appropriately follow the movement of the head, thereby enhancing the effect of reducing glare. From this perspective, it is possible to realize a simulation system that reduces the glare felt when used to perform a simulation.

[0084] Second Embodiment Next, a simulation system according to a second embodiment will be described. Note that the description of the second embodiment will focus on differences from the simulation system according to the first embodiment, and descriptions of similar points will be omitted.

[0085] That is, the simulation system of the second embodiment is as follows: A simulation system for projecting an image onto the head of a living thing, comprising an imaging unit, an information processing unit, and a projection unit, wherein the imaging unit is configured to capture images including the head continuously in time series, and the information processing unit is configured to calculate predetermined information regarding the head based on the captured images and generate an image to be projected, wherein the predetermined information includes the position, shape, and projected aspect of the head, and the image to be projected has a base portion and one or more object portions, and the projection unit is configured to project the image generated by the information processing unit based on the predetermined information in accordance with the position of the head, and to adjust the illuminance of ambient light at the position where the head is present to I １ [lux], and the illuminance of the base part in the projected image is I ２ When [lux] is used, I ２ / I １ is greater than or equal to 1.5 and less than or equal to 30.

[0086] That is, in the simulation system of the second embodiment, like the simulation system 1 of the first embodiment, it is equipped with an imaging unit 2, an information processing unit 3, and a projection unit 4, but it is optional whether one or more object parts included in the image generated by the information processing unit 3 include a part that masks the eye part included in the head, and it is also optional whether the latency is 50 ms or less.

[0087] On the other hand, in the simulation system of the second embodiment, ２ / I １ It is essential that I is 1.5 or more and 30 or less (the same applies to the simulation method of the second embodiment). ２ / I １ Since an appropriate range is set for , it is possible to realize a simulation system in which the user is less likely to feel glare when performing a simulation, also in the second embodiment.

[0088] [Modifications] In addition, the simulation system 1 may employ the following modifications.

[0089] Fig. 8 is a configuration diagram showing a first modified example of the simulation system 1. In the above-described embodiment, the image capturing unit 2, the information processing unit 3, and the projection unit 4 are each shown as independent units, but as in the simulation system 1 shown in Fig. 8, the image capturing unit 2 and the information processing unit 3 may be incorporated into the projection unit 4. In other words, a high-speed, high-performance projector may be implemented that integrally includes the functions of the image capturing unit 2, the information processing unit 3, and the projection unit 4.

[0090] Fig. 9 is a configuration diagram showing a second modified example of the simulation system 1. In the above embodiment, the illuminance adjustment unit 5 has been described as an arcade-shaped member located near the person P1, but a member may be disposed so as to surround the person P1, the image capture unit 2, the information processing unit 3, and the projection unit 4, as in the illuminance adjustment unit 5a in Fig. 9. In other words, the illuminance adjustment unit 5a may form a booth surrounding the person P1, the image capture unit 2, the information processing unit 3, and the projection unit 4, and the person P1 may perform a simulation in this booth.

[0091] In the above embodiment, the information processing unit 3 has been described as a computer having predetermined functions, but the information processing unit 3 may also be a control circuit that converts the information captured by the imaging unit 2 into a predetermined signal and transmits it to the projection unit 4.

[0092] In the above embodiment, the projection location includes the head of the living creature, but as a modified example, the projection location may not include the head. For example, the projection location may be the torso, hands, legs, etc. of the living creature. In this embodiment, it is possible to perform simulations of clothing, avatars, manicures, and pedicures.

[0093] Furthermore, it may be provided in the following aspects.

[0094] (1) A simulation system for projecting an image onto the head of a living organism, comprising an imaging unit, an information processing unit, and a projection unit, wherein the imaging unit is configured to continuously capture images including the head in a time series, and the information processing unit is configured to calculate predetermined information about the head based on the captured images and generate an image to be projected, wherein the predetermined information includes the position, shape, and projected manner of the head, and the image to be projected has a base portion and one or more object portions, wherein the one or more object portions include portions that mask the eye portions included in the head, and the projection unit is configured to project the image generated by the information processing unit based on the predetermined information to match the position of the head, and wherein the simulation system is configured so that the time from the time of a frame in which the imaging unit captures an image of the head to the time when the projection unit projects the image onto the head after information processing corresponding to this frame is performed is 50 ms or less.

[0095] (2) In the simulation system described in (1) above, the illuminance of the ambient light at the position where the head is present is set to I １ [lux], and the illuminance of the base part in the projected image is I ２ When [lux] is used, I ２ / I １is greater than or equal to 1.5 and less than or equal to 30.

[0096] (3) In the simulation system described in (2) above, １ is greater than or equal to 5 and less than or equal to 500.

[0097] (4) In the simulation system described in (2) or (3) above, the illuminance of a region corresponding to the black part of the eye in the image projected by the projection unit is set to I ３ When [lux] is used, I ３ / I １ is 10 or less.

[0098] (5) A simulation system according to any one of (1) to (4) above, further comprising an illuminance adjustment unit, wherein the illuminance adjustment unit is configured to adjust the illuminance of ambient light at the position where the head is located by a member arranged around the head.

[0099] (6) In the simulation system described in any one of (1) to (5) above, the imaging unit is a camera having an operating frequency of 300 fps or more.

[0100] (7) In the simulation system described in any one of (1) to (6) above, the information processing unit calculates the specified information for the current frame of the captured image using a specified area centered on the position of the head calculated from the immediately previous frame.

[0101] (8) In the simulation system described in any one of (1) to (7) above, the projection unit is a projector having an operating frequency of 300 Hz or more.

[0102] (9) A simulation system according to any one of (1) to (8) above, which simulates makeup application to the head.

[0103] (10) The simulation system according to (9) above, further comprising a makeup input unit, wherein the information processing unit outputs the contents of one or more of the object parts based on an input to the makeup input unit.

[0104] (11) A simulation system for projecting an image onto the head of a living thing, comprising an imaging unit, an information processing unit, and a projection unit, wherein the imaging unit is configured to capture images including the head continuously in time series, and the information processing unit is configured to calculate predetermined information regarding the head based on the captured images and generate an image to be projected, wherein the predetermined information includes the position, shape, and projection manner of the head, and the image to be projected has a base portion and one or more object portions, and the projection unit is configured to project the image generated by the information processing unit based on the predetermined information in accordance with the position of the head, and to adjust the illuminance of ambient light at the position where the head is present to I. １ [lux], and the illuminance of the base part in the projected image is I ２ When [lux] is used, I ２ / I １ is greater than or equal to 1.5 and less than or equal to 30.

[0105] (12) A simulation method using a simulation system that projects an image onto the head of a living organism, comprising an imaging step, an information processing step, and a projection step, wherein in the imaging step, images including the head are continuously captured in chronological order, and in the information processing step, predetermined information regarding the head is calculated based on the captured images and an image to be projected is generated, wherein the predetermined information includes the position, shape, and projected manner of the head, and the image to be projected has a base portion and one or more object portions, and the one or more object portions include portions that mask eye portions included in the head, and in the projection step, the image generated in the information processing step is projected to match the position of the head based on the predetermined information, and the time from the time of a frame in which the head is imaged in the imaging step to the time when information processing corresponding to this frame is performed and the time when the image is projected onto the head in the projection step is 50 ms or less.

[0106] (13) A simulation method using a simulation system that projects an image onto the head of a living thing, comprising: an imaging step, an information processing step, and a projection step; in the imaging step, captured images including the head are captured continuously in time series; in the information processing step, predetermined information regarding the head is calculated based on the captured images, and an image to be projected is generated; here, the predetermined information includes the position, shape, and projected manner of the head; the image to be projected has a base portion and one or more object portions; and in the projection step, the image generated in the information processing step is projected in accordance with the position of the head based on the predetermined information, and the illuminance of ambient light at the position where the head is present is calculated. １ [lux], and the illuminance of the base part in the projected image is I ２ When [lux] is used, I ２ / I １ A simulation method in which is 1.5 or more and 30 or less. Of course, this is not the only option.

[0107] Finally, while various embodiments of the present invention have been described, these are presented by way of example only and are not intended to limit the scope of the invention. The novel embodiments may be embodied in various other forms, and various omissions, substitutions, and modifications may be made without departing from the spirit of the invention. Such embodiments and modifications are intended to be included within the scope and spirit of the invention, as well as within the scope of the inventions and their equivalents as defined in the accompanying claims.

[0108] 1: Simulation system 2: Imaging unit 3: Information processing unit 4: Projection unit 5, 5a: Illumination adjustment unit 6: Makeup input unit 30: Communication bus 31: Communication unit 32: Memory unit 33: Control unit 41: Light emitting unit 60: Communication bus 61: Communication unit 62: Memory unit 63: Control unit 64: Display unit 65: Input unit 331: Imaging control unit 332: Projection control unit 333: Calculation unit 334: Input reception unit 335: Illumination control unit BS1: Base part D: Display screen IMG1: Image OBJ11 to OBJ14: Object parts OBJ21 to OBJ23, OBJ21a, OBJ21b: Object P1: Human

Claims

1. A simulation system for projecting an image onto the head of a living being, comprising: an imaging unit, an information processing unit, and a projection unit; the imaging unit is configured to continuously capture imaging images including the head in a time series; the information processing unit is configured to calculate predetermined information regarding the head based on the imaging image and generate an image for projection, where the predetermined information includes the position, shape, and projected mode of the head; the image for projection has a base portion and one or more object portions; one or more of the object portions include a portion that masks the eye portion included in the head; the projection unit is configured to project the image generated by the information processing unit in accordance with the predetermined information at the position of the head; A simulation system configured such that the time from the time when the imaging unit captures an image of the head to the time when the projection unit projects the image onto the head after information processing corresponding to this frame is performed is 50 ms or less.

2. In the simulation system according to Claim 1, when the illuminance of the ambient light at the position where the head is present is I １ [lux] and the illuminance of the base portion in the projected image is I ２ [lux], a simulation system in which I ２ / I １ is 1.5 or more and 30 or less.

3. In the simulation system according to Claim 2, the I １ is 5 or more and 500 or less.

4. In the simulation system according to Claim 2, The illuminance of the area corresponding to the black eye part in the video projected by the projection unit is I ３ [lux], when I ３ / I １ is 10 or less, a simulation system.

5. In the simulation system according to claim 1, further comprising an illuminance adjustment unit, The illuminance adjustment unit is configured to adjust the illuminance of the ambient light at the position where the head is located by a member disposed around the head, a simulation system.

6. In the simulation system according to claim 1, The imaging unit is a camera with an operating frequency of 300 fps or more, a simulation system.

7. In the simulation system according to claim 1, The information processing unit calculates the predetermined information using a predetermined area centered on the position of the head calculated from the immediately preceding frame with respect to the current frame of the captured image, a simulation system.

8. In the simulation system according to claim 1, The projection unit is a projector with an operating frequency of 300 Hz or more, a simulation system.

9. In the simulation system according to claim 1, Simulating makeup on the head, a simulation system.

10. In the simulation system according to claim 9, further comprising a makeup input unit, The information processing unit outputs the content of one or more of the object parts based on the input to the makeup input unit, a simulation system.

11. A simulation system for projecting an image onto the head of a living being, comprising: an imaging unit, an information processing unit, and a projection unit; the imaging unit is configured to continuously capture imaging images including the head in a time series; the information processing unit is configured to calculate predetermined information regarding the head based on the imaging image and generate an image for projection, where the predetermined information includes the position, shape, and projected manner of the head; the image for projection has a base portion and one or more object portions; the projection unit is configured to project the image generated by the information processing unit in accordance with the position of the head based on the predetermined information; wherein when the illuminance of the ambient light at the position where the head is present is I １ [lux] and the illuminance of the base portion in the projected image is I ２ [lux], I ２ / I １ is 1.5 or more and 30 or less, the simulation system.

12. A simulation method using a simulation system for projecting an image onto the head of a living being, comprising: an imaging step, an information processing step, and a projection step; in the imaging step, continuously capturing imaging images including the head in a time series; in the information processing step, calculating predetermined information regarding the head based on the imaging image and generating an image for projection, where the predetermined information includes the position, shape, and projected manner of the head; the image for projection has a base portion and one or more object portions; one or more of the object portions include a portion that masks the eye portion included in the head; In the projection step, based on the predetermined information, the video generated in the information processing step is projected in accordance with the position of the head. A simulation method, wherein the time from the time of the frame in which the head is imaged in the imaging step to the time when information processing corresponding to this frame is performed and the video is projected onto the head in the projection step is 50 ms or less.

13. A simulation method using a simulation system for projecting a video onto a head of a living organism, comprising: an imaging step, an information processing step, and a projection step; In the imaging step, an imaging image including the head is continuously imaged in time series; In the information processing step, based on the imaging image, predetermined information regarding the head is calculated and a video for projection is generated. Here, the predetermined information includes the position, shape, and projected mode of the head; the video for projection has a base portion and one or more object portions; In the projection step, based on the predetermined information, the video generated in the information processing step is projected in accordance with the position of the head; When the illuminance of the ambient light at the position where the head exists is I １ [lux], and the illuminance of the base portion in the projected video is I ２ [lux], a simulation method in which I ２ / I １ is 1.5 or more and 30 or less.