Simulation system and simulation method

Abstract

[Problem] To provide a simulation system, etc., with which glare at use time is not readily felt felt when a simulation is performed. [Solution] According an embodiment of the present invention, provided is a simulation system that projects a video to the head of a living being. This simulation system comprises an image-capturing unit, an information processing unit, and a projection unit. The image-capturing unit is configured to successively capture images in time series, the captured images including the head. The information processing unit is configured to calculate prescribed information relating to the head and generate a video to be projected, on the basis of the captured images. The prescribed information includes the position, shape, and projected form of the head. The video to be projected has a base portion and one or more object portions. The one or more object portions include a portion that masks the eyes included in the head. The projection unit is configured to project the video generated by the information processing unit, in conformance with the position of the head, on the basis of the prescribed information. A configuration is adopted such that a duration from the time of a frame in which the image-capturing unit captures an image of the head until the time at which information processing corresponding to the frame is carried out and the projection unit projects the video to the head is 50 ms or less.

Description

Technical Field

[0004] ,

[0006] , , , , ,

[0005] , , , , , ,

[0001] The present invention relates to a simulation system and a simulation method.

Background Art

[0002] In recent years, a technology of projecting an image onto a three-dimensional object, also called "projection mapping", has attracted attention. Further, by applying such a technology, the development of a technology of projecting a makeup image onto a human face or the like has also been made. As related technologies, there are the technologies disclosed in Patent Document 1 and Patent Document 2.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, as a result of the inventors' examination, it has been found that in the above-described projection technology, there are cases where the glare of the light of the projector used is felt.

[0005] In view of the above circumstances, the present invention aims to provide a simulation system or the like that is less likely to cause glare during use when performing a simulation.

Means for Solving the Problems

[0006] According to one aspect of the present invention, a simulation system for projecting an image onto the head of a living organism is provided. This simulation system comprises an imaging unit, an information processing unit, and a projection unit. The imaging unit is configured to continuously capture images including the head in a time series. The information processing unit is configured to calculate predetermined information about the head based on the captured images and to generate an image for projection. The predetermined information includes the position, shape, and projected appearance of the head. The projected image has a base portion and one or more object portions. The one or more object portions include a portion that masks the eye portion included in the head. The projection unit is configured to project the image generated by the information processing unit based on the predetermined information, aligned with the position of the head. The time from the time of the frame in which the imaging unit images the head to the time when the projection unit projects the image onto the head after the corresponding information processing is performed is configured to be 50 ms or less.

[0007] According to the above embodiment, a simulation system that minimizes glare during use is provided. [Brief explanation of the drawing]

[0008] [Figure 1] This is a configuration diagram showing the simulation system 1 according to the first embodiment. [Figure 2] This is a block diagram showing the hardware configuration of the information processing unit 3. [Figure 3] This is a functional block diagram showing the functions of the control unit 33 in the information processing unit 3. [Figure 4] A block diagram showing the hardware configuration of the tablet device (makeup input section 6). [Figure 5] This is an activity diagram illustrating the simulation method of the first embodiment. [Figure 6] This is a conceptual diagram illustrating the video generated in the information processing step. [Figure 7] This is an example of a display screen shown on a tablet device (makeup input unit 6). [Figure 8] This is a diagram showing the configuration of the first modified example of Simulation System 1. [Figure 9] This is a diagram showing a second modified example of Simulation System 1. [Modes for carrying out the invention]

[0009] Embodiments of the present invention will be described below with reference to the drawings. The various features shown in the embodiments below can be combined with each other.

[0010] Incidentally, the program for implementing the software appearing in this embodiment may be provided as a non-transitory computer-readable medium, or it may be provided so that it can be downloaded from an external server, or it may be provided so that the program is launched on an external computer and its functions are realized on a client terminal (so-called cloud computing).

[0011] Furthermore, in this embodiment, "part" may include, for example, hardware resources implemented by circuits in a broad sense, and the information processing of software that can be specifically realized by these hardware resources. In addition, various types of information are handled in this embodiment, and these types of information can be represented, for example, by the physical values ​​of signal values ​​representing voltage and current, the high or low values ​​of signal values ​​as a set of binary bits composed of 0s or 1s, or by quantum superposition (so-called qubits), and communication and calculations can be performed on circuits in a broad sense.

[0012] Furthermore, a circuit in a broad sense is a circuit realized by combining at least a suitable combination of circuits, circuits, processors, and memory. In other words, it includes application-specific integrated circuits (ASICs), programmable logic devices (for example, simple programmable logic devices (SPLDs), complex programmable logic devices (CPLDs), and field programmable gate arrays (FPGAs)), etc.

[0013] [First Embodiment] First, the simulation system of the first embodiment will be described. That is, the simulation system of the first embodiment is as follows. A simulation system that projects images onto the head of a living organism, It comprises an imaging unit, an information processing unit, and a projection unit, The imaging unit is configured to continuously capture images including the head in a time series. The information processing unit is configured to calculate predetermined information about the head and generate a projection image based on the captured image, and here, The predetermined information includes the position, shape, and projected appearance of the head, The projected image comprises a base portion and one or more object portions. One or more of the object parts include a part that masks the eye portion included in the head, The projection unit is configured to project the image generated by the information processing unit based on the predetermined information, aligned with the position of the head. A simulation system configured such that the time from the time of a frame in which the imaging unit images the head to the time when information processing corresponding to this frame is performed and the projection unit projects the video onto the head is 50 ms or less. Hereinafter, the configuration, applications, etc. of this simulation system will be described.

[0014] 1. Hardware configuration In this section, the hardware configuration of the first embodiment will be described.

[0015] 1.1 Simulation system 1 FIG. 1 is a configuration diagram showing a simulation system 1 according to the first embodiment. The simulation system 1 projects a video onto the head of a living being. In FIG. 1, a system that projects a video onto the head of a human P1 is shown. Note that this simulation system 1 can also be applied to animals different from humans. In addition, this simulation system 1 can be used for various applications. Typically, it is used to simulate makeup on the head. In this specification, "makeup" includes not only makeup applied to each part of the face such as foundation, highlighter, shading, eyeshadow, eyeliner, eyebrows, mascara, cheeks, and lips, but also coloring for the hair. Further, the simulation system 1 can be used for applications such as simulating accessories (decorations) for the head, simulating hairstyles, simulating increases and decreases in three-dimensional structures such as pores and wrinkles, simulating increases and decreases in skin color unevenness such as acne and scars, simulating cosmetic plastic surgery, visualizing the arrangement of subcutaneous tissues, etc., in addition to simulating makeup on the head. Also, the projection site may include parts other than the head. For example, as described above, while simulating makeup on the head, projection may be performed on the torso, hands, legs, etc. so that clothing that matches this makeup can be simulated.

[0016] The simulation system 1 comprises an imaging unit 2, an information processing unit 3, and a projection unit 4, which are connected via a circuit. Although not essential in this embodiment, Figure 1 shows a configuration in which the simulation system 1 includes, in addition to the above-described components, an illumination adjustment unit 5 and a cosmetic input unit 6. These components will be further explained.

[0017] 1.2 Imaging Unit 2 The imaging unit 2 is configured to continuously capture images including the head in a time series. The imaging unit 2 is, for example, a camera (visible light, infrared light, etc. can be used as appropriate) that is configured to acquire information from the outside world as an image. It is preferable that such a camera has a high operating frequency (frame rate), which is known as high-speed vision. The frame rate (operating frequency) of the camera is, for example, 300 fps or more, preferably 400 fps or more, and more preferably 500 fps or more or 1000 fps or more. Also, the camera's operating frequency is, for example, 100, 125, 150, 175, 200, 225, 250, 275, 300, 325, 350, 375, 400, 425, 450, 475, 500, 525, 550, 575, 600, 625, 650, 675, 700, 725, 750, 775, 800, 825, 850, 875, 900, 925, 950, 975, 1000, 1025, 1050, 1075, 1100, 1125, 1150, 1175, 1 The frame rates may also be 200, 1225, 1250, 1275, 1300, 1325, 1350, 1375, 1400, 1425, 1450, 1475, 1500, 1525, 1550, 1575, 1600, 1625, 1650, 1675, 1700, 1725, 1750, 1775, 1800, 1825, 1850, 1875, 1900, 1925, 1950, 1975, or 2000 fps, and may be within the range of any two of the values ​​exemplified here.

[0018] Furthermore, the imaging unit 2 is not limited to a camera; it may employ a measurement sensor capable of acquiring three-dimensional shape, or it may employ multiple sensors with different functions. It is assumed that the field of view position in the projection unit 4 (described later) and the field of view position in the imaging unit 2 are consistent through prior calibration. Although not shown, a coaxial optical system may also be employed to implement the system as a calibration-free system.

[0019] 1.3 Information Processing Unit 3 The information processing unit 3 is configured to calculate predetermined information about the head based on the captured image and to generate a projection image. Figure 2 is a block diagram showing the hardware configuration of the information processing unit 3. Figure 3 is a functional block diagram showing the functions of the control unit 33 in the information processing unit 3. The information processing unit 3 has a communication unit 31, a storage unit 32, and a control unit 33, and these components are electrically connected within the information processing unit 3 via a communication bus 30. In particular, with respect to the control unit 33, the information processing unit 3 includes an imaging control unit 331, a projection control unit 332, a calculation unit 333, an input receiving unit 334, and an illumination control unit 335. Each component will be described further below.

[0020] <Communications Department 31> The communication unit 31 preferably uses wired communication methods such as USB, IEEE1394, Thunderbolt®, and wired LAN network communication, but may also include wireless LAN network communication, mobile communication such as LTE / 3G, and Bluetooth® communication as needed. In other words, it is more preferable to implement it as a collection of these multiple communication methods.

[0021] For example, it is preferable that the communication unit 31 is configured to communicate with the imaging unit 2, projection unit 4, etc., using a predetermined high-speed communication standard. Specifically, the communication unit 31 is configured to transmit an image of the projection light emitted by the light-emitting unit 41 of the projection unit 4. In addition, the communication unit 31 is configured to receive the image captured by the imaging unit 2.

[0022] <Storage section 32> The memory unit 32 stores various types of information as defined above. This can be done, for example, as a storage device such as a solid-state drive (SSD), or as memory such as random access memory (RAM) that stores temporarily necessary information (arguments, arrays, etc.) related to program calculations. A combination of these may also be used.

[0023] In particular, the memory unit 32 stores imaging information acquired by the imaging unit 2 and received by the communication unit 31. The memory unit 32 also stores a sensor control program for controlling the imaging control unit 331 in the control unit 33 to receive imaging information from the imaging unit 2. The memory unit 32 also stores a light emission control program for controlling the projection control unit 332 in the control unit 33 to ensure that the light emission unit 41 emits light at a predetermined operating frequency. In addition, the storage unit 32 also stores various programs and other information related to the information processing unit 3, which are executed by the arithmetic unit 333 and other components in the control unit 33.

[0024] <Control Unit 33> The control unit 33 performs processing and control of the overall operation related to the information processing unit 3. The control unit 33 is, for example, a central processing unit (CPU) not shown. The control unit 33 realizes various functions related to the information processing unit 3 by reading predetermined programs stored in the memory unit 32. Specifically, these include imaging control functions, projection control functions, calculation functions, input reception functions, illumination control functions, etc. In other words, information processing by software (stored in the memory unit 32) is concretely realized by hardware (control unit 33) and can be executed as the imaging control unit 331, projection control unit 332, calculation unit 333, input reception unit 334, and illumination control unit 335. That is, the program can make the computer function as each part of the information processing unit 3.

[0025] Although Figure 2 shows a single control unit 33, in reality, it is not limited to this, and it may be implemented with multiple control units 33 for each function. A combination of these is also possible. The various functions that the control unit 33 can implement will be described in more detail below.

[0026] (Imaging control unit 331) The imaging control unit 331 is a system in which information processing performed by software (stored in the memory unit 32) is concretely implemented by hardware (control unit 33). The imaging control unit 331 controls the system to receive imaging information from the imaging unit 2. The operating frequency of the imaging unit 2 may also be adjusted, and the system may be configured to generate a 3D projection image model using the imaging information acquired by the imaging unit 2.

[0027] (Projection control unit 332) The projection control unit 332 is a system in which information processing by software (stored in the memory unit 32) is concretely realized by hardware (control unit 33). The projection control unit 332 is configured to control the light-emitting unit 41 in the projection unit 4 so that it emits light at a predetermined operating frequency. In other words, the projection control unit 332 generates a control signal to control the light-emitting unit 41, which is transmitted to the light-emitting unit 41 in the projection unit 4 via the communication unit 31. The light-emitting unit 41 then emits light to project projection light onto the target (human P1) based on this control signal.

[0028] (Computation unit 333) The arithmetic unit 333 is where information processing performed by software (stored in the memory unit 32) is concretely realized by hardware (the control unit 33). The arithmetic unit 333 is configured to perform various calculations, such as image processing and transformation of projected images. These are not particularly limited and can be performed as needed.

[0029] (Input reception unit 334) The input receiving unit 334 is where information processing by software (stored in the memory unit 32) is concretely realized by hardware (control unit 33). The input receiving unit 334 is configured to receive, for example, input content for the cosmetic input unit 6, which will be described later. These are not particularly limited and can be implemented as appropriate as needed.

[0030] (Illuminance control unit 335) The illuminance control unit 335 is a system in which information processing by software (stored in the memory unit 32) is concretely realized by hardware (control unit 33). The illuminance control unit 335 is configured to control the illuminance at the location of human P1 as needed. That is, it may be configured to adjust the output of light projected from the projection unit 4, and if the illuminance adjustment unit 5 is electrically connected to the information processing unit 3, the illuminance control unit 335 may be configured to operate the illuminance adjustment unit 5 appropriately.

[0031] 1.4 Projection section 4 The projection unit 4 is configured to project the image generated by the information processing unit 3, aligned with the position of the head. More specifically, the projection unit 4 typically includes a light-emitting unit 41, which is configured to emit light containing the image so that it can be projected onto the projection target (human P1). The appearance and internal configuration of the projection unit 4 are not particularly limited, but it is preferable to use a projector with a high refresh rate (operating frequency), which is known as a high-speed projector. The operating frequency of the projector is, for example, 300 Hz or higher, preferably 400 Hz or higher, and more preferably 500 Hz or higher or 1000 Hz or higher. Also, the projector's operating frequencies are, for example, 100, 125, 150, 175, 200, 225, 250, 275, 300, 325, 350, 375, 400, 425, 450, 475, 500, 525, 550, 575, 600, 625, 650, 675, 700, 725, 750, 775, 800, 825, 850, 875, 900, 925, 950, 975, 1000, 1025, 1050, 1075, 1100, 1125, 1150, 1175 ,1200,1225,1250,1275,1300,1325,1350,1375,1400,1425,1450,1475,1500,1525,1550,1575,1600,1625,1650,1675,1700,1725,1750,1775,1800,1825,1850,1875,1900,1925,1950,1975,2000Hz may also be, and may be within the range between any two of the values ​​exemplified here. Furthermore, the projection unit 4 may be configured to allow adjustment of the optical axis using mirrors or the like in accordance with the movement of the head. This makes it easier to project not only from the front of the head, but also from the sides and top and bottom. In addition, even if the position of human P1 moves, it becomes easier to project appropriately to the moved position.

[0032] 1.5 Illuminance adjustment section 5 The illuminance adjustment unit 5 is used to adjust the illuminance of ambient light at the location of the projection target (human P1). Typically, the illuminance adjustment unit 5 is configured to adjust the illuminance of ambient light at the location of the head of the projection target by a component disposed around the head of the projection target. Here, the adjustment of illuminance may be achieved by blocking light entering from surrounding light sources. Alternatively, the illuminance adjustment unit 5 may be achieved by emitting light to create the desired ambient light. Although Figure 1 shows an embodiment in which the illuminance adjustment unit 5 exists independently, if the illuminance adjustment unit 5 includes electrical elements, the illuminance adjustment unit 5 may be connected to the information processing unit 3, etc., by a circuit or the like.

[0033] 1.6 Makeup Input Section 6 The makeup input unit 6 is used to input details of the makeup to be simulated. The makeup input unit 6 is typically a portable terminal, and specific examples include tablet terminals and smartphones, but a PC (Personal Computer) may also be used, and the details are not limited. Furthermore, the makeup input unit 6 may be the same terminal as the information processing unit 3, and predetermined information processing may be performed by inputting to the information processing unit 3.

[0034] The following describes the hardware configuration of the makeup input unit 6, assuming it is a tablet device. Figure 4 is a block diagram showing the hardware configuration of the tablet device (makeup input unit 6). The tablet terminal (makeup input unit 6) has a communication unit 61, a storage unit 62, a control unit 63, a display unit 64, and an input unit 65, and these components are electrically connected within the tablet terminal (makeup input unit 6) via a communication bus 60. The description of the communication unit 61, storage unit 62, and control unit 63 is substantially the same as that of the communication unit 31, storage unit 32, and control unit 33 in the information processing unit 3 described above, so it will be omitted.

[0035] The display unit 64 may be included in the casing of the tablet terminal (cosmetic input unit 6), for example, or it may be an external component. The display unit 64 displays a graphical user interface (GUI) screen that can be operated by the user. This is preferably done by using different display devices such as CRT displays, liquid crystal displays, organic EL displays, and plasma displays, depending on the type of terminal. Here, the display unit 64 will be described as being included in the casing of the tablet terminal (cosmetic input unit 6).

[0036] The input unit 65 may be included in the casing of the tablet terminal (appliance input unit 6) or it may be an external component. For example, the input unit 65 may be integrated with the display unit 64 and implemented as a touch panel. If it is a touch panel, the user can input tap operations, swipe operations, etc. Of course, instead of a touch panel, a switch button, mouse, QWERTY keyboard, etc., may be used. In other words, the input unit 65 receives operation input made by the user. This input is transmitted as a command signal to the control unit 63 via the communication bus 60, and the control unit 63 can perform predetermined controls and calculations as needed.

[0037] Further details regarding the images displayed on the display unit 64 of the makeup input unit 6, and the content that can be input using those images, will be explained later.

[0038] 2. Simulation Method This section will describe each step of the simulation method using the aforementioned simulation system 1. Specifically, the simulation method of the first embodiment comprises an imaging step, an information processing step, and a projection step, and this section will describe each of these steps with reference to activity diagrams and the like.

[0039] Figure 5 is an activity diagram illustrating the simulation method of the first embodiment. First, in Activity A01, images including the head are acquired continuously in a time series (imaging step). This imaging step is achieved by the imaging unit 2 continuously acquiring images of the head of human P1 in a time series.

[0040] In Activity A02, based on the captured image acquired in the imaging step, predetermined information about the head is calculated and a projected image is generated (information processing step). Here, the predetermined information includes the position, shape, and projected form of the head, and the projected image has a base portion and one or more object portions, one or more of which includes a portion that masks the eye portion included in the head.

[0041] To determine the position of the head, a pre-trained model is used, which has been pre-trained to learn the positional relationships of parts that can serve as feature points within the head. Typical feature points include the eyes, nose, mouth, and ears. When the image captured in the imaging step is input to this pre-trained model, each feature point contained in the image is calculated. Alternatively, rule-based algorithms such as template matching may be used instead of the pre-trained model.

[0042] This information processing step is typically performed by the information processing unit 3 performing predetermined calculations or other operations. Here, we will explain the video generated in this information processing step, referring to Figure 6. Figure 6 is a conceptual diagram illustrating the video generated in the information processing step. In other words, the video IMG1 in Figure 6 corresponds to the video generated in the information processing step. Typically, the process of generating this video IMG1 involves combining the base part BS1 and the object parts OBJ11 to OBJ14.

[0043] In the example shown in Figure 6, the base portion BS1 is the part that constitutes the skin color (base color) of the head to be projected, but the color of the "base portion" is not necessarily limited to a single color. In other words, the color that constitutes the base portion may be a color that makes up a gradient or a color with a pattern applied to part of it. However, in discussing the relationship between ambient light illuminance and the illuminance of the base portion, as described later, the illuminance of this base portion can be defined as the color projected over the largest area of ​​the face on the head of human P1. Furthermore, in discussing the relationship of illuminance, the illuminance of the base portion refers only to the light emitted from the projection unit 4. The value of this illuminance of the base portion can be obtained by subtracting the illuminance of ambient light from the illuminance of the base portion when the projected light from the projection unit 4 and ambient light are simultaneously irradiated.

[0044] Object part OBJ11 is the part that masks the eye area included in the head, object part OBJ12 is the part that represents the shape of the cheeks, object part OBJ13 is the part that represents the shape of the eye makeup, and object part OBJ14 is the part that represents the shape of the lips.

[0045] This image IMG1 is generated by calculating predetermined information about the head based on the captured image taken in the imaging step. Specifically, this predetermined information includes the position, shape, and projected appearance of the head. In other words, the positional relationships of the various objects in the aforementioned image IMG1 are adjusted (optimized) based on the facial features (eyes, eyelids, cheeks, lips, etc.) present on the head captured in the imaging step.

[0046] In Figure 6, the video IMG1 is shown as a separate concept for each object part, but the method of constructing the video IMG1 is not limited to this. For example, when constructing the video IMG1, the content in which each object part is pre-combined may be combined with the base part BS1, or the video IMG1 may be adopted as a combination of the base part BS1 and each object part from the beginning.

[0047] Furthermore, among the various object parts mentioned above, the part that masks the eye area included in the head (object part OBJ11) has the function of reducing the glare of projected light entering the eye. However, this object part OBJ11 may mask the entire eye, including the white of the eye as well as the pupil (iris). Note that this part that masks the eye area included in the head (object part OBJ11) is usually composed of a dark color relative to the color that makes up the base part BS1.

[0048] Furthermore, while such information processing steps are typically performed by the information processing unit 3, the information processing unit 3 may be configured to output the contents of one or more object parts based on the input to the cosmetic input unit 6.

[0049] The method of outputting the contents of the object portion based on the input to the cosmetic input unit 6 will be explained with reference to Figure 7. Figure 7 is an example of a display screen shown on the tablet terminal (cosmetic input unit 6).

[0050] In the display screen D of Figure 7, object OBJ21 allows you to select the shape of various makeup, object OBJ22 allows you to select the color of various makeup, and object OBJ23 allows you to adjust the intensity of various makeup. In actual operation, pressing object OBJ21a selects the corresponding shape, and pressing object OBJ21b selects a different shape. However, for convenience, objects OBJ21, OBJ22, and OBJ23 will be described as collections of the same type of object.

[0051] In other words, a person (typically human P1, but also an operator other than human P1) who interacts with the display screen D displayed on the display unit 64 of the makeup input unit 6 selects the shape of the makeup they wish to apply to human P1 based on object OBJ21, and selects the color of the makeup they wish to apply to human P1 based on object OBJ22. Although object OBJ22 shows a color palette that allows selection of various colors, instead, objects of different shapes that allow selection of various colors may be provided. Furthermore, object OBJ23 allows selection of the intensity of each makeup based on the sliding operation of a bar.

[0052] The content input to the cosmetic input unit 6 is received by the input receiving unit 334 of the information processing unit 3. The calculation unit 333 of the information processing unit 3 then performs a predetermined calculation based on the received content to generate the image to be projected. Furthermore, the content input to the makeup input unit 6 is not limited to the configuration shown in Figure 7, and may include expressions that describe the atmosphere or image of the makeup, such as "gorgeous" or "soft." Alternatively, the input may include content representing a celebrity such as an actor or idol, and the information processing unit 3 may output an object portion that evokes this celebrity.

[0053] Furthermore, when such information processing steps are performed, the information processing unit 3 may be configured to calculate predetermined information using a predetermined region centered on the position of the head calculated from the previous frame, relative to the current frame of the captured image. In other words, in this configuration, the position of the head is calculated in the previous frame, and based on the area around the head, predetermined information is calculated and an image is generated for projection. To put it another way, the processing unit 333 does not perform information processing on the entire frame of the captured image, but rather only on a portion of the frame. Performing this type of processing can contribute to improving the information processing speed.

[0054] Furthermore, during the information processing step, the information processing unit 3 may adjust the projection color of the image based on the reflective properties of a specific part of the head. This specific part may be the skin, lips, hair, etc., on the head. The reflective properties of this specific part may be determined using a database acquired in advance, or they may be measured on-site. If measured on-site, the imaging unit 2 may measure the reflective properties of the specific part of the head. On the other hand, the measurement of reflective properties may be performed by a configuration different from that of the imaging unit 2. For example, a predetermined sensor (not shown in Figure 1) connected to the information processing unit 3 may detect the reflective properties of a specific part of the head, and the content of the image projected by the information processing unit 3 may be adjusted based on these detected reflective properties. To give a more typical example, the information processing unit 3 may be configured to detect errors from the target color during simulation based on the reflective characteristics of a specific part of the head, and may control the projection data and the output level of the projection unit 4 based on this detected error.

[0055] Returning to the explanation of the activity diagram in Figure 5, in the subsequent activity A03, the image generated in the information processing step is projected based on predetermined information, aligned with the position of the head (projection step). This projection step is typically performed by the projection unit 4. By performing this step, an image that fits the head of the target can be projected, and as a result, simulations such as makeup application can be performed. In this projection step, the optical axis may be adjusted using mirrors or the like in accordance with the movement of the head.

[0056] Subsequently, the same process (activities A01, A02, A03) is executed every other frame. This ensures that even if human P1 moves, the image follows this movement and the desired image is projected onto the head (the [Next] route in Figure 5). To terminate the simulation, one simply needs to stop the function provided in at least one of the imaging unit 2, information processing unit 3, and projection unit 4 (the [End] route in Figure 5). Note that this activity does not necessarily have to be repeated every frame; it can also be configured to be repeated every few frames.

[0057] Herein, the simulation method of the first embodiment has the following features. In other words, the time from the time of the frame in which the head is imaged in the imaging step to the time when the information processing corresponding to this frame is performed and the image is projected onto the head in the projection step (this may also be called "latency") is 50ms or less.

[0058] Furthermore, the latency mentioned above is preferably 45ms or less, more preferably 40ms or less, even more preferably 35ms or less, even more preferably 30ms or less, particularly preferably 25ms or less, especially preferably 20ms or less, especially preferably 15ms or less, even more preferably 10ms or less, and most preferably 5ms or less. Such a reduction in latency time may also be achieved by improving the operating frequency of the imaging unit 2 and projection unit 4, or by improving the information processing capability of the information processing unit 3. Furthermore, as in the example of the information processing step described above, calculating predetermined information using a predetermined region centered on the position of the head calculated from the previous frame for the current frame of the captured image can also contribute to such a reduction in latency. While there is no specific lower limit to the latency, one example would be 0.1ms or higher.

[0059] Furthermore, the simulation method of the first embodiment may have the following features. In other words, when the illuminance of ambient light at the position of the head is I1 [lux] and the illuminance of the base portion of the projected image is I2 [lux], the system may be configured such that I2 / I1 is between 1.5 and 30. The inventors have found that when the ambient light intensity at the head's position is extremely low, the projected image tends to appear dazzling, and when the ambient light intensity at the head's position is extremely high, the range of colors that can be expressed tends to be limited. Therefore, it is preferable to set the ratio between the ambient light intensity at the head's position and the illuminance of the base portion of the projected image within an appropriate range. By setting it within such a range, the perceived glare is reduced, and a wide range of colors that can be expressed can be ensured.

[0060] In this specification, "ambient light at the location of the head" is defined as the total light reaching the head, excluding the portion of light that the projection unit 4 directly projects onto the head. Here, this ambient light typically includes natural light such as sunlight and light emitted from flames, artificial light sources such as fluorescent lamps (daylight, neutral white, white, etc.), incandescent bulbs, and light emitted from optical devices other than the projection unit 4 (LEDs, ELs, lasers, etc.), as well as light that the projection unit 4 indirectly projects onto the head. Furthermore, this ambient light may have its color tone adjusted via a color balance adjustment filter or the like. In addition, this light indirectly projected onto the head typically includes light that reflects off other parts of the body and reaches the head, and light that reflects off the walls of the space where the simulation is performed and reaches the head. Therefore, the simulation system 1 of this embodiment can provide a makeup environment that simulates day and night, weather conditions, and various events (studio, stage, viewing, meeting, dining, anti-aging, etc.) by adjusting the type of light source and illuminance settings of the ambient light. Furthermore, "light reaching the head" refers to light that shines on the area of ​​the face enclosed by a curved line connecting the hairline, chin, and both ears.

[0061] Furthermore, from the viewpoint of reducing glare, it is preferable that the light from the ambient light source be projected as indirect light reflected from a wall or the like, rather than being projected directly onto the face of human P1. In addition, it is more preferable that the projection unit 4 is directly illuminated by the ambient light source.

[0062] The above-mentioned I2 / I1 ratio is preferably 3 or greater, more preferably 5 or greater, even more preferably 7 or greater, and particularly preferably 10 or greater. Furthermore, the I2 / I1 ratio is preferably 27 or less, more preferably 25 or less, even more preferably 22 or less, and particularly preferably 20 or less.

[0063] Table 1 below summarizes the results of screening I2 / I1 values ​​using the simulation system 1 shown in Figure 1. As shown in Table 1, setting the I2 / I1 values ​​within a range reduces glare and ensures a wider color gamut can be represented.

[0064] Note that in the results shown in Table 1, a white LED was used as the ambient light. More specifically, the evaluation was performed by adjusting the output of the white LED while fixing I2 to 500 lux and I3 (described later) to 0 lux.

[0065] Furthermore, the "lack of glare" in Table 1 is evaluated according to the following criteria. Specifically, the image was projected onto a panel of 10 people for 10 minutes, and the "lack of glare" was evaluated based on the number of people who reported experiencing eye strain. The specific criteria for evaluation are as follows: A: 1 person or less B: 2 to 4 people C: 5 or more people

[0066] The "range of colors that can be expressed" in Table 1 is evaluated according to the following criteria. Specifically, 30 colors were selected from eyeshadows currently on the market. These 30 selected eyeshadows were reproduced using Simulation System 1, projected onto a panel, and visually evaluated by five cosmetic evaluation specialists. The "range of colors that can be expressed" was evaluated based on the number of colors that three or more cosmetic evaluation specialists judged to be equivalent to the actual product. The specific evaluation criteria are as follows: A: More than 25 colors B: 20 ​​colors or more and 24 colors or less C: 15 colors or more and 19 colors or less D: 14 ​​colors or less

[0067] [Table 1]

[0068] Furthermore, I1 is preferably 5 or greater, more preferably 8 or greater, even more preferably 10 or greater, even more preferably 20 or greater, and particularly preferably 30 or greater. On the other hand, I1 is preferably 500 or less, more preferably 400 or less, even more preferably 300 or less, even more preferably 250 or less, and particularly preferably 200 or less. Setting the ambient light intensity at the head's position within this range offers advantages such as making it easier to secure the color gamut of the projected light and achieving natural-looking makeup.

[0069] Table 2 below summarizes the results of screening the I1 value using the simulation system 1 shown in Figure 1. As shown in Table 2, setting the I1 value within a range has advantages such as reducing glare, ensuring a wider color gamut for projected light, and facilitating the creation of natural-looking makeup.

[0070] In the results shown in Table 2, a white LED was used as ambient light, and the output of the white LED was adjusted while fixing I2 / I1 to 10 and I3 (described later) to 0 [lux] for evaluation.

[0071] Furthermore, the "range of colors that can be expressed" in Table 2 was evaluated according to the same criteria as those in Table 1.

[0072] The "naturalness" in Table 2 is evaluated according to the following criteria. Specifically, eyeshadow, blush, and lipstick were projected onto a panel, and five cosmetic product evaluators visually assessed them. The "naturalness" was evaluated based on the number of cosmetic product evaluators who responded that it looked "natural" compared to when the makeup was actually applied. The specific criteria for judgment are as follows: A: 4 or more people B: 2 to 3 people C: 1 person or less

[0073] [Table 2]

[0074] Furthermore, when the illuminance of the area corresponding to the pupil in the image projected by the projection unit 4 is I3 [lux], it is preferable that I3 / I1 be 10 or less, more preferably 5 or less, and even more preferably 3 or less. In other words, by setting the illuminance of the area corresponding to the pupil relatively low relative to the illuminance of ambient light in this way, glare can be reduced more easily. The illuminance in the area corresponding to the pupil is determined by measuring the illuminance at the center of the pupil (iris). Furthermore, there are no specific restrictions on the lower limit of the I3 / I1 value, but one example is that it should be 0.01 or greater, and it can also be set to 0.

[0075] Table 3 below summarizes the results of screening the I3 / I1 values ​​using the simulation system 1 shown in Figure 1. As shown in Table 3, setting the I3 / I1 values ​​appropriately makes it easier to reduce glare.

[0076] In the results shown in Table 3, I1 was fixed at 50 lux and I2 at 500 lux, while the illuminance in the area corresponding to the pupil was adjusted for evaluation. In this case, the ambient light used was also light emitted from a white LED.

[0077] Furthermore, the "lack of glare" in Table 3 was evaluated according to the same criteria as those in Table 1.

[0078] [Table 3]

[0079] Furthermore, when adjusting the various illuminance values ​​(numerical ranges) as described above, the functions of the illuminance adjustment unit 5 may be utilized. That is, if the ambient illuminance is excessively high relative to the head, the illuminance adjustment unit 5 may block the light emitted from surrounding light sources to achieve the desired ambient light illuminance. On the other hand, if the ambient illuminance is excessively low, the illuminance adjustment unit 5 may emit an appropriate amount of light to achieve the desired ambient light illuminance.

[0080] In an exemplary embodiment, the imaging unit 2 may detect the illuminance of ambient light at the location of the head, and the illuminance control unit 335 of the information processing unit 3 may control the illuminance adjustment unit 5 based on this detected illuminance. Note that the detection of ambient light illuminance may be performed by a configuration different from that of the imaging unit 2. For example, a predetermined sensor (not shown in Figure 1) connected to the information processing unit 3 may detect the illuminance at the location of the head, and the illuminance control unit 335 of the information processing unit 3 may control the illuminance adjustment unit 5 based on this detected illuminance.

[0081] Furthermore, the imaging unit 2 may detect the illuminance of the ambient light at the location of the head, and based on this detected illuminance, the output level of the image projected by the projection unit 4 may be adjusted as appropriate. That is, if the projection unit 4 is a projector, the illuminance of the base portion may be controlled to an appropriate range by adjusting the output of the projector. In addition, the values ​​of I3 / I1 may be controlled by using a highly light-blocking color for the mask portion corresponding to the eye portion of the image projected by the projection unit 4.

[0082] In addition, to further reduce the glare experienced by human P1, photochromic contact lenses may be fitted to human P1.

[0083] 3. Conclusion Thus, in the simulation system 1 of the first embodiment, a predetermined image is projected onto the head, while one or more object parts include a part that masks the eye area included in the head. In other words, this masking part functions to mitigate the glare of the light emitted during this projection. By shortening the latency time, the masking part can be made to appropriately follow the movement of the head, thereby enhancing the effect of reducing glare. From this perspective, it is possible to create a simulation system that minimizes glare during use when performing simulations.

[0084] [Second Embodiment] Next, we will describe the simulation system of the second embodiment. This description of the second embodiment will focus on the differences from the simulation system of the first embodiment, and similar matters will be omitted.

[0085] In other words, the simulation system of the second embodiment is as follows. A simulation system that projects images onto the head of a living organism, It comprises an imaging unit, an information processing unit, and a projection unit, The imaging unit is configured to continuously capture images including the head in a time series. The information processing unit is configured to calculate predetermined information about the head and generate a projection image based on the captured image, and here, The predetermined information includes the position, shape, and projected appearance of the head, The projected image comprises a base portion and one or more object portions. The projection unit is configured to project the image generated by the information processing unit based on the predetermined information, aligned with the position of the head. A simulation system in which, when the illuminance of ambient light at the location of the head is I1 [lux] and the illuminance of the base portion in the projected image is I2 [lux], I2 / I1 is between 1.5 and 30.

[0086] In other words, the simulation system of the second embodiment includes an imaging unit 2, an information processing unit 3, and a projection unit 4, similar to the simulation system 1 of the first embodiment. However, it is optional whether the one or more object parts included in the image generated by the information processing unit 3 include a part that masks the eye part included in the head, and it is also optional whether the latency is 50ms or less.

[0087] On the other hand, in the simulation system of the second embodiment, it is essential that I2 / I1 be between 1.5 and 30 (the same applies to the simulation method of the second embodiment). In this way, by setting an appropriate range for I2 / I1, the second embodiment also makes it possible to realize a simulation system that is less likely to cause glare during use when performing simulations.

[0088] [Differentiation] In addition, the following embodiments may be adopted for the simulation system 1.

[0089] Figure 8 is a configuration diagram showing a first modified example of the simulation system 1. In the previously described embodiment, the imaging unit 2, the information processing unit 3, and the projection unit 4 were shown as independent configurations. However, as shown in the simulation system 1 in Figure 8, the imaging unit 2 and the information processing unit 3 may be incorporated into the projection unit 4. In other words, a high-speed, high-performance projector that integrates the functions of the imaging unit 2, the information processing unit 3, and the projection unit 4 may be implemented.

[0090] Figure 9 is a configuration diagram showing a second modified example of the simulation system 1. In the above-described embodiment, the illuminance adjustment unit 5 was described as an arcade-like member located near the human P1, but as shown in the illuminance adjustment unit 5a of Figure 9, the member may be arranged to surround the human P1, the imaging unit 2, the information processing unit 3, and the projection unit 4. In other words, the illuminance adjustment unit 5a may constitute a booth surrounding the human P1, the imaging unit 2, the information processing unit 3, and the projection unit 4, and the human P1 may perform the simulation inside this booth.

[0091] In the embodiments described above, the information processing unit 3 was described as a computer having a predetermined function, but the information processing unit 3 may also be a control circuit that converts the information captured by the imaging unit 2 into a predetermined signal and transmits it to the projection unit 4.

[0092] In the embodiments described above, the projection area included the head of a living organism, but in modified versions, such a head may be excluded. For example, the projection area may be the torso, hands, legs, etc. of a living organism. In such embodiments, it is possible to simulate clothing, avatars, manicures, and pedicures.

[0093] Furthermore, they may be provided in the following embodiments.

[0094] (1) A simulation system for projecting an image onto the head of a living organism, comprising: an imaging unit; an information processing unit; and a projection unit, wherein the imaging unit is configured to continuously capture images including the head in a time series; the information processing unit is configured to calculate predetermined information relating to the head and generate an image to be projected based on the captured images, wherein the predetermined information includes the position, shape, and projected appearance of the head; the projected image has a base portion and one or more object portions, the one or more object portions include a portion that masks the eye portion included in the head; the projection unit is configured to project the image generated by the information processing unit based on the predetermined information, aligned with the position of the head; and the time from the time of the frame in which the imaging unit images the head to the time when the projection unit projects the image onto the head, corresponding to the frame, is 50 ms or less.

[0095] (2) A simulation system in which, in the simulation system described in (1) above, when the illuminance of ambient light at the position of the head is I1 [lux] and the illuminance of the base portion in the projected image is I2 [lux], I2 / I1 is 1.5 or more and 30 or less.

[0096] (3) A simulation system in which I1 is 5 or more and 500 or less, as described in (2) above.

[0097] (4) A simulation system according to (2) or (3) above, wherein when the illuminance of the area corresponding to the pupil portion of the image projected by the projection unit is I3 [lux], I3 / I1 is 10 or less.

[0098] (5) A simulation system according to any one of the above items (1) to (4), further comprising an illuminance adjustment unit, wherein the illuminance adjustment unit is configured to adjust the illuminance of ambient light at the location of the head by means of a member disposed around the head.

[0099] (6) A simulation system according to any one of the above items (1) to (5), wherein the imaging unit is a camera with an operating frequency of 300 fps or higher.

[0100] (7) A simulation system according to any one of the above items (1) to (6), wherein the information processing unit calculates the predetermined information for the current frame of the captured image using a predetermined region centered on the position of the head calculated from the previous frame.

[0101] (8) A simulation system according to any one of the above items (1) to (7), wherein the projection unit is a projector with an operating frequency of 300 Hz or higher.

[0102] (9) A simulation system for simulating the application of makeup to the head, in the simulation system described in any one of items (1) to (8) above.

[0103] (10) A simulation system as described in (9) above, further comprising a cosmetic input unit, wherein the information processing unit outputs the contents of one or more of the object parts based on the input to the cosmetic input unit.

[0104] (11) A simulation system for projecting an image onto the head of a living organism, comprising: an imaging unit; an information processing unit; and a projection unit, wherein the imaging unit is configured to continuously capture images including the head in a time series; the information processing unit is configured to calculate predetermined information relating to the head and generate an image to be projected based on the captured images, wherein the predetermined information includes the position, shape, and projected appearance of the head; the projected image has a base portion and one or more object portions; the projection unit is configured to project the image generated by the information processing unit based on the predetermined information, aligned with the position of the head; and when the illuminance of ambient light at the location of the head is I1 [lux] and the illuminance of the base portion in the projected image is I2 [lux], the simulation system wherein I2 / I1 is 1.5 or more and 30 or less.

[0105] (12) A simulation method using a simulation system for projecting an image onto the head of a living organism, comprising: an imaging step; an information processing step; and a projection step, wherein the imaging step involves continuously capturing images including the head in a time series; the information processing step involves calculating predetermined information about the head and generating a projection image based on the captured images, wherein the predetermined information includes the position, shape, and projected appearance of the head; the projection image comprises a base portion and one or more object portions, the one or more object portions include a portion that masks the eye portion included in the head; and the projection step involves projecting the image generated in the information processing step onto the head based on the predetermined information, aligned with the position of the head; and the time from the time of the frame in which the head is imaged in the imaging step to the time when the information processing corresponding to this frame is performed and the projection image is projected onto the head in the projection step is 50 ms or less.

[0106] (13) A simulation method using a simulation system for projecting images onto the head of a living organism, comprising: an imaging step; an information processing step; and a projection step, wherein the imaging step continuously captures images including the head in a time series; the information processing step calculates predetermined information relating to the head and generates a projection image based on the captured images, wherein the predetermined information includes the position, shape, and projected appearance of the head; the projection image comprises a base portion and one or more object portions; and the projection step projects the image generated in the information processing step onto the head at the position of the head based on the predetermined information, and when the illuminance of ambient light at the location of the head is I1 [lux] and the illuminance of the base portion in the projected image is I2 [lux], I2 / I1 is 1.5 or more and 30 or less. Of course, this is not always the case.

[0107] Finally, various embodiments of the present invention have been described, but these are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims and their equivalents. [Explanation of Symbols]

[0108] 1: Simulation System 2: Imaging Department 3: Information Processing Department 4: Projection section 5, 5a: Illuminance adjustment section 6: Makeup input section 30: Communications bus 31: Communications Department 32: Storage section 33: Control Unit 41: Light-emitting part 60: Communications bus 61: Communications Department 62: Storage section 63: Control Unit 64:Display section 65: Input section 331: Imaging control unit 332: Projection Control Unit 333: Arithmetic section 334: Input Reception Section 335: Illuminance Control Unit BS1: Base part D:Display screen IMG1: Video OBJ11~OBJ14: Object part OBJ21~OBJ23, OBJ21a, OBJ21b: Objects P1: Human

Claims

1. A simulation system that projects images onto the head of a living organism, It comprises an imaging unit, an information processing unit, and a projection unit, The imaging unit is configured to continuously capture images including the head in a time series. The information processing unit is configured to calculate predetermined information about the head and generate a projection image based on the captured image, and here, The predetermined information includes the position, shape, and projected appearance of the head, The projected image comprises a base portion and one or more object portions. One or more of the object parts include a part that masks the eye portion included in the head, The projection unit is configured to project the image generated by the information processing unit based on the predetermined information, aligned with the position of the head. The time from the time of the frame in which the imaging unit images the head until the time when the projection unit projects the image onto the head, corresponding to the information processing of that frame, is performed, is configured to be 50 ms or less. A simulation system in which, when the illuminance of ambient light at the location of the head is I1 [lux] and the illuminance of the area corresponding to the pupil in the image projected by the projection unit is I3 [lux], I3 / I1 is 10 or less.

2. In the simulation system described in claim 1, The illuminance of the base portion in the projected image is I ２ When [lux] is used, I ２ / I １ A simulation system in which the value is between 1.5 and 30.

3. In the simulation system described in claim 1, The above I １ A simulation system where the value is between 5 and 500.

4. In the simulation system described in claim 1, Furthermore, it is equipped with an illumination adjustment unit, The simulation system is configured such that the illuminance adjustment unit adjusts the illuminance of ambient light at the location of the head by means of a member disposed around the head.

5. In the simulation system described in claim 1, The aforementioned imaging unit is a simulation system in which the camera has an operating frequency of 300 fps or higher.

6. In the simulation system described in claim 1, The information processing unit is a simulation system that calculates predetermined information using a predetermined region centered on the position of the head calculated from the previous frame, with respect to the current frame of the captured image.

7. In the simulation system described in claim 1, The projection unit is a simulation system in which the projection unit is a projector with an operating frequency of 300 Hz or higher.

8. In the simulation system described in claim 1, A simulation system for simulating the application of makeup to the head.

9. In the simulation system described in claim 8, Furthermore, it is equipped with a makeup input section. The information processing unit is a simulation system that outputs the contents of one or more object parts based on the input to the cosmetic input unit.

10. A simulation system that projects images onto the head of a living organism, It comprises an imaging unit, an information processing unit, and a projection unit, The imaging unit is configured to continuously capture images including the head in a time series. The information processing unit is configured to calculate predetermined information about the head and generate a projection image based on the captured image, and here, The predetermined information includes the position, shape, and projected appearance of the head, The projected image comprises a base portion and one or more object portions. The projection unit is configured to project the image generated by the information processing unit based on the predetermined information, aligned with the position of the head. The illuminance of ambient light at the location of the head is I １ [lux], the illuminance of the base portion in the projected image is I ２ When [lux] is used, I ２ / I １ A simulation system in which the value is between 1.5 and 30.

11. A simulation method using a simulation system that projects images onto the head of a living organism, It comprises an imaging step, an information processing step, and a projection step. In the imaging step, images including the head are captured continuously in a time series. In the aforementioned information processing step, predetermined information relating to the head is calculated based on the captured image, and an image is generated for projection, where, The predetermined information includes the position, shape, and projected appearance of the head, The projected image comprises a base portion and one or more object portions. One or more of the object parts include a part that masks the eye portion included in the head, In the projection step, the image generated in the information processing step is projected based on the predetermined information, aligned with the position of the head. The time from the time of the frame in which the head is imaged in the imaging step to the time in which information processing corresponding to this frame is performed and the image is projected onto the head in the projection step is 50 ms or less. A simulation method in which, when the illuminance of ambient light at the location of the head is I1 [lux] and the illuminance of the area corresponding to the pupil in the image projected in the projection step is I3 [lux], I3 / I1 is 10 or less.

12. A simulation method using a simulation system that projects images onto the head of a living organism, It comprises an imaging step, an information processing step, and a projection step. In the imaging step, images including the head are captured continuously in a time series. In the aforementioned information processing step, predetermined information relating to the head is calculated based on the captured image, and an image is generated for projection, where, The predetermined information includes the position, shape, and projected appearance of the head, The projected image comprises a base portion and one or more object portions. In the projection step, the image generated in the information processing step is projected based on the predetermined information, aligned with the position of the head. Let the illuminance of the ambient light at the position where the head exists be I １ [lux], and the illuminance of the base portion in the projected image be I ２ [lux]. When I ２ / I １ is 1.5 or more and 30 or less, a simulation method.

Images