Information processing device and program for information processing device

Abstract

This information processing device includes: a communication unit that communicates with an image capturing device that continuously captures images of an object; and a control unit that processes the captured images obtained from the image capturing device. The control unit: executes a first processing step in which a feature portion of the object included in a first captured image is detected and a second processing step in which a new feature portion in a second captured image obtained subsequently to the first captured image during the first processing step is detected by using the feature portion in the previous captured image as an initial value and correcting the initial value; and outputs information corresponding to the new feature portion in the second captured image at the end of the second processing step.　

Description

Information Processing Apparatus and Program for Information Processing Apparatus Cross - reference to Related Applications

[0001] This application claims the priority of Japanese Patent Application No. 2024 - 182256 filed in Japan on October 17, 2024, and incorporates the entire disclosure of the prior application herein for reference.

[0002] This disclosure relates to an information processing apparatus and a program for the information processing apparatus.

[0003] Instead of actually applying makeup with cosmetics, painting with pigments, etc. to the user's face, projection mapping that projects visible light onto the user's face using a projection device is known. In projection mapping, the target part of the user's face is detected, and control of the projection device to project visible light onto the target part is performed. Patent Documents 1 to 4 disclose image processing techniques in images of the user's face and the like.

[0004] Japanese Patent Application Laid - Open No. 2021 - 128476, Japanese Patent Application Laid - Open No. 2007 - 299051, Japanese Patent Publication No. 2008 - 504606, Japanese Patent Application Laid - Open No. 2017 - 162409

[0005] In projection mapping, when the position of the user's face moves, the followability of the projection of visible light to the moving target part is low. If visible light is projected onto a part other than the target part or the time is long, the observer will feel uncomfortable. Therefore, it is desirable to improve the followability of the projection of visible light to the moving target part.

[0006] Hereinafter, an information processing apparatus and the like that enable improvement of the followability of the projection of visible light to a moving target part will be disclosed.

[0007] To solve the above problems, the information processing device in this disclosure includes a communication unit that communicates with an imaging device that continuously images an object, and a control unit that processes the captured images obtained from the imaging device. The control unit performs a first processing step of detecting characteristic parts of the object included in a first captured image, and a second processing step of detecting new characteristic parts in a second captured image obtained between the first and second processing steps, by modifying the initial value using the characteristic parts in the previous captured image as the initial value, and outputs information corresponding to the new characteristic parts in the second captured image at the end of the second processing step.

[0008] To solve the above problems, the program for the information processing device in this disclosure communicates with an imaging device that continuously images an object, and is executed by an information processing device that processes the images obtained from the imaging device, thereby causing the information processing device to perform a first processing step of detecting characteristic parts of the object included in the first image, and a second processing step of detecting new characteristic parts in a second image obtained between the first and second image captures and the first processing step, by modifying the initial value using the characteristic parts in the previous image capture as the initial value, and further, causing the information processing device to perform a step of outputting information corresponding to the new characteristic parts in the second image capture at the end of the second processing step.

[0009] The information processing device described in this disclosure makes it possible to improve the tracking performance of visible light projection onto a moving target part.

[0010] This is a diagram showing an example of the configuration of an information processing system. This is a flowchart showing an example of the operation procedure of an information processing device. This is a diagram explaining the preliminary detection process. This is a diagram explaining the regression detection process. This is a timing chart showing an example of the operation of an information processing device. This is a timing chart showing an example of the operation of an information processing device in a modified example. This is a timing chart showing an example of the operation of an information processing device in a modified example.

[0011] Embodiments of the present invention will be described below.

[0012] [System Configuration] Figure 1 is a diagram showing an example configuration of one embodiment of the present invention. The information processing system 1 is a system that supports projection mapping for makeup simulation, virtual painting, etc., on the face of a user 15. The information processing system 1 has an information processing device 10, an imaging device 12, and a projection device 13 that are connected to each other via a network 11 so as to be able to communicate with each other. The information processing device 10 is, for example, one computer or multiple computers that can communicate with each other. The computer includes a personal computer, a tablet terminal device, a smartphone, etc. The imaging device 12 includes a camera and its control device, which are installed in a position where the user 15 can be imaged. The camera of the imaging device 12 is a camera that captures images using light of a specific wavelength in the near-infrared, ultraviolet, or visible light range. The projection device 13 has a light source and an optical system for irradiating visible light and projects a makeup expression onto the face of the user 15 using visible light. The lighting device 14 is installed in a room such as a cosmetics store or amusement facility and has a light source that irradiates light of the wavelength necessary for imaging by the imaging device 12. Network 11 is, for example, a local area network (LAN) of a store, facility, or business establishment. Network 11 may also include the internet, an ad hoc network, a metropolitan area network (MAN), a mobile communication network, or other networks.

[0013] In the information processing system 1, when the projection device 13 projects visible light onto the face of the user 15, which is the target object, to represent makeup, painting, etc., the information processing device 10 controls the operation of the projection device 13 using the image of the user 15's face captured by the imaging device 12, that is, the image captured using near-infrared, ultraviolet, etc. Specifically, the information processing device 10 has a communication unit 101 that communicates with the imaging device 12 which continuously captures images of the target object, and a control unit 103 that processes the images obtained from the imaging device 12. The control unit 103 performs a first processing step (hereinafter referred to as the preliminary detection step) to detect feature parts of the target object included in the first captured image, and a second processing step (hereinafter referred to as the regression detection step) to detect new feature parts in a second captured image obtained between the first captured image and the preliminary detection step, by modifying the initial value using the feature parts in the previous captured image as the initial value. Here, the feature parts are landmark points on the face of the user 15 and are used to derive the target area for projecting visible light. The control unit 103 then outputs information corresponding to a new feature area in the last captured image during the regression detection process, that is, an instruction to project visible light onto the target area corresponding to the feature area, to the projection device 13. As the user's face 15 moves, the feature areas in the captured image are sequentially displaced. The information processing device 10 combines a preliminary detection process, which has relatively high detection accuracy for feature areas but requires a relatively long processing time, with a regression detection process, which has a relatively high possibility of false detection but can be processed in a relatively short processing time. This allows for rapid and accurate detection of the latest feature area. Therefore, even when the target area moves, it becomes possible to improve the tracking performance of the visible light projection onto the target area.

[0014] [Configuration of Information Processing Device 10] The information processing device 10 includes a communication unit 101, a storage unit 102, a control unit 103, an input unit 105, and an output unit 106. When the information processing device 10 is composed of two or more computers capable of communicating information with each other, these components are appropriately arranged in two or more computers.

[0015] The communication unit 101 includes one or more communication interfaces. The communication interface is, for example, an interface that supports wired or wireless LAN standards and connects to a nearby router device. The communication interface may have modules that support short-range wireless communication such as Bluetooth®, or mobile communication standards such as LTE (Long Term Evolution), 4G (4th Generation), or 5G (5th Generation). The communication unit 101 receives information used in the operation of the information processing device 10 and transmits information obtained through the operation of the information processing device 10. The information processing device 10 is connected to the network 11 by the communication unit 101 and communicates information with other devices via the network 11, or through direct peer-to-peer connections.

[0016] The storage unit 102 includes, for example, one or more semiconductor memories, one or more magnetic memories, one or more optical memories, or a combination of at least two of these, which function as main memory, auxiliary memory, or cache memory. The semiconductor memory is, for example, RAM (Random Access Memory) or ROM (Read Only Memory). The RAM is, for example, SRAM (Static RAM) or DRAM (Dynamic RAM). The ROM is, for example, EEPROM (Electrically Erasable Programmable ROM). The storage unit 102 stores information used in the operation of the control unit 103 and information obtained by the operation of the control unit 103.

[0017] The control unit 103 includes one or more processors, one or more dedicated circuits, or a combination thereof. The processors are, for example, general-purpose processors such as CPUs (Central Processing Units), or dedicated processors such as GPUs (Graphics Processing Units) specialized for specific processing. The dedicated circuits are, for example, FPGAs (Field-Programmable Gate Arrays), ASICs (Application Specific Integrated Circuits), etc. The control unit 103 controls each part of the information processing device 10 and executes information processing related to the operation of the information processing device 10.

[0018] The functions of the information processing device 10 are realized by the processor included in the control unit 103 executing a control program. The control program is a program that causes the processor to function as the control unit 103. In addition, some or all of the functions of the information processing device 10 may be realized by a dedicated circuit included in the control unit 103. Furthermore, the control program may be stored in a non-transient recording / storage medium readable by the control unit 103, and the control unit 103 may read it from the medium.

[0019] The input unit 105 includes one or more input interfaces. These input interfaces may include, for example, physical keys, capacitive keys, pointing devices, touchscreens integrated with displays, microphones for receiving voice input, or cameras for capturing images. Furthermore, the input interfaces may also include scanners or cameras for scanning image codes, or IC card readers. The input unit 105 receives operations from an operator inputting information used in the operation of the information processing device 10, and sends the input information to the control unit 103.

[0020] The output unit 106 includes one or more output interfaces. The output interfaces are, for example, a display or a speaker. The display is, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display. The output unit 106 outputs information obtained by the operation of the information processing device 10 to the user, operator, etc.

[0021] Next, the configurations of the imaging device 12, projection device 13, and illumination device 14 will be described.

[0022] The imaging device 12 includes one or more cameras and their control devices. The control device includes a processor that controls the operation of the cameras and a communication module for sending captured images to the information processing device 10 via the network 11 or peer-to-peer. The imaging device 12 uses the cameras to capture images of the user's face 15 at an arbitrary frame rate, for example, several hundred frames per second, and sends the captured images to the information processing device. The frame rate is preferably 300 fps or higher, more preferably 400 fps or higher, and more preferably 500 to 1000 fps. The cameras of the imaging device 12 may be cameras that capture non-visible light images, such as near-infrared cameras or ultraviolet cameras, or cameras that capture images of light in a specific wavelength range within the visible light range, that is, in a wavelength range different from the visible light range.

[0023] The projection device 13 has one or more light sources, an optical system, and a control device for these. The control device has a processor that controls the operation of the light source and the optical system, and a communication module for communicating information with the information processing device 10 via the network 11 or peer-to-peer. The light source is a lamp, laser, LED, etc., which emits light including visible light in the range of 380 nm to 780 nm. The optical system is a transmissive liquid crystal system, a reflective liquid crystal system, a DLP system, an RGB-LED system, etc. The projection device 13 emits visible light in a pattern corresponding to the information and instructions received from the information processing device 10 at an arbitrary frame rate, for example, several hundred frames per second. For example, it is 300 Hz or higher, preferably 400 Hz or higher, and more preferably 500 Hz or higher or 1000 Hz or higher.

[0024] The lighting device 14 includes a luminaire installed on the ceiling, wall, or any position that easily illuminates the face of the user 15 in a store, facility, etc., and a control device thereof. The luminaire of the lighting device 14 has, for example, a group of LED elements with different peak wavelengths as a light source, and is configured to emit ambient light including wavelengths of infrared light, ultraviolet light, etc., necessary for imaging by the imaging device 12 by selectively turning the group of LED elements on and off. The control device of the lighting device 14 controls the wavelength, intensity, etc., of the light from the luminaire. When visible light is projected by the projection device 13, it is preferable that the lighting device 14 is controlled to an arbitrary wavelength and intensity that does not interfere with the visible light.

[0025] [Operation of Information Processing Device 10] Figure 2 is a flowchart illustrating an example of the operation of the information processing device 10. Figures 3A and 3B illustrate the preliminary detection process and the regression detection process performed in part of the procedure shown in Figure 2. Figure 4 is a timing chart showing the timing of each process when the procedure shown in Figure 2 is pipelined.

[0026] The procedure shown in Figure 2 is the procedure for when the information processing device 10 controls the projection device 13 to perform projection mapping, for example, for makeup simulation, and is executed by the control unit 103 of the information processing device 10 in response to operator input. Each step in Figure 2 is pipelined, and two or more steps are executed in parallel by the control unit 103. As a result, the procedure in Figure 2 is executed in a total of, for example, a few milliseconds. Note that the procedure in Figure 2 may be executed by one information processing device or by different information processing devices.

[0027] In step S20, the control unit 103 acquires an image of the user 15's face. The control unit 103 acquires the image captured by the imaging device 12 via the communication unit 101.

[0028] In step S21, the control unit 103 detects characteristic facial features of the user 15 from the captured image. The control unit 103 performs a preliminary detection step and a regression detection step on the captured image to derive the positions of the characteristic features.

[0029] In the preliminary detection step, as shown in Figure 3A, the control unit 103 performs a process to detect landmark points 32, which are "feature areas" in this embodiment, on a single frame of captured image 30 that includes the face of the user 15. The landmark points 32 include points on the contours connecting the eyebrows, eyes, and mouth, which move characteristically when facial expressions change, the bridge of the nose, the contours of the nostrils, and the contours of the chin. The control unit 103 detects the landmark points 32 from the captured image 30 by performing arbitrary image recognition processing on the captured image 30, for example, landmark detection using a face feature extraction library on a rectangular region including the face. Here, an example of detecting landmark points 32 on the face of the user 15 captured from the front is shown, but even if the face of the user 15 is captured from an oblique direction other than the front, the same process is performed in the preliminary detection step.

[0030] In the regression detection process, the control unit 103 applies initial values ​​of landmark points to a single frame of captured image including the user's face 15, and modifies the initial values ​​by an arbitrary amount to detect landmark points that match the user's face, i.e., new landmark points. Here, as shown in state 31-1 of Figure 3B, when applying initial values ​​32a, which are a general-purpose landmark point template for a front view, to a single frame of captured image 31 including the user's face 15, and then, as shown in state 31-2, modifying the initial values ​​32a by an arbitrary amount to detect new landmark points 32c that match the user's face 15, the initial values ​​32a may not match the user's face 15 due to individual differences such as the angle and contour of the user's face, which may result in relatively low detection speed and accuracy of the new landmark points 32c. On the other hand, as shown in state 31-3, by setting the initial value 32b to a landmark point 32 that is closer to the new landmark point 32c to be ultimately detected, the detection speed of the new landmark point 32c can be increased, and the detection accuracy can be improved so that the detected new landmark point fits the face of the user 15. Therefore, in the regression detection step of this embodiment, the control unit 103 uses the landmark point detected in the preliminary detection step or regression detection step in the image captured in the image captured in the previous frame as an initial value to detect the landmark point in the image captured in the current frame, that is, the "new feature area" in this embodiment. As the face of the user 15 moves, the landmark point is displaced between frames. In the regression detection step, by using the landmark point in the previous frame, that is, a landmark point that is more likely to be closer to the new landmark point to be detected, as an initial value, it is possible to detect the new landmark point faster and more accurately compared to, for example, the case where a general-purpose landmark point in a front view is used as the initial value.

[0031] In the regression detection process, the control unit 103 can use a pre-trained machine learning model to detect new feature points from the initial values ​​of the landmark points. This model can be used, for example, to learn the relationship between the initial values ​​and the detection results when, for example, two captured images of the same person or two captured images of different people, the landmark points of one captured image are used as the initial values ​​and the landmark points of the other captured images are used as the detection results. Furthermore, in such machine learning, by using the midpoint between the landmark points of one captured image and the landmark points of the other captured image as the initial values, a model can be generated to obtain landmark point detection results with greater accuracy. Such a model is stored in the storage unit 102 in advance. In order to learn the displacement of landmark points between continuously captured images, it is preferable to perform machine learning using a series of consecutive captured images captured at a high frame rate as training data. However, the above method makes it possible to omit the acquisition and generation of such training data.

[0032] Returning to Figure 2, in step S22, the control unit 103 determines the target area. The target area is the area onto which visible light is projected by the projection device 13. The control unit 103 extracts the target area by arbitrary image processing, including, for example, pattern matching for landmark points. Here, pattern matching includes image processing such as deforming a 3D model of the face that represents the skeleton, facial expressions, etc., to fit the characteristic areas and thereby identifying the target area in the 3D model. The target areas are, for example, the eyes, cheeks, lips, etc. The control unit 103 extracts such target areas and derives the spatial coordinates of each target area.

[0033] In step S23, the control unit 103 acquires cosmetic film information. The cosmetic film information includes the type of cosmetic film to be applied to the target area and information on the color of the cosmetic film. The types of cosmetic films include foundation, eyeshadow, blush, lipstick, etc. The information on the color of the cosmetic film includes gradation values ​​of an arbitrary color system that represent the color of the cosmetic film. The information on the color of the cosmetic film may also include information on the reflectance spectrum, such as the spectral reflectance, diffusion coefficient, and absorption coefficient of the cosmetic film. The cosmetic film information is stored in the storage unit 102 by, for example, the user 15 or operator inputting the desired cosmetic film information into the information processing device 10 in advance, and the control unit 103 can acquire the cosmetic film information from the storage unit 102.

[0034] In step S24, the control unit 103 determines the color of the visible light to be projected onto the target area. Based on the cosmetic film information, the control unit 103 determines the color of the visible light to represent the cosmetic agent to be applied to the target area. The control unit 103 may determine the color of the visible light to be projected onto the target area by any method, for example, based on information such as the reflection spectrum of the target area, the reflection spectrum of the cosmetic film, the ambient light spectrum, and the projection light spectrum, which are stored in the storage unit 102 beforehand. The control unit 103 may also obtain and use the color of the visible light determined in a previous processing cycle from the storage unit 102.

[0035] In step S25, the control unit 103 sends an instruction to the projection device 13 to project visible light. The instruction includes information specifying the position of the target area and the color of the visible light. Based on the spatial coordinates of the projection device 13 and the spatial coordinates of the target area, which are stored in the memory unit 102 beforehand, the control unit 103 identifies the light sources for projecting visible light onto the target area and generates information specifying the intensity of each light source.

[0036] When the projection device 13 projects a modified color of visible light onto the target area in response to an instruction from the information processing device 10, the target area takes on the target color, and makeup is simulated on the user's face 15. The user 15 can, for example, see the makeup simulation on their own face by looking at their reflection in a mirror.

[0037] Steps S23 to S25 may be performed for each target area. If the control unit 103 detects multiple target areas in step S22 and the cosmetic film information acquired in step S23 targets different types of target areas, steps S23 to S25 may be performed for each target area. Since the reflection spectrum differs depending on the type of target area, it becomes possible to determine the color of visible light with greater accuracy.

[0038] This embodiment also includes the case in which the information processing device 10 operates in a standalone state without being connected to the network 11 during a part of the procedure shown in Figure 2.

[0039] Next, the timing of the pipelined steps S21 to S25 will be explained using Figure 4. In Figure 4, the horizontal axis represents the passage of time, and the timing of the imaging process by the imaging device 12, the preliminary detection process by the control unit 103 of the information processing device 10, the regression detection process, and the output process are shown.

[0040] Image acquisition by the imaging device 12 is performed at times T0-T1, T1-T2, ..., T5-T6, .... Each elapsed time corresponds to the frame rate of the imaging device 12. Then, at times T1, T2, ... T6, ..., the captured images F1, F2, ... F6, ... are acquired by the control unit 103. The timing at which the captured images are acquired corresponds to step S20 in Figure 2.

[0041] In the preliminary detection step included in step S21, the control unit 103 executes one processing cycle over a two-frame imaging time. Here, for example, an example is shown in which one processing cycle is executed at times T1-T3, T3-T5, etc. During times T1-T3, the preliminary detection step P(F1) is executed on the captured image F1, and during this time, the control unit 103 acquires captured images F2 and F3. During times T3-T5, the preliminary detection step P(F3) is executed on the captured image F3, and during this time, the control unit 103 acquires captured images F4 and F5. The control unit 103 performs detection of a rectangular region containing a face in each captured image using a pipeline process (not shown in this diagram), and executes each preliminary detection step and each regression detection step based on the information of the detected rectangular region. At that time, instead of detecting the rectangular region in every frame, the control unit 103 performs detection of the rectangular region in discrete frames, and estimates the rectangular region by interpolation or extrapolation in frames where detection of the rectangular region is not performed, thereby speeding up the processing. In order for the control unit 103 to interpolate or extrapolate a rectangular region, it can use, for example, a machine learning model that has been pre-programmed to determine the positional relationship between the rectangular region of one image and the rectangular region of another image in two images of the same person or two images of different people. Such a model is stored in the storage unit 102 in advance.

[0042] Furthermore, in the regression detection step included in step S21, the control unit 103 executes one or more processing cycles during the imaging time of one frame. Preferably, the processing cycle of the regression detection step is executed in 0.1 milliseconds to 0.5 milliseconds. Here, for example, an example is shown in which one or two processing cycles are executed during the imaging time of one frame.

[0043] Between times T3 and T5, the control unit 103 executes regression detection steps R(F2) and R(F3) on the captured images F2 and F3 obtained during the preliminary detection step P(F1) for the immediately preceding captured image F1. First, the control unit 103 executes a regression detection step R(F2) for one processing cycle on the captured image F2, using the landmark points of the captured image F1 detected in the preliminary detection step P(F1) as initial values, to detect new landmark points in the captured image F2. Furthermore, the control unit 103 executes a regression detection step R(F3) for one processing cycle on the captured image F3, using the detected landmark points of the captured image F2 as initial values, to detect new landmark points in the captured image F3. The processing of regression detection steps R(F2) and R(F3) for the captured images F2 and F3 is performed between times T3 and T31. Furthermore, the control unit 103 executes a regression detection step R'(F4) for one processing cycle on the captured image F4, using the detected landmark points of the captured image F3 as initial values, to detect new landmark points in the captured image F4. Then, based on the new landmark points detected from the captured image F4, the control unit 103 executes the processing corresponding to steps S22 to S25 and sends a projection instruction to the projection device 13. This processing on the captured image F4 is performed from time T4 to T41. In this way, immediately after capturing the captured image F4, projection of visible light onto the target area based on the landmark points in the captured image F4 is performed. Here, the landmark points of the captured image F4 are based on landmark points detected at high speed from the captured images F2 and F3 by regression detection steps R(F2) and R(F3) using landmark points detected with relatively high accuracy from the captured image F1 by the preliminary detection step P(F1), so that detection accuracy is guaranteed to a certain extent while being obtained at high speed. Therefore, even if the target area moves, it becomes possible to project visible light while tracking the target area at high speed and accurately.

[0044] Further, between times T5 to T51, the control unit 103 performs regression detection steps R(F4), R(F5), and R'(F5) on the captured images F4 and F5 obtained during the preliminary detection step P(F3) for the immediately preceding captured image F3, in the same manner as for the captured images F2 and F3. At this time, the control unit 103 performs the regression detection step R'(F5) on the acquired captured image F5 with the landmark points detected in the captured image F4 as the initial values, thereby detecting new landmark points of the captured image F5, and instructs the projection device 13 to project visible light onto the target part corresponding to the landmark points. By doing so, projection at a high tracking speed is ensured. On the other hand, the control unit 103 performs the regression detection step R(F4) on the captured image F4 with the landmark points detected with high accuracy from the captured image F3 by the preliminary detection step P(F3) as the initial values, and further performs the regression detection step R(F5) on F5 with the landmark points detected from the captured image F4 as the initial values. Then, the control unit 103 performs the regression detection step R(F6) on the subsequent captured image F6 at times T6 to T61 using the landmark points detected from the captured image F5 by the regression detection step R(F5). Even if a detection error of the landmark points occurs in the regression detection step due to a foreign object being captured in the captured image, etc., it is possible to recover from the error at any time by using the detection result of the highly accurate landmark points by the preliminary detection step.

[0045] [Modification Example 1] FIG. 5 shows a timing chart of the imaging step, preliminary detection step, regression detection step, and output step when the preliminary detection step corresponds to the imaging time for three frames.

[0046] At times T1 to T4, the control unit 103 executes the preliminary detection process P(F1) on the captured image F1, and during this period, the captured images F2, F3, and F4 are acquired. At times T4 to T41, the control unit 103 executes the regression detection processes R(F2), R(F3), and R(F4) on the captured images F2, F3, and F4 obtained during the preliminary detection process for the immediately previous captured image F1. First, the control unit 103 executes the regression detection process R(F2) for one processing cycle on the captured image F2 with the landmark points of the captured image F1 detected in the preliminary detection process P(F1) as the initial values to detect new landmark points of the captured image F2. Further, the control unit 103 executes the regression detection process R(F3) for one processing cycle on the captured image F3 with the detected landmark points of the captured image F2 as the initial values to detect new landmark points of the captured image F3. Further, the control unit 103 executes the regression detection process R(F4) for one processing cycle on the captured image F4 with the detected landmark points of the captured image F3 as the initial values to detect new landmark points of the captured image F4. Further, at times T5 to T51, the control unit 103 executes the regression detection process R'(F5) for one processing cycle on the captured image F5 with the detected landmark points of the captured image F4 as the initial values to detect new landmark points of the captured image F5. Then, the control unit 103 sends an instruction for projection to the projection device 13 based on the new landmark points detected from the captured image F5. In this way, immediately after the capture of the captured image F5, the projection of visible light onto the target site based on the landmark points in the captured image F4 is executed. Further, at times T6 to T61, the control unit 103 executes the regression detection process R'(F6) on the acquired captured image F6 with the landmark points detected in the captured image F5 as the initial values to detect new landmark points of the captured image F6, and instructs the projection device 13 to project visible light onto the target site corresponding to the landmark points. In this way, projection with high tracking performance onto the target site becomes possible.

[0047] [Modification Example 2] FIG. 6 shows a timing chart of the imaging process, preliminary detection process, regression detection process, and output process when the preliminary detection process corresponds to the imaging time for one frame.

[0048] Between times T1 and T2, the control unit 103 performs a preliminary detection process P(F1) on the captured image F1, and during this time, it acquires the captured image F2. Between times T2 and T21, the control unit 103 performs a regression detection process R'(F2) for one processing cycle on the captured image F2 obtained during the preliminary detection process P(F1) on the previous captured image F1, using the landmark points of the captured image F1 detected in the preliminary detection process P(F1) as initial values, and detects new landmark points in the captured image F2. Then, based on the new landmark points detected from the captured image F2, the control unit 103 sends a projection instruction to the projection device 13. Between times T2 and T3, the control unit 103 performs a preliminary detection process P(F2) on the captured image F2, and during this time, it acquires the captured image F3. Between times T3 and T31, the control unit 103 executes a regression detection step R'(F3) for one processing cycle on the captured image F3 obtained during the preliminary detection step P(F2) for the immediately preceding captured image F2, using the landmark points of the captured image F2 detected in the preliminary detection step P(F2) as initial values, thereby detecting new landmark points in the captured image F3. Then, based on the new landmark points detected from the captured image F3, the control unit 103 sends a projection instruction to the projection device 13. In this case, since the regression detection step based on the results of the preliminary detection step is executed once for each captured image to detect landmark points, it is possible to detect landmark points with higher accuracy than when the regression detection step is executed multiple times.

[0049] According to the procedure described above, the information processing device 10 can detect the latest feature area by combining a preliminary detection step, which has relatively high detection accuracy for feature areas but requires a relatively long processing time, and a regression detection step, which has a relatively high possibility of false detection but can be processed in a relatively short processing time. The combination of the required times for the preliminary detection step and the regression detection step, as described above, and the combination of the number of frames of the captured image processed in each step according to the required time for each step are arbitrary, but for example, it is possible to combine a regression detection step that can process less than 3 to 4 frames during a preliminary detection step of less than 10 milliseconds. In this way, even when the target area is moving, it is possible to improve the tracking performance of the projection of visible light onto the target area.

[0050] The above description showed an example where the target of the makeup simulation was a user. However, the target object may be something other than a person's face, such as hair or clothing, or it may be an animal other than a person, such as a pet. Furthermore, this embodiment can also be applied when projection mapping is performed as part of entertainment or a show other than makeup simulation, for example, when projection mapping is performed to mimic face painting. Alternatively, this embodiment can also be applied when projection mapping is performed on part or all of an inanimate object, such as a vehicle or a mechanical structure.

[0051] In the above, the processing and control program that defines the operation of the information processing device 10 may be stored on a cloud server or the like and downloaded to the information processing device 10 via the network 11, or it may be stored on a computer-readable non-transient recording and storage medium and read by the information processing device 10 from the medium.

[0052] As described above, embodiments have been explained based on various drawings and examples, but it should be noted that those skilled in the art will find it easy to make various modifications and alterations based on this disclosure. Therefore, it should be noted that these modifications and alterations are within the scope of this disclosure. For example, the functions included in each means, each step, etc., can be rearranged in a logically consistent manner, and multiple means, steps, etc., can be combined into one or divided.

[0053] Some embodiments of the present disclosure are illustrated below. However, it should be noted that embodiments of the present disclosure are not limited to these. [Note 1] Information processing device having a communication unit that communicates with an imaging device that continuously images an object, and a control unit that processes an image obtained from the imaging device, wherein the control unit performs a first processing step of detecting a feature portion of the object included in a first image, and a second processing step of detecting a new feature portion in a second image obtained between the first and second processing steps, by modifying the initial value using the feature portion in the previous image as the initial value, and outputs information corresponding to the new feature portion in the second image at the end of the second processing step. [Note 2] Information processing device in Note 1, wherein the communication unit further communicates with a projection device that projects light onto the object, and the information corresponding to the new feature portion is an instruction to the projection device to irradiate the projection device with light corresponding to the new feature portion. [Note 3] An information processing device in which, according to Note 1 or 2, the control unit derives the new feature portion in the second image by using the feature portion in the first image as the initial value of the second image at the beginning of the second processing step. [Note 4] An information processing device in which, according to Note 2, the control unit executes the first processing step for a processing time equal to or greater than the imaging cycle of the imaging device, and the second processing step for a processing time less than the imaging cycle. [Note 5] An information processing device in which, according to Note 4, the control unit instructs the projection device to project light corresponding to the new feature portion obtained by executing the first and second processing steps on N (N is a natural number of 2 or more) frames of images when the (N+1) frame is captured.[Note 6] A program for an information processing device that communicates with an imaging device that continuously images an object and processes the images obtained from the imaging device, thereby causing the information processing device to perform a first processing step of detecting a feature portion of the object included in the first image, and a second processing step of detecting a new feature portion in a second image obtained between the first and second processing steps, by modifying the initial value using the feature portion in the previous image as the initial value, and further causing the information processing device to perform a step of outputting information corresponding to the new feature portion in the second image at the end of the second processing step. [Note 7] In Note 6, the information processing device further communicates with a projection device that projects light onto the object, and the information corresponding to the new feature portion is an instruction to the projection device to irradiate the projection device with light corresponding to the new feature portion. [Note 8] A program for the information processing device, in which, according to Note 6 or 7, the information processing device derives the new feature portion in the second image by using the feature portion in the first image as the initial value of the second image at the beginning of the second processing step. [Note 9] A program for the information processing device, in which, according to Note 7, the information processing device executes the first processing step for a processing time equal to or greater than the imaging cycle of the imaging device, and the second processing step for a processing time less than the imaging cycle. [Note 10] A program for the information processing device, in which, according to Note 9, the information processing device instructs the projection device to project light corresponding to the new feature portion obtained by executing the first and second processing steps on N (N is a natural number of 2 or more) frames of images, when the (N+1) frame is captured.

[0054] 10: Information processing device 11: Network 12: Imaging device 13: Projection device 14: Lighting device 101: Communication unit 102: Storage unit 103: Control unit 105: Input unit 106: Output unit

Claims

1. An information processing device comprising: a communication unit that communicates with an imaging device that continuously images an object; and a control unit that processes the images obtained from the imaging device, wherein the control unit performs a first processing step of detecting characteristic parts of the object included in a first image, and a second processing step of detecting new characteristic parts in a second image obtained between the first and second processing steps, by modifying the initial value using the characteristic parts in the previous image as the initial value, and outputs information corresponding to the new characteristic parts in the second image at the end of the second processing step.

2. The information processing device according to claim 1, wherein the communication unit further communicates with a projection device that projects light onto the object, and the information corresponding to the new feature area is an instruction to the projection device to irradiate the projection device with light corresponding to the new feature area.

3. The information processing apparatus according to claim 1, wherein the control unit derives a new feature portion in the second image by using the feature portion in the first image as the initial value of the second image at the beginning of the second processing step.

4. The information processing apparatus according to claim 2, wherein the control unit executes the first processing step for a processing time equal to or greater than the imaging cycle of the imaging device, and the second processing step for a processing time equal to or less than the imaging cycle.

5. The information processing device according to claim 4, wherein the control unit instructs the projection device to project light corresponding to the new feature portion obtained by performing the first and second processing steps on N (N is a natural number of 2 or more) frames of captured images when the (N+1) frame of captured images is captured.

6. A program for an information processing device that communicates with an imaging device that continuously images an object and processes the images obtained from the imaging device, wherein the program is executed by the information processing device, causing the information processing device to perform a first processing step of detecting characteristic parts of the object included in the first image, and a second processing step of detecting new characteristic parts in a second image obtained between the first and second image captures and the first processing step, by modifying the initial value using the characteristic parts in the previous image capture as the initial value, and further, causing the information processing device to perform a step of outputting information corresponding to the new characteristic parts in the second image capture at the end of the second processing step.

7. The information processing device according to claim 6, further comprising a program for the information processing device, wherein the information processing device communicates with a projection device that projects light onto the object, and the information corresponding to the new feature area is an instruction to the projection device to irradiate the projection device with light corresponding to the new feature area.

8. A program for the information processing device according to claim 6, which causes the information processing device to derive a new feature portion in the second image by using the feature portion in the first image as the initial value of the second image at the beginning of the second processing step.

9. A program for the information processing device according to claim 7, which causes the information processing device to execute the first processing step for a processing time equal to or greater than the imaging cycle of the imaging device, and the second processing step for a processing time equal to or less than the imaging cycle.

10. A program for the information processing device according to claim 9, which causes the information processing device to instruct the projection device to project light corresponding to the new feature portion obtained by performing the first and second processing steps on N (where N is a natural number of 2 or more) frames of captured images when the (N+1) frame of captured images is captured.

Images