Information processing device and method of operating the information processing device

Abstract

Efficiently acquire data for machine learning to identify the characteristics of an object. [Solution] The information processing device has a communication unit and a control unit that communicates using the communication unit. The control unit acquires a first image of an object captured with light from a first region and a second image of the object captured with light from a second region from an imaging device. Using a model that has learned the correspondence between the first and second image pairs, the control unit generates a fourth image using light from the second region from a third image of another object captured with light from the first region, such that the difference in pixel values ​​between the third image and the fourth image is small or the edge similarity is large.

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus and an operation method thereof.

Background Art

[0002] There is known a technique of imaging an object such as a person with light in the visible light region or light in the non-visible light region according to the purpose. A visible imaging image is obtained by imaging with light in the visible light region. On the other hand, for example, in order to detect body temperature or the like, an imaging image by far infrared rays is used. Patent Document 1 discloses a technique of irradiating an object with visible light and infrared light, imaging reflected visible light and reflected infrared light, and performing face verification based on the reflected visible light image and biometric discrimination based on the reflected infrared light image. Further, Patent Document 2 discloses a technique of detecting feature points from a visible light image.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] When using machine learning to detect various features of an object based on a visible light image and a non-visible light image of the object, a huge amount of teacher data may be required. However, while visible light images that can be used as teacher data can be relatively easily acquired, non-visible light images are extremely rare and thus difficult to acquire.

[0005] In view of the above, an information processing apparatus and the like that can efficiently acquire data for machine learning of object features will be disclosed below.

Means for Solving the Problems

[0006] To solve the above problems, the information processing device in this disclosure includes a communication unit and a control unit that communicates using the communication unit, the control unit acquires a first image of an object captured with light from a first region and a second image of the object captured with light from a second region from an imaging device, and generates a fourth image using light from the second region from a third image of another object captured with light from the first region, using a model that has learned the correspondence between the first and second image pairs, such that the difference in pixel values ​​between the third image and the fourth image is small or the edge similarity is large.

[0007] To solve the above problems, the operation method of the information processing apparatus in this disclosure includes the steps of acquiring a first image obtained by imaging an object with light from a first region and a second image obtained by imaging the object with light from a second region from an imaging device, and generating a fourth image obtained by imaging another object with light from the second region using light from the first region, from a third image obtained by imaging another object with light from the first region, using a model that has learned the correspondence between the first and second image pairs, such that the difference in pixel values ​​between the third image and the fourth image is small or the edge similarity is large. [Effects of the Invention]

[0008] According to the information processing device etc. described in this disclosure, it becomes possible to efficiently acquire data for machine learning the characteristics of an object. [Brief explanation of the drawing]

[0009] [Figure 1] This is a diagram showing an example of the configuration of an information processing system. [Figure 2A] This is a flowchart illustrating an example of the operation procedure of an information processing device. [Figure 2B] This is a flowchart illustrating an example of the operation procedure of an information processing device. [Figure 2C] This is a flowchart illustrating an example of the operation procedure of an information processing device. [Figure 2D] This is a flowchart illustrating an example of the operation procedure of an information processing device. [Figure 3A] This is a diagram illustrating image processing using an information processing device. [Figure 3B] This is a diagram illustrating image processing using an information processing device. [Figure 4] This is a diagram illustrating image processing using an information processing device. [Figure 5] This is a flowchart illustrating an example of the operation procedure of an information processing device. [Figure 6] This figure shows an example of a target area. [Modes for carrying out the invention]

[0010] Embodiments of the present invention will be described below.

[0011] [System Configuration] Figure 1 shows an example configuration of one embodiment of the present invention. The information processing system 1 includes an information processing device 10, imaging devices 12-1 and 12-2, and a projection device 13, which are connected to each other via a network 11 to enable information communication. The information processing device 10 is, for example, one computer or multiple computers that can communicate with each other. The computer includes personal computers, tablet terminals, smartphones, etc. Imaging devices 12-1 and 12-2 each include a camera and its control device, which are positioned to capture images of the user 15. The camera of imaging device 12-1 is a camera that captures visible light images, and the camera of imaging device 12-2 is a camera that captures images using light of wavelengths in the non-visible light region, such as near-infrared and ultraviolet light. The projection device 13 has a light source and optical system for irradiating colored light and projects a makeup expression onto the user 15's face using visible light. The lighting device 14 is installed in a room such as a store or studio where the user 15 performs makeup simulations and has a light source that irradiates ambient light. Network 11 is, for example, a local area network (LAN) of a store, business, etc. Network 11 may also include the internet, an ad hoc network, a metropolitan area network (MAN), a mobile communication network, or other networks.

[0012] Information processing system 1 assists in the simulation of applying makeup using cosmetics to the face of user 15. User 15's face, as the object, is illuminated by invisible light from the illumination device 14 and receives a projection of visible light makeup representation from the projection device 13. Hereinafter, visible light used for makeup representation will be referred to as colored light. Information processing device 10 controls the operation of projection device 13 using images captured by imaging device 12-2, that is, images captured using near-infrared, ultraviolet, etc. Specifically, information processing device 10 detects feature parts from the auxiliary image using a model (hereinafter referred to as feature detection model) 108 that has learned the feature parts in the auxiliary image obtained by applying a predetermined process to the first type of image (a visible light image from imaging device 12-1, hereinafter referred to as the main image), one of two types of images obtained by imaging the object with light in different wavelength ranges. Here, feature areas are, for example, landmark points, corner points, edge points, etc. The feature detection model 108 detects feature areas of the object from the main image (hereinafter, the feature areas detected from the main image are called reference feature areas), and is pre-generated by machine learning using training data generated by applying the reference feature areas to the auxiliary image. Then, the information processing device 10 derives the target area onto which colored light will be projected based on the feature areas, and instructs the projection device 13 to project colored light onto the target area.

[0013] When attempting to detect feature regions from the main image of user 15's face, i.e., the visible light image, the detection accuracy of feature regions may decrease due to coloring from the projection device 13. Therefore, the information processing device 10 detects feature regions in auxiliary images such as near-infrared images and ultraviolet images, eliminating the effect of coloring. However, detecting feature regions in auxiliary images may result in lower detection accuracy compared to detecting feature regions in the main image without the influence of colored light. In this embodiment, the information processing device 10 improves the accuracy of detecting feature regions from auxiliary images by using a feature detection model 108 that has been pre-trained to detect feature regions in auxiliary images using training data obtained by applying reference feature regions detected from the main image to the auxiliary image.

[0014] When the information processing device 10 generates a feature detection model 108 using machine learning, it uses a large number of auxiliary images to which reference feature regions in the main image are applied as training data. Generating such training data requires a huge number of auxiliary images. Generating auxiliary images by capturing the faces of many users would require a huge amount of work. Furthermore, while many main images are available as open data, etc., auxiliary images are extremely rare and must be generated by capturing them one by one. In this embodiment, the information processing device 10 acquires a main image of an object captured in a first region, for example, the visible light region, and an auxiliary image of the same object captured in a second region, for example, the invisible light region, from the imaging devices 12-1 and 12-2. Using a model 109 that has learned the correspondence between main image and auxiliary image pairs (hereinafter referred to as "captured image pairs"), it generates an auxiliary image in the invisible light region from a main image of a different object captured in the visible light region, such that the difference in pixel values ​​with that main image is small or the edge similarity is large. Therefore, the information processing device 10 can generate auxiliary images from a large number of available main images, eliminating the need to generate auxiliary images individually. Consequently, it is possible to generate auxiliary images for machine learning more efficiently compared to capturing auxiliary images for a large number of users. Therefore, it becomes possible to efficiently acquire data for machine learning to learn the features of an object.

[0015] Next, the configuration of the information processing apparatus 10 will be described.

[0016] The information processing apparatus 10 includes a communication unit 101, a storage unit 102, a control unit 103, an input unit 105, and an output unit 106. When these components are configured by two or more computers capable of communicating with each other, they are appropriately arranged in the two or more computers. Any one of the two or more computers may be a server computer connected to be capable of information communication via the network 11.

[0017] The communication unit 101 includes one or more communication interfaces. The communication interface corresponds to, for example, a wired or wireless LAN standard and is an interface for connecting to a nearby router device. The communication interface may have a module corresponding to short-range wireless communication such as Bluetooth (registered trademark), or mobile communication standards such as LTE (Long Term Evolution), 4G (4th Generation), or 5G (5th Generation). The communication unit 101 receives information used in the operation of the information processing apparatus 10 and transmits information obtained by the operation of the information processing apparatus 10. The information processing apparatus 10 is connected to the network 11 by the communication unit 101 and performs information communication with other devices via the network 11 or by a direct peer-to-peer connection or the like.

[0018] The storage unit 102 includes, for example, one or more semiconductor memories, one or more magnetic memories, one or more optical memories, or a combination of at least two of these, which function as main memory, auxiliary memory, or cache memory. The semiconductor memory is, for example, RAM (Random Access Memory) or ROM (Read Only Memory). The RAM is, for example, SRAM (Static RAM) or DRAM (Dynamic RAM). The ROM is, for example, EEPROM (Electrically Erasable Programmable ROM). The storage unit 102 stores information used for the operation of the control unit 103 and information obtained by the operation of the control unit 103. The storage unit 102 also stores the feature detection model 108 and the image pair generation model 109. However, either or both of the feature detection model 108 and the image pair generation model 109 may be stored in a server device or the like that the information processing device 10 can communicate with via the network 11, and the information processing device 10 may use either or both of the feature detection model 108 and the image pair generation model 109 via the network 11.

[0019] The control unit 103 includes one or more processors, one or more dedicated circuits, or a combination thereof. The processors are, for example, general-purpose processors such as CPUs (Central Processing Units) or dedicated processors such as GPUs (Graphics Processing Units) specialized for specific processing. The dedicated circuits are, for example, FPGAs (Field-Programmable Gate Arrays) or ASICs (Application Specific Integrated Circuits). The control unit 103 controls each part of the information processing device 10 and executes information processing related to the operation of the information processing device 10.

[0020] The functions of the information processing device 10 are realized by the processor included in the control unit 103 executing a control program. The control program is a program that causes the processor to function as the control unit 103. In addition, some or all of the functions of the information processing device 10 may be realized by a dedicated circuit included in the control unit 103. Furthermore, the control program may be stored in a non-transient recording / storage medium readable by the control unit 103, and the control unit 103 may read it from the medium.

[0021] The input unit 105 includes one or more input interfaces. These input interfaces may include, for example, physical keys, capacitive keys, pointing devices, touchscreens integrated with displays, microphones for receiving voice input, or cameras for capturing images. Furthermore, the input interfaces may also include scanners or cameras for scanning image codes, or IC card readers. The input unit 105 receives operations from an operator inputting information used in the operation of the information processing device 10, and sends the input information to the control unit 103.

[0022] The output unit 106 includes one or more output interfaces. These output interfaces are, for example, a display or a speaker. The display is, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display. The output unit 106 outputs information obtained through the operation of the information processing device 10 to the user, operator, etc.

[0023] Next, the configurations of the imaging devices 12-1 and 12-2, the projection device 13, and the illumination device 14 will be described.

[0024] Each imaging device 12-1 and 12-2 has one or more cameras and their control devices. The control device has a processor that controls the operation of the cameras and a communication module for sending captured images to the information processing device 10 via the network 11 or peer-to-peer. Each imaging device 12-1 and 12-2 captures the face of the user 15 at an arbitrary frame rate, for example several hundred frames per second, using its respective camera, and sends the captured images to the information processing device. The frame rate is preferably 300 fps or higher, more preferably 400 fps or higher, and more preferably 500 fps or higher. The cameras of imaging devices 12-1 and 12-3 are cameras for capturing images of an object with light in different wavelength ranges to obtain captured images. The camera of imaging device 12-1 is a visible light camera, such as a monocular camera or a stereo camera. On the other hand, the camera of imaging device 12-2 is a camera that captures invisible light images, such as a near-infrared camera or an ultraviolet camera. Each camera in imaging devices 12-1 and 12-2 is positioned coaxially with respect to light from the object via a spectrometer and mirror. In this way, each camera in imaging devices 12-1 and 12-2 can capture the face of the user 15, illuminated by the illumination light from the illumination device 14, and generate a main image and an auxiliary image with the same field of view, which can then be sent to the information processing device 10. More preferably, each camera in imaging devices 12-1 and 12-2 can simultaneously capture images in response to instructions from the information processing device 10, for example, and generate a main image and an auxiliary image, which can then be sent to the information processing device 10.

[0025] The projection device 13 has one or more light sources, optical systems, and control devices thereof. The control device has a processor that controls the operation of the light sources and optical systems, and a communication module for communicating information with the information processing device 10 via the network 11 or peer-to-peer. The light source is a lamp, laser, LED, etc., which emits light including visible light in the range of 380 nm to 780 nm. The wavelength range of the light emitted by the light source of the projection device 13 overlaps with the wavelength range of the light emitted when the imaging device 12-1 images an object. The optical system is a transmissive liquid crystal system, a reflective liquid crystal system, a DLP system, an RGB-LED system, etc. The projection device 13 emits colored light in a pattern corresponding to the information and instructions received from the information processing device 10 at an arbitrary frame rate, for example, several hundred frames per second. For example, it is 300 Hz or higher, preferably 400 Hz or higher, and more preferably 500 Hz or higher or 1000 Hz or higher.

[0026] The lighting device 14 includes a lighting fixture installed on or near the ceiling of a store, studio, etc., and its control device. Alternatively, the lighting fixture may be set at any position that easily illuminates the face of the user 15. The lighting fixture of the lighting device 14 has, for example, a group of LED elements with different peak wavelengths as a light source, and is configured to emit ambient light including visible light necessary for capturing the main image and infrared and ultraviolet light necessary for capturing the backup image by selectively turning the group of LED elements on and off. The control device of the lighting device 14 controls the wavelength, intensity, etc. of the light from the lighting fixture. When projecting colored light with the projection device 13, it is preferable that the lighting device 14 is controlled to an arbitrary wavelength and intensity that does not interfere with the colored light. Furthermore, the lighting device 14 may output different illumination intensities or use different equipment when acquiring the main image and auxiliary image for generating the feature detection model 108 and when projecting with the projection device 13.

[0027] [Operation of Information Processing Device 10 - 1] Figures 2A to 2D are flowcharts illustrating examples of the operation of the information processing device 10. The steps in Figures 2A to 2D are the procedures for generating the feature detection model 108 and the image pair generation model 109.

[0028] The procedure shown in Figure 2A is a procedure for generating an image pair generation model 109, which is executed by the control unit 103 of the information processing device 10 in response to the operator's actions.

[0029] In step S201, the control unit 103 acquires a main image and an auxiliary image of the face of one or more users 15. The control unit 103 acquires the main image and the auxiliary image from the imaging devices 12-1 and 12-2, respectively. The imaging devices 12-1 and 12-2 capture images at an arbitrary frame rate and send the main image and the auxiliary image, respectively, to the information processing device 10. It is preferable that the frame rates of the imaging devices 12-1 and 12-2 are the same, or that frame rates are used such that frames are captured simultaneously. Furthermore, the acquired main image and auxiliary image may include images captured not only from the front of the user 15's face, but also from different angles corresponding to the rotation of the user 15's face, and may include images of the user 15's face with multiple different facial expressions.

[0030] In step S202, the control unit 103 creates a pair consisting of a main image and an auxiliary image (hereinafter referred to as an image capture pair) and stores it in the storage unit 102. For example, the control unit 103 creates a pair of main images and auxiliary images with the same acquisition time from multiple main images and auxiliary images using a timestamp, and stores them in the storage unit 102 as an image capture pair.

[0031] JPEG2026057316000002.jpg237170

[0032] In step S204, the control unit 103 stores the image pair generation model 109 in the storage unit 102.

[0033] This embodiment also includes cases where, instead of using a GAN, an image pair generation model 109 is configured to generate auxiliary images such that the difference between the main image and pixel values ​​is small or the edge similarity is large, for example, by using a diffusion model.

[0034] The procedure shown in Figure 2B is a procedure for generating data to be used for machine learning of the feature detection model 108, and is executed by the control unit 103 of the information processing device 10 in response to the operator's input.

[0035] In step S211, the control unit 103 acquires main facial images for multiple users 15. The acquired main facial images may include images taken not only from the front of the user 15's face, but also from different angles corresponding to the rotation of the user 15's face, and may include images of the user 15's face with different expressions. The control unit 103 acquires the main facial images by reading them from the storage unit 102, which have been previously acquired from open data or the like.

[0036] In step S212, the control unit 103 generates auxiliary images from the main images using the image pair generation model 109. For example, the control unit 103 inputs multiple main images into the image pair generation model 109 and obtains corresponding auxiliary images as outputs.

[0037] In step S213, the control unit 103 creates a training image pair including the main image and the generated auxiliary image and stores it in the storage unit 102.

[0038] In step S214, the control unit 103 performs the processing in steps S212 and S213 on the main image acquired in step S211 to determine whether training image pairs have been generated. If the processing for all main images is complete (Yes), the control unit 103 terminates the procedure in Figure 2B. If it is not complete (No), it returns to step S212 and performs the processing in steps S212 and S213 on the new main image.

[0039] The procedure shown in Figure 2B generates data for machine learning to create the feature detection model 108. By obtaining main images from various angles and facial expressions and generating corresponding auxiliary images, it becomes possible to perform machine learning to detect feature areas while tracking the user's facial movements.

[0040] The procedure shown in Figure 2C is the procedure for generating training data for generating the feature detection model 108, and is executed by the control unit 103 of the information processing device 10 in response to the operator's actions.

[0041] In step S21, the control unit 103 acquires a training image pair including a primary image and a secondary image of the user 15's face. The control unit 103 reads and acquires the training image pair generated in the procedure shown in Figure 2B from the storage unit 102.

[0042] In step S22, the control unit 103 performs a process to detect feature regions in the main image. Feature regions include landmark points, corner points, edge points, etc. The control unit 103 detects feature regions from the main image using any image processing procedure. The positional information of the detected feature regions, i.e., reference feature regions, is stored in the storage unit 102 for each main image. For example, in the example in Figure 4, a landmark point detection process is performed on the main image 301, and a main image 311 in which a landmark point as a reference feature region 300 is detected is generated and stored.

[0043] In step S23, the control unit 103 applies reference feature regions to the auxiliary image. The control unit 103 uses an arbitrary image processing procedure to associate the contour of the user 15's face, the center of the contour, or the centroid of the contour in the main image and the auxiliary image. By arranging the cameras of the imaging devices 12-1 and 12-2 coaxially, the processing load for associating the main image and the auxiliary image is reduced. The control unit 103 then applies the reference feature regions detected in the associated main image to the auxiliary image. In this way, the control unit 103 generates an auxiliary image for machine learning and stores it in the storage unit 102. For example, in the example in Figure 4, by transferring the reference feature region 300 contained in the main image 311 to the auxiliary image 302, an auxiliary image 321 containing the reference feature region 300 is generated and stored.

[0044] In step S24, the control unit 103 performs the processing in steps S22 and S23 on all training image pairs acquired in step S21 to determine whether auxiliary images constituting training data for machine learning have been generated. If the processing on all training image pairs is complete (Yes), the control unit 103 terminates the procedure in Figure 2C; otherwise, it returns to step S22 and performs the processing in steps S22 and S23 on new pairs.

[0045] The procedure shown in Figure 2C generates training data for machine learning to create the feature detection model 108. Furthermore, performing the procedure in Figure 2C on multiple different users generates training data that takes into account a wider variety of user facial features. Additionally, by rotating each user's face to obtain primary and secondary images from multiple angles, or by having each user change their facial expression to obtain primary and secondary images with different expressions, it becomes possible to perform machine learning to detect feature regions while tracking the user's facial movements.

[0046] The procedure shown in Figure 2D is for generating a feature detection model 108 using training data, and is executed by the control unit 103 of the information processing device 10 in response to operator input. Alternatively, the procedure in Figure 2D may be executed by another server device or the like using the training data generated in the procedure in Figure 2C. In that case, the following explanation will be executed by the control unit of the other server device or the like.

[0047] In step S25, the control unit 103 acquires training data. For example, the control unit 103 reads and acquires training data from the storage unit 102.

[0048] In step S26, the control unit 103 performs machine learning using training data to generate a feature detection model 108. Since the training data is data in which feature regions are annotated on auxiliary images, a feature detection model 108 that has already learned the feature regions in the auxiliary images is generated by performing machine learning on this training data. The generated feature detection model 108 is stored in the storage unit 102. Alternatively, the feature detection model 108 generated by the information processing device 10 may be stored in another server device or the like.

[0049] The feature detection model 108 is generated by following the steps shown in Figure 2D.

[0050] [Operation of Information Processing Device 10 - 2] Figure 5 is a flowchart illustrating an example of the operation of the information processing device 10.

[0051] The procedure shown in Figure 5 is the procedure by which the information processing device 10 controls the projection device 13 using the feature detection model 108 to perform a makeup simulation, and is executed by the control unit 103 of the information processing device 10 in response to operator input. Each step in Figure 5 is pipelined, and two or more steps are executed in parallel by the control unit 103. As a result, the procedure in Figure 4 is executed in a total of, for example, a few milliseconds. Note that the procedures in Figures 2A to 2D and Figure 5 may be executed by one information processing device or by different information processing devices.

[0052] In step S40, the control unit 103 acquires an auxiliary image of the user 15's face. The control unit 103 acquires each of the auxiliary images captured by the imaging device 12-2.

[0053] In step S41, the control unit 103 detects feature regions from the auxiliary image of the user 15's face. The control unit 103 inputs the auxiliary image to the feature detection model 108 and obtains an auxiliary image as output in which landmark points representing feature regions are superimposed.

[0054] In step S42, the control unit 103 determines the target area. The target area is the area onto which the projection device 13 projects colored light. The control unit 103 extracts the target area by arbitrary image processing, including, for example, pattern matching for landmark points. Here, pattern matching includes image processing such as deforming a 3D model of the face that represents the skeleton, facial expressions, etc., to fit the feature area and thereby identifying the target area in the 3D model. Figure 6 shows examples of target areas on the face 33 of user 15, namely the eye area 30, cheeks 31, and lips 32. The control unit 103 extracts such target areas and derives the spatial coordinates of each target area.

[0055] In step S43, the control unit 103 acquires cosmetic film information. The cosmetic film information includes the type of cosmetic film to be applied to the target area and information on the color of the cosmetic film. The types of cosmetic films include foundation, eyeshadow, blush, lipstick, etc. The information on the color of the cosmetic film includes gradation values ​​of an arbitrary color system that represent the color of the cosmetic film. The information on the color of the cosmetic film may also include information on the reflectance spectrum, such as the spectral reflectance, diffusion coefficient, and absorption coefficient of the cosmetic film. The cosmetic film information is stored in the storage unit 102, for example, by the user 15 or operator inputting the desired cosmetic film information into the information processing device 10 in advance, and the control unit 103 can acquire the cosmetic film information from the storage unit 102.

[0056] In step S44, the control unit 103 determines the color of the colored light to be projected onto the target area. Based on the cosmetic film information, the control unit 103 determines the color of the colored light to represent the cosmetic agent to be applied to the target area. The control unit 103 may determine the color of the colored light to be projected onto the target area by any method, for example, based on information such as the reflection spectrum of the target area, the reflection spectrum of the cosmetic film, the ambient light spectrum, and the projection light spectrum, which are stored in the storage unit 102 beforehand. The control unit 103 may also retrieve and use the color of the colored light determined in a previous processing cycle from the storage unit 102.

[0057] In step S45, the control unit 103 sends an instruction to the projection device 13 to project colored light. The instruction includes information specifying the position of the target area and the color of the colored light. Based on the spatial coordinates of the projection device 13 and the spatial coordinates of the target area, which are stored in the memory unit 102 beforehand, the control unit 103 identifies the light sources for projecting colored light onto the target area and generates information specifying the intensity of each light source.

[0058] When the projection device 13 projects colored light of the adjusted color onto the target area in response to instructions from the information processing device 10, the target area takes on the target color, and makeup is simulated on the user's face 15. The user 15 can, for example, see the makeup simulation on their own face by looking at their reflection in a mirror.

[0059] The control unit 103 may, for example, acquire the main image from the imaging device 12-1 in step S40 and display it on the output unit 106's display to present it to the user 15. In this way, the main image obtained by capturing the user 15's face, onto which colored light was projected in previous processing cycles, can be displayed. This allows the user 15 to visually confirm a simulation of makeup on their own face.

[0060] Steps S43 to S45 may be performed for each target area. If the control unit 103 detects multiple target areas in step S42 and the cosmetic film information acquired in step S43 targets different types of target areas, steps S43 to S45 may be performed for each target area. Since the reflection spectrum differs depending on the type of target area, it becomes possible to determine the color of colored light with greater accuracy.

[0061] This embodiment also includes the case in which the information processing device 10 operates in a standalone state without being connected to the network 11 during a part of the procedure shown in Figure 5.

[0062] According to the procedure described above, the information processing device 10 can detect feature regions in the auxiliary image, which is less affected by colored light than the main image, with an accuracy close to that of feature region detection on the main image. Therefore, since the target region can be accurately set based on the detected feature regions, it becomes possible to improve the accuracy of projecting colored light onto the target region of the object.

[0063] When the imaging device 12-2 is configured to perform imaging with light in a specific wavelength range of the visible light region, the control unit 103 can send an instruction to the projection device 13 or the illumination device 14 to selectively reduce the intensity of light in the wavelength range used for imaging from the visible light projected by the projection device 13 or the illumination device 14. Preferably, the wavelength range of the light used for imaging by the imaging device 12-2 is different from the wavelength range of the colored light irradiated onto the target area. In this case, by weakening the intensity of light of such color, the degree to which light different from the makeup color is mixed into the makeup expression on the user's face 15 can be reduced, making it possible to accurately simulate the makeup expression.

[0064] The above description showed an example where the target of the makeup simulation was a user. However, the target object may be something other than a person's face, such as hair or clothing, or it may be an animal other than a person, such as a pet. Furthermore, this embodiment is also applicable when projection mapping is performed as part of a show or other performance other than a makeup simulation. Alternatively, this embodiment is also applicable when projection mapping is performed on part or all of an inanimate object such as a vehicle or mechanical structure.

[0065] Furthermore, in cases where different information is detected using visible light images and invisible light images of an object, applying this embodiment makes it possible to acquire only the visible light image and generate the invisible light image from it, thereby reducing the amount of work required.

[0066] This embodiment is also applicable when the main image and auxiliary image are arbitrary images of light in different wavelength ranges. For example, this embodiment can be applied when generating an image in the visible light region from an image captured with light in the invisible light region. This embodiment can also be applied when generating an image using light in a part of the visible light region (for example, the blue light which is difficult to see) from a visible light image. Furthermore, this embodiment can also be applied when generating an image using light in a part of the invisible light region (for example, the mid-infrared) from an image using invisible light (for example, the near-infrared).

[0067] In the above, the processing and control program that defines the operation of the information processing device 10 may be stored on a cloud server or the like and downloaded to the information processing device 10 via the network 11, or it may be stored on a computer-readable non-transient recording and storage medium and read by the information processing device 10 from the medium.

[0068] As described above, embodiments have been explained based on various drawings and examples, but it should be noted that those skilled in the art will find it easy to make various modifications and alterations based on this disclosure. Therefore, it should be noted that these modifications and alterations are within the scope of this disclosure. For example, the functions, etc., included in each means, each step, etc., can be rearranged in a logically consistent manner, and multiple means, steps, etc., can be combined into one or divided. [Explanation of Symbols]

[0069] 10: Information Processing Devices 11: Network 12-1, 12-2: Imaging device 13: Projection device 14: Lighting equipment 101: Communications Department 102: Storage section 103: Control Unit 105: Input section 106: Output section

Claims

1. Communications Department and, The system includes a control unit that performs communication using the aforementioned communication unit, The control unit acquires a first image of an object captured with light from a first region and a second image of the object captured with light from a second region from the imaging device, and using a model that has learned the correspondence between the first and second image pairs, generates a fourth image using light from the second region from a third image of another object captured with light from the first region, such that the difference in pixel values ​​between the third image and the fourth image is small or the edge similarity is large. Information processing device.

2. In claim 1, The third and fourth images are used as training data for machine learning to detect feature parts of the other object from the third image and apply those feature parts to the fourth image. Information processing device.

3. In claim 2, The aforementioned feature portion is a landmark point in the other object. Information processing device.

4. In claim 1, The first and second image pairs are captured by a coaxial pair of imaging devices, respectively. Information processing device.

5. A method for operating an information processing device, A step of acquiring a first image of the object captured with light from a first region and a second image of the object captured with light from a second region from an imaging device, A step of generating a fourth image using light from the second region, from a third image taken with light from the first region of a different object using a model that has learned the correspondence between the first and second image pairs, such that the difference in pixel values ​​between the third image and the fourth image is small or the edge similarity is large. A method of operating an information processing device, including the operation of the device.

6. In claim 5, The third and fourth images are used as training data for machine learning to detect feature parts of the other object from the third image and apply those feature parts to the fourth image. The operation method of an information processing device.

7. In claim 6, The aforementioned feature portion is a landmark point in the other object. The operation method of an information processing device.

8. In claim 6, The first and second image pairs are captured by a coaxial pair of imaging devices, respectively. The operation method of an information processing device.

Images