Information processing device, system, and method of operating the system

Abstract

To improve the accuracy of projecting colored light onto the target area of ​​an object. [Solution] The information processing device has a communication unit and a control unit that communicates with an imaging device via the communication unit. The control unit detects the feature portions of another object from the second type of image of another object using a model that has learned the feature portions in the second type of image obtained by applying a predetermined process to the first type of image of the object, which is one of two types of image obtained by imaging an object with light in different wavelength regions.

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus, a system, and an operation method of the system.

Background Art

[0002] Techniques for simulating makeup have been proposed instead of actually applying makeup to a user's face with cosmetics. For example, projection mapping that projects colored light imitating makeup onto a user's face by a projection device has been proposed. In projection mapping, a target part of a user's face is detected, and control of the projection device is performed to project colored light onto the target part. Patent Documents 1 and 2 disclose image processing techniques in an image of a user's face or the like.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] In projection mapping of makeup, when projecting visible light representing the color of each cosmetic agent onto a target part of a user's face, for example, the eyelid or the outer corner of the eye if it is eye shadow, or the lip if it is lipstick, if there is a part where colored light is not projected on the target part or if the colored light spills out from the target area, the expression of the makeup is impaired. Therefore, improvement in the projection accuracy of colored light onto the target part of an object is desired.

[0005] Hereinafter, an information processing apparatus and the like capable of improving the projection accuracy of colored light onto the target part of an object will be disclosed.

Means for Solving the Problems

[0006] To solve the above problems, the information processing device in this disclosure includes a communication unit and a control unit that communicates with an imaging device via the communication unit, wherein the control unit applies a predetermined process to a first type of imaging image, one of two types of imaging images obtained by imaging an object with light in different wavelength regions, to the second type of imaging image, and uses a model that has learned the feature parts of the second type of imaging image of another object to detect the feature parts of that other object from the second type of imaging image of the other object.

[0007] An information processing device in another aspect of this disclosure includes a communication unit and a control unit that transmits and receives information via the communication unit, wherein the control unit acquires two types of captured images obtained by imaging an object with light in different wavelength regions from an imaging device via the communication unit, detects characteristic parts of the object from the first type of captured image, applies the characteristic parts to the second type of captured image, and generates training data for machine learning the characteristic parts in the second type of captured image.

[0008] The system in this disclosure is a system comprising an information processing device and an imaging device, wherein the imaging device sends two types of images obtained by imaging an object with light in different wavelength regions to the information processing device, and the information processing device applies the feature portions of the object detected by performing a predetermined process on the first type of imaging image to the second type of imaging image, and uses a model that has learned the feature portions in the second type of imaging image of another object to detect the feature portions of that other object from the second type of imaging image of the other object.

[0009] The system in this disclosure is a system comprising an information processing device and an imaging device, wherein the imaging device sends two types of images obtained by imaging an object with light in different wavelength regions to the information processing device, the information processing device detects feature parts of the object from the first type of imaging image, applies the feature parts to the second type of imaging image, and generates training data for machine learning the feature parts in the second type of imaging image.

[0010] The method of operating the system in this disclosure is a method of operating a system having an information processing device and an imaging device, wherein the imaging device sends two types of captured images obtained by imaging an object with light in different wavelength regions to the information processing device, and the information processing device applies the feature portions of the object detected by performing a predetermined process on the first type of captured image to the second type of captured image, and uses a model that has learned the feature portions in the second type of captured image of another object to detect the feature portions of that other object from the second type of captured image of the other object.

[0011] The method of operating the system in this disclosure is a method of operating a system having an information processing device and an imaging device, wherein the imaging device sends two types of images obtained by imaging an object with light in different wavelength regions to the information processing device, the information processing device detects feature parts of the object from the first type of imaging image, applies the feature parts to the second type of imaging image, and generates training data for machine learning the feature parts in the second type of imaging image. [Effects of the Invention]

[0012] The information processing device described in this disclosure makes it possible to improve the accuracy of projecting colored light onto the target area of ​​an object. [Brief explanation of the drawing]

[0013] [Figure 1] This is a diagram showing an example of the configuration of an information processing system. [Figure 2A] This is a flowchart illustrating an example of the operation procedure of an information processing device. [Figure 2B] This is a flowchart illustrating an example of the operation procedure of an information processing device. [Figure 3] This is a diagram illustrating image processing using an information processing device. [Figure 4] This is a flowchart illustrating an example of the operation procedure of an information processing device. [Figure 5] This figure shows an example of a target area. [Modes for carrying out the invention]

[0014] Embodiments of the present invention will be described below.

[0015] [System Configuration] Figure 1 shows an example configuration of one embodiment of the present invention. The information processing system 1 includes an information processing device 10, imaging devices 12-1 and 12-2, and a projection device 13, which are connected to each other via a network 11 to enable information communication. The information processing device 10 is, for example, one computer or multiple computers that can communicate with each other. The computer includes personal computers, tablet terminals, smartphones, etc. Imaging devices 12-1 and 12-2 each include a camera and its control device, which are positioned to capture images of the user 15. The camera of imaging device 12-1 is a camera that captures visible light images, and the camera of imaging device 12-2 is a camera that captures images using near-infrared, ultraviolet, or light of a specific wavelength in the visible light range. The projection device 13 has a light source and optical system for irradiating colored light and projects a makeup expression onto the user 15's face using colored light. The lighting device 14 is installed in a room such as a store or studio where the user 15 performs makeup simulations and has a light source that irradiates ambient light. Network 11 is, for example, a local area network (LAN) of a store, business, etc. Network 11 may also include the internet, an ad hoc network, a metropolitan area network (MAN), a mobile communication network, or other networks.

[0016] Information processing system 1 assists in the simulation of applying makeup using cosmetics to the face of user 15. User 15's face, as the object, is illuminated by ambient light from the lighting device 14 and receives a projection of visible light makeup representation from the projection device 13. Hereinafter, visible light used for makeup representation will be referred to as colored light. Information processing device 10 controls the operation of projection device 13 using images captured by imaging device 12-2, that is, images captured using near-infrared, ultraviolet, etc. Specifically, information processing device 10 detects feature parts from the auxiliary image using a model (hereinafter referred to as feature detection model) 108 that has learned the feature parts in the auxiliary image obtained by applying a predetermined process to the first type of image (a visible light image from imaging device 12-1, hereinafter referred to as the main image), one of two types of images obtained by imaging the object with light in different wavelength ranges. Here, feature areas are, for example, landmark points, corner points, edge points, etc. The feature detection model 108 detects feature areas of the object from the main image (hereinafter, the feature areas detected from the main image are called reference feature areas), and is pre-generated by machine learning using training data generated by applying the reference feature areas to the auxiliary image. Then, the information processing device 10 derives the target area onto which colored light will be projected based on the feature areas, and instructs the projection device 13 to project colored light onto the target area.

[0017] When attempting to detect the characteristic part from the main image of the face of user 15, that is, the visible light image, the detection accuracy of the characteristic part may decrease due to coloring by the colored light from the projection device 13. Therefore, the information processing device 10 detects the characteristic part by eliminating the influence of coloring in auxiliary images such as near-infrared images and ultraviolet images. However, when detecting the characteristic part in the auxiliary image, the detection accuracy may be inferior compared to the case of detecting the characteristic point without being affected by the colored light in the main image. In this regard, according to the present embodiment, the information processing device 10 uses the feature detection model 108 that has been pre-trained on the teacher data obtained by applying the reference characteristic part detected from the main image to the auxiliary image, to improve the accuracy when detecting the characteristic part from the auxiliary image. And by improving the detection accuracy of the characteristic part, the information processing device 10 can instruct the projection device 13 to project the colored light to a more accurate projection position. Therefore, it becomes possible to improve the projection accuracy of the colored light onto the target part of the object.

[0018] Next, the configuration of the information processing device 10 will be described.

[0019] The information processing device 10 includes a communication unit 101, a storage unit 102, a control unit 103, an input unit 105, and an output unit 106. These components are appropriately arranged in two or more computers when the information processing device 10 is composed of two or more computers capable of communicating with each other.

[0020] The communication unit 101 includes one or more communication interfaces. The communication interface corresponds to, for example, a wired or wireless LAN standard and is an interface for connecting to a nearby router device. The communication interface may have a module corresponding to short-range wireless communication such as Bluetooth (registered trademark), or mobile communication standards such as LTE (Long Term Evolution), 4G (4th Generation), or 5G (5th Generation). The communication unit 101 receives information used in the operation of the information processing device 10 and transmits information obtained by the operation of the information processing device 10. The information processing device 10 is connected to the network 11 by the communication unit 101 and performs information communication with other devices via the network 11 or by a direct peer-to-peer connection or the like.

[0021] The storage unit 102 includes, for example, one or more semiconductor memories that function as a main storage device, an auxiliary storage device, or a cache memory, one or more magnetic memories, one or more optical memories, or a combination of at least two of these. The semiconductor memory is, for example, a RAM (Random Access Memory) or a ROM (Read Only Memory). The RAM is, for example, a SRAM (Static RAM) or a DRAM (Dynamic RAM). The ROM is, for example, an EEPROM (Electrically Erasable Programmable ROM). The storage unit 102 stores information used in the operation of the control unit 103 and information obtained by the operation of the control unit 103. The storage unit 102 also stores the feature detection model 108. However, the feature detection model 108 may be stored in a server device or the like that the information processing device 10 can communicate with via the network 11, and the information processing device 10 may use the feature detection model 108 via the network 11.

[0022] The control unit 103 includes one or more processors, one or more dedicated circuits, or a combination thereof. The processors are, for example, general-purpose processors such as CPUs (Central Processing Units) or dedicated processors such as GPUs (Graphics Processing Units) specialized for specific processing. The dedicated circuits are, for example, FPGAs (Field-Programmable Gate Arrays) or ASICs (Application Specific Integrated Circuits). The control unit 103 controls each part of the information processing device 10 and executes information processing related to the operation of the information processing device 10.

[0023] The functions of the information processing device 10 are realized by the processor included in the control unit 103 executing a control program. The control program is a program that causes the processor to function as the control unit 103. In addition, some or all of the functions of the information processing device 10 may be realized by a dedicated circuit included in the control unit 103. Furthermore, the control program may be stored in a non-transient recording / storage medium readable by the control unit 103, and the control unit 103 may read it from the medium.

[0024] The input unit 105 includes one or more input interfaces. These input interfaces may include, for example, physical keys, capacitive keys, pointing devices, touchscreens integrated with displays, microphones for receiving voice input, or cameras for capturing images. Furthermore, the input interfaces may also include scanners or cameras for scanning image codes, or IC card readers. The input unit 105 receives operations from an operator inputting information used in the operation of the information processing device 10, and sends the input information to the control unit 103.

[0025] The output unit 106 includes one or more output interfaces. These output interfaces are, for example, a display or a speaker. The display is, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display. The output unit 106 outputs information obtained through the operation of the information processing device 10 to the user, operator, etc.

[0026] Next, the configurations of the imaging devices 12-1 and 12-2, the projection device 13, and the illumination device 14 will be described.

[0027] Each imaging device 12-1 and 12-2 has one or more cameras and their control devices. The control device has a processor that controls the operation of the cameras and a communication module for sending captured images to the information processing device 10 via the network 11 or peer-to-peer. Each imaging device 12-1 and 12-2 captures the face of the user 15 at an arbitrary frame rate, for example several hundred frames per second, using its respective camera, and sends the captured images to the information processing device. The frame rate is preferably 300 fps or higher, more preferably 400 fps or higher, and more preferably 500 fps or higher. The cameras of imaging devices 12-1 and 12-2 are cameras for capturing images of an object with light in different wavelength regions and obtaining captured images. The camera of imaging device 12-1 is, for example, a visible light camera such as a monocular camera or a stereo camera. On the other hand, the camera of imaging device 12-2 may be a camera that captures non-visible light images such as a near-infrared camera or an ultraviolet camera, or it may be a camera that captures images of light in a specific wavelength region within the visible light region, that is, a wavelength region different from the visible light region. Each camera in imaging devices 12-1 and 12-2 is positioned coaxially with respect to light from the object via a spectrometer and mirror. In this way, each camera in imaging devices 12-1 and 12-2 can capture the face of the user 15, illuminated by the illumination light from the illumination device 14, and generate a main image and an auxiliary image with the same field of view, which can then be sent to the information processing device 10. More preferably, each camera in imaging devices 12-1 and 12-2 can simultaneously capture images in response to instructions from the information processing device 10, for example, and generate a main image and an auxiliary image, which can then be sent to the information processing device 10.

[0028] The projection device 13 has one or more light sources, optical systems, and control devices thereof. The control device has a processor that controls the operation of the light sources and optical systems, and a communication module for communicating information with the information processing device 10 via the network 11 or peer-to-peer. The light source is a lamp, laser, LED, etc., which emits light including visible light in the range of 380 nm to 780 nm. The wavelength range of the light emitted by the light source of the projection device 13 overlaps with the wavelength range of the light emitted when the imaging device 12-1 images an object. The optical system is a transmissive liquid crystal system, a reflective liquid crystal system, a DLP system, an RGB-LED system, etc. The projection device 13 emits colored light in a pattern corresponding to the information and instructions received from the information processing device 10 at an arbitrary frame rate, for example, several hundred frames per second. For example, it is 300 Hz or higher, preferably 400 Hz or higher, and more preferably 500 Hz or higher or 1000 Hz or higher.

[0029] The lighting device 14 includes a lighting fixture installed on or near the ceiling of a store, studio, etc., and its control device. Alternatively, the lighting fixture may be set in any position that easily illuminates the face of the user 15. The lighting fixture of the lighting device 14 has, for example, a group of LED elements with different peak wavelengths as a light source, and is configured to emit ambient light including visible light necessary for capturing the main image and infrared and ultraviolet light necessary for capturing the auxiliary image by selectively turning the group of LED elements on and off. The control device of the lighting device 14 controls the wavelength, intensity, etc. of the light from the lighting fixture. When projecting colored light with the projection device 13, it is preferable that the lighting device 14 is controlled to an arbitrary wavelength and intensity that does not interfere with the colored light. Furthermore, the lighting device 14 may output different illumination intensities or use different equipment when acquiring the main image and auxiliary image for generating the feature detection model 108 and when projecting with the projection device 13.

[0030] [Operation of Information Processing Device 10 - 1] Figures 2A and 2B are flowcharts illustrating an example of the operation of the information processing device 10.

[0031] The procedure shown in Figure 2A is the procedure for generating training data for the feature detection model 108, and is executed by the control unit 103 of the information processing device 10 in response to operator input.

[0032] In step S20, the control unit 103 acquires a main image and an auxiliary image of the user 15's face. The control unit 103 acquires the main image and the auxiliary image from the imaging devices 12-1 and 12-2, respectively. The imaging devices 12-1 and 12-2 capture images at an arbitrary frame rate and send the main image and the auxiliary image, respectively, to the information processing device 10. It is preferable that the frame rates of the imaging devices 12-1 and 12-2 are the same, or that frame rates are used such that frames are captured simultaneously. In addition, the acquired main image and auxiliary image may include images captured not only from the front of the user 15's face, but also from different angles corresponding to the rotation of the user 15's face.

[0033] In step S21, the control unit 103 creates a pair of main images and auxiliary images and stores them in the storage unit 102. For example, the control unit 103 creates a pair of main images and auxiliary images with the same acquisition time from multiple main images and auxiliary images using a timestamp, and stores them in the storage unit 102. For example, as shown in Figure 3, the main image 301 and the auxiliary image 302 are stored as a pair.

[0034] In step S22, the control unit 103 performs a process to detect feature regions in the main image. Feature regions include landmark points, corner points, edge points, etc. The control unit 103 detects feature regions from the main image using any image processing procedure. The positional information of the detected feature regions, i.e., reference feature regions, is stored in the storage unit 102 for each main image. For example, in the example in Figure 3, a landmark point detection process is performed on the main image 301, and a main image 311 in which a landmark point as a reference feature region 300 is detected is generated and stored.

[0035] In step S23, the control unit 103 applies reference feature regions to the auxiliary image. The control unit 103 uses an arbitrary image processing procedure to associate the contour of the user 15's face, the position of the center of the contour, or the centroid of the contour in the main image and the auxiliary image. By arranging the cameras of the imaging devices 12-1 and 12-2 coaxially, the processing load for associating the main image and the auxiliary image is reduced. The control unit 103 then applies the reference feature regions detected in the associated main image to the auxiliary image. In this way, the control unit 103 generates an auxiliary image for machine learning and stores it in the storage unit 102. For example, in the example in Figure 3, by transferring the reference feature region 300 contained in the main image 311 to the auxiliary image 321, an auxiliary image 321 containing the reference feature region 300 is generated and stored.

[0036] In step S24, the control unit 103 performs the processing in steps S22 and S23 for all pairs of main and auxiliary images acquired in step S20 and associated in step S21 to determine whether auxiliary images constituting training data for machine learning have been generated. If the processing for all pairs is complete (Yes), the control unit 103 terminates the procedure in Figure 2A; otherwise, it returns to step S22 and performs the processing in steps S22 and S23 for new pairs.

[0037] The procedure shown in Figure 2A generates training data for machine learning to create the feature detection model 108. Furthermore, performing the procedure in Figure 2A on multiple different users generates training data that takes into account a wider variety of user facial features. Additionally, by rotating each user's face to obtain primary and secondary images from multiple angles, or by having each user change their facial expression to obtain primary and secondary images with different expressions, it becomes possible to perform machine learning to detect feature regions while tracking the user's facial movements.

[0038] The procedure in Figure 2B is for generating a feature detection model 108 using training data, and is executed by the control unit 103 of the information processing device 10 in response to operator input. Alternatively, the procedure in Figure 2B may be executed by another server device or the like using the training data generated in the procedure in Figure 2A. In that case, the following explanation will be executed by the control unit of the other server device or the like.

[0039] In step S25, the control unit 103 acquires training data. For example, the control unit 103 reads and acquires training data from the storage unit 102.

[0040] In step S26, the control unit 103 performs machine learning using training data to generate a feature detection model 108. Since the training data is data in which feature regions are annotated on auxiliary images, a feature detection model 108 that has already learned the feature regions in the auxiliary images is generated by performing machine learning on this training data. The generated feature detection model 108 is stored in the storage unit 102. Alternatively, the feature detection model 108 generated by the information processing device 10 may be stored in another server device or the like.

[0041] The feature detection model 108 is generated by following the procedure shown in Figure 2B.

[0042] [Operation of Information Processing Device 10 - 2] Figure 4 is a flowchart illustrating an example of the operation of the information processing device 10.

[0043] The procedure shown in Figure 4 is the procedure by which the information processing device 10 controls the projection device 13 using the feature detection model 108 to perform a makeup simulation, and is executed by the control unit 103 of the information processing device 10 in response to operator input. Each step in Figure 4 is pipelined, and two or more steps are executed in parallel by the control unit 103. As a result, the procedure in Figure 4 is executed in a total of, for example, a few milliseconds. Note that the procedures in Figures 2A, 2B and 4 may be executed by one information processing device or by different information processing devices.

[0044] In step S40, the control unit 103 acquires an auxiliary image of the user 15's face. The control unit 103 acquires each of the auxiliary images captured by the imaging device 12-2.

[0045] In step S41, the control unit 103 detects feature regions from the auxiliary image of the user 15's face. The control unit 103 inputs the auxiliary image to the feature detection model 108 and obtains an auxiliary image as output in which landmark points representing feature regions are superimposed.

[0046] In step S42, the control unit 103 determines the target area. The target area is the area onto which the projection device 13 projects colored light. The control unit 103 extracts the target area by arbitrary image processing, including, for example, pattern matching for landmark points. Here, pattern matching includes image processing such as deforming a 3D model of the face that represents the skeleton, facial expressions, etc., to fit the feature area and thereby identifying the target area in the 3D model. Figure 5 shows examples of target areas on the user's face 33, including the eye area 30, cheeks 31, and lips 32. The control unit 103 extracts such target areas and derives the spatial coordinates of each target area.

[0047] In step S43, the control unit 103 acquires cosmetic film information. The cosmetic film information includes the type of cosmetic film to be applied to the target area and information on the color of the cosmetic film. The types of cosmetic films include foundation, eyeshadow, blush, lipstick, etc. The information on the color of the cosmetic film includes gradation values ​​of an arbitrary color system that represent the color of the cosmetic film. The information on the color of the cosmetic film may also include information on the reflectance spectrum, such as the spectral reflectance, diffusion coefficient, and absorption coefficient of the cosmetic film. The cosmetic film information is stored in the storage unit 102, for example, by the user 15 or operator inputting the desired cosmetic film information into the information processing device 10 in advance, and the control unit 103 can acquire the cosmetic film information from the storage unit 102.

[0048] In step S44, the control unit 103 determines the color of the colored light to be projected onto the target area. Based on the cosmetic film information, the control unit 103 determines the color of the colored light to represent the cosmetic agent to be applied to the target area. The control unit 103 may determine the color of the colored light to be projected onto the target area by any method, for example, based on information such as the reflection spectrum of the target area, the reflection spectrum of the cosmetic film, the ambient light spectrum, and the projection light spectrum, which are stored in the storage unit 102 beforehand. The control unit 103 may also retrieve and use the color of the colored light determined in a previous processing cycle from the storage unit 102.

[0049] In step S45, the control unit 103 sends an instruction to the projection device 13 to project colored light. The instruction includes information specifying the position of the target area and the color of the colored light. Based on the spatial coordinates of the projection device 13 and the spatial coordinates of the target area, which are stored in the memory unit 102 beforehand, the control unit 103 identifies the light sources for projecting colored light onto the target area and generates information specifying the intensity of each light source.

[0050] When the projection device 13 projects colored light of the adjusted color onto the target area in response to instructions from the information processing device 10, the target area takes on the target color, and makeup is simulated on the user's face 15. The user 15 can, for example, see the makeup simulation on their own face by looking at their reflection in a mirror.

[0051] The control unit 103 may, for example, acquire the main image from the imaging device 12-1 in step S40 and display it on the output unit 106's display to present it to the user 15. In this way, the main image obtained by capturing the user 15's face, onto which colored light was projected in previous processing cycles, can be displayed. This allows the user 15 to visually confirm a simulation of makeup on their own face.

[0052] Steps S43 to S45 may be performed for each target area. If the control unit 103 detects multiple target areas in step S42 and the cosmetic film information acquired in step S43 targets different types of target areas, steps S43 to S45 may be performed for each target area. Since the reflection spectrum differs depending on the type of target area, it becomes possible to determine the color of colored light with greater accuracy.

[0053] This embodiment also includes the case in which the information processing device 10 operates in a standalone state without being connected to the network 11 during part of the procedure shown in Figure 4.

[0054] According to the procedure described above, the information processing device 10 can detect feature regions in the auxiliary image, which is less affected by colored light than the main image, with an accuracy close to that of feature region detection in the main image when no projected light is applied. Therefore, since the target region can be accurately set based on the detected feature regions, it becomes possible to improve the accuracy of projecting colored light onto the target region of the object.

[0055] When the imaging device 12-2 is configured to perform imaging with light in a specific wavelength range of the visible light region, the control unit 103 can send an instruction to the projection device 13 to selectively reduce the intensity of the light in the wavelength range used for imaging among the visible light projected by the projection device 13. Preferably, the wavelength range of the light used for imaging by the imaging device 12-2 is different from the wavelength range of the colored light irradiated onto the target area. In this case, by weakening the intensity of the light of that color, the degree to which light different from the makeup color is mixed into the makeup expression on the user's face 15 can be reduced, making it possible to accurately simulate the makeup expression.

[0056] The above description showed an example where the target of the makeup simulation was a user. However, the target object may be something other than a person's face, such as hair or clothing, or it may be an animal other than a person, such as a pet. Furthermore, this embodiment is also applicable when projection mapping is performed as part of a show or other performance other than a makeup simulation. Alternatively, this embodiment is also applicable when projection mapping is performed on part or all of an inanimate object such as a vehicle or mechanical structure.

[0057] In the above, the processing and control program that defines the operation of the information processing device 10 may be stored on a cloud server or the like and downloaded to the information processing device 10 via the network 11, or it may be stored on a computer-readable non-transient recording and storage medium and read by the information processing device 10 from the medium.

[0058] As described above, embodiments have been explained based on various drawings and examples, but it should be noted that those skilled in the art will find it easy to make various modifications and alterations based on this disclosure. Therefore, it should be noted that these modifications and alterations are within the scope of this disclosure. For example, the functions, etc., included in each means, each step, etc., can be rearranged in a logically consistent manner, and multiple means, steps, etc., can be combined into one or divided. [Explanation of Symbols]

[0059] 10: Information Processing Devices 11: Network 12-1, 12-2: Imaging device 13: Projection device 14: Lighting equipment 101: Communications Department 102: Storage section 103: Control Unit 105: Input section 106: Output section

Claims

1. Communications Department and, The system includes a control unit that communicates with an imaging device via the aforementioned communication unit, The control unit applies a predetermined process to the first type of image obtained by imaging an object with light in different wavelength regions to detect the feature regions of the object, and then applies these feature regions to the second type of image obtained by imaging the object with light in different wavelength regions. Using a model that has learned the feature regions in the second type of image obtained by imaging the object, the control unit detects the feature regions of another object from the second type of image of the object. Information processing device.

2. In claim 1, The control unit sends an instruction to the projection device to project the first light, which is included in the wavelength range of light when the first type of image is captured, onto a target area based on the characteristic area of ​​the object. Information processing device.

3. In claim 1, The light when the first type of image is captured has wavelengths in the visible light region. The light used when the second type of image is captured has wavelengths in the near-infrared or ultraviolet region. Information processing device.

4. Communications Department and, It has a control unit that sends and receives information via the aforementioned communication unit, The control unit acquires two types of captured images obtained by imaging an object with light in different wavelength regions from the imaging device via the communication unit, detects characteristic parts of the object from the first type of captured image, applies these characteristic parts to the second type of captured image, and generates training data for machine learning the characteristic parts in the second type of captured image. Information processing device.

5. In claim 4, The control unit generates a model for detecting the feature region from the second type of captured image by machine learning using the training data. Information processing device.

6. In claim 4, The first and second types of captured images are each captured by a coaxial pair of imaging devices. Information processing device.

7. In claim 6, The first and second types of captured images are captured simultaneously by the coaxial pair of imaging devices, Information processing device.

8. A system comprising an information processing device and an imaging device, The imaging device sends two types of images obtained by imaging an object with light in different wavelength regions to the information processing device. The information processing device applies the feature portion of the object detected by performing a predetermined process on the first type of captured image to the second type of captured image, and uses a model that has learned the feature portion in the second type of captured image of another object to detect the feature portion of that other object from the second type of captured image of that other object. system.

9. In claim 8, The information processing device sends an instruction to the projection device to project the first light, which is included in the wavelength range of light at the time the first type of image was captured, onto a target area based on the characteristic area of ​​the object. system.

10. In claim 8, The light when the first type of image is captured has wavelengths in the visible light region. The light used when the second type of image is captured has wavelengths in the near-infrared or ultraviolet region. system.

11. A system comprising an information processing device and an imaging device, The imaging device sends two types of images obtained by imaging an object with light in different wavelength regions to the information processing device. The information processing device detects characteristic parts of the object from a first type of captured image, applies the characteristic parts to a second type of captured image, and generates training data for machine learning the characteristic parts in the second type of captured image. system.

12. In claim 11, The information processing device generates a model for detecting the feature portion from the second type of captured image by machine learning using the training data. system.

13. In claim 11, The first and second types of captured images are each captured by a coaxial pair of imaging devices. system.

14. In claim 13, The first and second types of captured images are captured simultaneously by the coaxial pair of imaging devices, system.

15. A method for operating a system having an information processing device and an imaging device, The imaging device sends two types of images obtained by imaging an object with light in different wavelength regions to the information processing device. The information processing device applies the feature portion of the object detected by performing a predetermined process on the first type of captured image to the second type of captured image, and uses a model that has learned the feature portion in the second type of captured image of another object to detect the feature portion of that other object from the second type of captured image of that other object. How the system works.

16. In claim 15, The information processing device sends an instruction to the projection device to project the first light, which is included in the wavelength range of light at the time the first type of image was captured, onto a target area based on the characteristic area of ​​the object. How the system works.

17. In claim 15, The light when the first type of image is captured has wavelengths in the visible light region. The light used when the second type of image is captured has wavelengths in the near-infrared or ultraviolet region. How the system works.

18. A method for operating a system having an information processing device and an imaging device, The imaging device sends two types of images obtained by imaging an object with light in different wavelength regions to the information processing device. The information processing device detects characteristic parts of the object from a first type of captured image, applies the characteristic parts to a second type of captured image, and generates training data for machine learning the characteristic parts in the second type of captured image. How the system works.

19. In claim 18, The information processing device generates a model for detecting the feature portion from the second type of captured image by machine learning using the training data. How the system works.

20. In claim 18, The first and second types of captured images are each captured by a coaxial pair of imaging devices. How the system works.

21. In claim 20, The first and second types of captured images are captured simultaneously by the coaxial pair of imaging devices, How the system works.

Images