Image processing device, model generation device, skin state measurement system, image processing method, model generation method, determination method, and train data creation method

Abstract

The image processing apparatus includes a model storage unit that stores a learned model, the learning completion model is obtained by using a learning captured image obtained by imaging the skin of a person irradiated with visible light and a positive solution captured image obtained by imaging the skin of a person irradiated with ultraviolet light to generate fluorescence caused by porphyrin. A machine learning unit that performs machine learning so as to generate an image of the human skin in which fluorescence caused by porphyrin is generated on the basis of a captured image obtained by capturing the human skin irradiated with visible light; and an output image acquisition unit that acquires, using the learned model, an output image on the basis of a captured subject image obtained by capturing the skin of the subject irradiated with visible light, the output image being an image of the skin of a person in which fluorescence caused by porphyrin is generated.

Description

Technical Field

[0001] The present invention relates to an image processing device, a model generating device, a skin condition measuring system, an image processing method, a model generating method, a determination method and a training data producing method.

[0002] This application claims priority to Japanese Patent Application No. 2022-127815 filed in Japan on August 10, 2022, and the contents are incorporated herein by reference. Background Art

[0003] In the past, the presence of Propionibacterium acnes (Propionibacterium acnes) which is the cause of acne, and the condition of the skin were investigated by detecting porphyrin on the skin generated as a metabolite. As a method for detecting porphyrin on the skin, for example, a porphyrin detection method described in Patent Document 1 is known. In the porphyrin detection method described in Patent Document 1, weak ultraviolet rays including visible light are irradiated onto the skin surface, and the skin surface is photographed using a CCD (charge coupled device) color camera having a cooling mechanism, so that a color image signal of the skin surface output from the CCD color camera is projected onto a television monitor as a still image, and the still image is observed.

[0004] Prior art literature

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent Application Laid-Open No. 5-103771 Summary of the invention

[0007] Problem that the invention aims to solve

[0008] However, the porphyrin detection method described in Patent Document 1 requires irradiation of the subject's skin surface with ultraviolet rays, so skin damage caused by ultraviolet irradiation is a problem, and a method for detecting porphyrins without irradiation with ultraviolet rays is desired.

[0009] The present invention was completed in consideration of such a situation, and its purpose is to provide an image processing device, a model generating device, a skin condition measuring system, an image processing method and a model generating method that can detect porphyrin on the skin of a subject even without irradiating the subject with ultraviolet rays.

[0010] Solutions for solving problems

[0011] One embodiment of the present invention is an image processing device comprising: a model storage unit that stores a learned model, wherein the learned model is obtained by machine learning in a manner of generating an image of human skin that produces fluorescence due to porphyrin based on a learning video image obtained by photographing the skin of a person irradiated with visible light and a correct video image obtained by photographing the skin of a person that has produced fluorescence due to porphyrin by irradiating ultraviolet light; a subject video image acquisition unit that acquires a subject video image obtained by photographing the skin of the subject that has been irradiated with visible light; and an output image acquisition unit that uses the learned model to acquire an output image based on the subject video image, wherein the output image is an image of human skin that has produced fluorescence due to porphyrin.

[0012] One embodiment of the present invention is that in the above-mentioned image processing device, in the machine learning of the learned model, a compressed learning camera image obtained by compressing a part of the learning camera image and a compressed correct answer camera image obtained by compressing a part of the correct answer camera image are used.

[0013] According to one embodiment of the present invention, in the above-mentioned image processing device, the skin is facial skin, and the part is a forehead, a nose, or a periphery of a mouth.

[0014] According to one aspect of the present invention, in the above-mentioned image processing device, an enhanced correct answer captured image in which a red component is enhanced with respect to the correct answer captured image is used in the machine learning of the learned model.

[0015] One embodiment of the present invention is that, in the above-mentioned image processing device, the correct answer camera image is an RGB image composed of an R image, a G image and a B image, and the enhanced correct answer camera image is an RGB image generated by adding a differential image obtained by subtracting the G image from the R image to the RGB image.

[0016] One embodiment of the present invention is that, in the above-mentioned image processing device, the learned model is composed of layers of an encoder part and layers of a decoder part.

[0017] One embodiment of the present invention is that, in the above-mentioned image processing device, the learned model has a U-Net structure in which the layers of the encoder part and the layers of the decoder part are symmetrically structured and connected by skip connections.

[0018] One embodiment of the present invention is a model generating device comprising: a learning camera image acquiring unit, which acquires a learning camera image obtained by photographing the skin of a person irradiated with visible light; a correct camera image acquiring unit, which acquires a correct camera image obtained by photographing the skin of a person irradiated with ultraviolet light and producing fluorescence caused by porphyrin; and a model generating unit, which uses the learning camera image and the correct camera image to perform machine learning of the model in a manner of generating an image of the skin of a person producing fluorescence caused by porphyrin based on the camera image obtained by photographing the skin of the person irradiated with visible light, thereby generating a learned model.

[0019] One embodiment of the present invention is that, in the above-mentioned model generating device, the model generating unit generates a compressed learning camera image obtained by compressing a part of the learning camera image and a compressed correct answer camera image obtained by compressing a part of the correct answer camera image, and uses the compressed learning camera image and the compressed correct answer camera image for machine learning of the model.

[0020] According to one embodiment of the present invention, in the above-mentioned model generating device, the skin is facial skin, and the part is the forehead, nose, or the periphery of the mouth.

[0021] According to one embodiment of the present invention, in the above-mentioned model generation device, the model generation unit generates an enhanced correct answer camera image in which a red component is enhanced for the correct answer camera image, and uses the enhanced correct answer camera image for machine learning of the model.

[0022] One embodiment of the present invention is that, in the above-mentioned model generating device, the correct answer camera image is an RGB image composed of an R image, a G image and a B image, and the enhanced correct answer camera image is an RGB image generated by adding a differential image obtained by subtracting the G image from the R image to the RGB image.

[0023] One embodiment of the present invention is that, in the above-mentioned model generation device, the model is composed of layers of an encoder part and layers of a decoder part.

[0024] One embodiment of the present invention is that, in the above-mentioned model generation device, the model has a U-Net structure in which the layers of the encoder part and the layers of the decoder part are symmetrically structured and connected by skip connections.

[0025] One aspect of the present invention is a skin condition measurement system including: the above-mentioned image processing device; and the above-mentioned model generation device.

[0026] One embodiment of the present invention is an image processing method, comprising the following steps: obtaining a camera image of a subject obtained by photographing the skin of the subject irradiated with visible light; and using a learned model to obtain an output image based on the camera image of the subject, wherein the output image is an image of human skin that has produced fluorescence due to porphyrin, wherein the learned model is a model obtained by machine learning in a manner of generating an image of human skin that has produced fluorescence due to porphyrin based on a camera image obtained by photographing the skin of the subject irradiated with visible light, using a learning camera image obtained by photographing the skin of the subject irradiated with visible light and a correct camera image obtained by photographing the skin of the subject that has produced fluorescence due to porphyrin by being irradiated with ultraviolet rays.

[0027] One embodiment of the present invention is that in the above-mentioned image processing method, in the machine learning of the learned model, a compressed learning camera image obtained by compressing a part of the learning camera image and a compressed correct answer camera image obtained by compressing a part of the correct answer camera image are used.

[0028] According to one embodiment of the present invention, in the above-mentioned image processing method, the skin is facial skin, and the part is the forehead, nose, or the periphery of the mouth.

[0029] One aspect of the present invention is that, in the above-mentioned image processing method, an enhanced correct answer captured image in which a red component is enhanced with respect to the correct answer captured image is used in the machine learning of the learned model.

[0030] One embodiment of the present invention is that, in the above-mentioned image processing method, the correct answer camera image is an RGB image composed of an R image, a G image and a B image, and the enhanced correct answer camera image is an RGB image generated by adding a differential image obtained by subtracting the G image from the R image to the RGB image.

[0031] One embodiment of the present invention is that, in the above-mentioned image processing method, the learned model is composed of layers of an encoder part and layers of a decoder part.

[0032] One embodiment of the present invention is that, in the above-mentioned image processing method, the learned model has a U-Net structure in which the layers of the encoder part and the layers of the decoder part are symmetrically structured and connected by skip connections.

[0033] One embodiment of the present invention is a model generation method, comprising the following steps: obtaining a learning camera image obtained by photographing the skin of a person irradiated with visible light; obtaining a correct camera image obtained by photographing the skin of a person who has been irradiated with ultraviolet light and has produced fluorescence caused by porphyrin; and a model generation step, using the learning camera image and the correct camera image to perform machine learning of the model in a manner such that an image of the skin of a person who has produced fluorescence caused by porphyrin is generated based on the camera image obtained by photographing the skin of a person irradiated with visible light, to generate a learned model.

[0034] One embodiment of the present invention is that in the above-mentioned model generation method, in the model generation step, a compressed learning camera image obtained by compressing a part of the learning camera image and a compressed correct answer camera image obtained by compressing a part of the correct answer camera image are generated, and the compressed learning camera image and the compressed correct answer camera image are used for machine learning of the model.

[0035] One embodiment of the present invention is that, in the above-mentioned model generation method, the skin is facial skin, and the part is the forehead, nose, or the periphery of the mouth.

[0036] One embodiment of the present invention is that in the above-mentioned model generation method, in the model generation step, an enhanced correct answer camera image in which the red component is enhanced is generated for the correct answer camera image, and the enhanced correct answer camera image is used for machine learning of the model.

[0037] One embodiment of the present invention is that, in the above-mentioned model generation method, the correct answer camera image is an RGB image composed of an R image, a G image and a B image, and the enhanced correct answer camera image is an RGB image generated by adding a differential image obtained by subtracting the G image from the R image to the RGB image.

[0038] One embodiment of the present invention is that, in the above-mentioned model generation method, the model is composed of layers of an encoder part and layers of a decoder part.

[0039] One embodiment of the present invention is that, in the above-mentioned model generation method, the model has a U-Net structure in which the layers of the encoder part and the layers of the decoder part are symmetrically structured and connected by skip connections.

[0040] According to one aspect of the present invention, in the above-mentioned image processing method, in the step of acquiring the learning captured image and the correct answer captured image, at least one mapping reference point is set.

[0041] One aspect of the present invention is a determination method for determining acne susceptibility by comparing the output image acquired by the above-mentioned image processing device with a standard image, wherein the standard image is an image of the skin of a normal person who does not develop acne.

[0042] One embodiment of the present invention is a determination method, which is an evaluation method for evaluating the effect of skin care before and after the subject performs skin care, wherein a first acquisition step is to acquire the output image acquired by the above-mentioned image processing device based on the camera image of the subject before the subject performs skin care; a second acquisition step is to acquire the output image acquired by the above-mentioned image processing device based on the camera image of the subject after the subject performs skin care; and the output image acquired by the first acquisition step is compared with the output image acquired by the second acquisition step to determine the degree of improvement obtained by skin care.

[0043] One embodiment of the present invention is a method for producing training data, comprising: a learning camera image acquisition step, in which a learning camera image is acquired, wherein the learning camera image is an image obtained by photographing the skin of a person irradiated with visible light and a portion having a reference point; a correct answer camera image acquisition step, in which a correct answer camera image is acquired, wherein the correct answer camera image is an image obtained by photographing the same subject as the learning camera image and the skin of a person who has been irradiated with ultraviolet rays and has produced fluorescence caused by porphyrin using a camera device different from a camera device that photographed the learning camera image; a first extraction step, in which a portion of the learning camera image is extracted based on the reference point; a second extraction step, in which a portion of the correct answer camera image is extracted based on the reference point; and a training data production step, in which training data is produced by storing images extracted from the learning camera image and the correct answer camera image in correspondence with each other.

[0044] One embodiment of the present invention is that the above-mentioned training data production method also includes an image processing step, in which an image matching the learning camera image is generated by performing image processing on the correct camera image, and in the second extraction step, a partial area based on the reference point is extracted from the correct camera image after the image processing is performed by the image processing step.

[0045] Effects of the Invention

[0046] According to the present invention, an effect of being able to detect porphyrin on the skin of a subject is achieved without irradiating the subject with ultraviolet rays. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 This is a block diagram showing a configuration example of a skin condition measurement system according to one embodiment.

[0048] Figure 2 This is a structural diagram showing an example of the outline of a model according to one embodiment.

[0049] Figure 3 This is a flowchart showing an example of a procedure of a model generation method according to one embodiment.

[0050] Figure 4 It is a diagram showing an example of a cutout portion of a learning captured image and a correct answer captured image according to one embodiment.

[0051] Figure 5 This is an example of an image used to illustrate pre-processing 2 of a correct image captured according to one embodiment.

[0052] Figure 6 This is a flowchart showing an example of a procedure of an image processing method according to one embodiment.

[0053] Figure 7 This is an image diagram for explaining the difference between the shooting of a learning captured image and a correct answer captured image according to one embodiment.

[0054] Figure 8 This is an image diagram of reference points used for creating training data according to one embodiment.

[0055] Fig. 9 It is a diagram showing an example of pre-processing of a correct answer captured image and cutting out of a learning captured image and a correct answer captured image according to one embodiment. DETAILED DESCRIPTION

[0056] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

[0057] Figure 1 This is a block diagram showing a configuration example of a skin condition measurement system according to one embodiment. Figure 1 The skin condition measurement system shown includes an image processing device 10 and a model generation device 20. The image processing device 10 and the model generation device 20 may transmit and receive data online through communication, or may input and output data offline.

[0058] The subject U is a person who uses the skin condition measurement system and receives skin condition measurement. The subject U uses the camera of the smartphone 30 to photograph the skin of the part to be measured, and uses the smartphone 30 to send the photographed image (subject camera image) A to the image processing device 10. For example, the subject U uses the camera of the smartphone 30 to photograph his or her entire face, or the forehead, nose, and mouth area that he or she is worried about developing acne, and uses the smartphone 30 to send the photographed subject camera image A of the face to the image processing device 10.

[0059] The subject video image A is a video image obtained by photographing the skin of the subject U irradiated with visible light. When photographing the subject video image A, it is sufficient to irradiate the skin of the subject U with visible light, and it is not necessary to irradiate the skin of the subject U with ultraviolet light when photographing the subject video image A. The light source irradiating the visible light includes a wavelength of 400nm to 770nm. Since it is not necessary to irradiate ultraviolet light, in order to reduce damage to the skin of the subject U, the ultraviolet (UV) intensity of the light source is preferably as weak as possible, for example, preferably below the detection limit of a UV meter. As the light source irradiating the visible light, natural light such as sunlight and moonlight, incandescent lamps, fluorescent lamps, xenon lamps, LED bulbs and other artificial lights are listed. The light source may also include ultraviolet light, but artificial light with weak UV intensity is preferred, and xenon lamps and LED bulbs are more preferred among artificial lights.

[0060] In addition, in the present embodiment, the subject U utilizes a smartphone 30 with a camera (camera), but is not limited to this. For example, a mobile terminal device such as a tablet computer (tablet PC) with a camera can also be utilized. In addition, the camera image captured by a digital camera can also be sent to the image processing device 10 by a communication terminal such as a smartphone. In the case of shooting with the camera of a smartphone 30, the porphyrin detection accuracy can be further improved by shooting in a manner that can perform color correction and position correction. As a color correction method, for example, a color chart is mapped when shooting the subject U. As a position correction method, for example, ArUco marks or stickers that become marks are used as mapping reference points when shooting the subject U and projection transformation is performed by image editing software. In addition, in the case of utilizing a mobile terminal device such as a smartphone 30, a tablet PC, etc. with a camera, the image processing device 10 described later can also be integrated into the mobile terminal device. In this case, there is no need to send the captured face image A of the subject to the image processing device 10 using the smartphone 30 , and there is no need to send the output image B described later from the image processing device 10 to the smartphone 30 of the subject U.

[0061] [Image processing device]

[0062] The image processing device 10 includes a model storage unit 11 , a subject captured image acquisition unit 12 , and an output image acquisition unit 13 .

[0063] The image processing device 10 has computer hardware such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and a memory, and the various functions of the image processing device 10 are realized by the CPU executing a computer program stored in the memory. In addition, as the image processing device 10, it can be constructed using a general-purpose computer device, or it can be constructed as a dedicated hardware device. For example, the image processing device 10 can also be constructed using a server computer connected to a communication network such as the Internet. In addition, the various functions of the image processing device 10 can also be realized by cloud computing. In addition, the image processing device 10 can be realized by a single computer, or the functions of the image processing device 10 can be dispersed to multiple computers. In addition, as the image processing device 10, for example, it can also be constructed to open a website using a WWW system.

[0064] The model storage unit 11 stores the learned model MDa. The learned model MDa is provided by the model generation device 20. The learned model MDa is a model obtained by machine learning in a manner of generating an image of human skin that has generated fluorescence due to porphyrin based on a camera image obtained by photographing human skin that has been irradiated with visible light, using a learning camera image obtained by photographing human skin that has been irradiated with visible light and a correct camera image obtained by photographing human skin that has generated fluorescence due to porphyrin by irradiating ultraviolet light.

[0065] The subject captured image acquisition unit 12 acquires the subject captured image A. The subject captured image A is an image obtained by capturing the skin of the subject U irradiated with visible light. The subject captured image acquisition unit 12 receives the subject captured image A transmitted from the smartphone 30 of the subject U via the communication line.

[0066] Furthermore, the subject image acquisition unit 12 may compress the subject image A into a predetermined compression size and convert it into a subject compressed image A'. For example, when the subject image size is "3456×5184" pixels, it is compressed into "256×256" pixels.

[0067] In addition, in order to minimize the blur of the output image caused by the difference in the position of the face due to the movement of the subject U during shooting, it is also possible to cut the image size of "768×768" pixels from the subject's camera image A of "3456×5184" pixels, and compress the image of the cut part of "768×768" pixels to "256×256" pixels. Regarding the cut part, the camera image of the entire face of the subject can also be cut at equal intervals, and it is also preferred to cut the forehead, nose or mouth area where the probability of the presence of porphyrin is high. The "256×256" pixel image is the compressed camera image A' of the subject.

[0068] In addition, the numerical values ​​of the image sizes such as "3456×5184" pixels, "768×768" pixels, and "256×256" pixels described above are examples of image sizes, and are not limited thereto.

[0069] The output image acquisition unit 13 acquires an output image B or an output image B′, which is an image of human skin generating fluorescence due to porphyrin, from the subject captured image A or the subject compressed captured image A′ using the learned model MDa.

[0070] When the learned model MDa receives the subject camera image A as input, it outputs the output image B. The output image B is an image of the skin of the subject U where the fluorescence due to porphyrin is generated, which is inferred by the learned model MDa based on the subject camera image A of the subject U. For example, the output image B is an image of the face of the subject U where the fluorescence due to porphyrin is generated, which is inferred by the learned model MDa based on the subject camera image A of the face of the subject U.

[0071] When the learned model MDa is input with the compressed camera image A' of the subject, it outputs the output image B'. The output image B' is an image of the skin of the subject U where the fluorescence caused by porphyrin is generated, which is inferred by the learned model MDa based on the compressed camera image A' of the subject U. For example, the output image B' is an image of the face of the subject U where the fluorescence caused by porphyrin is generated, which is inferred by the learned model MDa based on the compressed camera image A' of the face of the subject U. Since the output image B' is a compressed image, the output image acquisition unit 13 restores the output image B' to the output image B by expanding it to the same size as the camera image A of the subject. When the camera image of the entire face of the subject is cut at equal intervals and compressed respectively, the output image acquisition unit 13 restores the output image B' to the output image B, and obtains the output image B of the same size as the camera image A of the subject by connecting the output images B.

[0072] The output image acquisition unit 13 transmits the output image B to the smartphone 30 of the subject U via the communication line.

[0073] The subject U receives the output image B sent from the image processing device 10 via the communication line through the smartphone 30. The subject U can visually confirm the state of porphyrin on his / her skin using the output image B by displaying the output image B received by the smartphone 30 on the display screen of the smartphone 30. For example, the subject U sends a photographed image A of his / her face that he / she is worried about developing acne to the image processing device 10 through the smartphone 30, and in response, the output image B received by the smartphone 30 from the image processing device 10 is displayed on the display screen of the smartphone 30, thereby visually confirming the state of porphyrin on his / her face that he / she is worried about developing acne using the output image B.

[0074] [Model generation device]

[0075] The model generation device 20 includes a learning captured image acquisition unit 21 , a correct answer captured image acquisition unit 22 , a model generation unit 23 , and a model output unit 24 .

[0076] The model generation device 20 has computer hardware such as a CPU, a GPU, and a memory, and the CPU executes a computer program stored in the memory, thereby realizing the various functions of the model generation device 20. In addition, as the model generation device 20, it can be constructed using a general-purpose computer device, or it can be constructed as a dedicated hardware device. For example, the model generation device 20 can also be constructed using a server computer connected to a communication network such as the Internet. In addition, the various functions of the model generation device 20 can also be realized by cloud computing. In addition, the model generation device 20 can be realized by a single computer, or the functions of the model generation device 20 can be distributed to multiple computers for realization.

[0077] The learning camera image acquisition unit 21 acquires the learning camera image C. The correct answer camera image acquisition unit 22 acquires the correct answer camera image D. In the learning camera image C and the correct answer camera image D, the person (selected subject) selected to obtain the learning data is the acquisition object. The learning camera image C is a camera image obtained by photographing the skin of the selected subject irradiated with visible light. For example, the learning camera image C is a camera image obtained by photographing the face of the selected subject irradiated with visible light. The correct answer camera image D is an image obtained by photographing the skin of the selected subject irradiated with ultraviolet rays to generate fluorescence caused by porphyrin. For example, the correct answer camera image D is an image obtained by photographing the face of the selected subject irradiated with ultraviolet rays to generate fluorescence caused by porphyrin.

[0078] Since ultraviolet rays will be irradiated to the skin of the selected subject when shooting the corrected camera image D, it is preferable to obtain the consent of the selected subject regarding the irradiation of ultraviolet rays and the effects caused by the irradiation of ultraviolet rays. The light source for irradiating ultraviolet rays includes a wavelength of 100nm to 400nm. As the light source for irradiating ultraviolet rays, for example, a mercury lamp, an ultraviolet LED lamp, a black light lamp, etc. are cited. In addition, it is preferred that the viewing angle and position of the learning camera image C and the corrected camera image D are the same. As a camera for acquiring the learning camera image C and the corrected camera image D with the same viewing angle and position, for example, "VISIA (registered trademark), manufactured by Canfield Scientific" is cited. In addition, as a camera for acquiring the learning camera image C, a smartphone can also be used instead of the above-mentioned VISIA. In this case, in order to obtain the learning camera image C and the corrected camera image D with the same viewing angle and position, at least one, preferably 4 to 8 reference points of the same place can also be specified in the camera screen.

[0079] The model generation unit 23 uses the learning camera image C and the correct camera image D to perform machine learning of the model MD in a manner to generate an image of human skin that produces fluorescence due to porphyrin based on a camera image obtained by photographing the skin of a person irradiated with visible light, thereby generating a learned model MDa. When the learned model MDa is input with the subject camera image A or the subject compressed camera image A', it outputs the output image B or the output image B'.

[0080] The model output unit 24 outputs the learned model MDa generated by the model generation unit 23. The learned model MDa can be output by, for example, writing to a computer-readable recording medium, transmitting data by communication, etc. The learned model MDa output by the model output unit 24 is provided to the image processing device 10. In the image processing device 10, the model storage unit 11 stores the provided learned model MDa.

[0081] Furthermore, the image processing device 10 and the model generating device 20 may be implemented using the same information processing device, or may be implemented using different information processing devices.

[0082] Figure 2 This is a model configuration diagram showing an example of the schematic configuration of the model MD according to the present embodiment. Figure 2The model MD shown has a U-Net structure in which the layers of the encoder 110 and the layers of the decoder 120 are symmetrically structured, and each layer is connected by a jump connection 130. The encoder 110 encodes the input image Pin in stages through each layer. The decoder 120 decodes the encoding result of the encoder 110 in stages through each layer. The decoding result of the decoder 120 is output from the model MD as an output image Pout. Each layer of the encoder 110 and each corresponding layer of the decoder 120 are connected by a jump connection 130.

[0083] In addition, in this embodiment, using Figure 2 The U-Net model is used as the model MD, but it is not limited to this. For example, the model MD can also be Figure 2 The model does not include the skip connection 130. In addition, in this embodiment, GAN (Generative Adversarial Networks) can also be applied.

[0084] Next, the model generation method and image processing method according to the present embodiment will be described.

[0085] [Model generation method]

[0086] Reference Figure 3 The model generation method involved in this embodiment is described. Figure 3 This is a flowchart showing an example of the procedure (model learning phase S100 ) of the model generation method according to the present embodiment.

[0087] The model learning phase S100 is a phase executed by the model generation device 20 to perform machine learning of the model MD using the learning captured images C and the correct answer captured images D to generate a learned model MDa.

[0088] (Step S101) The learning camera image acquisition unit 21 acquires a learning camera image C for a plurality of selected subjects. The correct answer camera image acquisition unit 22 acquires a correct answer camera image D for the plurality of selected subjects. The learning camera image C and the correct answer camera image D are RGB images obtained by color-photographing the faces of the selected subjects. The RGB image is composed of an image of a red component (R image), an image of a green component (G image), and an image of a blue component (B image). The model generation device 20 maintains the learning camera image C and the correct answer camera image D of the same selected subject in association. In the learning camera image C and the correct answer camera image D of the same selected subject, the position of the face appearing in the camera image is consistent.

[0089] In addition, when the learning camera image C is acquired by the camera of the smartphone, position correction is performed to make the position of the face consistent. As a method of position correction, the above-mentioned method is cited. The following is a detailed description.

[0090] As a position correction method, for example, an image irradiated with visible light and photographed by a smartphone, and an image photographed in a state where fluorescence caused by porphyrin is generated by irradiating ultraviolet rays are projected for position alignment. As a specific process, at least one mapping reference point, preferably more than 4 and less than 8, of the same place is specified between the two images, and the position alignment is achieved by matching with the reference point of one party. As the above-mentioned designated mapping reference points, moles, pseudo moles using magic pens, stickers, etc. are cited. As the size of the reference point, from the perspective of more consistent position offset in pixels, smaller is more preferred. As a specific example, it is preferred that a sticker with extremely small holes pre-opened is pasted on the cheeks surrounding the forehead, nose, and mouth where acne is mainly formed, and after shooting with a smartphone and VISIA, the hole in the sticker is used as a reference point for projection transformation, thereby performing position alignment. For example, when the pixel value correlation coefficient between the generated image and the correct image when the position alignment is performed visually is set to 1, the pixel value correlation coefficient when marking is made by painting on the sticker with a paint pen is "1.10", the pixel value correlation coefficient when marking is made by making a hole in the sticker with a needle (thickness of 0.5mm) is "1.23", and the pixel value correlation coefficient when marking is made by making a hole in the sticker with an extremely fine needle (thickness of 0.4mm) is "3.20".

[0091] (Step S102) The model generation unit 23 performs preprocessing on the learning captured image C and the correct answer captured image D. Next, the preprocessing on the learning captured image C and the correct answer captured image D will be described.

[0092] (Preprocessing 1)

[0093] In the preprocessing step 1, the model generation unit 23 generates a compressed learning camera image obtained by compressing a part of the learning camera image C and a compressed correct camera image obtained by compressing a part of the correct camera image D as images for machine learning of the model MD. Figure 4 , respectively, portions (cropped portions) cut out from the learning camera image C and the correct answer camera image D are shown. Figure 4In the example, the cutout portion is cut out from the forehead, nose and mouth area of ​​the face. This is because the forehead, nose and mouth area are areas where acne is particularly prone to form, that is, areas where Proteobacillus acnes, which is the cause of acne, is easily produced and a large amount of porphyrin is easily detected. In addition, the cutout portion can also be cut out from at least any one of the forehead, nose and mouth area of ​​the face. In addition, the cutout portion can be cut out from other parts of the face (such as cheeks, etc.) except the forehead, nose and mouth area of ​​the face, or when the learning camera image C and the correct camera image D are camera images of the entire face of the selected subject, each image can be cut out at equal intervals.

[0094] The model generation unit 23 compresses the image of the cutout portion to a predetermined compression size. For example, when the image size of the original learning camera image C and the correct answer camera image D is "3456×5184" pixels, the image size of the cutout portion is set to "768×768" pixels, and the image of the cutout portion of "768×768" pixels is compressed to "256×256" pixels. The image of "256×256" pixels obtained by compressing the image of the cutout portion of "768×768" pixels cut out from the original learning camera image C is a compressed learning camera image. The image of "256×256" pixels obtained by compressing the image of the cutout portion of "768×768" pixels cut out from the original correct answer camera image D is a compressed correct answer camera image. In addition, regarding the image size of the cropped parts of the learning camera image C and the correct camera image D, in order to minimize the blur of the output image caused by the difference in the position of the face due to the movement of the same selected subject during the shooting period of the learning camera image C and the correct camera image D, "768×768" pixels are preferably used.

[0095] In addition, the numerical values ​​of the image sizes such as "3456×5184" pixels, "768×768" pixels, and "256×256" pixels described above are examples of image sizes, and are not limited thereto.

[0096] (Preprocessing 2)

[0097] In preprocessing No. 2, the model generation unit 23 generates an enhanced correct solution camera image in which the red component is emphasized for the correct solution camera image D. In this embodiment, preprocessing No. 2 is performed on the compressed correct solution camera image generated in the above-mentioned preprocessing No. 1. In addition, as a preferred embodiment of preprocessing No. 2, the enhanced correct solution camera image is an RGB image generated by adding a differential image obtained by subtracting the G image from the R image among the R image, G image, and B image constituting the RGB image to the RGB image of the correct solution camera image D.

[0098] Figure 5This is an example of an image for explaining the second pre-processing of the correct solution captured image D according to the present embodiment. Figure 5 (1) is a compressed corrected camera image (RGB image) generated in preprocessing step 1. Figure 5 (2), (3), and (4) are the R image, G image, and B image that constitute the compressed correct image (RGB image). Figure 5 The difference image of Figure (5) is added to the RGB image of Figure (1) to generate the RGB image of Figure (6) (emphasized correct camera image). Figure 5 The RGB image of (6) (emphasized correct camera image) is compared to Figure 5 In the original RGB image (compressed rectified camera image) of (1), the red component is emphasized.

[0099] The above is a description of the pre-processing of the learning captured image C and the correct answer captured image D.

[0100] In the present embodiment, the model generation unit 23 uses the compressed learning camera images and the emphasized correct answer camera images for machine learning of the model MD. In addition, in the present embodiment, the compressed learning camera images and the emphasized correct answer camera images are used for machine learning of the model MD, but the compressed learning camera images and the compressed correct answer camera images can also be used for machine learning of the model MD without preprocessing.

[0101] (Step S103) The model generation unit 23 inputs the compressed correct solution image C' as the input image Pin into the model MD (refer to Figure 2 ). When the model MD receives the compressed correct image (input image Pin), it generates an output image Pout and outputs the output image Pout (refer to Figure 2 ).

[0102] (Step S104) The model generation unit 23 compares the output image Pout output from the model MD with the enhanced correct answer camera image D', and performs feedback control on the model MD in such a way that the difference value of the comparison result becomes smaller. The enhanced correct answer camera image D' is an enhanced correct answer camera image generated from the correct answer camera image D of the same selected subject corresponding to the compressed correct answer camera image C' used as the input image Pin.

[0103] In the present embodiment, as the difference value between the output image Pout and the enhanced correct solution captured image D′, for example, MSE (mean squared error) of the following formula is used.

[0104] [Number 1]

[0105]

[0106] Wherein, N is the number of pixels, i is the pixel number, xi is the i-th pixel value of the emphasized correct solution camera image D', and yi is the i-th pixel value of the output image Pout.

[0107] The model generation unit 23 uses the learning camera images C (compressed correct answer camera images C') and correct answer camera images D (enhanced correct answer camera images D') acquired from a plurality of selected subjects to repeatedly perform machine learning of the model MD until a predetermined termination condition is satisfied. The predetermined termination condition may be a predetermined number of times, or may be that the difference between the output image Pout and the enhanced correct answer camera image D' becomes smaller than a predetermined value.

[0108] The model generation unit 23 delivers the model MD after the machine learning as a learned model MDa to the model output unit 24. The model output unit 24 outputs the learned model MDa.

[0109] It was confirmed through experiments that by using the compressed learning camera image and the compressed correct camera image generated by the above-mentioned preprocessing 1 for machine learning of the model MD, the reproducibility of the part caused by porphyrin fluorescence in the output image Pout generated by the learned model MDa is improved. Specifically, nine subjects were selected, and regarding the subject's camera images, a subject's camera image A' was obtained by cutting the image A of subject A of "3456×5184" pixels with an image size of "768×768" pixels and compressing the image of the cut portion of "768×768" pixels into "256×256" pixels. Without performing preprocessing No. 1, the pixel value correlation coefficient between the output image and the correct camera image was "0.22". On the contrary, when performing preprocessing No. 1, in which an image of "768×768" pixels was cut out from the learning camera image C and the correct camera image D of "3456×5184" pixels and compressed into a compressed learning camera image and a compressed correct camera image of "256×256" pixels.

[0110] In addition, it was confirmed through experiments that by using the emphasized correct answer camera image generated by the above-mentioned preprocessing 2 for machine learning of the model MD, the reproducibility of the part of the fluorescence caused by porphyrin in the output image Pout generated by the learned model MDa was improved. Specifically, 9 subjects were selected, and as for the subject camera image, the subject camera image A' obtained by cutting the image size of "768×768" pixels from the subject camera image A of "3456×5184" pixels and compressing the image of the cut portion of "768×768" pixels to "256×256" pixels was used. When the above-mentioned preprocessing 1 was performed and the preprocessing 2 was not performed, the pixel value correlation coefficient between the output image and the correct answer camera image was "0.35", while in contrast, when the above-mentioned preprocessing 1 and the preprocessing 2 were performed, the pixel value correlation coefficient between the output image and the correct answer camera image was "0.61".

[0111] [Image processing method]

[0112] Reference Figure 6 The image processing method according to this embodiment will be described. Figure 6 1 is a flowchart showing an example of the procedure (skin condition measurement step S200 ) of the image processing method according to the present embodiment.

[0113] The skin condition measurement stage S200 is a stage executed by the image processing device 10 to generate an output image B based on the subject camera image A of the subject U or the subject compressed camera image A' converted from the subject camera image A using the learned model MDa. The learned model MDa generated by the model generation device 20 in the above-mentioned model learning stage S100 is stored in the model storage unit 11 of the image processing device 10.

[0114] (Step S201) The subject U uses, for example, the camera of the smartphone 30 to photograph the skin of the part to be measured (e.g., the face), and uses the smartphone 30 to send the photographed subject image A to the image processing device 10. The subject image acquisition unit 12 receives the subject image A sent from the smartphone 30 of the subject U via a communication line or the like. Here, the subject image acquisition unit 12 may compress the subject image A into a predetermined compression size and convert it into a compressed subject image A'.

[0115] Furthermore, when photographing using the camera of the smartphone 30, the photographing may be performed so that color correction and position correction can be performed. The above-mentioned method is cited as a specific method of color correction and position correction.

[0116] (Step S202) The output image acquisition unit 13 inputs the subject's captured image A or the subject's compressed captured image A' as an input image Pin to the learned model MDa (refer to Figure 2 When the learned model MDa is input with the subject's camera image A or the subject's compressed camera image A' (input image Pin), it generates an output image Pout and outputs the output image Pout (refer to Figure 2 ). The output image acquisition unit 13 acquires the output image Pout output from the learned model MDa as the output image B or the output image B'. The output image B or the output image B' is an image of the skin of the subject U that produces fluorescence due to porphyrin, which is inferred by the learned model MDa based on the subject's photographic image A or the subject's compressed photographic image A' of the subject U. For example, the output image B or the output image B' is an image of the face of the subject U that produces fluorescence due to porphyrin, which is inferred by the learned model MDa based on the subject's photographic image A or the subject's compressed photographic image A' of the face of the subject U. In addition, since the output image B' is a compressed image, the output image acquisition unit 13 expands the output image B' to the same size as the subject's photographic image A and restores it to the output image B.

[0117] The output image acquisition unit 13 sends the output image B to the smartphone 30 of the subject U via the communication line. The subject U receives the output image B sent from the image processing device 10 through the smartphone 30 via the communication line. The subject U can use the output image B to visually confirm the state of porphyrin on his skin (for example, face) by displaying the output image B received by the smartphone 30 on the display screen of the smartphone 30. In addition, when using a mobile terminal device such as a smartphone 30 or a tablet PC that has a camera and is integrated into the image processing device 10, there is no need to use the smartphone 30 to send the captured face of the subject A to the image processing device 10, and to send the output image B from the image processing device 10 to the smartphone 30 of the subject U.

[0118] [Variation of the training data creation method]

[0119] Next, refer to Figures 7 to 9A variation of the training data production method is described below. First, the problem to be solved by this variation is described. For example, in the learning stage, "VISIA (registered trademark), manufactured by Canfield Scientific" is used as a camera for acquiring learning camera images C and correct camera images D. By using such a device, the viewing angle and position of the learning camera image C and the correct camera image D are the same, and no position alignment is required (or the amount of position alignment is very small). However, in the inference stage, it is assumed that the subject U uses a smartphone 30 to take a subject camera image A. In such a case, since the number of pixels of the learning image is greatly different from the number of pixels of the inference image, it is sometimes impossible to obtain a sufficient inference.

[0120] Therefore, in the learning stage, it is also considered to use images captured by the smartphone 30 in the same way as in the inference stage. The camera image C for learning is a camera image obtained by capturing the skin of the selected subject irradiated with visible light, and thus can be easily captured using the smartphone 30. On the other hand, the correct camera image D is an image obtained by capturing the face of the selected subject irradiated with ultraviolet light to produce fluorescence caused by porphyrin, and thus cannot be easily captured using the smartphone 30. Therefore, it is considered to set the camera image C for learning to an image captured by the smartphone 30, and set the correct camera image D to an image captured using a device such as "VISIA (registered trademark), manufactured by Canfield Scientific", and align these images to create training data.

[0121] Figure 7 This is an image diagram for explaining the difference between the shooting of a learning captured image and a correct captured image according to one embodiment. With reference to this diagram, a method for acquiring images in a modified example of the training data creation method will be described. Figure 7 (A) is an example of a method for obtaining a correct answer camera image D, and is an example of a case where the image is taken using a device such as "VISIA (registered trademark), manufactured by Canfield Scientific". As shown in the figure, the face of the subject is fixed to the device, so there is no hand shaking, etc. If the learning camera image C is also taken at the same time, the position of the subject's face does not change, so there is no need for position alignment (if not at the same time, the amount of position alignment is also very small). In addition, if Figure 7 As shown in (A), when photographing is performed using a device such as "VISIA (registered trademark), manufactured by Canfield Scientific", the photographing device itself also uses a device with higher image quality than the smartphone 30.

[0122] Figure 7(B) is an example of a method for acquiring a camera image C for learning, and is an example of a case where a smartphone 30 is used for shooting. As shown in the figure, the face of the subject is not fixed, and hand shaking occurs. In addition, the image quality of the camera device itself is also lower than that of devices such as "VISIA (registered trademark), manufactured by Canfield Scientific". In order to use the correct camera image D and the camera image C for learning, which are respectively captured in this way, as training data, in the present embodiment, high-precision position alignment is performed. In addition, even when using a smartphone 30, it is possible to shoot with the face fixed to suppress the occurrence of hand shaking. However, in the present embodiment, training data based on the same image as the image in the inference stage is deliberately produced, so that shooting can also be performed in a situation where shooting is performed in the inference stage. The situation where shooting is performed in the inference stage may also refer to a situation where the hand is stretched out to shoot oneself (so-called selfie) using the built-in camera of the smartphone 30.

[0123] Figure 8 This is an image diagram of reference points used for training data generation according to an embodiment. Four reference points are marked on the face of the subject. The reference points may also be reference points such as stickers. In order to further improve the accuracy, the stickers may be marked with marks such as oil-based pens, or holes may be opened on the stickers for position determination.

[0124] Fig. 9 This figure shows an example of preprocessing of a correct answer camera image and cutting of a learning camera image and a correct answer camera image according to an embodiment. With reference to this figure, an example of preprocessing of a correct answer camera image D and cutting of a learning camera image C and a correct answer camera image D according to a variation of a training data production method is described.

[0125] first, Fig. 9 (A) is an example of a correct answer captured image D. Since the correct answer captured image D is captured with the face fixed, it shows a normal view image. Fig. 9 (B) is an example of a learning captured image C. The learning captured image C is an image captured by using the smartphone 30 without fixing the face, and therefore may not be a straight view. In the present embodiment, these images are aligned.

[0126] Before position alignment, preprocessing is performed on the correct solution captured image D. In this preprocessing, the correct solution captured image D is transformed to match the learning captured image C using a conventional technique such as projection transformation. This preprocessing may also be performed by image processing. Fig. 9(C) shows the result of preprocessing the correct answer camera image D. Here, in this embodiment, the correct answer camera image D is not transformed to match the correct answer camera image D, but is transformed to match the learning camera image C. This is to perform learning with the same image as the image in the inference stage by performing the transformation to match the image in the inference stage.

[0127] Then, by Fig. 9 The image shown in (B) and Fig. 9 A portion of the image is extracted (cut) from each of the images shown in (C) to create training data. Figure 8 In addition, the reference points do not need to be specially marked for the purpose of creating training data, and can also be feature points of the human body such as moles and corners of the mouth.

[0128] According to the modified example of the training data preparation method as described above, the learning camera image acquisition step is provided to obtain a learning camera image C, which is an image obtained by photographing the skin of a person irradiated with visible light and a portion having a reference point, and the correct answer camera image acquisition step is provided to obtain an image obtained by photographing the same subject as the learning camera image C and the skin of a person that has generated fluorescence due to porphyrin by being irradiated with ultraviolet rays by a camera device different from the camera device that photographed the learning camera image C, which is a correct answer camera image D. In addition, the first extraction step is provided to extract a partial region based on the reference point in the learning camera image C, and the second extraction step is provided to extract a partial region based on the reference point in the correct answer camera image D. In addition, the training data preparation step is provided to store the images extracted from the learning camera image C and the correct answer camera image D in correspondence with each other, thereby preparing training data. By preparing training data in this way, it is possible to prepare training data that is the same as the image in the inference stage. That is, according to the present embodiment, learning can be performed with high accuracy, and inference can be performed with high accuracy.

[0129] In addition, according to a variation of the training data production method as described above, an image processing step is further included to perform image processing on the correct camera image D (rather than the learning camera image C) to generate an image that matches the learning camera image C. The image processing step refers to preprocessing. In addition, in the second extraction step, a portion of the area based on the reference point in the correct camera image D that has been image processed by the image processing step is extracted. By producing training data in this way, it is possible to produce training data that is the same as the image in the inference stage. That is, according to this embodiment, learning can be performed with high precision, and inference can be performed with high precision.

[0130] In the image processing step, in addition to performing image processing on the correct answer captured image D, image processing may also be performed on the learning captured image C.

[0131] The above is the description of the model generation method and the image processing method involved in this embodiment.

[0132] In addition, the visible light used when shooting the learning video image C and the subject video image A is preferably light of the same wavelength and intensity. In addition, as the visible light used when shooting the learning video image C and the subject video image A, blue light (for example, with a wavelength of 380nm to 550nm) may also be used. It has been confirmed through experiments that by using blue light when shooting the learning video image C and the subject video image A, the reproducibility of the part caused by the fluorescence of porphyrin in the output image Pout generated by the learning model MDa is improved compared with the case of using white light. Specifically, nine subjects were selected, and as for the photographic images of the subjects, a photographic image A' of the subjects was obtained by cutting the photographic image A of the subjects of "3456×5184" pixels to an image size of "768×768" pixels and compressing the image of the cut portion of "768×768" pixels to "256×256" pixels. When the above-mentioned preprocessing 1 and preprocessing 2 were performed, the pixel value correlation coefficient between the output image and the correct photographic image was "0.61" when white light (400nm~770nm) was used, while the pixel value correlation coefficient between the output image and the correct photographic image was "0.65" when blue light (380nm~550nm) was used. In addition, as an embodiment using blue light, for example, a blue film is covered on artificial light such as an incandescent lamp, a fluorescent lamp, or an LED bulb, or irradiation with an LED bulb that only emits blue light, or irradiation by displaying one surface of a smartphone or a monitor in blue.

[0133] According to the above-mentioned embodiment, the image processing device 10 uses the learned model MDa to obtain an output image B as an image of human skin that has generated fluorescence due to porphyrin based on a subject camera image A obtained by photographing the skin of the subject U irradiated with visible light. The learned model MDa is obtained by machine learning in a manner of generating an image of human skin that has generated fluorescence due to porphyrin based on a camera image obtained by photographing the skin of the subject U irradiated with visible light, using a learning camera image C obtained by photographing the skin of the subject irradiated with visible light and a correct camera image D obtained by photographing the skin of the subject that has generated fluorescence due to porphyrin by irradiating ultraviolet light. Thus, the following effect is obtained: even if the subject U is not irradiated with ultraviolet light, porphyrin on the skin of the subject U can be detected. The subject camera image A can also be converted into a subject compressed camera image A' obtained by compressing a part of the subject camera image A for use. As a result, the reproducibility of the part of the output image B generated by the learning model MDa caused by the fluorescence of porphyrin is improved. In addition, the part can also be the forehead, nose or mouth area of ​​the face. Based on the compressed camera image A' of the subject, an output image B' is obtained as an image of the skin of a person who has generated the fluorescence caused by porphyrin. The output image B' is a compressed image, which is expanded to the same size as the camera image A of the subject and restored to the output image B, thereby obtaining the output image B.

[0134] In addition, in the machine learning of the learned model MDa, a compressed learning camera image obtained by compressing a part of the learning camera image C and a compressed correct camera image obtained by compressing a part of the correct camera image D can also be used. As a result, the reproducibility of the part caused by the fluorescence of porphyrin in the output image B generated by the learned model MDa is improved. In addition, the part can also be the forehead, nose or mouth area of ​​the face.

[0135] In addition, in the machine learning of the learned model MDa, an emphasized correct-answer camera image in which the red component is emphasized can also be used for the correct-answer camera image D. As a result, the reproducibility of the part of the output image B generated by the learned model MDa caused by the fluorescence of porphyrin is improved. In addition, the correct-answer camera image D can also be an RGB image composed of an R image, a G image, and a B image, and the emphasized correct-answer camera image can also be an RGB image generated by adding a differential image obtained by subtracting the G image from the R image to the RGB image.

[0136] In addition, the learned model MDa may also be composed of layers of the encoder part and layers of the decoder part. In addition, the learned model MDa may also have a U-Net structure in which the layers of the encoder part and the layers of the decoder part are symmetrically structured and connected by jump connections.

[0137] According to the above-mentioned embodiment, the model generation device 20 uses the learning camera image C obtained by photographing the skin of a person irradiated with visible light and the correct camera image D obtained by photographing the skin of a person who has generated fluorescence due to porphyrin by being irradiated with ultraviolet light, and performs machine learning of the model MD in a manner of generating an image of the skin of a person who has generated fluorescence due to porphyrin based on the camera image obtained by photographing the skin of a person irradiated with visible light, thereby generating a learned model MDa. If the learned model MDa is used, the following effect can be obtained: even if the ultraviolet light is not irradiated to the subject U, porphyrin on the skin of the subject U can be detected.

[0138] In addition, the model generation device 20 may also generate a compressed learning camera image obtained by compressing a part of the learning camera image C and a compressed correct camera image obtained by compressing a part of the correct camera image D, and use the compressed learning camera image and the compressed correct camera image for machine learning of the model MD. As a result, the reproducibility of the part caused by the fluorescence of porphyrin in the output image B generated by the learned model MDa is improved. In addition, the part may also be the forehead, nose or mouth area of ​​the face.

[0139] In addition, the model generation device 20 can also generate an emphasized correct-answer camera image that emphasizes the red component for the correct-answer camera image D, and use the emphasized correct-answer camera image for machine learning of the model MD. As a result, the reproducibility of the part of the fluorescence caused by porphyrin in the output image B generated by the learned model MDa is improved. In addition, the correct-answer camera image D can also be an RGB image composed of an R image, a G image, and a B image, and the emphasized correct-answer camera image can also be an RGB image generated by adding a differential image obtained by subtracting the G image from the R image to the RGB image.

[0140] In addition, the model MD may be composed of layers of the encoder part and layers of the decoder part. In addition, the model MD may have a U-Net structure in which the layers of the encoder part and the layers of the decoder part are symmetrically structured and connected by skip connections.

[0141] In addition, the image processing device 10 as described above can also be used to determine the susceptibility of acne. For example, by comparing the output image obtained by the image processing device 10 with the standard image, the susceptibility of acne can be determined. The standard image may also refer to an image of an ordinary person who does not have acne. In addition, the standard image may also be a predetermined appropriate image prepared according to the age, gender, nationality, etc. of the determination object (subject U). In addition, the standard image may not be an actually captured image, but may be an image generated by image processing.

[0142] In addition, the image processing device 10 as described above can also be used to evaluate the effect of skin care before and after the subject performs skin care. In this case, as a first acquisition step, an output image output by the image processing device 10 is acquired based on a photographic image A of the subject before the subject performs skin care. In addition, as a second acquisition step, an output image output by the image processing device 10 is acquired based on a photographic image A of the subject after the subject performs skin care. Then, the output image acquired by the first acquisition step is compared with the output image acquired by the second acquisition step to determine the effect of the skin care. This determination can be performed by comparing the output images before and after the skin care, or by comparing the output images before and after the skin care with the standard images respectively.

[0143] In addition, a computer program for realizing the functions of each of the above-mentioned devices may be recorded in a computer-readable recording medium, and the computer system may read and execute the program recorded in the recording medium. In addition, the "computer system" mentioned here may also include hardware such as OS and peripheral devices.

[0144] In addition, in the case of utilizing the WWW system, the "computer system" also includes a homepage providing environment (or display environment).

[0145] The “computer-readable recording medium” refers to a storage device such as a flexible disk, a magneto-optical disk, a writable nonvolatile memory such as a ROM or a flash memory, a removable medium such as a DVD (Digital Versatile Disc), or a hard disk built into a computer system.

[0146] Furthermore, "computer-readable recording media" also include media that retain programs for a certain period of time, such as volatile memory (e.g., DRAM (Dynamic Random Access Memory)) inside a computer system that serves as a server or client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line.

[0147] In addition, the above-mentioned program can also be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by transmission waves in the transmission medium. Here, the "transmission medium" for transmitting the program refers to a medium having the function of transmitting information, such as a network (communication network) such as the Internet, or a communication line (communication line) such as a telephone line.

[0148] The above-mentioned program may be used to realize a part of the above-mentioned functions, and may be a so-called difference file (difference program) that can realize the above-mentioned functions by combining with a program already recorded in the computer system.

[0149] Although the embodiments of the present invention have been described in detail with reference to the drawings, the specific configuration is not limited to the embodiments and design changes and the like are also included within the scope that does not depart from the gist of the present invention.

[0150] Description of Reference Numerals

[0151] 10: image processing device; 11: model storage unit; 12: subject camera image acquisition unit; 13: output image acquisition unit; 21: learning camera image acquisition unit; 22: correct answer camera image acquisition unit; 23: model generation unit; 24: model output unit; MD: model; MDa: learned model; 30: smart phone; U: subject.

Claims

1. An image processing device comprising: a model storage unit that stores a learned model, the learned model being obtained by machine learning in a manner of generating an image of human skin that has produced fluorescence due to porphyrin based on the camera image obtained by photographing the skin of a person irradiated with visible light, using a learning camera image obtained by photographing the skin of a person irradiated with visible light and a correct camera image obtained by photographing the skin of a person irradiated with ultraviolet light and producing fluorescence due to porphyrin; a subject captured image acquisition unit that acquires a subject captured image obtained by capturing the skin of the subject irradiated with visible light; and An output image acquisition unit acquires an output image from the captured image of the subject using the learned model, the output image being an image of human skin in which fluorescence due to porphyrin is generated.

2. The image processing device according to claim 1, wherein: In the machine learning of the learned model, a compressed learning captured image obtained by compressing a part of the learning captured image and a compressed correct answer captured image obtained by compressing a part of the correct answer captured image are used.

3. The image processing device according to claim 2, wherein: The skin is facial skin, The part is the forehead, nose or mouth area.

4. The image processing device according to claim 1, wherein: In the machine learning of the learned model, an enhanced correct answer captured image in which a red component is enhanced with respect to the correct answer captured image is used.

5. The image processing device according to claim 4, wherein: The correct camera image is an RGB image composed of an R image, a G image, and a B image. The enhanced correct-answer captured image is an RGB image generated by adding a difference image obtained by subtracting the G image from the R image to the RGB image.

6. The image processing device according to any one of claims 1 to 5, wherein: The learned model is composed of layers of the encoder part and layers of the decoder part.

7. A model generation device comprising: a learning camera image acquisition unit that acquires a learning camera image obtained by photographing the skin of a person irradiated with visible light; a correct-answer photographed image acquisition unit that acquires a correct-answer photographed image obtained by photographing human skin that has generated fluorescence due to porphyrin by being irradiated with ultraviolet rays; and A model generating unit uses the learning camera image and the correct camera image to perform machine learning of the model in a manner that generates an image of human skin that produces fluorescence caused by porphyrin based on a camera image obtained by photographing human skin irradiated with visible light, thereby generating a learned model.

8. A skin condition measurement system comprising: The image processing device according to any one of claims 1 to 5; and The model generating device according to claim 7.

9. An image processing method comprising the following steps: acquiring a photographic image of the subject obtained by photographing the skin of the subject irradiated with visible light; as well as using the learned model, acquiring an output image from the photographed image of the subject, the output image being an image of human skin in which fluorescence due to porphyrin is generated; Among them, the learned model is a model obtained by machine learning in a manner of generating an image of human skin that produces fluorescence caused by porphyrin based on the camera image obtained by photographing the skin of a person irradiated with visible light, using a learning camera image obtained by photographing the skin of a person irradiated with visible light and a correct camera image obtained by photographing the skin of a person that produces fluorescence caused by porphyrin by irradiating ultraviolet light.

10. The image processing method according to claim 9, wherein: In the step of acquiring the learning captured image and the correct captured image, at least one mapping reference point is set.

11. A model generation method comprising the following steps: acquiring a learning camera image obtained by photographing the skin of a person irradiated with visible light; acquiring a correct image obtained by photographing human skin which has generated fluorescence due to porphyrin by being irradiated with ultraviolet rays; and The learning camera image and the correct camera image are used to perform machine learning of the model in a manner that generates an image of human skin that produces fluorescence due to porphyrin based on a camera image obtained by photographing human skin irradiated with visible light, thereby generating a learned model.

12. A determination method for determining susceptibility to acne by comparing the output image acquired by the image processing apparatus according to any one of claims 1 to 5 with a standard image, the standard image being an image of the skin of an ordinary person who does not develop acne.

13. A determination method is an evaluation method for evaluating the effect of skin care before and after a subject performs skin care, wherein: a first acquisition step of acquiring the output image acquired by the image processing device according to any one of claims 1 to 5 based on a photographed image of the subject before the subject performs skin care; a second acquisition step of acquiring the output image acquired by the image processing device according to any one of claims 1 to 5 based on the photographed image of the subject after the subject has performed skin care; as well as The output image acquired by the first acquisition step is compared with the output image acquired by the second acquisition step to determine the degree of improvement obtained by skin care.

14. A method for preparing training data, comprising: a learning camera image acquisition step of acquiring a learning camera image, wherein the learning camera image is an image obtained by photographing the skin of a person irradiated with visible light and a portion having a reference point; A correct answer camera image acquisition step, wherein the correct answer camera image is an image obtained by photographing the same subject as the learning camera image with a camera device different from the camera device that photographed the learning camera image, and the correct answer camera image is an image of human skin that has been irradiated with ultraviolet rays and has generated fluorescence caused by porphyrin; A first extraction step of extracting a partial area based on the reference point in the learning camera image; A second extraction step of extracting a portion of the area based on the reference point in the correct camera image; as well as The training data creating step creates training data by storing images extracted from the learning captured images and the correct answer captured images in association with each other.

Images