Wrinkle grade determination device, wrinkle grade determination method, wrinkle grade determination program, and wrinkle grade determination system
The AI-based wrinkle grade determination system addresses accuracy and reliability issues by providing precise wrinkle grade assessments and user-friendly explanations, enhancing user understanding and convenience.
Patent Information
- Application Number
- JP2025077441
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2025-05-07
- Publication Date
- 2026-11-19
AI Technical Summary
Existing methods for determining wrinkle grades lack accuracy, reliability, and fail to provide explanations or answers to user questions regarding skin surface images, making them inconvenient for users.
A wrinkle grade determination system utilizing artificial intelligence (AI) trained on multiple training data, including skin surface images and their features, to determine wrinkle grades with high precision and reliability, and provide responses to user queries or explanations about the determination process.
The system enables accurate and reliable wrinkle grade determination, enhances user understanding through explanations and answers, and improves convenience compared to conventional methods.
Smart Images

Figure 2026181781000001_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a wrinkle grade determination device, a wrinkle grade determination method, a wrinkle grade determination program, and a wrinkle grade determination system that use artificial intelligence (AI) trained on a plurality of pieces of training data including skin surface images, wrinkle grades of the skin surface images, descriptions of features of the skin surface images, and / or determination reasons for the wrinkle grades, to enable high-precision and high-reliability determination of the wrinkle grade of a skin surface image, and can present response sentences to user questions, description sentences of features of the skin surface image, and / or description sentences of the determination reasons for the wrinkle grade.
Background Art
[0002] Wrinkles (may also be referred to as "shiwa" or "zhou") are one of the skin aging symptoms caused by factors such as aging, stress, and ultraviolet exposure, and the determination of the progression degree of wrinkles (wrinkle grade) is regarded as important from medical and cosmetic perspectives. Therefore, the wrinkle grade is comprehensively determined from the results of direct observation by a person with advanced determination skills such as a dermatologist or a beauty professional, direct measurement of the subject's outer canthus by an instrument, or making a mold of the subject's outer canthus and performing instrument measurement on the mold.
[0003] On the other hand, with the widespread use of smartphones equipped with cameras, techniques for determining wrinkle grade by computer processing through the provision of images have been proposed (for example, see Patent Document 1). Further, a technique has been proposed that generates a composite image from a facial image and outputs a wrinkle-enhanced image color-coded for each depth based on frequency components (for example, see Patent Document 2).
Prior Art Documents
Patent Documents
[0004]
Patent Document 1
Patent Document 2
[0005] However, the prior art described in Patent Document 1 above was not sufficient in terms of the accuracy of wrinkle grade determination. Furthermore, the prior art described in Patent Document 2 above does not classify wrinkles into wrinkle grades based on wrinkle characteristics (wrinkle depth, number of wrinkles, etc.) in skin surface images, and therefore is not sufficiently reliable in terms of wrinkle grade determination. Furthermore, the prior art described in Patent Documents 1 and 2 above does not provide answers to user questions, descriptions of the characteristics of the skin surface image, and / or explanations of the reasons for determining the wrinkle grade, in addition to the wrinkle grade determination result of the skin surface image. Therefore, it is difficult for users to determine the wrinkle grade, and it has the problem of being less convenient than conventional methods.
[0006] The present invention has been made in view of the above circumstances, and aims to provide a wrinkle grade determination device, a wrinkle grade determination method, a wrinkle grade determination program, and a wrinkle grade determination system that can determine the wrinkle grade of a skin surface image with high accuracy and reliability using artificial intelligence (AI) trained on multiple training data including skin surface images, the wrinkle grade of the skin surface image, a description of the features of the skin surface image, and / or the reason for determining the wrinkle grade, and that can present the wrinkle grade of the skin surface image, an answer to a user's question, a description of the features of the skin surface image, and / or an explanation of the reason for determining the wrinkle grade. [Means for solving the problem]
[0007] The wrinkle grade determination device of the present invention, as a means for solving the above-mentioned problems, An acquisition unit that acquires a skin surface image entered by the user, or a skin surface image entered by the user and a question text, The acquired skin surface image is processed with an image encoder and converted into image features. If a question exists, the question is processed using a text encoder to convert it into text features, and the text features take the lead in interacting with the image features. Cross-modal processing is performed to merge features suitable for determining the acquired skin surface image and answering the question. If no question exists, cross-modal processing is performed to merge features suitable for explaining the skin surface image obtained from the image features. A control unit generates text that includes an answer to a question, a description of the features of the skin surface image, and / or an explanation of the reason for determining the wrinkle grade of the skin surface image, based on features merged by cross-modal processing. It includes an output unit that outputs the wrinkle grade of a skin surface image and the generated text as the answer.
[0008] In the wrinkle grade determination device, it is preferable that the cross-modal processing in the control unit is performed using parameters that have been updated by learning using a plurality of training data including a skin surface image, the wrinkle grade of the skin surface image, and a description of the features of the skin surface image and / or the reason for determining the wrinkle grade.
[0009] The wrinkle grade determination method of the present invention involves a computer, An acquisition process to acquire a skin surface image entered by the user, or a skin surface image entered by the user and a question text, The acquired skin surface image is processed with an image encoder and converted into image features. If a question exists, the question is processed using a text encoder to convert it into text features, and the text features take the lead in interacting with the image features. Cross-modal processing is performed to merge features suitable for determining the acquired skin surface image and answering the question. If no question exists, cross-modal processing is performed to merge features suitable for explaining the skin surface image obtained from the image features. A control step that generates text including an answer to a question, a description of the features of the skin surface image, and / or an explanation of the reason for determining the wrinkle grade of the skin surface image, based on features merged by cross-modal processing, The system performs an output process that outputs the wrinkle grade of the skin surface image and the generated text as the answer.
[0010] The wrinkle grade determination program of the present invention is a computer program that provides a computer A data acquisition process to obtain a skin surface image entered by the user, or a skin surface image entered by the user and a question text, The acquired skin surface image is processed with an image encoder and converted into image features. If a question exists, the question is processed using a text encoder to convert it into text features, and the text features take the lead in interacting with the image features. Cross-modal processing is performed to merge features suitable for determining the acquired skin surface image and answering the question. If no question exists, cross-modal processing is performed to merge features suitable for explaining the skin surface image obtained from the image features. A control process that generates text including an answer to a question, a description of the features of the skin surface image, and / or an explanation of the reason for determining the wrinkle grade of the skin surface image, based on features merged by cross-modal processing. The system performs output processing that outputs the wrinkle grade of the skin surface image and the generated text as the answer.
[0011] The wrinkle grade determination system of the present invention is A learning device that updates parameters by learning using multiple training data including skin surface images, wrinkle grades of said skin surface images, descriptions of the features of the skin surface images and / or reasons for determining the wrinkle grade, and A wrinkle grade determination system having a wrinkle grade determination device that performs wrinkle grade determination, The learning device is An acquisition unit that acquires skin surface images, the wrinkle grade of said skin surface images, and text describing the features of the skin surface images and / or the reason for determining the wrinkle grade from training data, The acquired skin surface image is processed with an image encoder and converted into image features. The acquired text is processed with a text encoder and converted into text features. A control unit performs cross-modal processing to fuse image features and text features, and updates parameters by learning based on the fused features. A storage unit that stores parameters updated through learning, Equipped with, The wrinkle grade determination device is An acquisition unit that acquires a skin surface image entered by the user, a skin surface image entered by the user and a question, a skin surface image entered by the user and updated parameters, or a skin surface image entered by the user, a question, and updated parameters. The acquired skin surface image is processed with an image encoder and converted into image features. If a question exists, the question is processed using a text encoder to convert it into text features. If acquired parameters exist, those parameters are used. If a question exists, the text features of the question primarily influence the interaction with the image features. Cross-modal processing is performed to merge features suitable for determining the acquired skin surface image and answering the question. If no question exists, cross-modal processing is performed to merge features suitable for explaining the acquired skin surface image from the image features. A control unit generates text that includes an answer to a question, a description of the features of the skin surface image, and / or an explanation of the reason for determining the wrinkle grade of the skin surface image, based on features merged by cross-modal processing. It includes an output unit that outputs the wrinkle grade of a skin surface image and the generated text as the answer. [Effects of the Invention]
[0012] According to the present invention, it is possible to provide a wrinkle grade determination device, a wrinkle grade determination method, a wrinkle grade determination program, and a wrinkle grade determination system that can determine the wrinkle grade of a skin surface image with high accuracy and high reliability using artificial intelligence (AI) trained on a plurality of training data including a skin surface image, the wrinkle grade of the skin surface image, a description of the features of the skin surface image and / or a reason for determining the wrinkle grade, and that can present a response sentence to a user's question, a description sentence of the features of the skin surface image, and / or a description sentence of the reason for determining the wrinkle grade. [Brief Description of the Drawings]
[0013] [Figure 1] Figure 1 is a schematic diagram showing an example of the configuration of the wrinkle grade determination device of the present invention. [Figure 2] Figure 2 is a schematic diagram showing an example of wrinkle grade determination processing of the wrinkle grade determination device. [Figure 3] Figure 3 is a flowchart showing an example of the flow of wrinkle grade determination processing in the wrinkle grade determination method of the present invention. [Figure 4] Figure 4 is a schematic diagram showing an example of the configuration of the wrinkle grade determination system of the present invention. [Figure 5] Figure 5 is a schematic diagram showing an example of the configuration of a learning device. [Figure 6] Figure 6 is a schematic diagram showing an example of training data. [Figure 7] Figure 7 is a flowchart showing an example of the flow of learning processing. [Figure 8] Figure 8 is a schematic diagram showing the processing outline of the first embodiment of the wrinkle grade determination method of the present invention using the wrinkle grade determination device of the present invention. [Figure 9] Figure 9 is a schematic diagram showing the processing outline of the second embodiment of the wrinkle grade determination method of the present invention using the wrinkle grade determination device of the present invention. [Mode for Carrying Out the Invention]
[0014] (Wrinkle Grade Determination Device and Wrinkle Grade Determination Method) The wrinkle grade determination device of the present invention comprises an acquisition unit, a control unit, and an output unit, and further comprises other units as necessary. The wrinkle grade determination method of the present invention includes an acquisition step, a control step, and an output step, and further includes other steps as necessary.
[0015] The wrinkle grade determination method of the present invention can be suitably carried out by the wrinkle grade determination apparatus of the present invention, the acquisition and output process can be performed by the acquisition unit, the control process can be performed by the control unit, the output process can be performed by the output unit, and other processes can be performed by other means.
[0016] The wrinkle grade determination device of the present invention preferably determines the wrinkle grade using parameters obtained by learning with a plurality of training data including a skin surface image, the wrinkle grade of the skin surface image, a description of the features of the skin surface image, and / or the reason for determining the wrinkle grade. Publicly available parameters can also be used.
[0017] According to the wrinkle grade determination device and wrinkle grade determination method of the present invention, the wrinkle grade of a skin surface image can be determined with high accuracy and reliability using artificial intelligence (AI) trained with multiple training data including a skin surface image, the wrinkle grade of the skin surface image, a description of the features of the skin surface image, and / or the reason for determining the wrinkle grade. Furthermore, since answers to user questions, descriptions of the features of the skin surface image, and / or explanations of the reason for determining the wrinkle grade can be presented, it becomes easier for users to determine the wrinkle grade, and convenience is greatly improved compared to conventional methods.
[0018] Characteristics of skin surface images include, for example, the clarity of the skin surface image, the presence or absence of shadows in the skin surface image, the color of the skin surface, the texture of the skin surface (smoothness, roughness, etc.), the presence and condition of pores, wrinkles, sagging, redness caused by inflammation, and pigmentation such as blemishes.
[0019] The training data consists of multiple sets of training data in which dermatologists, researchers skilled in wrinkle evaluation, or experienced beauticians determine the wrinkle grade of skin surface images, in accordance with wrinkle grade evaluation criteria such as "Guidelines for the Evaluation of Anti-Wrinkle Products for Obtaining New Efficacy Claims" (Journal of the Japan Society of Cosmetic Scientists (Vol.30, No.4, pp.316-332, 2006)) and Skin Atlas (L'Oréal Japan R&I Skin Aging Atlas, Japanese Edition). These sets contain descriptions of the characteristics of the skin surface images and / or explanations of the reasons for the wrinkle grade determination. An AI trained using such training data can achieve highly accurate and reliable wrinkle grade determination of skin surface images. Furthermore, since it can provide answers to user questions, descriptions of the characteristics of the skin surface images, and / or explanations of the reasons for the wrinkle grade determination, it becomes easier for users to determine the wrinkle grade, significantly improving convenience compared to conventional methods.
[0020] -Acquisition process and acquisition unit- The acquisition process involves acquiring a skin surface image entered by the user, or a skin surface image entered by the user along with a question, and is performed by the acquisition unit.
[0021] User input of skin surface images is mandatory, while inputting the question text is optional.
[0022] The acquisition unit is not particularly limited as long as it can receive various requests from users regarding the wrinkle grade determination device, and can use publicly known means as appropriate, such as a keyboard, mouse, touch panel, microphone, USB memory, SSD (Solid-State Drive), HDD (Hard Disk Drive), smartphone, etc.
[0023] It is preferable that the cross-modal processing in the control unit is performed using parameters that have been updated by learning from multiple training data, including a skin surface image, the wrinkle grade of the skin surface image, and a description of the features of the skin surface image and / or the reason for determining the wrinkle grade. Parameters refer to the weights and biases that are adjusted during the model's learning process. The model has a vast number of parameters, ranging from hundreds of millions to trillions. Updating all parameters based on skin surface images and text requires enormous computational resources. Therefore, by fixing a certain percentage of parameters and updating the others, it is possible to create a model that can efficiently provide detailed answers about wrinkle grades with fewer computational resources. As parameters, you can use parameters that have been trained and updated by a learning device, or publicly available parameters.
[0024] -Control process and control unit- The various processes performed by the wrinkle grade determination method of the present invention are carried out by the control unit of the wrinkle grade determination device of the present invention. The control unit may be, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a part of the machine used in the learning device of the present invention, or a combination thereof. The control unit implements various functions by executing a program (for example, the wrinkle detection program of the present invention) loaded into the main memory or the like.
[0025] The main memory stores various programs as well as data necessary to execute those programs. For example, a main memory system can be one that includes at least one of ROM (Read Only Memory) and RAM (Random Access Memory). ROM stores various programs, such as the BIOS (Basic Input / Output System). There are no particular restrictions on the type of ROM; it can be appropriately selected depending on the purpose. Examples include mask ROM and PROM (Programmable ROM). RAM functions as a working area that is expanded when various programs stored in ROM or auxiliary storage devices are executed by the control unit. There are no particular restrictions on the type of RAM, and it can be appropriately selected according to the purpose. Examples include DRAM (Dynamic Random Access Memory), SRAM (Static Random Access Memory), and VRAM (Video Random Access Memory).
[0026] The control unit of the wrinkle grade determination device of the present invention specifically comprises a cross-modal processing unit and a text generation unit. The control step of the wrinkle grade determination method of the present invention specifically includes a cross-modal processing step and a text generation step.
[0027] -Cross-modal processing step and cross-modal processing section- The cross-modal processing step involves processing the acquired skin surface image with an image encoder to convert it into image features, and if a question exists, processing the question with a text encoder to convert it into text features. When a question exists, the text features of the question primarily influence the interaction with the image features, and cross-modal processing is performed to fuse features suitable for determining the acquired skin surface image and answering the question. When no question exists, cross-modal processing is performed to fuse features suitable for explaining the acquired skin surface image from the image features, and this process is carried out by the cross-modal processing unit. If the acquired parameters exist, cross-modal processing will be performed using those parameters.
[0028] If a question exists, it serves to guide which parts of the image features to focus on when cross-modal processing is used to associate image features with text features. If no question is present, the model interprets the image content based on its learned knowledge without any prompting from a question, and cross-modal processing is performed to output features necessary to generate a suitable explanatory text.
[0029] If no question text is obtained (only a skin surface image is obtained), the AI has been trained in advance using multiple training data sets that include skin surface images and their wrinkle grades, as well as descriptions of the features of the skin surface images and / or the reasons for determining the wrinkle grade. Therefore, the AI will provide an explanation of the features of the skin surface image and the reasons for determining the wrinkle grade based on the obtained skin surface image. However, if only a skin surface image is obtained, there is a possibility that the explanation will be in a state that is not intended by the user or is incomplete. The advantage of obtaining an answer that matches the user's intent is that if a question text is present, an answer that matches the user's intent will be obtained.
[0030] The questions are not pre-written; users can freely enter their own questions. Examples of questions include: "What is the wrinkle grade of this skin surface image?", "Are there any noticeable wrinkles in this skin surface image?", and "Is the wrinkle grade of this skin surface image 3?".
[0031] Cross-modal processing is performed by applying an attention mechanism. Furthermore, applying a cross-attention mechanism is even more preferable for cross-modal processing. The cross-attention mechanism is the core component that allows the cross-modal processing unit to understand image features and extract information necessary for tasks such as text generation and maximizing the similarity between image and text features. Moreover, the cross-modal processing unit has parameters that can be updated, enabling the construction of models tailored to the learning content. With fixed queries, parameters cannot be updated, making it impossible to build models tailored to the learning content.
[0032] Examples of cross-modal processing include Contrastive Learning used in CLIP (Contrastive Language-Image Pre-training), Image-Text Matching, Image-Grounded Text Generation, and Image-Text Contrastive Learning used in BLIP (Bootstrapping Language-Image Pre-training)-2, and multimodal embedding learning used in VisualBERT and CLIP.
[0033] -Text generation process and text generation unit- The text generation process is a process that generates text, including answers to questions, descriptions of the features of skin surface images, and / or explanations of the reasons for determining the wrinkle grade of skin surface images, based on features merged by cross-modal processing, and is performed by a text generation unit.
[0034] The text generation unit uses a language model (LM). A language model (LM) is a type of artificial intelligence (AI) that has been trained on multiple text datasets, and it is a technology that enables the understanding and creation of text. Examples of language models (LMs) include Char-RNN, Bidirectional LSTM, Seq2Seq (Google), GPT-3 (OpenAI), GPT-4 (OpenAI), Open Pre-trained Transformer (Meta), Llama (Meta), Flan-T5 (Google), and PaLM (Google).
[0035] -Output process and output section- The output process is a process in which the output unit outputs generated text as an answer, which includes the wrinkle grade of the skin surface image, a description of the characteristics of the skin surface image, a description of the reason for determining the wrinkle grade, and / or an answer to the question. There are no particular restrictions on the output unit; any known output unit can be used as appropriate, such as a printer, liquid crystal display, organic EL display, email, or audio.
[0036] The output unit, if it obtains a question, outputs an answer to the question including the wrinkle grade of the skin surface image and / or a description of the skin surface image. If no question exists, it outputs a description of the skin surface image including the wrinkle grade of the skin surface image as the answer. Furthermore, if the input question is something like "What is the wrinkle grade of this skin surface image?", the output unit can also output only the wrinkle grade of the skin surface image as the answer.
[0037] -Other processes and other parts- Other processes are not particularly restricted and can be selected as appropriate depending on the purpose, such as data storage processes and communication processes. Other sections are not particularly restricted and can be selected as appropriate depending on the purpose; examples include data storage sections and communication sections.
[0038] There are no particular restrictions on the data storage unit, as long as it can store generated text and other data. Examples include SSDs (Solid-State Drives) and HDDs (Hard Disk Drives).
[0039] As for the communication unit, there are no particular restrictions as long as it is capable of communicating with the outside of the wrinkle grade determination device, and publicly known devices can be used as appropriate, such as transceivers.
[0040] (Wrinkle Grade Determination System) The wrinkle grade determination system of the present invention comprises a wrinkle grade determination device for determining wrinkle grades, and a learning device that learns from a plurality of training data including skin surface images, the wrinkle grade of the skin surface images, and descriptions of the features of the skin surface images and / or the reasons for determining the wrinkle grade, and further comprises other parts as necessary.
[0041] The learning device updates its parameters by learning using multiple training data sets, which include skin surface images, the wrinkle grade of the skin surface images, a description of the features of the skin surface images, and / or the reason for determining the wrinkle grade. The learning device can be provided to users who perform wrinkle grade determination by independently learning and updating parameters, without being incorporated into the wrinkle grade determination system of the present invention.
[0042] The wrinkle grade determination device in the wrinkle grade determination system of the present invention can determine the wrinkle grade using parameters obtained by learning using a plurality of training data including a skin surface image, the wrinkle grade of the skin surface image, a description of the features of the skin surface image and / or the reason for determining the wrinkle grade. The wrinkle grade determination device can perform wrinkle grade determination independently, for example, using publicly available parameters, without being incorporated into the wrinkle grade determination system of the present invention. A wrinkle grade determination device similar to that of the present invention can be used, so a detailed explanation thereof will be omitted.
[0043] <Learning device> The learning device updates its parameters by learning using multiple training data, including skin surface images, the wrinkle grade of the skin surface images, a description of the features of the skin surface images, and / or the reason for determining the wrinkle grade, and stores the updated parameters in the storage unit. The learning device comprises an acquisition unit, a control unit, and a storage unit, and further includes other units as needed.
[0044] -Acquisition process and acquisition unit- The acquisition process involves acquiring skin surface images, the wrinkle grade of the skin surface images, and text describing the features of the skin surface images and / or the reasons for determining the wrinkle grade, from the training data, and this process is performed by the acquisition unit. Since the acquisition unit of the learning device can be the same as the acquisition unit of the wrinkle grade determination device of the present invention, a detailed explanation thereof will be omitted.
[0045] -Control process and control unit- The processing performed by the learning device is carried out by the control unit of the learning device. Since the control unit of the learning device is the same as that of the wrinkle grade determination device of the present invention, a detailed explanation thereof will be omitted.
[0046] The control unit of the learning device specifically comprises a cross-modal processing unit and a learning unit. The control process of the learning device specifically includes a cross-modal processing process and a learning process.
[0047] -Cross-modal processing step and cross-modal processing section- The cross-modal processing step involves processing acquired skin surface images with an image encoder to convert them into image features, processing acquired text with a text encoder to convert it into text features, and then performing cross-modal processing to fuse the image features and text features. This process is carried out by a cross-modal processing unit.
[0048] -Learning Process and Learning Department- The learning process involves updating parameters by learning based on features fused through cross-modal processing, and this is performed by the learning unit.
[0049] In the learning unit, learning is preferably performed using a loss function. Parameter updates continue until a predetermined number of training iterations are reached or the loss function converges. It is preferable that the parameter update status be saved periodically. The administrator specifies when to stop updating parameters in the learning device. For example, they might specify the number of training iterations. Note that the appropriate number of training iterations will vary depending on the model and training content.
[0050] Examples of loss functions include cross-entropy loss, binary cross-entropy loss, negative log-likehood loss, and contrastive loss.
[0051] In the learning device according to the present invention, a model pre-trained with multiple pairs of skin surface images and image descriptions is used to train the model with multiple training data including wrinkle grades, reasons for determining the wrinkle grade, and / or features of the skin surface image. Before training, the model can only provide general answers about wrinkle grades, but after training, it becomes able to provide detailed answers about wrinkle grades.
[0052] -Preservation process and preservation section- The storage unit stores parameters that have been updated through learning. The parameters stored in the storage unit are used, as needed, for wrinkle grade determination in the wrinkle grade determination device of the present invention.
[0053] As for the storage device, there are no particular restrictions as long as it can save parameters, and it can be appropriately selected according to the purpose. Examples include CD-ROM (Compact Disc ROM), DVD-ROM (Digital Versatile Disk ROM), MO disk (Magneto-Optical Disk), SD card, USB memory (USB (Universal Serial Bus) Flash Drive), SSD (Solid-State Drive), and HDD (Hard Disk Drive).
[0054] -Other processes and other parts- Other departments are not particularly restricted and can be selected as appropriate according to their purpose; for example, a communications department is one such example.
[0055] As for the communication unit, there are no particular restrictions as long as it is capable of communicating with the outside of the learning device, and publicly known devices can be used as appropriate, such as transceivers.
[0056] (Wrinkle Grade Assessment Program) The wrinkle grade determination program of the present invention is a computer program that, A data acquisition process to obtain a skin surface image entered by the user, or a skin surface image entered by the user and a question text, The acquired skin surface image is processed with an image encoder and converted into image features. If a question exists, the question is processed using a text encoder to convert it into text features, and the text features take the lead in interacting with the image features. Cross-modal processing is performed to merge features suitable for determining the acquired skin surface image and answering the question. If no question exists, cross-modal processing is performed to merge features suitable for explaining the skin surface image obtained from the image features. A control process that generates text including an answer to a question, a description of the features of the skin surface image, and / or an explanation of the reason for determining the wrinkle grade of the skin surface image, based on features merged by cross-modal processing. The system performs output processing that outputs the wrinkle grade of the skin surface image and the generated text as the answer.
[0057] The wrinkle grade determination program of the present invention can be, for example, a program that causes a computer to execute the wrinkle grade determination method of the present invention using the wrinkle grade determination device of the present invention. Furthermore, a preferred embodiment of the wrinkle grade determination program of the present invention can be, for example, the same as a preferred embodiment of the wrinkle grade determination method of the present invention.
[0058] The wrinkle grade determination program of the present invention can be created using various known programming languages, depending on the configuration of the computer system used and the type and version of the operating system.
[0059] The wrinkle grade determination program of the present invention may be recorded on a recording medium such as a CD-ROM (Compact Disc ROM), DVD-ROM (Digital Versatile Disk ROM), MO disk (Magneto-Optical Disk), SD card, USB memory (USB (Universal Serial Bus) Flash Drive), SSD (Solid-State Drive), or HDD (Hard Disk Drive).
[0060] Furthermore, when recording the wrinkle grade determination program of the present invention on the above-mentioned recording medium, it can be used either directly or by installing it on a hard disk via a recording medium reader in a computer system, as needed. Furthermore, the wrinkle grade determination program of the present invention may be stored in an external storage area (such as another computer) accessible from a computer system via an information and communication network. In this case, the wrinkle grade determination program of the present invention stored in the external storage area can be used by installing it directly from the external storage area via an information and communication network, the Web, or the cloud, or by installing it on an HDD (Hard Disk Drive) or SSD (Solid-State Drive) as needed. The wrinkle grade determination program of the present invention may be recorded on multiple recording media, divided into sections for each arbitrary process.
[0061] <Computer-readable recording medium> A computer-readable recording medium according to the present invention records the wrinkle grade determination program of the present invention. There are no particular limitations on the computer-readable recording medium according to the present invention, and it can be appropriately selected depending on the purpose. Examples include CD-ROM (Compact Disc ROM), DVD-ROM (Digital Versatile Disk ROM), MO disk (Magneto-Optical Disk), SD card, USB memory [USB (Universal Serial Bus) Flash Drive], SSD (Solid-State Drive), HDD (Hard Disk Drive), and the like. Furthermore, the computer-readable recording medium according to the present invention may be a plurality of recording media in which the wrinkle grade determination program of the present invention is divided and recorded for each arbitrary processing step.
[0062] The following describes in more detail an example of the technology disclosed in this invention, using a configuration example and flowchart of the wrinkle grade determination device of the present invention.
[0063] Figure 1 is a schematic diagram of the wrinkle grade determination device of the present invention. As shown in Figure 1, the wrinkle grade determination device 11 includes an acquisition unit 13, a control unit 14, and an output unit 15. Figure 2 is a schematic diagram showing an example of the wrinkle grade determination process in the wrinkle grade determination device 11.
[0064] The acquisition unit 13 of the wrinkle grade determination device acquires (1) a skin surface image input by the user, (2) a skin surface image input by the user and a question, (3) parameters that have been updated by learning using multiple training data including a skin surface image input by the user, the wrinkle grade of the skin surface image, a description of the characteristics of the skin surface image and / or the reason for determining the wrinkle grade, or (4) a skin surface image input by the user, a question, and the updated parameters.
[0065] The control unit 14 of the wrinkle grade determination device comprises a cross-modal processing unit 19 and a text generation unit 20. The cross-modal processing unit 19 processes the acquired skin surface image with an image encoder to convert it into image features, processes the question text with a text encoder to convert it into text features if a question text exists, uses the acquired parameters if they exist, and performs cross-modal processing so that the text features of the question text take the lead in interacting with the image features and fuses features suitable for determining the acquired skin surface image and answering the question text. On the other hand, if no question text exists, performs cross-modal processing so that features suitable for explaining the acquired skin surface image are fused from the image features.
[0066] The text generation unit 20 generates text that includes an answer to a question, a description of the features of the skin surface image, and / or an explanation of the reason for determining the wrinkle grade of the skin surface image, based on the feature quantities merged by cross-modal processing.
[0067] The output unit 15 outputs generated text as the answer, which includes the wrinkle grade of the skin surface image, a description of the features of the skin surface image, a description of the reason for determining the wrinkle grade, and / or an answer to the question.
[0068] Figure 4 is a schematic diagram of the wrinkle grade determination system of the present invention. The wrinkle grade determination system 100 shown in Figure 4 comprises a wrinkle grade determination device 11 that performs wrinkle grade determination, and a learning device 12 that updates parameters by learning using multiple training data.
[0069] As shown in Figure 5, the learning device 12 includes an acquisition unit 16, a control unit 17, and a storage unit 18.
[0070] The acquisition unit 16 of the learning device acquires skin surface images, wrinkle grades of the skin surface images, and text describing the features of the skin surface images and / or the reasons for determining the wrinkle grade from training data as shown in Figure 6.
[0071] Figure 6 is a schematic diagram showing an example of training data. The "Image No." data item is used to identify skin surface images, is pre-configured, and is linked to the corresponding skin surface image data. The "wrinkle grade" data item indicates the wrinkle grade of the skin surface image. The "Image Description" data item is text data that includes a description of the features of the skin surface image and / or the reason for determining the wrinkle grade.
[0072] The control unit 17 of the learning device comprises a cross-modal processing unit 21 and a learning unit 22. The learning device's cross-modal processing unit 21 processes the acquired skin surface image with an image encoder to convert it into image features, processes the acquired text with a text encoder to convert it into text features, and performs cross-modal processing to fuse the image features and text features.
[0073] The learning unit 22 performs a decision using a loss function based on the features fused and output by cross-modal processing, and updates the parameters. Parameter updates continue until a predetermined number of training iterations are reached or the loss function converges. The parameter update status is saved periodically.
[0074] The learning device's storage unit 18 stores the parameters that have been updated through learning.
[0075] <Flowchart showing the processing flow of the wrinkle grade determination method> Here, Figure 3 is a flowchart showing an example of the wrinkle grade determination process in the wrinkle grade determination method of the present invention using the wrinkle grade determination device of the present invention. The process flow in the wrinkle grade determination method will be described below with reference to Figures 1 and 2.
[0076] In step S11, the acquisition unit 13 of the wrinkle grade determination device 11 acquires a skin surface image input by the user or a skin surface image input by the user and a question, and parameters that have been updated by learning using multiple training data including the skin surface image, the wrinkle grade of the skin surface image, and a description of the features of the skin surface image and / or the reason for determining the wrinkle grade, and then proceeds to S12. Parameters can be acquired as needed. When the learning device 12 acquires updated parameters through learning using multiple training data, including skin surface images, the wrinkle grade of the skin surface images, and descriptions of the features of the skin surface images and / or the reasons for determining the wrinkle grade, an accurate wrinkle grade determination answer can be obtained. However, if the parameters are not acquired, an accurate wrinkle grade determination answer may not be obtained.
[0077] In step S12, the control unit 14 of the wrinkle grade determination device 11 determines whether or not a skin surface image exists in the input information. If a skin surface image exists, the process proceeds to S13. If a skin surface image does not exist, the process ends.
[0078] In step S13, the control unit 14 of the wrinkle grade determination device 11 determines whether or not a question statement exists in the input information. If no question statement exists, the process moves to S14; if a question statement exists, the process moves to S15.
[0079] In step S14, the cross-modal processing unit 19 of the control unit 14 of the wrinkle grade determination device 11 processes the acquired skin surface image using an image encoder and converts it into image features, then proceeds to processing S16.
[0080] In step S15, the cross-modal processing unit 19 of the control unit 14 of the wrinkle grade determination device 11 processes the acquired skin surface image using an image encoder and converts it into image features. The acquired question text is also processed using a text encoder and converted into text features, after which the process proceeds to S16.
[0081] In step S16, the cross-modal processing unit 19 of the control unit 14 of the wrinkle grade determination device fuses image features and, if present, text features using parameters stored in the storage unit 18 of the learning device 12, and then proceeds to processing S17.
[0082] In step S17, the control unit 14 of the wrinkle grade determination device passes the feature quantities fused by the cross-modal processing unit 19 to the text generation unit 20. The language model of the text generation unit 20 generates text that includes an answer to the question, a description of the features of the skin surface image, and / or an explanation of the reason for determining the wrinkle grade of the skin surface image. Then, the process moves on to S18.
[0083] In step S18, the output unit 15 of the wrinkle grade determination device outputs the text generated by the text generation unit 20 as the answer, and this process ends.
[0084] <Flowchart showing the processing flow of the learning device> Here, Figure 7 is a flowchart showing an example of the learning process flow of the learning device in the wrinkle grade determination system of the present invention. The processing flow in the learning device 12 will be described below with reference to Figure 5.
[0085] In step S21, the acquisition unit 16 of the learning device 12 receives training data including a skin surface image, the wrinkle grade of the skin surface image, a description of the features of the skin surface image, and / or the reason for determining the wrinkle grade, and then proceeds to process S22.
[0086] In step S22, the cross-modal processing unit 21 of the control unit 17 of the learning device 12 processes the input training data—skin surface images and descriptive text for the skin surface images—using an image encoder and a text encoder, respectively, to convert them into image features and text features, and then proceeds to S23.
[0087] In step S23, the cross-modal processing unit 21 of the control unit 17 of the learning device 12 performs cross-modal processing on image features and text features, and then the process moves on to S24.
[0088] In step S24, the learning unit 22 of the control unit 17 of the learning device 12 updates the parameters by performing learning based on the features fused by the cross-modal processing unit 21, and then proceeds to S25. Learning in the learning unit 22 is performed using a loss function. Examples of loss functions include cross-entropy loss, binary cross-entropy loss, negative log-likehood loss, and contrastive loss.
[0089] In step S25, the control unit 17 of the learning device 12 determines whether or not to terminate the learning process. If the learning process is to be terminated, it proceeds to S26; if the learning process is not to be terminated, it proceeds to S21. The administrator specifies when to stop updating parameters in the learning device. For example, they might specify the number of training iterations. Note that the appropriate number of training iterations will vary depending on the model and training content.
[0090] In step S26, the updated parameters are saved by the storage unit 18 of the learning device 12, and the process ends. The parameters stored in the storage unit 18 are used, as needed, for wrinkle grade determination by the wrinkle grade determination device.
[0091] The following describes embodiments of the wrinkle grade determination device and the wrinkle grade determination method of the present invention.
[0092] <First Embodiment> As shown in Figure 8, when a user inputs a skin surface image and the question "Please tell me the grade of the wrinkles in this photograph and the reason why" into the acquisition unit 13 of the wrinkle grade determination device 11, the control unit 14 of the wrinkle grade determination device 11 generates text including the answer to the question, "The wrinkle grade is 0 because there are no wrinkles around the eyes," and this text is output by the output unit 15 of the wrinkle grade determination device 11.
[0093] According to the first embodiment, since there is a skin surface image and a question entered by the user, there is an advantage in that the accuracy of generating the answer intended by the user is improved.
[0094] <Second Embodiment> As shown in Figure 9, when a user inputs a skin surface image and the question "none" into the acquisition unit 13 of the wrinkle grade determination device 11, the control unit 14 of the wrinkle grade determination device 11 generates text that includes an explanation of the reason for the wrinkle grade determination of the skin surface image, such as "wrinkle grade 0" and "there are no noticeable wrinkles around the eyes, so this is a wrinkle grade 0 image," and this text is output by the output unit 15 of the wrinkle grade determination device 11.
[0095] If no question text exists (only a skin surface image is input), the artificial intelligence (AI) has been trained in advance using multiple training data sets that include skin surface images and their wrinkle grades, as well as descriptions of the skin surface image features and / or the reasons for determining the wrinkle grade. Therefore, an explanatory text, including descriptions of the skin surface image features and / or the reasons for determining the wrinkle grade, is generated from the input skin surface image. Therefore, in the second embodiment, compared to the first embodiment, only skin surface images are input from the user, which means that responses that are unintended or incomplete by the user may be output.
[0096] While several embodiments of the present invention have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These embodiments can be carried out in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims of the invention and its equivalents. [Explanation of symbols]
[0097] 11 Wrinkle grade determination device 12 Learning device 13 Acquisition Department 14 Control Unit 15 Output section 16 Acquisition Department 17 Control Unit 18 Preservation Department 19 Cross-modal processing unit 20 Text generation unit 21 Cross-modal processing unit 22 Learning Department 100 Wrinkle Grade Determination System
Claims
1. An acquisition unit that acquires a skin surface image entered by the user, or a skin surface image entered by the user and a question text, The acquired skin surface image is processed with an image encoder and converted into image features. If the aforementioned question exists, the question is processed using a text encoder to convert it into text features, and cross-modal processing is performed so that the text features play a leading role in the interaction with the image features, and features suitable for determining the acquired skin surface image and answering the question are fused. On the other hand, if the aforementioned question does not exist, cross-modal processing is performed so that features suitable for explaining the skin surface image acquired from the image features are fused. A control unit that generates text including an answer to a question, a description of the features of the skin surface image, and / or an explanation of the reason for determining the wrinkle grade of the skin surface image, based on the feature quantities fused by the cross-modal processing, An output unit that outputs the wrinkle grade of the skin surface image and the generated text as the answer, A wrinkle grade determination device characterized by comprising the following:
2. The wrinkle grade determination device according to claim 1, wherein the cross-modal processing in the control unit is performed using parameters that have been learned and updated using a plurality of training data including a skin surface image, the wrinkle grade of the skin surface image, and a description of the features of the skin surface image and / or the reason for determining the wrinkle grade.
3. Computers An acquisition step to acquire a skin surface image entered by the user, or a skin surface image entered by the user and a question text, The acquired skin surface image is processed with an image encoder and converted into image features. If the aforementioned question exists, the question is processed using a text encoder to convert it into text features, and cross-modal processing is performed so that the text features play a leading role in the interaction with the image features, and features suitable for determining the acquired skin surface image and answering the question are fused. On the other hand, if the aforementioned question does not exist, cross-modal processing is performed so that features suitable for explaining the skin surface image acquired from the image features are fused. A control step that generates text including an answer to a question, a description of the features of the skin surface image, and / or an explanation of the reason for determining the wrinkle grade of the skin surface image, based on the feature quantities fused by the cross-modal processing, Output process includes outputting the wrinkle grade of the skin surface image and the generated text as the answer, A method for determining wrinkle grade, characterized by performing the following.
4. On the computer, A data acquisition process to obtain a skin surface image entered by the user, or a skin surface image entered by the user and a question text, The acquired skin surface image is processed with an image encoder and converted into image features. If the aforementioned question exists, the question is processed using a text encoder to convert it into text features, and cross-modal processing is performed so that the text features play a leading role in the interaction with the image features, and features suitable for determining the acquired skin surface image and answering the question are fused. On the other hand, if the aforementioned question does not exist, cross-modal processing is performed so that features suitable for explaining the skin surface image acquired from the image features are fused. A control process that generates text including an answer to a question, a description of the features of the skin surface image, and / or an explanation of the reason for determining the wrinkle grade of the skin surface image, based on the feature quantities fused by the cross-modal processing, Output processing that outputs the wrinkle grade of the skin surface image and the generated text as the answer, A wrinkle grade determination program characterized by causing the following to be executed.
5. A learning device that updates parameters by learning using multiple training data including skin surface images, wrinkle grades of said skin surface images, descriptions of the features of said skin surface images and / or reasons for determining the wrinkle grade, and Wrinkle grade determination device that performs wrinkle grade determination, A wrinkle grade determination system having, The learning device is An acquisition unit that acquires a skin surface image, the wrinkle grade of the skin surface image, and text describing the features of the skin surface image and / or the reason for determining the wrinkle grade from the aforementioned training data. The acquired skin surface image is processed with an image encoder and converted into image features. The acquired text is processed with a text encoder and converted into text features. A control unit performs cross-modal processing to fuse the image features and the text features, and updates the parameters by learning based on the fused features. A storage unit that stores parameters updated through learning, Equipped with, The wrinkle grade determination device described above is An acquisition unit that acquires a skin surface image entered by the user, a skin surface image entered by the user and a question, a skin surface image entered by the user and the updated parameters, or a skin surface image entered by the user, a question, and the updated parameters. The acquired skin surface image is processed with an image encoder and converted into image features. If a question exists, the question is processed using a text encoder to convert it into text features. If acquired parameters exist, those parameters are used. If a question exists, the text features of the question primarily influence the interaction with the image features. Cross-modal processing is performed to merge features suitable for determining the acquired skin surface image and answering the question. If no question exists, cross-modal processing is performed to merge features suitable for explaining the acquired skin surface image from the image features. A control unit generates text that includes an answer to a question, a description of the features of the skin surface image, and / or an explanation of the reason for determining the wrinkle grade of the skin surface image, based on features merged by cross-modal processing. A wrinkle grade determination system characterized by comprising: an output unit that outputs a wrinkle grade of a skin surface image and generated text as the answer.
Citation Information
Patent Citations
Method of predicting skin condition using skin resident flora analysis
JP2019062887A
Wrinkle detection device, wrinkle detection method, and wrinkle detection program
JP2023044515A
