Information processing method, program, and information processing device

Abstract

To provide an information processing method, etc. capable of appropriately performing analysis on corneocytes from a captured image of a skin tissue.SOLUTION: An information processing method causes a computer to acquire an image obtained by imaging a skin tissue of a subject, identify a contour of each corneocyte by inputting the acquired image to a first model trained to identify a contour of each corneocyte including a portion overlapping with another corneocyte when the image is input, and calculate an overlap area ratio representing a ratio of an area of a portion where the corneocyte overlaps with another corneocyte to an area of the corneocyte, or an overlap distance representing a width of a portion where the corneocyte overlaps with another corneocyte on the basis of an identification result of the contour of each corneocyte.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to an information processing method, a program, and an information processing device. [Background technology]

[0002] With the advancement of artificial intelligence technology, methods have been proposed for detecting cells from images of biological tissues using machine learning models. For example, Patent Document 1 discloses a machine learning system that can distinguish and identify adjacent cells by constructing a semantic segmentation model using training data in which artificial markers are attached to cells that do not overlap with other cells for training images of multiple cells in culture. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-18531 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the invention of Patent Document 1 only learns images in which correct labels (artificial markers) are assigned only to cells that do not overlap with other cells, and therefore cannot detect cells that overlap with other cells.

[0005] In one aspect, an object of the present invention is to provide an information processing method and the like that can suitably perform analysis of stratum corneum cells from captured images of skin tissue. [Means for solving the problem]

[0006] In one aspect of the information processing method, a computer acquires an image of a subject's skin tissue, inputs the acquired image into a first model that has been trained to identify the contour of each stratum corneum cell, including the overlapping portion with other stratum corneum cells, and identifies the contour of each stratum corneum cell.Based on the identification results of the contour of each stratum corneum cell, a computer calculates an overlap area ratio, which represents the ratio of the area of ​​the stratum corneum cell that overlaps with other stratum corneum cells to the area of ​​the stratum corneum cell, or an overlap distance, which represents the width of the overlapping portion of the stratum corneum cell with other stratum corneum cells. [Effects of the Invention]

[0007] In one aspect, analysis of stratum corneum cells can be suitably performed from captured images of skin tissue. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is an explanatory diagram showing an example of the configuration of a stratum corneum cell analysis system. [Figure 2] FIG. 2 is a block diagram illustrating an example of the configuration of a server. [Figure 3] FIG. 2 is a block diagram illustrating an example of the configuration of a terminal. [Figure 4] FIG. 2 is an explanatory diagram showing an example of the record layout of an image DB and a subject DB. [Figure 5] FIG. 1 is an explanatory diagram of a first model. [Figure 6] FIG. 1 is an explanatory diagram of defective cells. [Figure 7] FIG. 1 is an explanatory diagram regarding the analysis process of stratum corneum cells. [Figure 8] FIG. 10 is an explanatory diagram showing an example of a display screen for the analysis results of stratum corneum cells. [Figure 9] FIG. 10 is an explanatory diagram showing an example of a display screen for the analysis results of stratum corneum cells. [Figure 10] FIG. 10 is an explanatory diagram showing an example of a display screen for the analysis results of stratum corneum cells. [Figure 11] 10 is a flowchart showing a procedure for generating a first model. [Figure 12]10 is a flowchart showing a procedure for generating a second model. [Figure 13] 10 is a flowchart showing the procedure of a process for analyzing stratum corneum cells. [Figure 14] FIG. 10 is an explanatory diagram showing an overview of a second embodiment. [Figure 15] 10 is a flowchart showing a processing procedure executed by a server according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] The present invention will be described in detail below with reference to the drawings showing embodiments thereof. (Embodiment 1) FIG. 1 is an explanatory diagram showing an example of the configuration of a stratum corneum cell analysis system. In this embodiment, a stratum corneum cell analysis system is described that analyzes the state of stratum corneum cells in an image captured from a sample of skin tissue from a subject. The stratum corneum cell analysis system includes an information processing device 1 and a terminal 2. Each device is connected to each other via a network N such as the Internet for communication.

[0010] The information processing device 1 is an information processing device capable of various information processing and sending and receiving information, such as a server computer, a personal computer, etc. In this embodiment, the information processing device 1 is assumed to be a server computer, and for the sake of brevity, will be referred to as server 1 below.

[0011] In this embodiment, server 1 functions as a generating device that generates a machine learning model that identifies the contours of each stratum corneum cell when an image of skin tissue is input by learning predetermined training data. Specifically, as described below, server 1 learns training data in which the contours of each stratum corneum cell, including portions overlapping with other stratum corneum cells, are labeled for a group of training images of skin tissue, thereby generating a first model 51 (see FIG. 5) that can identify the contours of stratum corneum cells even in portions overlapping with other stratum corneum cells. By making it possible to identify the contours even in portions overlapping with other stratum corneum cells, it becomes possible to accurately analyze the area, circularity, etc. of stratum corneum cells, as described below.

[0012] The server 1 also functions as an analyzer that identifies the contour of each stratum corneum cell from an image of a skin tissue sample taken from any subject and analyzes the state of the stratum corneum cells using the first model 51. Specifically, as described below, the server 1 calculates various parameters that represent the state of the stratum corneum cells (area, circularity, number of corners, etc. of the stratum corneum cells) based on the contour of the stratum corneum cells identified by the first model 51.

[0013] Terminal 2 is an information processing terminal of a user of the system, such as a personal computer or tablet terminal. The user of the system is assumed to be a person who observes the skin condition of a subject (e.g., a cosmetics manufacturer, a researcher, etc.), but the user may also be the subject himself / herself. For example, terminal 2 is pre-installed with first model 51 generated by server 1, and terminal 2 functions as an analysis device that uses first model 51 to identify the contours of each stratum corneum cell from an image of a skin tissue sample taken from a subject and analyze the condition of the stratum corneum cell. Specifically, as described below, terminal 2 calculates various parameters representing the condition of the stratum corneum cell (e.g., area, circularity, number of corners, etc.) based on the contours of the stratum corneum cell identified by first model 51.

[0014] For example, a user collects skin tissue from a subject using a stratum corneum collecting tape, stains the collected sample, and captures an image using a microscope 3. The terminal 2 identifies the contours of each stratum corneum cell by inputting the image captured by the microscope 3 into the first model 51. The terminal 2 then calculates various parameters based on the identified contours of the stratum corneum cells and displays them as analysis results.

[0015] In this embodiment, the generation (learning) of the first model 51 and the identification of stratum corneum cells using the first model 51 are performed by the server 1 and the terminal 2, respectively, but both processes may be performed by the same computer. For example, the server 1 may obtain a captured image from the terminal 2 via the network N, identify the contours of the stratum corneum cells using the first model 51, and output the identification result to the terminal 2.

[0016] In addition, in this embodiment, the subject's skin tissue is sampled using tape, but the means for sample collection is not limited to tape. In this embodiment, the stratum corneum cells are observed after staining, but a method that does not require staining (e.g., autofluorescence observation) may also be used. Instead of sampling skin tissue, the subject's skin (skin tissue) may be directly imaged (in which case, the observation method may be adjusted accordingly).

[0017] 2 is a block diagram showing an example of the configuration of the server 1. The server 1 includes a control unit 11, a main memory unit 12, a communication unit 13, and an auxiliary memory unit . The control unit 11 is a processor such as one or more CPUs (Central Processing Units), MPUs (Micro-Processing Units), GPUs (Graphics Processing Units), etc., and performs various information processing, control processing, etc. by reading and executing a program P1 stored in the auxiliary storage unit 14. The main storage unit 12 is a temporary storage area such as SRAM (Static Random Access Memory), DRAM (Dynamic Random Access Memory), or flash memory, and temporarily stores data required for the control unit 11 to execute arithmetic processing. The communication unit 13 is a communication module for performing communication-related processing, and transmits and receives information to and from the outside. The auxiliary storage unit 14 is a non-volatile storage area such as a large-capacity memory or a hard disk, and stores the program P1 and other data required for the control unit 11 to execute processing.

[0018] The auxiliary storage unit 14 may be an external storage device connected to the server 1. The server 1 may be a multi-computer consisting of multiple computers, or may be a virtual machine virtually constructed by software.

[0019] The server 1 may also include a reading unit that reads a non-transitory computer-readable recording medium 1a and reads the program P1 (program product) from the recording medium 1a. The program P1 may be executed on a single computer or on multiple computers interconnected via a network N.

[0020] 3 is a block diagram showing an example of the configuration of the terminal 2. The terminal 2 includes a control unit 21, a main memory unit 22, a communication unit 23, a display unit 24, an input unit 25, and an auxiliary memory unit . The control unit 21 has one or more processors such as CPUs, and performs various information processing by reading and executing a program P2 stored in the auxiliary storage unit 26. The main storage unit 22 is a temporary storage area such as RAM, and temporarily stores data necessary for the control unit 21 to execute arithmetic processing. The communication unit 23 is a communication module for performing communication-related processing, and transmits and receives information to and from the outside. The display unit 24 is a display screen such as a liquid crystal display, and displays images. The input unit 25 is an operation interface such as a keyboard or mouse, and receives operation input from the user.

[0021] Auxiliary storage unit 26 is a non-volatile storage area such as a hard disk or large-capacity memory, and stores program P2 and other data required for control unit 21 to execute processing. Auxiliary storage unit 26 also stores first model 51, second model 52, image DB 261, and subject DB 262.

[0022] The first model 51 is a machine learning model that has been trained with predetermined training data, and when an image of skin tissue is input, the model identifies the contours of each stratum corneum cell in the image. The second model 52 is a machine learning model that has been trained with predetermined training data, and when an image of skin tissue is input, the model determines whether each stratum corneum cell in the image is a stratum corneum cell with a shape defect (hereinafter referred to as a "defective cell"). The first model 51 and the second model 52 are expected to be used as program modules that constitute part of artificial intelligence software.

[0023] The image DB 261 is a database that stores image data of the skin tissue of the subject. The subject DB 262 is a database that stores data related to the subject. Specifically, as described below, the subject DB 262 stores skin data of the subject (such as responses to a questionnaire about skin conditions and measurement data of the skin tissue).

[0024] The terminal 2 may include a reading unit that reads a non-transitory computer-readable recording medium 2a and reads the program P2 (program product) from the recording medium 2a. The program P2 may be executed on a single computer or on multiple computers interconnected via a network N.

[0025] FIG. 4 is an explanatory diagram showing an example of the record layout of the image DB 261 and the subject DB 262. The image DB 261 includes an image ID column, an image column, and an analysis data column. The image ID column stores an image ID, which is an identifier for a captured image. The image column and the analysis data column store, in association with the image ID, an image of skin tissue and various parameters of stratum corneum cells analyzed from the image. For example, the analysis data column stores parameters calculated on an image-by-image basis (e.g., the number of stratum corneum cells in the image, total area, average area, etc.) and parameters calculated on a cell-by-cell basis (e.g., the area, perimeter, circularity of each stratum corneum cell). Furthermore, the analysis data column stores a flag (a value of 0 or 1) for each stratum corneum cell indicating whether the stratum corneum cell is a defective cell.

[0026] The subject DB 262 includes a subject ID column, a subject name column, and a skin data column. The subject ID column stores the subject ID, which is an identifier for the subject. The subject name column and the skin data column store the subject name and the subject's skin data, respectively, in association with the subject ID. The skin data is an index representing the condition of the subject's skin tissue, and is data that can be determined from sources other than an image. For example, the skin data column stores responses to a questionnaire given to the subject (age, gender, etc.) and data obtained by measuring the subject's skin (moisture content, TEWL (Trans Epidermal Water Loss), etc.).

[0027] Fig. 5 is an explanatory diagram of the first model 51. Fig. 5 conceptually illustrates how the contours of each stratum corneum cell in an image are identified when an image of skin tissue is input into the first model 51. The first model 51 will be described with reference to Fig. 5.

[0028] In Figure 5, for convenience, each stratum corneum cell is depicted as a regular hexagon, but in reality, stratum corneum cells are observed in a variety of shapes. Also, in Figure 5, only the outlines of the stratum corneum cells are depicted, but in reality, the stratum corneum cells are stained and observed in a different color from the rest of the stratum corneum.

[0029] The first model 51 is a machine learning model that has learned predetermined training data, and is, for example, a neural network constructed by deep learning. In this embodiment, the server 1 constructs, as the first model 51, a Mask R-CNN (Region-based Convolutional Neural Network) that can simultaneously classify objects appearing in an image and identify (segment) the objects on a pixel-by-pixel basis.

[0030] The first model 51 may be a neural network other than Mask R-CNN. Also, the first model 51 may be a machine learning model other than a neural network, such as a decision tree, a random forest, or a support vector machine (SVM).

[0031] The server 1 generates a first model 51 using training data in which correct labels indicating the contours of each stratum corneum cell are assigned to a group of training images of skin tissue. The training images are images of samples of skin tissue from one or more people (e.g., past test subjects). The correct labels are labels (lines) assigned to each stratum corneum cell in the images, tracing the contours of the stratum corneum cell. The training labels are assigned (drawn) by a designated operator.

[0032] The correct labels may be assigned not by lines but by filling the entire interior of each stratum corneum cell outline with a different color, etc. Also, the correct labels do not need to be assigned completely manually; for example, an operator may manually correct the outline of a stratum corneum cell extracted by image analysis software.

[0033] Here, labels of correct contour lines in the training data are also assigned to the contours of portions that overlap with other stratum corneum cells. The left and right sides of Figure 5 illustrate how stratum cells overlap. As shown in Figure 5, some stratum cells do not overlap with other stratum cells, while others do. In this embodiment, labels are also assigned to the contours of portions hidden under other stratum cells, making it possible to identify the contours of those portions.

[0034] The server 1 inputs training images into the first model 51 to identify the contours of each stratum corneum cell in the image. The server 1 compares the identified contours with the correct labels and updates parameters such as the weights between neurons so that the two are similar. The server 1 sequentially supplies each training image to the first model 51 to update the parameters, and finally generates the first model 51 with optimized parameters. The first model 51 generated by the server 1 is installed on the terminal 2.

[0035] When analyzing the skin (stratum corneum cells) of a subject, terminal 2 inputs a captured image of the subject's skin tissue into first model 51 to identify the contour of each stratum corneum cell. The right side of Figure 5 conceptually illustrates the results of identifying the contour of the stratum corneum cell. First model 51 outputs a label (shown by a thick line) representing the contour of the stratum corneum cell and a probability value (a number between 0 and 1; not shown in Figure 5) representing the likelihood of the contour identification result. As shown in Figure 5, first model 51 identifies the contour of each stratum corneum cell, including any overlapping portions with other stratum corneum cells.

[0036] In this case, terminal 2 excludes the stratified stratum corneum from the analysis target based on the probability value output from first model 51. The stratified stratum corneum refers to a portion of the collected sample where stratum corneum cells are overlapped in multiple layers. In this specification, the terms "overlap" and "overlapping" mean that stratum corneum cells are partially overlapping, and "stratified" means that two or more stratum corneum cells are overlapping in a hierarchical relationship.

[0037] In Figure 5, the stratum corneum is indicated by a dotted line. The stratum corneum is observed when a large number of stratum corneum cells are collected together during skin tissue collection (exfoliation). In actual images, the stratum corneum appears black and cloudy, making it difficult to distinguish between individual stratum corneum cells. For example, server 1 compares the probability value output from first model 51 with a predetermined threshold, and determines that stratum corneum cells below the threshold are stratum corneum cells, thereby excluding the stratum corneum from the analysis target.

[0038] Terminal 2 analyzes the state of the stratum corneum cells based on the contours identified above. Prior to this, terminal 2 uses second model 52 to determine whether the stratum corneum cells identified by first model 51 are defective cells, and excludes stratum corneum cells determined to be defective cells from the analysis target.

[0039] Fig. 6 is an explanatory diagram of defective cells. Fig. 6A shows normal stratum corneum cells with no shape defects, and Figs. 6B and 6C show defective cells. The process of determining defective cells using second model 52 will be described with reference to Fig. 6.

[0040] Defective cells are keratinocytes with a defective shape, such as keratinocytes that have been torn into two or more pieces (Figure 6B) or partially folded (Figure 6C). While Figure 6B illustrates keratinocyte fragments that have been torn into two or more pieces, there are cases in which only a portion of the cell fragments remains in the image due to reasons such as only a portion of the cell fragments adhering to the tape. In addition to the examples shown in Figures 6B and 6C, defective cells also include keratinocytes with partial holes or wrinkled keratinocytes. Defective keratinocytes occur due to factors such as the procedure used when collecting skin tissue (tape removal). These defective cells do not accurately represent the condition of the subject's skin (stratum corneum), as can be seen by comparing them with normal keratinocytes (Figure 6A).

[0041] Therefore, terminal 2 excludes defective cells from the analysis target and performs analysis on the remaining stratum corneum cells. Specifically, when a captured image of skin tissue (stratum corneum cells) is input, terminal 2 determines whether the stratum corneum cells in the image are defective cells using second model 52 that has been trained to determine whether the stratum corneum cells in the image are defective cells.

[0042] The second model 52 is a machine learning model that has learned predetermined training data, and is, for example, a decision tree (a binary tree in this embodiment). Similar to the first model 51, the second model 52 is generated by the server 1 learning the training data. Note that the second model 52 is not limited to a decision tree, and may be another machine learning model such as an SVM, a random forest, or a neural network.

[0043] Second model 52 may be a model that treats the determination of whether or not a stratum corneum cell is a defective cell as a classification problem, or may be a model that treats it as a regression problem that calculates the probability that the cell is a defective cell.

[0044] Furthermore, second model 52 may be a model that not only determines whether or not a cell is defective, but also determines the type of defective cell (torn, bent, etc.).

[0045] The training data is data in which a group of training images of skin tissue are assigned correct labels indicating whether or not the stratum corneum cells shown in the images are defective cells. The training images are a group of skin tissue images of one or more individuals (e.g., past subjects). The correct labels are binary labels indicating whether or not the cells are defective. As will be described later, in this embodiment, when the stratum corneum cells of a subject are analyzed, the user viewing the analysis results designates defective cells to perform labeling (see FIG. 10).

[0046] In this embodiment, features of stratum corneum cells that can be extracted (calculated) from a captured image are used as input to the second model 52. These features may include parameters that represent the state of the stratum corneum cells, such as the area and circularity of the stratum corneum cells, as well as pixel values ​​(color, etc.) of the image. Parameters such as area and circularity will be described later. These are only examples of features of stratum corneum cells, and other parameters (such as diameter, circularity, and number of corners, which will be described later) may also be included. Alternatively, the image itself, rather than the features, may be used as input to the second model 52.

[0047] The server 1 extracts features of each stratum corneum cell from a group of training images and uses them as a dataset for constructing a decision tree. The server 1 divides the dataset based on a predetermined reference value (such as impurity) to construct a decision tree structure consisting of multiple nodes. The server 1 generates a second model 52 by repeatedly constructing the decision tree structure recursively. The second model 52 generated by the server 1 is installed on the terminal 2.

[0048] When the contours of stratum corneum cells are identified by the first model 51, the terminal 2 extracts the feature values ​​of each stratum corneum cell whose contours have been identified. The terminal 2 then inputs the feature values ​​into the generated second model 52 to determine whether each stratum corneum cell is a defective cell. When the terminal 2 determines that each stratum corneum cell is a defective cell, it excludes the stratum corneum cell determined to be a defective cell from the analysis target (the stratum corneum cell used as the basis for calculating parameters).

[0049] In this embodiment, defective cells are excluded from the analysis target, but defective cells may also be included in the analysis target. For example, terminal 2 may predict the shape of defective cells identified from an image when no defect occurs using a machine learning model (second model 52 or another model) that has learned the shape (contour) of a cell when no defect occurs, and perform analysis based on the predicted results. In this way, not only may defective cells be excluded from the analysis target, but also defective cells may be included in the analysis.

[0050] Fig. 7 is an explanatory diagram of the analysis process of stratum corneum cells. Terminal 2 calculates various parameters that represent the state of the subject's stratum corneum cells based on the outline of the remaining stratum corneum cells excluding the defective cells. Each parameter calculated by terminal 2 will be explained using Fig. 7.

[0051] Based on the contours of individual stratum corneum cells, terminal 2 calculates parameters representing the state of each stratum corneum cell, specifically the area, diameter (see FIG. 7A), circularity (see FIG. 7A), circularity, number of corners, tilt angle (see FIG. 7B), and stratum corneum detection rate (see FIG. 7C). Based on the contours of multiple overlapping stratum corneum cells, terminal 2 also calculates parameters representing the degree of overlap of stratum corneum cells, specifically the overlap area rate (see FIG. 7D) and overlap distance (see FIG. 7E).

[0052] It should be noted that the parameters exemplified above are merely examples, and the parameters may be any parameters that represent the state of stratum corneum cells.

[0053] The area of ​​the stratum corneum cells is the area of ​​the image region of the stratum corneum cells that occupies the image (μm 2 ) The terminal 2 calculates the area of ​​each stratum corneum cell and the image average of the stratum corneum cell area (the average value of all stratum corneum cells in the image). Note that, although an example of calculating the area has been shown in this embodiment, the present invention is not limited to this. The terminal 2 may be configured to output the number of pixels of each layer cell. The terminal 2 may also calculate the area of ​​each layer cell by multiplying the number of pixels by the area per pixel calculated in advance from the lens magnification, image resolution, etc.

[0054] The diameter of a stratum corneum cell is the diameter (or radius) of the circumscribing circle and / or inscribing circle of the stratum corneum cell. The circumscribing circle and inscribing circle refer to the circumscribing circle and inscribing circle that are the smallest distance between two concentric circles when the stratum corneum cell is sandwiched between the inside and outside of the two concentric circles. In Figure 7A, the diameter of the circumscribing circle is shown as the "major diameter" and the diameter of the inscribing circle is shown as the "minor diameter." Alternatively, the stratum corneum cell can be approximated as an ellipse, and the major and minor diameters of the ellipse can be considered as the major and minor diameters of the stratum corneum cell. Terminal 2 calculates the major and minor diameters of each stratum corneum cell and the image average of the major and minor diameters.

[0055] Circularity is a numerical value that represents the degree of deviation from a geometrically correct circle. As shown in FIG. 7A, terminal 2 calculates circularity by dividing the difference in diameter between the circumscribing circle and the inscribing circle (the two concentric circles that sandwich a stratum corneum cell from the inside and outside, and determine the minimum distance between them) by 2. Alternatively, circularity can be determined as the ratio or difference between the major and minor axes of an elliptical stratum corneum cell. Server 1 calculates the circularity of each stratum corneum cell and the image average of circularity.

[0056] Circularity is a numerical value that indicates the complexity of the shape, and is calculated as follows: Circularity = 4π × S ÷ L 2 (S is the area of ​​the stratum corneum cell, and L is the perimeter of the stratum corneum cell). Terminal 2 calculates the circularity of each stratum corneum cell and the image average of the circularity.

[0057] The number of corners is the number of vertices when each stratum corneum cell is approximated as a polygon. For example, terminal 2 approximates the stratum corneum cell as a polygon using the Dauglas-Peucker algorithm and counts the number of corners. Terminal 2 calculates the number of corners for each stratum corneum cell and the ratio of each corner number in the image (the ratio of the number of corners = n (n is an integer of 3 or more) stratum corneum cells to the total number of stratum corneum cells).

[0058] The tilt angle is the angle between the line connecting the two most distant points on the contour of the stratum corneum cell and the horizon (a line parallel to the horizontal axis of the image), and represents the directionality of the stratum corneum cell. The tilt angle is illustrated in Figure 7B. Alternatively, the tilt angle can be determined as the angle between the long axis of the stratum corneum cell, which is approximated as an ellipse, and the horizon. Device 2 calculates the tilt angle for each stratum corneum cell and the image average of the tilt angle.

[0059] The stratum corneum detection rate is the percentage of the area of ​​stratum corneum cells detected (identified) as normal cells among the stratum corneum cells in an image. In Figure 7C, the outlines of stratum corneum cells detected as normal cells ("detected cells" in Figure 7C) are shown with solid lines, and the outlines of other stratum corneum cells ("non-detected cells") are shown with dotted lines. Terminal 2 calculates the stratum corneum detection rate by dividing the total area of ​​normal stratum corneum cells in the image that do not fall into the category of stratified stratum corneum, defective cells, etc. by the total area of ​​all stratum corneum cells. The area of ​​all stratum corneum cells is extracted by comparing the pixel value with a predetermined threshold.

[0060] The overlap area ratio is the ratio of the area of ​​the portion overlapping with other stratum corneum cells to the area of ​​the stratum corneum cells. Figure 7D illustrates how the overlap area ratio is calculated. When calculating the overlap area ratio of the stratum corneum cells on the right side of Figure 7D, the ratio of the area of ​​the overlapping portion indicated by symbol B to the area of ​​the stratum corneum cells indicated by symbol A is calculated. Alternatively, the overlap area ratio can be calculated as the ratio of the area of ​​the overlapping portion to the total area of ​​the two overlapping cells. Terminal 2 calculates the overlap area ratio for each stratum corneum cell and the image average of the overlap area ratio.

[0061] The overlap distance is the distance representing the width of the overlapping portion of a corneocyte with another corneocyte. Specifically, when a line is drawn connecting the two points where two corneocytes begin to overlap, the overlap distance is the distance between two parallel lines that are parallel to the line and pass through the two furthest overlapping points in a direction perpendicular to the line. Figure 7E illustrates how the overlap distance is calculated. The terminal 2 identifies the line connecting the two points where the two corneocytes begin to overlap (shown as a dotted line in Figure 7E) and identifies two parallel lines parallel to the line. In this case, the terminal 2 identifies the two furthest points (the lower left and upper right points in Figure 7E) from the overlapping portion of the two corneocytes in a direction perpendicular to the original line (dotted line) and identifies parallel lines passing through the two points. The terminal 2 calculates the distance between the two parallel lines as the overlap distance. The terminal 2 calculates the overlap distance for each stratum corneum cell and the image average of the overlap distance.

[0062] Furthermore, the terminal 2 calculates the variation (standard deviation, variance, coefficient of variation) for each of the above parameters within one image and among multiple images.

[0063] In this way, terminal 2 calculates parameters that represent the state of each individual stratum corneum cell (area, diameter, circularity, etc.), as well as parameters that represent the degree of overlap between stratum corneum cells (overlap area ratio, overlap distance). As described above, first model 51 identifies the contour of a stratum corneum cell, including the portions that overlap with other stratum corneum cells. Therefore, when calculating the area, diameter, circularity, etc. of each individual stratum corneum cell, each parameter can be suitably calculated taking into account the portions that overlap with other stratum corneum cells. Furthermore, by calculating the overlap area ratio, overlap distance, etc., it is possible to not only analyze the state of each individual stratum corneum cell, but also to analyze the degree to which stratum corneum cells overlap with each other.

[0064] 8 to 10 are explanatory diagrams showing examples of display screens of the analysis results of stratum corneum cells. In Fig. 8 to Fig. 10, examples of screens of the analysis results of stratum corneum cells are shown as screens displayed by terminal 2.

[0065] For example, terminal 2 displays a screen including a list display field 81 and a parameter display field 82. The list display field 81 is a display field showing a list of analysis targets, the analysis results of which are displayed on the right side of the screen. In this system, the analysis targets are folders that store a group of images of skin tissue (for example, multiple images taken of one subject in a single examination), the images in the folders, and each stratum corneum cell in the images. Terminal 2 accepts a selection input for the display target of the analysis results via the list display field 81, and displays the analysis results for the selected target. Figures 8 to 10 respectively show example screens when a folder, an image, and a stratum corneum cell are selected.

[0066] 8, when a folder is selected, terminal 2 displays the analysis parameters of the image group stored in that folder in parameter display field 82. Specifically, terminal 2 displays a list of the number of stratum corneum cells in the image, the total area of ​​the stratum corneum cells in the image, the average area of ​​the stratum corneum cells (image average), the average perimeter of the stratum corneum cells, etc. for each image.

[0067] 9, when an image in a folder is selected, terminal 2 displays the analysis parameters of the stratum corneum cells in the image in parameter display field 82. Specifically, terminal 2 displays a list of the area, perimeter, circularity, etc. of each stratum corneum cell.

[0068] 9, terminal 2 displays captured images in which labels with different display styles are superimposed on each identified stratum corneum cell in an image display field 91 below the list display field 81. Specifically, terminal 2 superimposes labels with different display colors (and bounding boxes surrounding the stratum corneum cells) on the area inside the contour of each stratum corneum cell. For convenience of illustration, in FIG. 9, the colors are shown hatched, and each stratum corneum cell is hatched with the same hatching. Terminal 2 superimposes the labels according to the output from first model 51, and generates and displays an image related to so-called segmentation.

[0069] 9 is an example, and the present embodiment is not limited to this. For example, terminal 2 may not only superimpose a label on each stratum corneum cell, but also display a probability value indicating the likelihood of the identification result for the contour of each stratum corneum cell. This allows the user to know how likely the identification result is.

[0070] 10, when a stratum corneum cell is selected in the image, terminal 2 displays the analysis parameters of the stratum corneum cell in parameter display field 82. Specifically, similar to the example screen of FIG. 9, terminal 2 displays a list of the area, perimeter, circularity, etc. of the stratum corneum cell.

[0071] Terminal 2 also displays a defect designation field 101 below the parameter display field 82. The defect designation field 101 is an input field for designating (registering) a stratum corneum cell whose contour has been identified by first model 51 as a defective cell (negative example). For example, as shown in FIG. 10, terminal 2 displays, in the defect designation field 101, an enlarged image (unlabeled) of the stratum corneum cell selected in list display field 81 and an enlarged image of the stratum corneum cell with a label attached. Terminal 2 then accepts designation input as to whether or not the stratum corneum cell shown in the enlarged image corresponds to a defective cell in response to an operation input to a registration button 1011. Note that multiple registration buttons 1011 are provided, and by accepting an operation input to any of the registration buttons 1011, terminal 2 also accepts designation input of the type of defective cell ("reason 1" or "reason 2" in FIG. 10).

[0072] skin 10 shows an example in which an enlarged image is displayed, but it is also possible to simply display an image in which labels are superimposed on stratum corneum cells without enlarging the image. Also, while FIG. 10 shows two images, one unlabeled and one labeled, terminal 2 may simply display the labeled image, or may be able to switch the label display on and off in response to a user operation input. That is, terminal 2 is only required to display at least an image in which labels are superimposed on stratum corneum cells and to accept input specifying missing cells, and the specific display and operation methods are not particularly limited.

[0073] When the terminal 2 receives the input specifying a defective cell, the terminal 2 associates the identified stratum corneum cells from the image with a flag indicating whether the stratum corneum cells are defective cells and stores the flag in the image DB 261 (see FIG. 4). The server 1 labels the captured image as a defective cell based on the above specification, and uses the label as training data for the second model 52. That is, the server 1 assigns a label to the image indicating that the stratum corneum cells specified as defective cells are defective cells, and uses the label as training data for generating the second model 52. The server 1 generates the second model 52 based on the training data stored in response to the user's specification.

[0074] As described above, according to this embodiment, stratum corneum cells can be suitably analyzed while taking into consideration the overlap of stratum corneum cells.

[0075] 11 is a flowchart showing the procedure of the process for generating the first model 51. The process for generating the first model 51 by machine learning will be described with reference to FIG. The control unit 11 of the server 1 acquires training data for generating the first model 51 (step S11). The training data is data in which labels are added to the contours of each stratum corneum cell, including portions that overlap with other stratum corneum cells, for a group of images of skin tissue.

[0076] Based on the training data, the control unit 11 generates a first model 51 that identifies the contour of each stratum corneum cell, including overlapping portions with other stratum corneum cells, when an image of skin tissue is input (step S12). Specifically, the control unit 11 generates Mask R-CNN as the first model 51. The control unit 11 inputs training images to the first model 51 to identify the contour of each stratum corneum cell. The control unit 11 optimizes parameters such as weights between neurons so that the stratum corneum cell identification results approximate the correct labels, and generates the first model 51. The control unit 11 then terminates the series of processes. Note that in this embodiment, the first model 51 and the second model 52 are configured separately, but similar estimation may also be performed using a single first model 51. Specifically, when an image is input, the first model 51 outputs a cell contour region and a missing cell region. In this case, the control unit 11 uses the label data of the contour region and the label data of the cell-defective region, and the image, to train the first model 51 using a segmentation network such as Mask R-CNN or U-NET.

[0077] 12 is a flowchart showing the procedure of the process for generating the second model 52. The process for generating the second model 52 will be described with reference to FIG. The control unit 11 of the server 1 acquires training data for generating the second model 52 (step S31). The training data is data in which a group of images of skin tissue is labeled with a label indicating whether each stratum corneum cell in the image is a defective cell having a shape defect. As described above, the label is assigned by a user who views the stratum corneum cell identification result and designates a defective cell.

[0078] Based on the training data, the control unit 11 generates a second model 52 that, when an image of skin tissue is input, determines whether each stratum corneum cell in the image is a defective cell (step S32). For example, the control unit 11 generates a decision tree as the second model 52. The control unit 11 extracts features (area, circularity, pixel value, etc.) of each stratum corneum cell from the training image and divides the data set consisting of the features of each stratum corneum cell according to a reference value such as impurity, thereby constructing a decision tree structure. The control unit 11 generates the second model 52 by recursively repeating the construction of the decision tree structure. The control unit 11 ends the series of processes.

[0079] 13 is a flowchart showing the procedure for analyzing stratum corneum cells. The processing steps for analyzing stratum corneum cells from captured images of stratum corneum cells will be described with reference to FIG. The control unit 21 of the terminal 2 acquires an image of the subject's skin tissue from the terminal 2 (step S51). The control unit 21 inputs the acquired image into the first model 51, thereby identifying the contour of each stratum corneum cell, including the portion where the stratum corneum cell overlaps with other stratum corneum cells (step S52).

[0080] The control unit 21 determines whether each stratum corneum cell whose contour has been identified is a defective cell or not, and excludes the defective cell from the analysis target (step S53). Specifically, the control unit 21 extracts a feature amount of each stratum corneum cell from the image, and inputs the extracted feature amount into the second model 52 to determine whether the cell is a defective cell or not.

[0081] Based on the results of identifying the contours of each stratum corneum cell other than the defective cells excluded in step S53, control unit 21 calculates parameters representing the state of the stratum corneum cells (step S54). These parameters include parameters representing the state of each stratum corneum cell, such as area and circularity, as well as parameters representing the degree of overlap between stratum corneum cells, such as overlap area ratio and overlap distance.

[0082] Control unit 21 displays the results of identifying the contours of the stratum corneum cells and the parameters calculated in step S54 (step S55). Specifically, as described above, control unit 21 displays an image in which labels are superimposed on each stratum corneum cell, and also displays a list of various parameters. Control unit 21 accepts input specifying stratum corneum cells that correspond to defective cells (step S56). Control unit 21 stores the image acquired in step S51, the parameters calculated in step S54, and a flag indicating whether or not the cell is defective in image DB 261 (step S57), and ends the series of processes.

[0083] As described above, according to the first embodiment, analysis of stratum corneum cells can be suitably performed from an image of the skin tissue of a subject.

[0084] (Embodiment 2) In the first embodiment, a configuration was described in which parameters representing the state of stratum corneum cells were calculated from a captured image of a subject's skin tissue. In the present embodiment, a configuration is described in which the correlation between the stratum corneum cell parameters calculated from the captured image and the subject's skin data other than the image is analyzed. Note that the same reference numerals are used to designate content that overlaps with the first embodiment, and description thereof will be omitted.

[0085] Fig. 14 is an explanatory diagram showing an overview of embodiment 2. Fig. 14 illustrates a state in which a graph 1402 showing the correlation between a parameter of a stratum corneum cell (the area of ​​the stratum corneum cell in Fig. 14) and skin data (TEWL) is displayed on the screen shown in Fig. 8. An overview of this embodiment will be described based on Fig. 14.

[0086] The skin data represents the condition of the subject's skin tissue. In this embodiment, the skin data includes responses to a questionnaire administered to the subject in advance and measurement data of the skin tissue. The questionnaire is a questionnaire about the condition of the skin, and includes questions about, for example, age, gender, frequency and method of skin care, skin type (oily skin, dry skin, etc.), stress level (stress check questions), and other skin concerns. The measurement data is data obtained by measuring the condition of the skin tissue using a predetermined measurement method, and includes, for example, the amount of moisture in the stratum corneum (measured from the electrical properties (electrical conductivity, capacitance) and optical properties (the movement of water molecules measurable by Raman spectroscopy, etc.) of the skin), TEWL (measured by a TEWA meter), wrinkles (measured by a wrinkle measurement device), age spots, skin gloss, transparency, brightness, redness, yellowness, pores, texture, dullness, and uneven skin tone (measured by facial photography, a colorimeter, etc.). Note that these are examples of skin data, and the skin data is not limited to the above.

[0087] Terminal 2 stores these skin data in advance in the subject DB 262. Terminal 2 analyzes the correlation between the skin data and parameters of stratum corneum cells (area, circularity, etc.) analyzed from an image of skin tissue. While the specific analysis method is not particularly limited, for example, terminal 2 analyzes the correlation between the two by performing a regression analysis using the parameters of stratum corneum cells as explanatory variables and the skin data as a response variable.

[0088] For example, when terminal 2 displays the analysis results of stratum corneum cells as in Fig. 8 etc., it also displays the skin data of the subject as shown in Fig. 14. Specifically, terminal 2 displays a list of skin data in a skin data display field 1401 located below parameter display field 82. For example, when terminal 2 receives a selection input of one or more folders (subjects) in list display field 81, it causes skin data display field 1401 to appear and displays a list of skin data of one or more subjects corresponding to the selected folders.

[0089] Terminal 2 also displays the analysis results of the correlation between the skin data and the analysis parameters of the stratum corneum cells. For example, terminal 2 displays graph 1402 showing the correlation between the two, as shown in Fig. 14. Graph 1402 has the stratum corneum cell parameters on the horizontal axis and the skin data on the vertical axis. Each plot in graph 1402 represents a subject, and the straight line in graph 1402 represents an approximation line based on the analysis results.

[0090] For example, the terminal 2 accepts selection inputs for selecting parameters to be used as explanatory variables and skin data to be used as objective variables from the parameter display field 82 and the skin data display field 1401. The terminal 2 displays a graph 1402 based on the analysis results of the correlation between the selected parameters and skin data. Note that while Fig. 14 illustrates a case where an analysis (simple regression) is performed using a single explanatory variable and objective variable, an analysis (multiple regression) using a plurality of explanatory variables and / or objective variables may also be performed.

[0091] As described above, according to this embodiment, by combining the analysis results (parameters) of stratum corneum cells using first model 51 with skin data, it is possible to examine the correlation between the state of stratum corneum cells and the state of skin.

[0092] 15 is a flowchart showing the processing procedure executed by the server 1 according to embodiment 2. After calculating the parameters representing the state of the stratum corneum cells (step S54), the terminal 2 executes the following processing. The control unit 21 of the terminal 2 acquires skin data of the subject from the subject DB 262 (step S201). The skin data is data that represents the condition of the subject's skin tissue, and includes the subject's responses to a questionnaire regarding the skin condition and / or measurement data of the skin tissue.

[0093] The control unit 21 displays the results of identifying the contours of the stratum corneum cells, various parameters analyzed from the image, etc. (Step S202) In this case, the control unit 21 also displays a list of the skin data of the subject.

[0094] The control unit 21 analyzes the correlation between the subject's skin data and each parameter and displays the analysis results (step S203). Specifically, as described above, the control unit 21 accepts a selection input for selecting the stratum corneum cell parameters as explanatory variables and the skin data as objective variables, and analyzes (e.g., regression analysis) the correlation between the selected parameters and the skin data. The control unit 21 displays a graph 1402 showing the correlation between the two. The control unit 21 proceeds to step S56.

[0095] As described above, according to the second embodiment, it is possible to objectively examine the influence of the state of stratum corneum cells on the state of the skin.

[0096] (Variation) In the second embodiment, the application of the analysis parameters of stratum corneum cells to the analysis of skin conditions has been described, but it is also possible to apply the analysis parameters to other uses.

[0097] For example, the terminal 2 may perform a simple measurement (estimation) of the turnover rate of the subject's skin from the analysis parameters of the stratum corneum cells. Turnover refers to the metabolic cycle in which cells or tissues in a living body are regenerated.

[0098] Specifically, terminal 2 estimates the turnover rate (turnover cycle) based on the area of ​​the stratum corneum cells, which is one of the parameters analyzed from the captured image. When turnover is delayed, the stratum corneum cells flatten over time, increasing their area. Therefore, terminal 2 estimates the turnover rate from the area of ​​the stratum corneum cells. For example, terminal 2 estimates the turnover rate from the average area of ​​the stratum corneum cells of the subject, and displays the estimated result on terminal 2. Note that terminal 2 may use other parameters in addition to area, such as circularity and number of corners.

[0099] Since the other points are the same as those in the first embodiment, a detailed description of the flowchart and other details will be omitted in this embodiment.

[0100] The embodiments disclosed herein are to be considered as illustrative in all respects and not restrictive. The scope of the present invention is defined by the claims, not by the above meaning, and is intended to include all modifications within the meaning and scope of the claims. [Explanation of symbols]

[0101] 1. Server (information processing device) 11 Control section 12 Main memory 13 Communications Department 14 Auxiliary storage P1 Program 51 1st model 52 2nd model 261 Image DB 262 Subject DB 2. Terminal 21 Control section 22 Main memory 23 Communications Department 24 Display section 25 Input section 26 Auxiliary storage P2 Program

Claims

1. Acquire an image of the subject's skin tissue; identifying the contours of each stratum corneum cell by inputting the acquired image into a first model that has been trained to identify the contours of each stratum corneum cell, including portions that overlap with other stratum corneum cells, when the image is input; Based on the results of identifying the contours of each stratum corneum cell, an overlap area ratio representing the ratio of the area of ​​the stratum corneum cell overlapping with another stratum corneum cell to the area of ​​the stratum corneum cell, or an overlap distance representing the width of the area of ​​the stratum corneum cell overlapping with another stratum corneum cell, is calculated. An information processing method in which processing is performed by a computer.

2. acquiring skin data representing a condition of the subject's skin tissue; Analyzing the correlation between the skin data and the overlap area ratio or the overlap distance The information processing method according to claim 1 .

3. Based on the image, it is determined whether each of the stratum corneum cells whose contours have been identified by the first model is a defective cell.

3. The information processing method according to claim 1 or 2.

4. When the image is input, the acquired image is input to a second model that has been trained to determine whether the stratum corneum cells in the image are defective cells, thereby determining whether each of the stratum corneum cells is a defective cell. The information processing method according to claim 3 .

5. displaying the image on a display unit in which labels are superimposed on the stratum corneum cells based on the result of identifying the contours of the stratum corneum cells using the first model; receiving a designation input as to whether each of the stratum corneum cells on which the label is superimposed corresponds to a defective cell having a shape defect; The second model is generated based on training data in which labels representing the stratum corneum cells designated as defective cells are assigned to the image. The information processing method according to claim 4.

6. extracting a feature amount of each of the stratum corneum cells from the image; Based on the extracted feature amount, it is determined whether each of the stratum corneum cells is a defective cell. The information processing method according to any one of claims 3 to 5.

7. Acquire an image of the subject's skin tissue; identifying the contours of each stratum corneum cell by inputting the acquired image into a first model that has been trained to identify the contours of each stratum corneum cell, including portions that overlap with other stratum corneum cells, when the image is input; Based on the results of identifying the contours of each stratum corneum cell, an overlap area ratio representing the ratio of the area of ​​the stratum corneum cell overlapping with another stratum corneum cell to the area of ​​the stratum corneum cell, or an overlap distance representing the width of the area of ​​the stratum corneum cell overlapping with another stratum corneum cell, is calculated. A program that causes a computer to perform a process.

8. an acquisition unit that acquires an image of the skin tissue of the subject; an identification unit that identifies the contour of each stratum corneum cell by inputting the acquired image into a first model that has been trained to identify the contour of each stratum corneum cell, including portions that overlap with other stratum corneum cells, when the image is input; a calculation unit that calculates an overlap area ratio that represents the ratio of the area of ​​a portion where the cornified cell overlaps with another cornified cell to the area of ​​the cornified cell, or an overlap distance that represents the width of the portion where the cornified cell overlaps with another cornified cell, based on the result of identifying the contour of each cornified cell; An information processing device comprising:

Images