Information processing method, information processing device, and program

Abstract

To provide an information processing method, an information processing device, and a program capable of estimating information accurately even when there is a gap between an actual phenomenon and a simulation result.SOLUTION: An information processing method includes: acquisition of first observation data, which is observation data on an object event; generation of a first feature amount from the first observation data using deep layer distance learning; matching of the first feature amount with a second feature amount generated by using deep layer distance learning from second observation data, which is the observation data on a reference event generated from a second component amount as a known component amount of the reference event, the second observation data being in a non-linear relation with the second component amount; and estimation of the second component amount corresponding to the second feature amount matched with the first feature amount as a first component amount which is the component amount of the object event.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The embodiments disclosed in the present specification and drawings relate to an information processing method, an information processing device, and a program. [Background technology]

[0002] In diagnosis, it is important to estimate the amount of a component in a subject from an image of the subject. In this regard, a technique for measuring the amount of a component in skin from an image of the skin is known.

[0003] However, when a discrepancy occurs between the actual phenomenon and the simulation results, the internal biological components estimated may be far from the correct answer. In particular, this discrepancy may be a black box that cannot be expressed mathematically or where the mathematical expression is not specified, making it difficult to correct the discrepancy. As a result, a condition different from the actual condition of the subject may be estimated, making it impossible to make a correct diagnosis.

[0004] Furthermore, situations where a black-box discrepancy occurs between actual phenomena and simulation results are not limited to the medical field, but can also occur in any field, such as physics, chemistry, engineering, biology, earth science, information, finance, economics, etc. Therefore, the above problem is common to all fields where a discrepancy may occur between actual phenomena and simulation results. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2012-231938 [Patent Document 2] Japanese Patent Application Laid-Open No. 2010-223609 Summary of the Invention [Problem to be solved by the invention]

[0006] The problem to be solved by the embodiments disclosed in this specification and the drawings is to accurately estimate information even when a discrepancy occurs between the actual phenomenon and the simulation results. However, the problem to be solved by the embodiments disclosed in this specification and the drawings is not limited to the above problem. Problems corresponding to the effects of each configuration shown in the embodiments described below can also be positioned as other problems. [Means for solving the problem]

[0007] An information processing method according to an embodiment includes acquiring first observation data, which is observation data of a target event; generating a first feature from the first observation data using deep metric learning; matching the first feature to a second feature generated using deep metric learning from second observation data, which is observation data of a reference event generated from a second component quantity, which is a known component quantity of the reference event, and which has a nonlinear relationship with the second component quantity; and estimating the second component quantity corresponding to the second feature quantity matched to the first feature quantity as the first component quantity, which is a component quantity of the target event. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a diagram illustrating an example of the configuration of an information processing device 100 according to an embodiment. [Figure 2] 3 is a flowchart showing the flow of a series of processes performed by a processing circuit 120 according to the embodiment. [Figure 3] 4A and 4B are diagrams for explaining a method for obtaining actual spectral reflectance R. FIG. [Figure 4] FIG. 10 is a diagram for explaining a method for generating an actual feature vector. [Figure 5] FIG. 10 is a diagram for explaining a method for generating a pseudo spectral reflectance R^. [Figure 6] FIG. 10 is a diagram for explaining a method for generating a pseudo feature vector. [Figure 7] FIG. 10 is a diagram for explaining matching of feature vectors. [Figure 8] FIG. 10 is a diagram illustrating an example of a matching result. [Figure 9] FIG. 11 is a diagram showing an example of a screen of the display 113a on which a map of the amount of components in a living body is displayed. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, an information processing method, an information processing device, and a program according to an embodiment will be described with reference to the drawings.

[0010] [overview] An information processing device according to an embodiment acquires observation data obtained when a target event is observed. For example, the target event may be "edema" in the medical field, and the observation data may be "spectral reflectance" obtained from an image of skin with edema. The target event is not limited to edema, but may be other symptoms or diseases. Furthermore, the target event is not limited to the medical field, but may be an object to be analyzed in other fields, such as physics, chemistry, engineering, biology, earth science, information, finance, and economics. In this embodiment, the target event is described as "edema" as an example.

[0011] When an information processing device according to an embodiment acquires an image of skin with edema (hereinafter also referred to as an edema image), it uses deep metric learning to generate a feature vector from the spectral reflectance (hereinafter referred to as actual spectral reflectance) corresponding to each pixel value of the edema image. Hereinafter, a feature vector derived from the actual spectral reflectance is referred to as an actual feature vector. The actual spectral reflectance obtained from the edema image is an example of "first observation data," and the actual feature vector is an example of "first feature amount."

[0012] The information processing device of the embodiment uses a simulation model (decoder MDL2 described below) to generate a pseudo spectral reflectance obtained when a reference event (e.g., edema) is observed, based on a known internal component amount of the reference event. Hereinafter, the known internal component amount is referred to as a pseudo internal component amount, and the pseudo-generated spectral reflectance is referred to as a pseudo spectral reflectance. This pseudo spectral reflectance is labeled with the pseudo internal component amount. The pseudo internal component amount is an example of a "second component amount," and the pseudo spectral reflectance is an example of "second observation data."

[0013] The amount of components in the body includes, for example, C Hb ,C H2O ,C mel It may contain various amounts of ingredients such as: C Hb is the hemoglobin concentration in the dermis [g / L], and C H2O is the water concentration in the subcutaneous tissue [g / L], and C mel is the melanin concentration [g / L] contained in the epidermis. In other words, the amount of a component in a living body is a set of component amounts that is a combination of multiple component amounts. The unit of each component amount in a living body is not limited to [g / L], and various units indicating percentages such as [%] may also be used.

[0014] In this embodiment, the spectral reflectance and the amount of components in the living body have a nonlinear relationship. Hb The lower the hemoglobin concentration C, the more rapidly the spectral reflectance changes. Hb The higher the value, the more gradually the spectral reflectance changes.

[0015] The information processing device according to the embodiment uses deep metric learning to generate a feature vector from pseudo-spectral reflectances labeled with pseudo-internal component amounts. Hereinafter, the feature vector derived from the pseudo-spectral reflectances is referred to as a pseudo-feature vector. The pseudo-feature vector is an example of a "second feature."

[0016] The information processing device of the embodiment moves real feature vectors from a first feature space in which the real feature vectors are distributed to a second feature space in which pseudo feature vectors are distributed.

[0017] The information processing apparatus of the embodiment matches real feature vectors with pseudo feature vectors in a second feature space.

[0018] Then, the information processing device of the embodiment estimates the pseudo in-vivo component amount corresponding to the pseudo feature vector matched with the real feature vector as the in-vivo component amount of the target event (hereinafter referred to as the actual in-vivo component amount). The actual in-vivo component amount is an example of the "first component amount".

[0019] By embedding the real spectral reflectance and the pseudo-spectral reflectance as feature vectors in a feature space using deep metric learning and then matching the real feature vectors with the pseudo-feature vectors in the feature space, it is possible to estimate information with high accuracy even when the relationship between the actual phenomenon and the simulation results cannot be mathematically expressed. For example, it is possible to estimate the amount of a biological component, which indicates the degree of edema in a living body, while reducing the influence of the discrepancy between the actual phenomenon and the simulation results.

[0020] In other words, by using deep distance learning to embed the real spectral reflectance and the pseudo-spectral reflectance as feature vectors in a feature space, and then matching the real feature vectors with the pseudo-feature vectors in the feature space, it is possible to accurately estimate the actual amount of components in a patient's body from an image of the patient's edema, and furthermore, to more accurately diagnose the patient based on the actual amount of components in the body.

[0021] [Configuration of information processing device] 1 is a diagram illustrating an example of the configuration of an information processing device 100 according to an embodiment. The information processing device 100 includes, for example, a communication interface 111, an input interface 112, an output interface 113, a memory 114, and a processing circuit 120.

[0022] The communication interface 111 communicates with external devices via a communication network NW. The communication network NW may refer to any information communication network that uses electrical communication technology. For example, the communication network NW includes wireless / wired LANs such as a hospital backbone LAN (Local Area Network) and the Internet, as well as telephone communication networks, optical fiber communication networks, cable communication networks, and satellite communication networks. The communication interface 111 includes, for example, a network interface card (NIC) and an antenna for wireless communication.

[0023] The input interface 112 accepts various input operations from an operator, converts the accepted input operations into electrical signals, and outputs the electrical signals to the processing circuit 120. For example, the input interface 112 includes a mouse, a keyboard, a trackball, a switch, a button, a joystick, a touch panel, etc. The input interface 112 may also be a user interface that accepts audio input from a microphone, etc. If the input interface 112 is a touch panel, the input interface 112 may also have the display function of a display 113a included in the output interface 113, which will be described later.

[0024] In this specification, the input interface 112 is not limited to an interface having physical operation parts such as a mouse, keyboard, etc. For example, an example of the input interface 112 also includes an electrical signal processing circuit that receives an electrical signal corresponding to an input operation from an external input device provided separately from the device and outputs this electrical signal to a control circuit.

[0025] The output interface 113 includes, for example, a display 113a and a speaker 113b. The display 113a displays various types of information. For example, the display 113a displays images generated by the processing circuit 120, a GUI (Graphical User Interface) for receiving various input operations from an operator, and the like. For example, the display 113a is an LCD (Liquid Crystal Display), a CRT (Cathode Ray Tube) display, an organic EL (Electro Luminescence) display, or the like. The speaker 113b outputs information input from the processing circuit 120 as sound.

[0026] The memory 114 is realized by, for example, a semiconductor memory element such as a RAM (Random Access Memory), a flash memory, a hard disk, or an optical disk. These non-transitory storage media may be realized by other storage devices connected via a communication network NW, such as a NAS (Network Attached Storage) or an external storage server device. The memory 114 may also include other non-transitory storage media such as a ROM (Read Only Memory) or a register. The memory 114 stores programs executed by the hardware processor of the processing circuit 120, various calculation results by the processing circuit 120, model information, and the like.

[0027] The model information is information (program or algorithm) that defines an autoencoder including an encoder MDL1 and a decoder MDL2 (described later), and a first deep metric learning model MDL3-1 and a second deep metric learning model MDL3-2 for generating feature vectors. MDL is simply a code that represents the abbreviation of MODEL.

[0028] The processing circuit 120 includes, for example, an acquisition function 121, a calculation function 122, a generation function 123, a matching function 124, an estimation function 125, and an output control function 126. The processing circuit 120 realizes these functions by, for example, a hardware processor (computer) executing a program stored in the memory 114 (storage circuit). The acquisition function 121 is an example of an "acquisition unit," the calculation function 122 is an example of a "calculation unit," the generation function 123 is an example of a "generation unit," the matching function 124 is an example of a "matching unit," and the estimation function 125 is an example of an "estimation unit."

[0029] The hardware processor in the processing circuit 120 may be, for example, a central processing unit (CPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a programmable logic device (e.g., a simple programmable logic device (SPLD) or a complex programmable logic device (CPLD)), a field programmable gate array (FPG), or the like. The term "hardware processor" refers to a circuit such as a FPGA (Field Programmable Gate Array). Instead of storing a program in memory 114, the program may be directly embedded in the circuit of the hardware processor. In this case, the hardware processor realizes its functions by reading and executing the program embedded in the circuit. The program may be stored in memory 114 in advance, or may be stored in a non-transitory storage medium such as a DVD or CD-ROM, and installed into memory 114 from the non-transitory storage medium when the non-transitory storage medium is inserted into a drive device (not shown) of information processing device 100. The hardware processor is not limited to being configured as a single circuit, but may be configured as a single hardware processor by combining multiple independent circuits to realize each function. Furthermore, multiple components may be integrated into a single hardware processor to realize each function.

[0030] [Processing flow of information processing device] A series of processes performed by the processing circuit 120 of the information processing device 100 will be described below with reference to a flowchart. Fig. 2 is a flowchart showing the flow of a series of processes performed by the processing circuit 120 according to the embodiment. The processes of S100 to S108 in this flowchart correspond to pre-processing for matching feature vectors, which will be described later, and the processes of S110 to S114 correspond to main processing including matching feature vectors.

[0031] First, the acquisition function 121 acquires an edema image IMG of the target patient (step S100).

[0032] The edema image IMG is, for example, an image of skin captured by a camera that visualizes the real spectral reflectance R of the skin at a site where edema may be present. The edema image IMG may be, for example, a three-dimensional image represented by a width x, a height y, and a wavelength λ used for visualization. The pixel value of each pixel in the edema image IMG is the real spectral reflectance R.

[0033] The camera used to generate the edema image IMG is typically a multispectral camera that visualizes the real spectral reflectance R of multiple wavelength bands (spectra), but is not limited to this and may be a camera that visualizes only the real spectral reflectance R of a single wavelength band.

[0034] For example, the acquisition function 121 may access a database, which is an external device, via the communication interface 111 and acquire an edema image IMG from the database. Furthermore, when the patient's doctor or the like inputs an edema image IMG into the input interface 112, the acquisition function 121 may acquire the edema image IMG from the input interface 112. Furthermore, when the edema image IMG is stored in the memory 114, the acquisition function 121 may acquire the edema image IMG from the memory 114. Furthermore, when the patient takes an image of his or her edema with a camera at home or the like, the acquisition function 121 may access the camera via the communication interface 111 and acquire the edema image IMG from the camera.

[0035] In addition to or instead of acquiring an edema image IMG captured by a camera, the acquisition function 121 may acquire the absorbance of the skin surface detected by a wearable sensor attached to the patient's arm, leg, etc. as an edema image IMG.

[0036] When the acquisition function 121 acquires the edema image IMG, it may perform image processing such as smoothing filtering or edge extraction on the edema image IMG, thereby removing palm prints, body hair, and the like that may appear in the edema image IMG, and more accurately extracting the characteristics of edema from the edema image IMG.

[0037] Next, the acquisition function 121 acquires the actual spectral reflectance R from each of the multiple pixels on the edema image IMG (step S102).

[0038] FIG. 3 is a diagram for explaining a method for acquiring the actual spectral reflectance R. As shown in the figure, an ROI (Region of Interest) may be set on the edema image IMG. The ROI may be set manually by a user or automatically based on the characteristics of the edema (such as edges or brightness). Once the ROI is set on the edema image IMG, the acquisition function 121 acquires the actual spectral reflectance R from each of the many pixels included in the ROI. For example, if the ROI includes nine pixels, nine actual spectral reflectances R are acquired.

[0039] Returning to the description of the flowchart, the generation function 123 then converts each of the multiple real spectral reflectances R obtained from the ROIs on the edema image IMG into a real feature vector using deep metric learning (step S104). In other words, the generation function 123 generates a real feature vector from each of the multiple real spectral reflectances R using deep metric learning.

[0040] FIG. 4 is a diagram illustrating a method for generating a real feature vector. For example, the generation function 123 may use a first deep metric learning model MDL3-1 that has been trained in advance to embed each of the multiple real spectral reflectances R into a first feature space as a real feature vector. In the illustrated example, the first feature space is represented as a two-dimensional space of f1 and f2, but is not limited to this. For example, the first feature space may be a high-dimensional space of three or more dimensions.

[0041] The first deep metric learning model MDL3-1 is a machine learning model that is trained to generate a real feature vector from the real spectral reflectance R when the real spectral reflectance R is input (to embed the real spectral reflectance R in a first feature space). When embedding the real spectral reflectance R in the first feature space, the first deep metric learning model MDL3-1 is trained to arrange similar real feature vectors close to each other and dissimilar real feature vectors far from each other in the first feature space. For example, the first deep metric learning model MDL3-1 is implemented by a neural network.

[0042] Generally, in deep metric learning, for a measurement value (e.g., spectral reflectance), positive pairs (e.g., the amount of two identical biological components) and negative pairs (e.g., the amount of two different biological components) are prepared as labels, and the system learns to move feature vectors labeled with positive pairs closer together in the feature space and move feature vectors labeled with negative pairs farther apart.

[0043] In this embodiment, the actual internal constituent amounts of the living body corresponding to the actual spectral reflectance R are unknown and cannot be used as labels. Therefore, the first deep metric learning model MDL3-1 may be trained by self-supervised learning using data augmentation to add another set of data sets of the actual spectral reflectance R and the actual internal constituent amounts of the living body, with these two data sets being a positive pair.

[0044] Signal processing that reproduces noise that occurs during imaging may be applied to the real spectral reflectance R that has increased due to data expansion. For example, to reproduce the appearance of shadows, the real spectral reflectance R may be uniformly reduced at all wavelengths. Also, to reproduce camera sensor noise, the real spectral reflectance R may be changed only at specific wavelengths.

[0045] To the first deep metric learning model MDL3-1 trained in this way, the generation function 123 inputs the real spectral reflectance R. As a result, the first deep metric learning model MDL3-1 generates real feature vectors in response to the input of the real spectral reflectance R, and further arranges similar real feature vectors close to each other in the first feature space.

[0046] Generally, the actual spectral reflectance R is prone to contain noise due to various factors such as reflected light from the skin surface, shadows, and variations in melanin concentration. When such noise is present, the hemoglobin concentration C in the skin Hb Even if the hemoglobin concentration C is the same, the actual spectral reflectance R may be large or small. Hb Even if the sensitivity of the actual spectral reflectance R is different, the sensitivity of the actual spectral reflectance R will be different.

[0047] In contrast, in this embodiment, the real spectral reflectance R is converted into a real feature vector by deep distance learning (the real spectral reflectance R is embedded in the first feature space), so the influence of noise can be reduced. As a result, by matching feature vectors described later, the hemoglobin concentration C is obtained from the edema image IMG. Hb The amount of such components in the living body can be estimated with high accuracy.

[0048] Note that if the edema image IMG is captured after removing factors such as reflected light from the skin surface, shadows, and variations in melanin concentration, the resulting edema image IMG will contain no noise. In this case, the real spectral reflectance R of the edema image IMG may be used directly for matching, which will be described later, without being converted into a real feature vector.

[0049] Returning to the explanation of the flowchart, the calculation function 122 uses a simulation model to calculate a plurality of pseudo spectral reflectances R^ labeled with known amounts of constituents in a living body (that is, pseudo amounts of constituents in a living body) (step S106).

[0050] The simulation model may be, for example, the decoder MDL2 at the latter stage of the encoder MDL1 and the decoder MDL2 included in the autoencoder.

[0051] The encoder MDL1 is a machine learning model in the front stage of the autoencoder, and converts the real spectral reflectance R into the pseudo-internal component amount. The pseudo-internal component amount output by the encoder MDL1 corresponds to a so-called latent variable.

[0052] The encoder MDL1 may be implemented, for example, by a neural network trained to make the output result of the decoder MDL2 (the pseudo-spectral reflectance R^ of each pixel of the patient's edema image) approach the real spectral reflectance R of each pixel of the patient's edema image. The encoder MDL1 may also be implemented using a genetic algorithm instead of a neural network. Furthermore, the encoder MDL1 may also be implemented using other optimization methods such as Bayesian optimization, grid search, random search, CMA-ES, the Nelder-Mead algorithm, or the quasi-Newton algorithm.

[0053] Decoder MDL2 is a device that detects the amount of a patient's internal components (C Hb ,C H2O ,C mel , ...) to calculate or simulate the pseudo-spectral reflectance R̂ of each pixel of the edema image of the patient. The decoder MDL2 may be implemented, for example, based on the Monte Carlo method, the Kubelka-Munk theory, or the Beer-Lambert law.

[0054] FIG. 5 is a diagram for explaining a method for generating the pseudo spectral reflectance R̂. For example, the calculation function 122 outputs a known pseudo amount of components in a living body (C Hb ,C H2O ,C mel , ...) is input. In response to this, the decoder MDL2 outputs the pseudo internal component amount (C Hb ,C H2O ,C melIn response to the input of the pseudo spectral reflectance R^, the calculation function 122 simulates and outputs the pseudo internal component amount (C Hb ,C H2O ,C mel , ...) to restore (decode) the pseudo-spectral reflectance R^. The decoder MDL2 is used to restore (decode) the pseudo-internal component amount (C Hb ,C H2O ,C mel Restoring the pseudo-spectral reflectance R^ from the spectral reflectances R^ may be interpreted as "reconstruction."

[0055] In the example shown, the hemoglobin concentration C Hb Therefore, the pseudo-spectral reflectance R^ is calculated for each hemoglobin concentration C, such as 3[%], 2[%], ..., 8[%]. Hb is labeled.

[0056] Returning to the description of the flowchart, the generation function 123 then converts each of the pseudo spectral reflectances R^ labeled with the pseudo internal biological component amounts into a pseudo feature vector using deep metric learning (step S108). In other words, the generation function 123 generates a pseudo feature vector from each of the pseudo spectral reflectances R^ using deep metric learning.

[0057] FIG. 6 is a diagram illustrating a method for generating a pseudo feature vector. For example, the generation function 123 may use a pre-trained second deep metric learning model MDL3-2 to embed each of the multiple pseudo spectral reflectances R^ into a second feature space as a pseudo feature vector. The second feature space may be a space different from the first feature space described above, and may have a different number of dimensions or a different base from the first feature space, for example. In the illustrated example, the second feature space is represented as a two-dimensional space of g1 and g2, but is not limited to this. For example, the second feature space may be a high-dimensional space of three or more dimensions.

[0058] The second deep metric learning model MDL3-2 is a machine learning model that is trained to generate a pseudo feature vector from the pseudo spectral reflectance R^ when the pseudo spectral reflectance R^ is input (to embed the pseudo spectral reflectance R^ in a second feature space). When embedding the pseudo spectral reflectance R^ in the second feature space, the second deep metric learning model MDL3-2 is trained to place pseudo feature vectors with similar pseudo-in-vivo component amounts close to each other in the second feature space, and to place pseudo feature vectors with dissimilar pseudo-in-vivo component amounts far from each other. For example, the second deep metric learning model MDL3-2 is implemented using a neural network.

[0059] For example, based on the pseudo-in-vivo component amounts, which are labels of the pseudo-spectral reflectance R^, identical pseudo-in-vivo component amounts are determined as positive pairs, and different pseudo-in-vivo component amounts are determined as negative pairs. The second deep metric learning model MDL3-2 then undergoes supervised learning based on the determined positive and negative pairs. That is, the second deep metric learning model MDL3-2 is trained to bring pseudo feature vectors labeled with positive pairs closer together and to move pseudo feature vectors labeled with negative pairs farther apart in the second feature space.

[0060] The generation function 123 inputs the pseudo-spectral reflectance R^ to the second deep metric learning model MDL3-2 trained in this way. As a result, the second deep metric learning model MDL3-2 outputs a pseudo-feature vector in response to the input of the pseudo-spectral reflectance R^, and further arranges similar pseudo-feature vectors close to each other in the second feature space.

[0061] As with the first deep metric learning model MDL3-1, the second deep metric learning model MDL3-2 may be trained by self-supervised learning using data augmentation to add another set of data sets of pseudo-spectral reflectance R^ and pseudo-internal biological component amounts, with these two data sets being a positive pair.

[0062] As described above, signal processing that reproduces noise that occurs during imaging (such as uniformly reducing the pseudo spectral reflectance R^) may be applied to the pseudo spectral reflectance R^ that has increased due to data expansion.

[0063] Returning to the explanation of the flowchart, when the real spectral reflectance R is embedded as a real feature vector in the first feature space and the pseudo spectral reflectance R^ is embedded as a pseudo feature vector in the second feature space, the matching function 124 matches the real feature vector with the pseudo feature vector (step S110).

[0064] 7 is a diagram for explaining matching of feature vectors. The matching function 124 moves the real feature vectors embedded in the first feature space to the second feature space using, for example, optimal transport. As optimal transport, for example, Gromov-Wasserstein optimal transport may be used. Equation (1) represents the objective function of Gromov-Wasserstein optimal transport.

[0065]

number

[0066] In the formula, D is a distance matrix (difference between feature vectors), and P is a transport matrix. The matching function 124 calculates the distance between real feature vectors (D i,i´ -D j,j´ ) is minimized. In other words, the matching function 124 transports the real feature vectors embedded in the first feature space to the second feature space. i,i´ and the distance D between the real feature vectors in the second feature space after transporting them from the first feature space to the second feature space. j,j´The real feature vectors embedded in the first feature space are transported to the second feature space so that they are the same. Then, the matching function 124 matches the real feature vectors with the pseudo feature vectors in the second feature space.

[0067] Instead of Gromov-Wasserstein optimal transport, the matching function 124 may use other optimal transports, such as unbalanced OT or barycenter OT, to match feature vectors.

[0068] Furthermore, the matching function 124 may use a gradient method such as steepest descent or stochastic gradient descent instead of optimal transport to match feature vectors.

[0069] Returning to the explanation of the flowchart, the estimation function 125 then estimates the actual internal component amount of each of the multiple pixels (actual spectral reflectance R) included in the ROI on the edema image IMG based on the matching result between the actual feature vector and the pseudo feature vector (step S112).

[0070] 8 is a diagram showing an example of a matching result. For example, the estimation function 125 estimates the pseudo-in vivo component amount corresponding to the pseudo feature vector matched with the real feature vector as the real in vivo component amount. As shown in the figure, the hemoglobin concentration C labeled with the pseudo spectral reflectance R^ used to generate the pseudo feature vector is Hb In this case, the estimation function 125 calculates the hemoglobin concentration C corresponding to the real spectral reflectance R used to generate the real feature vector matched to the pseudo feature vector. Hb is estimated to be 3%. In this way, the actual amount of the component in the living body for each pixel included in the ROI is estimated.

[0071] Returning to the explanation of the flowchart, next, when the actual internal component amounts of all pixels of the ROI are estimated, the output control function 126 calculates the pixel values ​​of all pixels as the actual internal component amounts (C Hb ,C H2O ,Cmel , . . . ) (hereinafter referred to as an internal organism constituent amount map) is generated, and this internal organism constituent amount map is output via the output interface 113 (step S114).

[0072] For example, the output control function 126 may display the map of the amount of constituents in a living body on the display 113a. The output control function 126 may also transmit the map of the amount of constituents in a living body to an external device (for example, a computer used by the attending physician of the patient to be diagnosed) via the communication interface 111. This completes the processing of this flowchart.

[0073] 9 is a diagram showing an example of the screen of the display 113a on which the internal component amount map is displayed. For example, the screen of the display 113a may display the ROI (left in the figure) of the edema image IMG and the internal component amount map of that ROI (right in the figure) side by side. This allows doctors and others to diagnose patients by comparing the actual spectral reflectance R with the actual internal component amount.

[0074] According to the embodiment described above, the processing circuit 120 of the information processing device 100 acquires an edema image IMG and acquires the actual spectral reflectance R (an example of "first observation data") from each of the multiple pixels included in the ROI of the edema image IMG.

[0075] The processing circuit 120 generates an actual feature vector (an example of a "first feature") from the actual spectral reflectance R using the first deep metric learning model MDL3-1.

[0076] The processing circuit 120 uses the second deep metric learning model MDL3-2 to generate a pseudo feature vector (an example of a "second feature") from a pseudo spectral reflectance R^ (an example of "second observation data") that has a nonlinear relationship with the pseudo internal biological component amount (an example of a "second component amount").

[0077] The processing circuit 120 matches the real feature vector with the pseudo feature vector. Then, the processing circuit 120 estimates the pseudo in-vivo component amount corresponding to the pseudo feature vector matched with the real feature vector as the real in-vivo component amount (an example of the "first component amount") corresponding to the real feature vector.

[0078] In this way, by using deep metric learning to embed the real spectral reflectance R and the pseudo-spectral reflectance R^ as feature vectors in a feature space and then matching the real feature vectors with the pseudo-feature vectors in the feature space, it is possible to estimate information with high accuracy even when the relationship between the actual phenomenon and the simulation results cannot be mathematically expressed. For example, it is possible to estimate the amount of a biological component, which indicates the degree of edema in a living body, while reducing the influence of the discrepancy between the actual phenomenon and the simulation results.

[0079] Generally, the actual spectral reflectance R is prone to contain noise due to various factors such as reflected light from the skin surface, shadows, and variations in melanin concentration. When such noise is present, the hemoglobin concentration C in the skin Hb Even if the hemoglobin concentration C is the same, the actual spectral reflectance R may be large or small. Hb Even if the sensitivity of the actual spectral reflectance R is different, the sensitivity of the actual spectral reflectance R will be different.

[0080] In contrast, in this embodiment, the real spectral reflectance R is converted into a real feature vector by deep metric learning (the real spectral reflectance R is embedded in the first feature space), thereby reducing the influence of noise. As a result, by matching feature vectors, the real amount of components in a living body can be estimated with high accuracy from the edema image IMG, and further, the patient can be diagnosed more accurately based on the real amount of components in a living body.

[0081] (Other embodiments) Other embodiments will be described below. In the above-described embodiments, the decoder MDL2 has been described as being implemented based on the Monte Carlo method, the Kubelka-Munk theory, or the Beer-Lambert law, but this is not limiting. For example, the decoder MDL2 may be implemented using a neural network or a genetic algorithm, similar to the encoder MDL1.

[0082] In the above-described embodiment, C is calculated from the actual spectral reflectance R of each pixel in the edema image IMG. Hb ,C H2O ,C mel However, the present invention is not limited to this. For example, the processing circuitry 120 may estimate the amount of components including at least one of soft tissue, calcium, and iodine from the number of photons detected by a photon-counting CT device.

[0083] More specifically, the processing circuitry 120 acquires the number of photons detected by the photon-counting CT apparatus (another example of "first observation data").

[0084] The processing circuit 120 uses the first deep metric learning model MDL3-1 to generate an actual feature vector from the measured photon count.

[0085] The processing circuit 120 uses the second deep metric learning model MDL3-2 to generate a pseudo feature vector from pseudo photon counts (another example of "second observation data") that have a nonlinear relationship with known component amounts such as soft tissue, calcium, and iodine (another example of "second component amounts").

[0086] The processing circuit 120 matches the pseudo feature vector with the real feature vector. Then, the processing circuit 120 estimates the component amount corresponding to the pseudo feature vector matched with the real feature vector as the component amount corresponding to the real feature vector (another example of the "first component amount"). By this processing, the component amounts of soft tissue, calcium, iodine, etc. can be estimated with high accuracy from the actually measured photon count.

[0087] Furthermore, the processing circuitry 120 may estimate the elastic modulus of the biological tissue irradiated with the ultrasonic waves from the echo signals measured by the ultrasonic diagnostic device.

[0088] More specifically, the processing circuitry 120 acquires echo signals (another example of "first observation data") measured by an ultrasound diagnostic device.

[0089] The processing circuit 120 uses the first deep metric learning model MDL3-1 to generate an actual feature vector from the actually measured echo signal.

[0090] The processing circuit 120 uses the second deep metric learning model MDL3-2 to generate a pseudo feature vector from a pseudo echo signal (another example of "second observation data") that has a nonlinear relationship with the elastic modulus of known biological tissue (another example of "second component quantity").

[0091] The processing circuit 120 matches the pseudo feature vector with the real feature vector. Then, the processing circuit 120 estimates the elastic modulus corresponding to the pseudo feature vector matched with the real feature vector as the elastic modulus corresponding to the real feature vector (another example of the "first component amount"). By such processing, the elastic modulus of biological tissue can be estimated with high accuracy from the actually measured echo signal.

[0092] The processing circuitry 120 may also estimate electrical property parameters (such as conductivity and permittivity) of biological tissue from the magnetic field observed by the magnetic resonance imaging apparatus.

[0093] More specifically, the processing circuitry 120 acquires signals or data (e.g., magnetic resonance signals, magnetic resonance data, k-space data) indicating a magnetic field observed by a magnetic resonance imaging apparatus. The magnetic resonance signals, magnetic resonance data, and k-space data are other examples of "first observation data."

[0094] The processing circuit 120 uses the first deep metric learning model MDL3-1 to generate actual feature vectors from actually measured magnetic resonance signals, etc.

[0095] The processing circuit 120 uses the second deep metric learning model MDL3-2 to generate a pseudo feature vector from a pseudo magnetic resonance signal (another example of "second observation data") that has a nonlinear relationship with a known electrical characteristic parameter (another example of "second component quantity").

[0096] The processing circuit 120 matches the pseudo feature vector with the real feature vector. Then, the processing circuit 120 estimates the electrical characteristic parameters corresponding to the pseudo feature vector matched with the real feature vector as the electrical characteristic parameters (another example of the "first component amount") corresponding to the real feature vector. By performing such processing, it is possible to estimate with high accuracy the electrical characteristic parameters (such as conductivity and permittivity) of biological tissue from the actually measured magnetic resonance signals, magnetic resonance data, k-space data, etc.

[0097] Furthermore, the processing circuitry 120 may estimate the amounts of components of an object of non-destructive testing from echo signals of electromagnetic waves or ultrasonic waves observed by non-destructive testing using electromagnetic waves, ultrasonic waves, etc. For example, the object of non-destructive testing may be concrete, and the amounts of its components may be hardness, strain, carbon content, vibration propagation velocity, etc.

[0098] More specifically, the processing circuitry 120 acquires echo signals of electromagnetic waves or ultrasonic waves observed by non-destructive testing (another example of "first observation data").

[0099] The processing circuit 120 uses the first deep metric learning model MDL3-1 to generate an actual feature vector from the actually measured echo signal.

[0100] The processing circuit 120 uses the second deep metric learning model MDL3-2 to generate a pseudo feature vector from a pseudo echo signal (another example of "second observation data") that has a nonlinear relationship with known component quantities such as hardness, strain, and carbon content (another example of "second component quantities").

[0101] The processing circuit 120 matches the pseudo feature vector with the real feature vector. Then, the processing circuit 120 estimates the component amount corresponding to the pseudo feature vector matched with the real feature vector as the component amount corresponding to the real feature vector (another example of the "first component amount"). By performing such processing, the component amounts of concrete and the like can be estimated with high accuracy from the echo signals of electromagnetic waves or ultrasound waves observed by non-destructive testing.

[0102] Although several embodiments have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, as well as within the scope of the invention and its equivalents as defined in the claims. [Explanation of symbols]

[0103] 100...information processing device, 111...communication interface, 112...input interface, 113...output interface, 113a...display, 114...memory, 120...processing circuit, 121...acquisition function, 122...calculation function, 123...generation function, 124...matching function, 125...estimation function, 126...output control function

Claims

1. Obtaining first observation data that is observation data of a target event; generating a first feature from the first observation data using deep metric learning; Matching the first feature with a second feature generated using deep metric learning from second observation data, the second observation data being observation data of the reference event generated from a second component amount that is a known component amount of the reference event, the second observation data having a nonlinear relationship with the second component amount; the second component amount corresponding to the second feature amount matched with the first feature amount is estimated as a first component amount that is a component amount of the target event; Information processing methods.

2. Matching the first feature to the second feature using optimal transport; The information processing method according to claim 1 .

3. the optimal transportation includes transporting the first feature values ​​from a first feature space in which the first feature values ​​are distributed to a second feature space in which the second feature values ​​are distributed such that distances between the first feature values ​​are the same before and after the transportation of the first feature values; matching the first feature amount to the second feature amount in the second feature space; The information processing method according to claim 2 .

4. matching the first feature quantity to the second feature quantity using a gradient method; The information processing method according to claim 1 .

5. the first observation data and the second observation data include spectral reflectance; the first component amount and the second component amount include any one of a hemoglobin concentration, a water concentration, and a melanin concentration; 3. The information processing method according to claim 1 or 2.

6. the first observation data and the second observation data include the number of photons detected by a photon-counting CT device; The first component amount and the second component amount include any one of soft tissue, calcium, and iodine.

3. The information processing method according to claim 1 or 2.

7. the first observation data and the second observation data include echo signals measured by an ultrasound diagnostic device; The first component amount and the second component amount include the elastic modulus of biological tissue.

3. The information processing method according to claim 1 or 2.

8. the first observation data and the second observation data include a magnetic field observed by a magnetic resonance imaging apparatus; the first component amount and the second component amount include electrical characteristic parameters of biological tissue; 3. The information processing method according to claim 1 or 2.

9. an acquisition unit that acquires first observation data that is observation data of a target event; a generation unit that generates a first feature from the first observation data using deep metric learning; a matching unit that matches the first feature with a second feature generated using deep metric learning from second observation data that is observation data of the reference event generated from a second component amount that is a known component amount of the reference event, the second observation data having a nonlinear relationship with the second component amount; an estimation unit that estimates the second component amount corresponding to the second feature amount matched with the first feature amount as a first component amount that is a component amount of the target event; An information processing device comprising:

10. A program to be executed by a computer, acquiring first observation data that is observation data of a target event; generating first features from the first observation data using deep metric learning; Matching the first feature to a second feature generated using deep metric learning from second observation data, the second observation data being observation data of the reference event generated from a second component amount that is a known component amount of the reference event, the second observation data having a nonlinear relationship with the second component amount; estimating the second component amount corresponding to the second feature amount matched to the first feature amount as a first component amount that is a component amount of the target event; Programs including.

Images