Parameter processing for estimating tissue cell parameter from images obtained by imaging affected area

Abstract

A tissue cell parameter processing device (101) estimates a tissue cell parameter from images obtained by imaging an affected area. In this estimation, a reception unit (102) receives a low-load image obtained by imaging the affected area under a low-load condition. A generation unit (103) feeds the low-load image as input into a trained generative model, and thus generates a high-load image that would be obtained by imaging the affected area under a high-load condition. An estimation unit (104) estimates a tissue cell parameter pertaining to the affected area from the high-load image. A control unit (105), which can be omitted, executes the processing by the reception unit (102), the generation unit (103), and the estimation unit (104) on a plurality of low-load images respectively obtained by imaging the affected area at different times, so that the tissue cell parameter at the different times is estimated. A prediction unit (106), which can be omitted, feeds the tissue cell parameter at the different times as input into a trained predictive model, and thus interpolates and predicts temporal changes in the tissue cell parameter.

Description

Parameter processing to estimate tissue cell parameters from images of the affected area

[0001] The present invention relates to a parameter processing device, a parameter processing method, a program, and an information recording medium for estimating tissue and cell parameters from an image of an affected area.

[0002] A super-resolution technology has been proposed that estimates an output image with improved resolution from an input image. Super-resolution performs estimation based on the characteristics of the object captured in the input image, and a neural network trained by deep learning or the like can be used. Training a neural network often involves preparing a photograph of a subject that is expected to be processed, applying noise, mosaicing, blurring, or resolution reduction to the photograph, which is then used as an input training image. This training image is then used as the output (correct answer) training image (see, for example, Patent Document 1 and Non-Patent Document 2). Super-resolution can be interpreted as an image estimation technology that, when an image captured under certain conditions is input, estimates and outputs an image that should have been captured under different conditions.

[0003] Various proposals have also been made for image registration (image alignment) techniques for associating feature points of the subject in two images of the same subject and obtaining a mapping representing the transformation from one image to the other or the inverse transformation (see, for example, Non-Patent Document 1). Such registration techniques can be used to create training data in image estimation techniques that estimate, from an image taken under certain conditions, an image that should be taken under different conditions.

[0004] Meanwhile, the inventors of the present application have proposed a technology for convolving attribute parameters representing the characteristics of a target, such as the patient's age, sex, height, weight, medical history, blood test results, test results for cytology specimens, biopsy results, and test results performed under low-stress conditions, into a target image, such as a medical image of a patient's affected area (Patent Document 2).This technology convolves attribute parameters into an input to a neural network that processes the target image or into a feature map created during the process, and replaces it with a feature map of the same size, making it possible to extract features and classify targets while taking into account both the target image and attribute parameters, while maintaining the configuration of the existing image processing network as much as possible.

[0005] In the field of cancer treatment, pathological specimens are obtained by performing invasive procedures such as resection of the affected area, and the pathological specimens are observed to obtain tissue cell parameters such as the number of tumor cells in the affected area, the number of early lesion cells in the affected area, the number of immune cells in the affected area, and the amount of blood in the tumor (vascular area) in the affected area.Based on the obtained tissue cell parameters, doctors diagnose the progression and severity of the patient's disease at the time of the invasive procedure.

[0006] Furthermore, in the field of mathematical biology of tumors, the inventors are conducting research to express mathematical models of tissue cell parameters related to cancer and tumors using differential equations (Non-Patent Document 3).

[0007] In addition, a technology has been proposed that uses deep learning to interpolate and predict data when a mathematical model of a physical quantity is expressed using a nonlinear difference equation (Non-Patent Document 4).

[0008] Furthermore, a super-resolution technology called SR3 has been proposed for improving image resolution (Non-Patent Document 2).

[0009] JP 2022-056769 A Patent No. 6345332 A

[0010] Jeremy Joslove and Emna Kamoun, "Image Registration: From SIFT to Deep Learning", Sicara's blog, [online], https: / / medium.com / sicara / image-registration-sift-deep-learning-3c794d794b7a (overview and introduction), https: / / www.sicara.fr / blog-technique / 2019-07-16-image-registration-deep-learning (main text), July 16, 2019 Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J. Fleet, and Mohammad Norouzi, "Image Super-Resolution via Iterative Refinement", Google Research, Brain Team [online], https: / / doi.org / 10.48550 / arXiv.2104.07636, April 15, 2021 Yamamoto, Y., Maziar Raissi, Paris Perdikaris, George Em Karniadakis, "Physics Informed Deep Learning (Part I): Data-driven Solutions of Nonlinear Partial Differential Equations", [online], https: / / doi.org / 10.48550 / arXiv.1711.10561, November 28, 2017.

[0011] However, if surgical or invasive procedures are required to calculate various tissue cell parameters, this places a heavy mental and physical burden on the patient, and the time, cost, and labor required for imaging are also significant.

[0012] Therefore, there is a strong demand for estimating tissue cell parameters from images taken under low-stress conditions, rather than under high-stress conditions that impose a heavy burden on patients and medical institutions.

[0013] To achieve this, it is conceivable to apply image estimation technology that estimates an image that should be captured under high-stress conditions from an image captured under low-stress conditions.

[0014] The present invention is devised to solve the above-mentioned problems, and aims to provide a parameter processing device, a parameter processing method, a program, and an information recording medium that estimate tissue cell parameters from images of an affected area taken under low-stress conditions.

[0015] The parameter processing device according to the present invention is configured to: accept low-stress images of an affected area taken under low-stress conditions; generate high-stress images that are estimated to be obtained by photographing the affected area under high-stress conditions by providing the accepted low-stress images as input to a trained generative model; and estimate tissue cell parameters related to the affected area from the generated high-stress images.

[0016] According to the present invention, it is possible to provide a parameter processing device, a parameter processing method, a program, and an information recording medium for estimating tissue and cell parameters from an image of an affected area taken under low-stress conditions.

[0017] FIG. 1 is an explanatory diagram showing a schematic configuration of a parameter processing device according to an embodiment of the present invention. FIG. 2 is an explanatory diagram showing an example of an image conversion neural network used by the parameter processing device according to an embodiment of the present invention. FIG. 3 is a drawing-substitute photograph showing, in monochrome, an example of a CT image, which is a low-burden image. FIG. 4 is a drawing-substitute photograph showing, in monochrome, an example of a pathology image, which is a high-burden image. FIG. 5 is a drawing-substitute photograph showing, in monochrome, an example of a CT image input to the parameter processing device. FIG. 6 is a drawing-substitute photograph showing, in monochrome, an example of a pathology image estimated from a CT image. FIG. 7 is a drawing-substitute photograph showing the distribution of color pixels associated with tissue cell parameters in the estimated pathology image. FIG. 8 is a flowchart showing the control flow of a training process for training a generative model according to an embodiment of the present invention. FIG. 9 is a flowchart showing the control flow of an estimation process for estimating tissue cell parameters from a low-burden image in a parameter processing device according to an embodiment of the present invention. FIG. 10 is an explanatory diagram showing a schematic configuration of a parameter processing device according to another embodiment of the present invention. FIG. 11 is a flowchart showing the control flow of a prediction process for interpolating and predicting time changes in tissue cell parameters from a low-burden image in a parameter processing device according to another embodiment of the present invention. Graphs showing interpolation and prediction of time changes in tissue cell parameters from tissue cell parameters at 20 different time points. Graphs showing interpolation and prediction of time changes in tissue cell parameters from tissue cell parameters at 6 different time points. Graphs showing interpolation and prediction of time changes in tissue cell parameters from tissue cell parameters at 3 different time points. Graphs showing interpolation and prediction of time changes in tissue cell parameters from tissue cell parameters at 2 different time points.

[0018] The following describes embodiments of the present invention. Note that these embodiments are for illustrative purposes only and do not limit the scope of the present invention. Therefore, those skilled in the art can adopt embodiments in which each or all of the elements of the present embodiments are replaced with equivalents. Furthermore, elements described in each example can be omitted as appropriate depending on the application. In this way, all embodiments constructed in accordance with the principles of the present invention are included in the scope of the present invention.

[0019] (Configuration) The parameter processing device according to this embodiment is typically realized by a computer running a program. The computer is connected to various output devices and input devices, and transmits and receives information to and from these devices.

[0020] A program executed by a computer can be distributed or sold by a server to which the computer is connected for communication, or it can be recorded on a non-transitory information recording medium such as a CD-ROM (Compact Disk Read Only Memory), flash memory, or EEPROM (Electrically Erasable Programmable ROM), and then the information recording medium can be distributed, sold, etc.

[0021] The program is installed on a non-transitory information recording medium such as a hard disk, solid-state drive, flash memory, or EEPROM of the computer. The parameter processing device of this embodiment is then realized by the computer. Generally, the computer's central processing unit (CPU) reads the program from the information recording medium into random access memory (RAM) under the control of the computer's operating system (OS), and then interprets and executes the code contained in the program. However, in an architecture in which the information recording medium can be mapped within a memory space accessible by the CPU, explicit loading of the program into RAM may not be necessary. Various pieces of information required during program execution can be temporarily stored in RAM.

[0022] Furthermore, as mentioned above, it is desirable for computers to be equipped with a GPU (Graphics Processing Unit) to perform various image processing calculations at high speed. By using a GPU and libraries such as TensorFlow, it becomes possible to utilize the learning and classification functions in various artificial intelligence processes under the control of the CPU.

[0023] It is also possible to configure the parameter processing device of this embodiment using a dedicated electronic circuit, rather than using a general-purpose computer. In this configuration, an electronic circuit that satisfies the specifications defined in the program is configured using an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit), and the electronic circuit functions as a dedicated device that performs the functions defined in the program, thereby realizing the parameter processing device of this embodiment.

[0024] (Low-burden conditions, high-burden conditions) In this application, the conditions under which the affected area is photographed are referred to as "high-burden conditions" and "low-burden conditions" based on the degree of mental and physical burden on the patient, the presence or absence of surgical or invasive procedures, and the amount of time, cost, and effort required, and photographic images taken under these conditions are referred to as "high-burden images" and "low-burden images."

[0025] Low-burden conditions refer to conditions that require little time, cost, and effort, little mental and physical burden on the patient, and are minimally invasive and do not require surgical procedures.Examples of such imaging conditions include computed tomography (CT) imaging using X-rays, magnetic resonance imaging (MRI), ultrasound imaging, endoscopic imaging, and camera imaging of the skin or fundus of the eye.

[0026] High-burden conditions include those that require a lot of time, money, and effort, that place a heavy mental and physical burden on the patient, and that require highly invasive surgical procedures, such as imaging of pathological specimens. Obtaining spatial omics analysis images or immunohistochemical staining images also falls under high-burden conditions.

[0027] For ease of understanding, the following explanation will be given using as examples a photographic image of a cut surface obtained by CT scanning as a low-stress image taken under low-stress conditions, and a pathological photographic image of a pathological specimen obtained by slicing an affected area excised from a patient and staining or otherwise processing it as necessary as a high-stress image taken under high-stress conditions.

[0028] Note that whether a condition is low or high is determined relatively based on two imaging conditions, so it is possible to associate, for example, a pathology image of a pathology specimen with a low burden and a spatial omics analysis image generated based on the specimen with a high burden. Therefore, this embodiment can be applied to any combination of two imaging conditions with different degrees of burden that are associated with a low burden and a high burden.

[0029] (Parameter Processing Device) For ease of understanding, the following description will be given assuming that the parameter processing device is realized by a computer executing a program. Fig. 1 is an explanatory diagram showing the general configuration of a parameter processing device according to an embodiment of the present invention. This parameter processing device estimates tissue and cell parameters at a certain point in time from a low-stress image captured under low-stress conditions at the time.

[0030] As shown in the figure, a parameter processing device 101 according to this embodiment includes a receiving unit 102, a generating unit 103, and an estimating unit 104.

[0031] First, the receiving unit 102 receives a low-stress image of the affected area captured under low-stress conditions. As described above, in the following description, a CT image showing a cut surface of the affected area of ​​a patient is assumed as the low-stress image.

[0032] The generation unit 103 then provides the received low-stress image as an input to the trained generative model, thereby generating a high-stress image that is estimated to be obtained by photographing the affected area under high-stress conditions.

[0033] The generative model can be, for example, an image transformation neural network with a simple filter configuration, or an image transformation neural network related to complex generative AI (generative artificial intelligence).More generally, the generative model can be any model that performs the process of transforming one image into another based on learning.

[0034] 2 is an explanatory diagram showing an example of an image conversion neural network used by a parameter processing device according to an embodiment of the present invention. The image conversion neural network shown in this figure is a so-called fully connected neural network, which is a network with a simple filter configuration that can be used for super-resolution processing to improve resolution in image conversion.

[0035] Alternatively, a simple autoencoder (AE) or a variational autoencoder (VAE) that adds stochastic noise to an autoencoder may be used.

[0036] The following configurations can be used as complex networks: SR3 (see Non-Patent Document 2), CycleGAN (Cycle-Consistent Generative Adversarial Networks), MUNIT (Multimodal UNsupervised Image-to-image Translation), and JointVAE.

[0037] In this way, any image conversion neural network that processes an input image and outputs a different image can be employed in the tissue and cell parameter processing device 101 according to this embodiment.

[0038] The training data for training the image conversion neural network can be low-stress images of the training area taken under low-stress conditions as input data, and high-stress images of the training area taken under high-stress conditions as correct answer data.

[0039] In particular, under low-stress conditions, the training lesion is imaged before a surgical procedure on the lesion, and under high-stress conditions, the training lesion is imaged after surgical resection and slices, and it is desirable that the cut surfaces of the training lesion captured in the low-stress image (input data) and the high-stress image (correct data) match or approximate each other. In this way, low-stress images and high-stress images with matching or approximating cut surfaces are prepared, and an image conversion neural network is trained. While matching the cut surfaces in both images improves estimation accuracy, super-resolution techniques based on unsupervised learning, such as GANs, do not necessarily require the cut surfaces to match; it is believed that the desired accuracy can be achieved as long as the cut surfaces are similar.

[0040] For this training, a certain amount of training data must be prepared. The training data can be prepared as follows:

[0041] First, for patients who are scheduled to have the affected area removed through invasive surgical procedures, a CT scan is taken before the procedure.

[0042] After the procedure, the excised affected area is sliced, stained appropriately to prepare a pathological specimen, and photographed using a microscope.

[0043] Then, from the CT scan results, an image of the cut surface that matches or is similar to the cut surface of the pathology specimen is extracted. Figure 3 is a monochrome photograph that shows an example of a low-stress CT image. Figure 4 is a monochrome photograph that shows an example of a high-stress pathology image.

[0044] Because the results of a CT scan contain three-dimensional information, it is possible to generate CT images for each of a number of cut surfaces from a single result. Therefore, by extracting the CT image with the highest similarity to the pathological image from the many CT images that can be generated, a CT image whose cut surface matches or is most similar to the pathological image can be obtained.

[0045] In this embodiment, CT images with matching or similar cut surfaces correspond to low-burden images that serve as input data for training, and pathological images of pathological specimens correspond to high-burden images that serve as output data for training.

[0046] In this way, a certain amount of training data is prepared and the generative model is trained to obtain pathological images from CT images.

[0047] If a CT image (low-stress image) of the affected area of ​​a new patient (a patient who has not undergone a highly stressful procedure) is given as input to the trained generative model, the generative model will estimate and output a pathology image (high-stress image) based on the assumption that "the affected area was invasively and surgically removed to create a pathology specimen, and then the pathology specimen was photographed to obtain the pathology image."

[0048] Finally, the estimation unit 104 estimates tissue cell parameters related to the affected area from the generated high-stress image.

[0049] Examples of tissue cell parameters that can be used include the number of tumor cells or early-stage lesion cells in the affected area on the cut surface of the specimen (cell count based on morphological information or cell-specific expression information, or Ki67, N), the number of immune cells in the affected area (CD8, I), and the amount of blood in the tumor (CD31, V).The tissue cell parameter may be the number on the entire cut surface of the affected area, or the number per unit area of ​​the cut surface.

[0050] These tissue cell parameters can be calculated mechanically by counting the number of pixels that are colored with the appropriate dye in a pathology image taken of a pathology specimen in which tumor cells, immune cells, vascular cells, etc. have been stained with an appropriate dye.

[0051] Although Ki67 is an immunostaining agent for general proliferating cells, it can be used as a substitute even when a simple immunostaining agent that reacts only to tumor cells does not exist.

[0052] FIG. 5 is a monochrome photograph showing an example of a CT image input to a parameter processing device. FIG. 6 is a monochrome photograph showing an example of a pathology image estimated from a CT image. FIG. 7 is a monochrome photograph showing the distribution of color pixels associated with tissue cell parameters in the estimated pathology image. The following description will be made with reference to these figures.

[0053] That is, in this example, a CT image (low-stress image) is taken while injecting a contrast agent into the affected area to obtain an image such as that shown in Figure 5, and then unnecessary surrounding parts (parts other than the affected area) are removed manually or automatically, and the aspect ratio is corrected as necessary, before being provided to the tissue cell parameter processing device 101 according to this embodiment.

[0054] Then, the tissue and cell parameter processing device 101 estimates a pathological image (high-burden image) as shown in FIG.

[0055] After this, in order to illustrate the distribution of the degree of staining, the tissue and cell parameter processing device 101 divides the image into grids, and for each grid, calculates the similarity between the pixels belonging to that grid and the color of the staining material, and determines the color intensity of that grid. The result obtained in this way is shown in Figure 7.

[0056] As shown in these figures, tissue cell parameters can be estimated by determining the distribution of pixels drawn in colors (colors of staining materials) associated with tissue cell parameters in a high-stress image estimated from a low-stress image.

[0057] That is, the estimation unit 104 estimates tissue and cell parameters from the distribution or density of pixels whose pixel values ​​satisfy a predetermined condition in the generated high-burden image.

[0058] As a result, the tissue cell parameter processing device 101 can estimate "tissue cell parameters that can only be obtained after surgical / invasive procedures" by simply taking low-impact CT images without performing any surgical / invasive procedures on the patient.

[0059] In this embodiment, it is also possible to apply the technology disclosed in Patent Document 2. That is, the receiving unit 102 is provided with low-stress images of the patient's affected area taken under low-stress conditions, as well as patient specifications related to the affected area, such as the patient's age, sex, height, weight, name of target organ, results of whole-body examination (such as examination for the presence or absence of cancer metastasis using PET, etc.), medical history, whether or not treatment has been received, the effects of treatment, blood test results, test results for cytology samples, biopsy results, and results of examinations performed under low-stress conditions, and when training the image conversion neural network, the patient specifications related to the affected area for training are provided as input data and convolved (not shown in FIG. 1).

[0060] In this embodiment, the pixel values ​​of each pixel of a low-load image provided to the neural network, or an image generated intermediately within the neural network, are convolved with the parameters to generate an image of the same size, and the image convolved with the parameters is provided to the subsequent layer.

[0061] In convolution, the pixel values ​​of each pixel in the image are calculated as a weighted linear sum of the pixel values ​​and the parameters, and the weights of the linear sum are adjusted to appropriate values ​​through training and learning of the neural network.

[0062] In this aspect, by taking into account the patient's specifications, it is believed that the likelihood of the high-stress image being output will improve, and therefore it is expected that the accuracy of the estimated tissue cell parameters will improve.

[0063] In addition, in a pathological specimen, if tumor cells, immune cells, and vascular cells are stained simultaneously using different staining materials on the same specimen and the staining materials have different colors, one pathological image can be estimated from one CT image, and three pathological cell parameters can be obtained from one pathological image based on the colors of each staining material.

[0064] Furthermore, when three adjacent slices are stained with different staining materials, three pathological images for estimating each pathological cell parameter may be estimated from one CT image.

[0065] (Training of Generative Model) The following describes the flow of processing for training a generative model that uses an image conversion network. Fig. 8 is a flowchart showing the control flow of the training processing for training a generative model according to an embodiment of the present invention.

[0066] To train a generative model, first, multiple sets of patient information for training are prepared (step S301), and the following process is repeated for each patient (step S302).

[0067] First, high-stress images of the affected area of ​​each patient taken under high-stress conditions are acquired from the set of information (step S303).

[0068] Next, a low-stress image of the affected area of ​​the patient taken under low-stress conditions, which has a cut surface that matches or is similar to that of the high-stress image and was taken at a similar time, is obtained from the set of information (step S304).

[0069] Then, after performing processing such as aligning the acquired high-burden image and low-burden image (step S305), training data is generated using the processed low-burden image as input data and the high-burden image as correct data (step S306). It is also possible to include patient specifications in the input data.

[0070] The above process is repeated for the affected areas of all patients (step S307), and when a plurality of training data are prepared, the training data is provided to the image conversion network for learning (step S308), and this process is terminated.

[0071] By learning in this way, a trained generative network using a trained image transformation network will estimate and output a high-stress image when a low-stress image (and patient specifications) is given as input.

[0072] (Tissue cell parameter estimation process) The following describes the flow of a process for estimating tissue cell parameters from a single low-stress image using a trained generative model. Fig. 9 is a flowchart showing the control flow of an estimation process for estimating tissue cell parameters from a low-stress image in a parameter processing device according to an embodiment of the present invention.

[0073] After learning the generative model by the process shown in Figure 8, the reception unit 102 of the tissue cell parameter processing device 101 receives low-stress images (and the specifications of the patient) taken under low-stress conditions of the affected area of ​​a patient who has not yet undergone high-stress treatment (step S401).

[0074] Then, the generating unit 103 uses the trained generative model to estimate and generate a high-stress image from the received low-stress image (and the specifications of the patient) (step S402).

[0075] Thereafter, the estimation unit 104 estimates tissue cell parameters from the generated high-stress image (step S403), and ends this process.

[0076] As described above, in this embodiment, it is possible to estimate tissue cell parameters at the time of imaging under low-stress conditions from a single low-stress image of the affected area of ​​a patient taken under low-stress conditions, without exposing the patient to high-stress conditions.

[0077] (Temporal Changes in Tissue and Cell Parameters) The following describes an aspect of interpolating and predicting temporal changes in tissue and cell parameters of an affected area. Fig. 10 is an explanatory diagram showing the general configuration of a parameter processing device according to another embodiment of the present invention. This parameter processing device estimates tissue and cell parameters at each imaging time point from a plurality of low-stress images taken under low-stress conditions at different times, and also interpolates and predicts tissue and cell parameters at intermediate and future times.

[0078] The parameter processing device 101 shown in this figure further includes a control unit 105 and a prediction unit 106 in addition to the receiving unit 102, generating unit 103, and estimating unit 104 disclosed in FIG.

[0079] In the embodiment shown in this figure, a high-stress image can be estimated using the patient's specifications together with the low-stress image. However, for ease of understanding, the patient's specifications are omitted from the illustration, as in Figure 1.

[0080] Here, the control unit 105 executes processing by the reception unit 102, the generation unit 103, and the estimation unit 104 on multiple low-stress images of the affected area taken at different times, and estimates tissue cell parameters at different times.

[0081] On the other hand, the prediction unit 106 interpolates and predicts the time changes of the tissue cell parameters by providing the estimated tissue cell parameters at different time points as input to the trained prediction model.

[0082] Here, the prediction model is described by a differential equation or a difference equation. For example, for the above-mentioned tumor cell count / tumor cell mass / (ultra-) early lesion cell count / (ultra-) early lesion cell mass N, immune response / immune cell count / immune cell mass I, and tumor blood volume / tumor blood vessel volume / tumor blood vessel area V, it is possible to construct a prediction model using the following differential equations by applying the technology disclosed in Non-Patent Document 3.

[0083]

[0084] In this embodiment, estimated values ​​of N, I, and V at multiple time points are obtained under the control of the control unit 105. Therefore, by calculating the unknown coefficients, i.e., the tumor cell growth rate r, the tumor cell-immune cell interaction rate a, the immune cell induction rate c, the maximum number of immune cells K, the immune cell outflow rate w, the tumor-induced neovascularization rate m, and the vascular reduction rate b, based on these estimated values, it becomes possible to interpolate and predict the time changes of tissue cell parameters.

[0085] Generally, a prediction model can be expressed by a differential equation or a difference equation. By converting the differential equation into a difference equation as needed, it is possible to perform precise predictions and forecasts for such time-varying physical quantities using a Physics Informed Deep Learning Neural Network (PINN) as disclosed in Non-Patent Document 4, even when the number of different time points is small.

[0086] Here, minute oscillations, i.e., extreme values ​​and inflection points, in the time course of tissue cell parameters provide important information for determining the progress of a patient's treatment. Therefore, by predicting tissue cell parameters from a small number of low-stress images taken under low-stress conditions at different times and identifying their extreme values, various monotherapies and combination therapies can be optimized.

[0087] Furthermore, when the inventors performed numerical simulations based on the above differential equations for different r, m, and a, it was found that the number of extreme points and inflection points differed even when the time-varying changes in the tumor cell count N and immune response I were minute. Figure 12 is a graph showing the results of a numerical simulation of the coefficient values ​​and number of extreme points in the prediction model and the time-varying trends of tissue cell parameters.

[0088] This figure shows the time course of N, I, and V for different r values ​​(0.010, 0.064, 0.119, 0.173, 0.228, 0.282, 0.337, 0.391, 0.446, 0.500) under two conditions (cond.1, cond.2). The black line corresponds to the number of tumor cells N, the white line with an upper border corresponds to the immune response I, and the white line with a lower border corresponds to the tumor blood volume V.

[0089] Comparing cond.1 and cond.2, at r=0.010 and r=0.064, the extreme value is one ("1" on the graph) and the number of tumor cells N does not increase thereafter. On the other hand, at r=0.119 and 0.173, the extreme values ​​are two ("1" and "2" on the graph). For r=0.119, there is no increase in the number of tumor cells N in cond.1, but it increases in cond.2. Furthermore, at r=0.173, the number of tumor cells N increases explosively in both cond.1 and cond.2.

[0090] Therefore, even if the change in the tissue cell parameters is small, it is possible to estimate whether or not an explosive increase will occur thereafter based on the estimated unknown coefficient values ​​and the number of extreme points.

[0091] Comparing cond.1 and cond.2 at r=0.119 and 0.173, the changes up to the extreme value "1" are almost the same, but the position of extreme value "2" is shifted, and the presence or absence of subsequent increase and the rate of increase are significantly different. Therefore, by looking at the shift in the position of the extreme values, it is expected that changes in tissue and cell parameters can be predicted in more detail.

[0092] Thus, even if the changes over time in tumor cell count N, immune response (immune cell mass) I, and tumor blood volume (vascular area) V are minimal, when the unknown coefficients are calculated using a prediction model, these minute changes are amplified and reflected in the coefficients, and this trend is also reflected in the number and cycle of extreme points and inflection points. Therefore, by referring to this information, it is thought that it will be possible to more accurately predict a patient's future tissue cell parameters and to optimize the effectiveness of various therapies.

[0093] (Prediction of Time-Varying Tissue and Cell Parameters) FIG. 11 is a flowchart showing the control flow of a prediction process for interpolating and predicting time-variing tissue and cell parameters from a low-stress image in a parameter processing device according to another embodiment of the present invention.

[0094] As shown in this figure, the control unit 105 of the tissue cell parameter processing device 101 prepares low-stress images (and the patient's specifications at the time of imaging) of the patient's affected area taken under low-stress conditions at multiple time points (step S501), and repeats the following processing for each of them (step S502).

[0095] That is, the control unit 105 causes the receiving unit 102 to receive low-stress images (and the patient's specifications at each time point) (step S503).

[0096] Then, the control unit 105 causes the generating unit 103 to generate a high-stress image at each time point (step S504), and causes the estimating unit 104 to estimate tissue cell parameters at each time point (step S505).

[0097] After the repetition (step S506) is completed, the prediction unit 106 provides the tissue cell parameters estimated for each of the multiple time points as input to the trained prediction model, and interpolates and predicts the time changes of the tissue cell parameters (step S507).

[0098] Furthermore, the prediction unit 106 may identify extreme points of the time variation (step S508).

[0099] Once the time variations of the tissue cell parameters have been obtained in this way, this process is terminated.

[0100] According to this embodiment, it is possible to predict temporal changes in tissue cell parameters at an extremely early stage, thereby enabling optimization of therapy that places less strain on the patient.

[0101] It is expected that the unknown coefficients will change if a different therapy is administered to the patient. However, by using a combination of low-stress images and PINN, etc., it is possible to estimate with high accuracy even if the number of time points at which low-stress images are taken after changing the therapy is small.

[0102] Figure 13 is a graph showing the interpolated and predicted time changes of tissue cell parameters from tissue cell parameters at 20 different time points. Figure 14 is a graph showing the interpolated and predicted time changes of tissue cell parameters from tissue cell parameters at six different time points. Figure 15 is a graph showing the interpolated and predicted time changes of tissue cell parameters from tissue cell parameters at three different time points. Figure 16 is a graph showing the interpolated and predicted time changes of tissue cell parameters from tissue cell parameters at two different time points. In these figures, the time points and values ​​of the tissue cell parameters, tumor cell count N, immune response I, and vascular area V, are represented in the graph by "●", "×", and "★", and the dotted lines on the graphs represent the interpolated and predicted time changes.

[0103] Comparing these figures, it can be seen that even when the number of time points at which low-stress images were taken was small (2 points, 3 points), the temporal changes in tissue cell parameters could be predicted with accuracy comparable to when the number of time points at which low-stress images were taken was large (6 points, 20 points).

[0104] Therefore, by taking low-impact images at least two or three times at different time points after changing therapy, it is possible to predict the effectiveness of the new therapy.

[0105] (Other parameters, etc.) In the above explanation, the number of cells can be directly used as the parameter N, but other numerical values ​​that are roughly proportional to the number of cells, such as the volume or capacity of a tumor or very early stage lesion, or the size in a photographic image, may also be used.

[0106] Parameter I is an immune response, and is a value corresponding to the number of immune cells such as lymphocyte count. Similarly, the number of cells may be used directly, or test values ​​such as PDL1 and CD8 may be used.

[0107] Furthermore, although the blood vessel area and tumor blood volume at the affected area are used as parameter V, interstitial components such as fibroblasts, for example, the amount of collagen, can also be used instead.

[0108] Furthermore, this embodiment can be applied to prediction models that use various parameters and employ various differential equations, in addition to the prediction model based on the differential equation shown in [Equation 1].

[0109] For example, this embodiment can be applied to the following differential equation in which changes in parameters N, I, and V are determined based on coefficients a, b, c, e, K, f, m, and n:

[0110]

[0111] Furthermore, applicable differential equations are not limited to those relating to the parameters N, I, and V. This embodiment can be applied to differential equations relating to any parameter.

[0112] In these prediction models, pathological phenomena are expressed using initial values ​​and boundary conditions, and the time series can be estimated based on CT images, blood test results, etc. Therefore, in such a configuration, data synchronization during inference is not required.

[0113] (Summary) As described above, the tissue parameter processing device of this embodiment is configured to include: a reception unit that receives low-stress images of the affected area taken under low-stress conditions; a generation unit that generates high-stress images that are estimated to be obtained by photographing the affected area under high-stress conditions by providing the received low-stress images as input to a trained generative model; and an estimation unit that estimates tissue cell parameters related to the affected area from the generated high-stress images.

[0114] Furthermore, in the tissue cell parameter processing device of this embodiment, imaging under the high-stress conditions can be configured to impose a greater mental or physical burden on the patient related to the affected area, or to require more time, effort, or cost, than imaging under the low-stress conditions.

[0115] In addition, the tissue cell parameter processing device of this embodiment can be configured to further include: a control unit that executes processing by the reception unit, the generation unit, and the estimation unit on multiple low-stress images of the affected area taken at different times, thereby estimating tissue cell parameters at the different times; and a prediction unit that interpolates and predicts the changes in the tissue cell parameters over time by providing the estimated tissue cell parameters at the different times as input to a trained prediction model.

[0116] In addition, in the tissue cell parameter processing device according to this embodiment, the prediction model can be configured to be described by a differential equation or a difference equation.

[0117] Furthermore, in the tissue and cell parameter processing device according to this embodiment, the prediction model can be configured to be realized by a PINN (Physics Informed Deep Learning Neural Network).

[0118] In addition, in the tissue and cell parameter processing device according to this embodiment, the prediction unit can be configured to further identify extreme points of the predicted time variation.

[0119] Furthermore, in the tissue cell parameter processing device according to this embodiment, the training data for learning the generative model can be configured so that low-stress images of the training affected area photographed under the low-stress conditions are used as input data, and high-stress images of the training affected area photographed under the high-stress conditions are used as correct answer data.

[0120] Furthermore, in the tissue cell parameter processing device of this embodiment, under the low-stress conditions, the training affected area is photographed before a surgical procedure on the affected area, and under the high-stress conditions, the training affected area that has been excised and sliced ​​by the surgical procedure is photographed, and the cut surfaces of the training affected area photographed in the low-stress image, which is the input data, and the high-stress image, which is the correct data, can be configured to match or be close to each other.

[0121] Furthermore, in the tissue cell parameter processing device according to this embodiment, the receiving unit can be configured to further receive specifications of the patient related to the affected area, and the input data for the generative model can further include the received specifications.

[0122] Furthermore, in the tissue cell parameter processing device according to this embodiment, the patient's specifications can be configured to include one or more of the patient's age, sex, height, weight, name of target organ, results of a whole-body examination, medical history, whether or not treatment has been received, the effects of treatment, results of blood tests, results of tests on cytological specimens, results of biopsy, and results of tests conducted under the low-stress conditions.

[0123] Furthermore, the tissue and cell parameter processing device according to this embodiment can be configured to estimate the tissue and cell parameters from the distribution or density of pixels whose pixel values ​​satisfy a predetermined condition in the generated high-burden image.

[0124] Furthermore, in the tissue cell parameter processing device according to this embodiment, the affected area is a tumor, and the tissue cell parameters can be configured to include at least the number of tumor cells, the number of early lesion cells, or the number of proliferating cells in the affected area, the number of immune cells in the affected area, the vascular area of ​​the affected area, the amount of tumor blood, or the amount of interstitial components.

[0125] The tissue cell parameter processing method of this embodiment is configured so that the tissue cell parameter processing device includes: a reception step for receiving low-stress images of the affected area taken under low-stress conditions; a generation step for generating high-stress images that are estimated to be obtained by photographing the affected area under high-stress conditions by providing the received low-stress images as input to a trained generative model; and an estimation step for estimating tissue cell parameters related to the affected area from the generated high-stress images.

[0126] The program according to this embodiment configures a computer to function as: a reception unit that receives low-stress images of the affected area taken under low-stress conditions; a generation unit that generates high-stress images that are estimated to be obtained by photographing the affected area under high-stress conditions by providing the received low-stress images as input to a trained generative model; and an estimation unit that estimates tissue cell parameters related to the affected area from the generated high-stress images.

[0127] The program according to this embodiment can be recorded on a non-transitory computer-readable information recording medium and distributed or sold, or can be distributed or sold via a temporary transmission medium such as a computer communication network.

[0128] The present invention allows for various embodiments and modifications without departing from the broad spirit and scope of the present invention. Furthermore, the above-described embodiments are intended to illustrate the present invention and do not limit the scope of the present invention. That is, the scope of the present invention is defined by the claims, not the embodiments. Various modifications made within the scope of the claims and their equivalents are deemed to be within the scope of the present invention. This application claims priority based on patent application No. 2024-088688, filed in Japan on Friday, May 31, 2024, and the contents of that basic application are incorporated herein to the extent permitted by the laws and regulations of the designated countries.

[0129] According to the present invention, it is possible to provide a parameter processing device, a parameter processing method, a program, and an information recording medium for estimating tissue and cell parameters from an image of an affected area taken under low-stress conditions.

[0130] 101 Tissue cell parameter processing device 102 Reception unit 103 Generation unit 104 Estimation unit 105 Control unit 106 Prediction unit

Claims

1. A tissue and cell parameter processing device comprising: a reception unit that receives low-stress images of an affected area taken under low-stress conditions; a generation unit that generates high-stress images that are estimated to be obtained by photographing the affected area under high-stress conditions by providing the received low-stress images as input to a trained generative model; and an estimation unit that estimates tissue and cell parameters related to the affected area from the generated high-stress images.

2. The tissue and cell parameter processing device according to claim 1, characterized in that imaging under the high-stress conditions places a greater mental or physical burden on the patient related to the affected area, or requires more time, effort or cost, than imaging under the low-stress conditions.

3. The tissue and cell parameter processing device according to claim 1, further comprising: a control unit that executes processing by the reception unit, the generation unit, and the estimation unit on a plurality of low-stress images of the affected area taken at different times, thereby estimating tissue and cell parameters at the different times; and a prediction unit that interpolates and predicts the time changes of the tissue and cell parameters by providing the estimated tissue and cell parameters at the different times as input to a trained prediction model.

4. The tissue and cell parameter processing device according to claim 3, wherein the prediction model is described by a differential equation or a difference equation.

5. The tissue and cell parameter processing device according to claim 4, wherein the prediction model is realized by a PINN (Physics Informed Deep Learning Neural Network).

6. The tissue and cell parameter processing device according to claim 3, wherein the prediction unit further identifies extreme points of the predicted time variation.

7. The tissue and cell parameter processing device according to claim 1, characterized in that the training data for learning the generative model is low-stress images of the training affected area taken under the low-stress conditions as input data, and high-stress images of the training affected area taken under the high-stress conditions as correct answer data.

8. The tissue cell parameter processing device according to claim 7, characterized in that under the low-stress conditions, the training affected area is photographed before a surgical procedure on the affected area, and under the high-stress conditions, the training affected area that has been excised and sliced ​​by the surgical procedure is photographed, and the cut surfaces of the training affected area photographed in the low-stress image, which is the input data, and the high-stress image, which is the correct data, match or are approximate to each other.

9. The tissue and cell parameter processing device according to claim 1 or 8, characterized in that the receiving unit further receives patient specifications related to the affected area, and the input data for the generative model further includes the received specifications.

10. The tissue cell parameter processing device according to claim 9, characterized in that the patient's specifications include one or more of the patient's age, sex, height, weight, name of target organ, results of a whole-body examination, medical history, whether or not treatment has been received, the effects of treatment, results of blood tests, results of tests on cytology samples, results of biopsy, and results of tests performed under the low-stress conditions.

11. The tissue and cell parameter processing device according to claim 1 or 8, characterized in that the tissue and cell parameters are estimated from the distribution or density of pixels whose pixel values ​​satisfy a predetermined condition in the generated high-burden image.

12. The tissue cell parameter processing device according to claim 11, characterized in that the affected area is a tumor, and the tissue cell parameters include at least the number of tumor cells, the number of early lesion cells, or the number of proliferating cells in the affected area, the number of immune cells in the affected area, the vascular area of ​​the affected area, the amount of tumor blood, or the amount of interstitial components.

13. A tissue cell parameter processing method characterized in that the tissue cell parameter processing device comprises: a reception step for receiving low-stress images of an affected area taken under low-stress conditions; a generation step for generating high-stress images that are estimated to be obtained by photographing the affected area under high-stress conditions by providing the received low-stress images as input to a trained generative model; and an estimation step for estimating tissue cell parameters related to the affected area from the generated high-stress images.

14. A program that causes a computer to function as: a reception unit that receives low-stress images of an affected area taken under low-stress conditions; a generation unit that generates high-stress images that are estimated to be obtained by photographing the affected area under high-stress conditions by providing the received low-stress images as input to a trained generative model; and an estimation unit that estimates tissue cell parameters related to the affected area from the generated high-stress images.

Images