Image generation apparatus, image generation method, and image generation program

Abstract

To realize an image generation apparatus capable of more accurately predicting changes in a lesion over time.SOLUTION: An image generation apparatus includes: an acquisition section for acquiring a medical image of a patient; a detection section for detecting a lesion candidate from the medical image; a differentiation section for generating, from information indicating the lesion candidate, a differentiation result of the lesion candidate regarding a disease condition of the patient; an image generation section for generating, on the basis of the information indicating the lesion candidate and the differentiation result of the lesion candidate, a prediction image indicating a state of the lesion candidate after a lapse of time; and an output section for outputting the prediction image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an image generation device, an image generation method, and an image generation program. [Background technology]

[0002] Technologies for assisting diagnosis are known. One example of the technology for assisting diagnosis is the technology described in Patent Document 1. Patent Document 1 describes inputting a plurality of endoscopic images in a time series into an endoscopic image trained model, and generating the next endoscopic image (the next to the last endoscopic image in the time series). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2021 / 014584 Summary of the Invention [Problem to be solved by the invention]

[0004] The technology described in Patent Document 1 has room for improvement in terms of more accurately predicting changes in lesions over time.

[0005] The present disclosure has been made in view of the above-mentioned problems, and an exemplary purpose thereof is to provide a technology that can more accurately predict changes in a lesion over time. [Means for solving the problem]

[0006] An image generating device according to an exemplary aspect of the present disclosure includes an acquisition means for acquiring a medical image of a patient, a detection means for detecting lesion candidates from the medical image, a discrimination means for generating a lesion candidate discrimination result relating to the patient's condition from information representing the lesion candidate, a predicted image generating means for generating a predicted image representing the state of the lesion candidate after the passage of time based on the information representing the lesion candidate and the lesion candidate discrimination result, and an output means for outputting the predicted image.

[0007] An image generation method according to an exemplary aspect of the present disclosure includes an acquisition process in which at least one processor acquires a medical image of a patient; a detection process in which the at least one processor detects lesion candidates from the medical image; a differentiation process in which the at least one processor generates a lesion candidate differentiation result relating to the patient's medical condition from information representing the lesion candidates; a predicted image generation process in which the at least one processor generates a predicted image representing the state of the lesion candidate after a time has elapsed based on the information representing the lesion candidates and the lesion candidate differentiation result; and an output process in which the at least one processor outputs the predicted image.

[0008] An image generation program according to an exemplary aspect of the present disclosure is a program for causing a computer to function as an image generation device, and causes the computer to function as an acquisition means for acquiring medical images of a patient, a detection means for detecting lesion candidates from the medical images, a discrimination means for generating a lesion candidate discrimination result relating to the patient's condition from information representing the lesion candidate, an image generation means for generating a predicted image representing the state of the lesion candidate after the passage of time based on the information representing the lesion candidate and the lesion candidate discrimination result, and an output means for outputting the predicted image. [Effects of the Invention]

[0009] According to an exemplary aspect of the present disclosure, an exemplary effect is achieved in that a technique for predicting changes in a lesion over time with higher accuracy can be provided. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a block diagram illustrating a configuration of an image generating device according to the present disclosure. [Figure 2] FIG. 1 is a flow diagram showing the flow of an image generation method according to the present disclosure. [Figure 3] 1 is a block diagram illustrating a configuration of an information processing device according to the present disclosure. [Figure 4] FIG. 2 is a diagram illustrating a functional configuration of a control unit according to the present disclosure. [Figure 5] FIG. 10 is a diagram showing a specific example of a predicted image output by an output control unit according to the present disclosure. [Figure 6] FIG. 10 is a diagram showing a specific example of a predicted image output by an output control unit according to the present disclosure. [Figure 7] FIG. 1 is a flow diagram illustrating an example of the flow of an image generating method according to the present disclosure. [Figure 8] FIG. 1 is a block diagram illustrating a configuration of a computer that functions as an image generating device or an information processing device according to the present disclosure DETAILED DESCRIPTION OF THE INVENTION

[0011] The following are examples of embodiments of the present invention. However, the present invention is not limited to the exemplary embodiments shown below, and various modifications are possible within the scope of the claims. For example, embodiments obtained by appropriately combining the technologies (part or all of the products or methods) employed in the exemplary embodiments shown below may also be included in the scope of the present invention. Furthermore, embodiments obtained by appropriately omitting some of the technologies employed in the exemplary embodiments shown below may also be included in the scope of the present invention. Furthermore, the effects mentioned in the exemplary embodiments shown below are examples of effects expected in the exemplary embodiments, and do not define the scope of the present invention. In other words, embodiments that do not exhibit the effects mentioned in the exemplary embodiments shown below may also be included in the scope of the present invention.

[0012] First Exemplary Embodiment A first exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. This exemplary embodiment is a basic form for each of the exemplary embodiments described below. Note that the scope of application of each technique employed in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technique employed in this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical obstacles arise. Furthermore, each technique shown in the drawings referenced to explain this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical obstacles arise.

[0013] (Configuration of image generating device) The configuration of the image generating device 1 will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the configuration of the image generating device 1. As shown in Fig. 1, the image generating device 1 includes an acquisition unit 11, a detection unit 12, a discrimination unit 13, an image generating unit 14, and an output unit 15.

[0014] The acquisition unit 11 acquires medical images of a patient. The detection unit 12 detects lesion candidates from the medical images. The differentiation unit 13 generates a lesion candidate differentiation result relating to the patient's condition from information representing the lesion candidates. The image generation unit 14 generates a predicted image representing the state of the lesion candidate after the passage of time, based on the information representing the lesion candidates and the lesion candidate differentiation result. The output unit 15 outputs the predicted image.

[0015] (Effect of image generation device) As described above, image generating device 1 employs a configuration including acquisition unit 11 that acquires medical images of a patient, detection unit 12 that detects lesion candidates from the medical images, discrimination unit 13 that generates lesion candidate discrimination results related to the patient's condition from information representing the lesion candidates, image generating unit 14 that generates a predicted image representing the state of the lesion candidate after the passage of time based on the information representing the lesion candidates and the lesion candidate discrimination results, and output unit 15 that outputs the predicted image. Therefore, image generating device 1 has the effect of being able to more accurately predict changes in lesions over time.

[0016] (Image generation method flow) The flow of the image generation method S1 will be described with reference to Fig. 2. Fig. 2 is a flow diagram showing the flow of the image generation method S1. As shown in Fig. 2, the image generation method S1 includes an acquisition process S11, a detection process S12, a discrimination process S13, an image generation process S14, and an output process S15.

[0017] In acquisition processing S11, at least one processor acquires medical images of a patient. In detection processing S12, at least one processor detects lesion candidates from the medical images. In differentiation processing S13, at least one processor generates a lesion candidate differentiation result related to the patient's condition from information representing the lesion candidates. In image generation processing S14, at least one processor generates a predicted image representing the state of the lesion candidate after the passage of time, based on the information representing the lesion candidates and the lesion candidate differentiation result. In output processing S15, at least one processor outputs the predicted image.

[0018] (Effect of image generation method) As described above, the image generation method S1 employs a configuration including an acquisition process S11 in which at least one processor acquires a medical image of a patient, a detection process S12 in which at least one processor detects lesion candidates from the medical image, a discrimination process S13 in which at least one processor generates a lesion candidate discrimination result relating to the patient's condition from information representing the lesion candidates, an image generation process S14 in which at least one processor generates a predicted image representing the state of the lesion candidate after time has passed based on the information representing the lesion candidates and the lesion candidate discrimination result, and an output process S15 in which at least one processor outputs the predicted image. Therefore, the image generation method S1 has the effect of being able to more accurately predict changes in lesions over time.

[0019] Second Exemplary Embodiment A second exemplary embodiment, which is one example of an embodiment of the present invention, will be described in detail with reference to the drawings. Components having the same functions as those described in the above exemplary embodiment will be assigned the same reference numerals, and their description will be omitted as appropriate. The scope of application of each technology employed in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technology employed in this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical hindrance occurs. Furthermore, each technology shown in each drawing referenced to explain this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical hindrance occurs.

[0020] (Configuration of information processing device) The configuration of the information processing device 1A will be described with reference to Fig. 3. Fig. 3 is a block diagram showing the configuration of the information processing device 1A. The information processing device 1A includes a control unit 10A, a storage unit 20A, a communication unit 30A, an input unit 40A, and an output unit 50A.

[0021] (Communications Department) The communication unit 30A communicates with devices external to the information processing device 1A via a communication line. While the specific configuration of the communication line does not limit the present exemplary embodiment, examples of the communication line include a wireless LAN (Local Area Network), a wired LAN, a WAN (Wide Area Network), a public line network, a mobile data communication network, or a combination thereof. The communication unit 30A transmits data supplied from the control unit 10A to other devices, and supplies data received from other devices to the control unit 10A.

[0022] (Input section) The input unit 40A is configured to receive input to the information processing device 1A, and includes, for example, input devices such as an endoscope, keyboard, mouse, touch panel, camera, and microphone. The input unit 40A is configured to receive endoscopic images or signals from an endoscope. The input unit 40A may also be configured to receive data from the input device via an interface such as a USB (Universal Serial Bus).

[0023] (output section) The output unit 50A is a component for performing output from the information processing device 1A, and includes, for example, output devices such as a display, a printer, a touch panel, a speaker, etc. The output unit 50A may also be configured to include, for example, an interface such as a USB, and to output data to the output device via the interface.

[0024] (Storage part) The storage unit 20A stores various types of information referenced by the control unit 10A. Examples of such information include medical information 201, medical images 202, lesion candidate information 203, discrimination results 204, and predicted images 205.

[0025] (Medical Information) The medical information 201 is information related to the patient's medical care. Examples of the medical information 201 include chart information. The medical information 201 includes, for example, at least one of the patient's personal information, examination findings, and medical history information. The patient's personal information is the patient's personal information and includes, for example, information indicating the patient's age, gender, smoking amount, etc. The patient's examination findings information is information indicating findings by a medical professional such as a doctor and includes, for example, information obtained by the medical professional through a medical interview or visual examination. The information obtained through a medical interview or visual examination may be, for example, a sentence such as "There is pain when pressing the abdomen." The patient's medical history information is information related to the patient's medical history and includes, for example, information indicating the patient's medical history, family medical history, and smoking amount, etc. The medical information 201 may also include information indicating the measurement results of the patient's vital signs. The information indicating the measurement results of the vital signs may be, for example, information indicating "37.8°C / heart rate 90 / respiratory rate 30."

[0026] (Medical Imaging) The medical images 202 are images of patients used for diagnosis and treatment. The medical images 202 include endoscopic images. The medical images 202 may also include at least one of X-ray images, endoscopic images, pathological images, MRI images, and CT images. The storage unit 20A stores medical information 201 and medical images 202 for each of a plurality of patients.

[0027] (lesion candidate information) Lesion candidate information 203 is information representing lesion candidates detected from medical image 202. Lesion candidate information 203 is, for example, information representing the region of a lesion (coordinate information, etc.), or image data representing an image in which the location of a lesion candidate is marked in medical image 202. Lesion candidate information 203 may also be data in text format.

[0028] (Identification results) The discrimination result 204 is a result of discrimination of a lesion candidate related to the patient's condition. Here, the discrimination result of a lesion candidate is a pseudo-diagnosis result candidate generated by the discrimination unit 13A. The discrimination result of a lesion candidate is, for example, a qualitative diagnosis result (classification of neoplastic / non-neoplastic). The discrimination result of a lesion candidate may also include information indicating the name of the disease. The discrimination result of a lesion candidate 204 is, for example, data in text format, but is not limited to this.

[0029] (Predicted image) Predicted image 205 is an image that represents the state of the lesion candidate indicated by lesion candidate information 203 after it has changed over time. It can also be said that predicted image 205 represents a prediction of how the lesion will change over time. As an example, predicted image 205 is used by medical professionals who provide medical care to make decisions.

[0030] (Control unit) The control unit 10A includes an acquisition unit 11A, a detection unit 12A, a classification unit 13A, an image generation unit 14A, and an output control unit 15A. The acquisition unit 11A is an example of an acquisition means according to the present disclosure. The detection unit 12A is an example of a detection means according to the present disclosure. The classification unit 13A is an example of a classification means according to the present disclosure. The image generation unit 14A is an example of an image generation means according to the present disclosure. The output control unit 15A is an example of an output means according to the present disclosure.

[0031] (Acquisition Department) The acquisition unit 11A acquires a medical image 202 of a patient to be diagnosed. The acquisition unit 11A acquires an endoscopic video consisting of a plurality of still images or moving images via the input unit 40A, and acquires the plurality of still images included in the endoscopic video as the medical image 202. As an example, the acquisition unit 11A may acquire the medical image 202 by reading the medical image 202 from a storage destination (which may be a storage device within the information processing device 1A or a storage device external to the information processing device 1A) specified by a user of the information processing device 1A. The acquisition unit 11A may also acquire the medical image 202 by receiving the medical image 202 from another device via the communication unit 30A. The acquisition unit 11A may also acquire the medical image 202 input to the input unit 40A.

[0032] The acquiring unit 11A may also acquire medical information 201 in addition to the medical image 202. In this case, the acquiring unit 11A may acquire the medical information 201 by reading the medical information 201 from a storage location (which may be a storage device within the information processing device 1A or a storage device external to the information processing device 1A) designated by the user of the information processing device 1A. The acquiring unit 11A may also acquire the medical information 201 by receiving the medical information 201 from another device via the communication unit 30A. The acquiring unit 11A may also acquire the medical information 201 input to the input unit 40A.

[0033] (Detection unit) The detection unit 12A detects lesion candidates from the medical image 202. As an example, the detection unit 12A detects lesion candidates from the medical image 202 using a detection model that detects lesions from an image. The detection model may be stored in the storage unit 20A of the information processing device 1A, or may be stored in a device other than the information processing device 1A. Note that the detection model being stored in the storage unit 20A means that parameters that define the detection model are stored in the storage unit 20A.

[0034] The detection model is, for example, a trained model generated by machine learning. For example, a trained neural network obtained by training with reference to a plurality of training images can be used as the detection model, but is not limited to this.

[0035] An example of the input to the detection model is a medical image 202. The input to the detection model may also include information other than the medical image 202. An example of the output of the detection model is lesion candidate information 203 that indicates lesion candidates. The output of the detection model may also include information other than the lesion candidate information 203.

[0036] When the detection model is stored in a device other than the information processing device 1A, the detection unit 12A, for example, inputs the medical image 202 to the detection model by transmitting the medical image 202 to the device storing the detection model via the communication unit 30A. In this case, the detection unit 12A receives information output by the detection model from the device via the communication unit 30A.

[0037] (Identification Department) The discrimination unit 13A generates a discrimination result of lesion candidates related to the patient's medical condition. As an example, the discrimination unit 13A generates a discrimination result of lesion candidates using a discrimination model generated by machine learning. The discrimination model may be stored in the memory unit 20A of the information processing device 1A, or may be stored in a device other than the information processing device 1A. Note that the discrimination model being stored in the memory unit 20A means that parameters defining the discrimination model are stored in the memory unit 20A.

[0038] The discrimination model is, for example, a trained model generated by machine learning. Examples of the discrimination model include, but are not limited to, trained models generated by supervised learning using techniques such as neural networks, or models in which generative AI such as ChatGPT (Chat Generative Pre-trained Transformer) and GPT-4 (Generative Pre-trained Transformer 4) is fine-tuned using medical information.

[0039] As an example, the input information input to the discrimination model by discrimination unit 13A includes lesion candidate information 203. The input information may also include patient medical information 201. If lesion candidate information 203 is coordinate information, discrimination unit 13A may use medical image 202 and the coordinate information to generate an image representing the lesion candidate (for example, an image in which the location of the lesion candidate is marked), and include the generated image in the input information.

[0040] The output of the discrimination model is information indicating the discrimination result of the lesion candidate. The discrimination result of the lesion candidate may be information indicating, for example, "neoplastic" or "non-neoplastic."

[0041] When the classification model is stored in a device other than the information processing device 1A, the classification unit 13A, for example, inputs the input information to the detection model by transmitting the input information to the device storing the classification model via the communication unit 30A. In this case, the classification unit 13A acquires the information output by the classification model by receiving it from the device via the communication unit 30A.

[0042] Furthermore, classification unit 13A may generate classification results 204 of multiple lesion candidates from lesion candidate information 203. Classification unit 13A may generate classification results 204 of multiple lesion candidates, for example, by using multiple classification models that are different from each other.

[0043] (Image generation section) Image generation unit 14A generates predicted image 205 that represents the state of the lesion candidate after it has changed over time, based on lesion candidate information 203 and lesion candidate discrimination result 204. As an example, image generation unit 14A uses generative model M1 generated by machine learning to generate predicted image 205 based on lesion candidate information 203 and lesion candidate discrimination result 204.

[0044] Here, the generative model M1 is, for example, a model trained to generate a predicted image representing the state of a lesion after change over time, using the time-lapse results of the lesion as ground truth data. Examples of the generative model M1 include, but are not limited to, models in which a generative AI such as ChatGPT or GPT-4 is fine-tuned using medical information.

[0045] The input information input by image generation unit 14A to generative model M1 includes lesion candidate information 203 and lesion candidate discrimination result 204. The input information may also include medical information 201. In this case, in other words, acquisition unit 11A acquires medical information, and image generation unit 14A inputs lesion candidate information 203, lesion candidate discrimination result 204, and medical information 201 to generative model M1, thereby generating predicted image 205.

[0046] Moreover, as an example, the output of the generative model M1 includes a predicted image 205. Moreover, the output of the generative model M1 may include the accuracy of the predicted image 205. In this case, in other words, the image generating unit 14A can be said to generate the accuracy of the predicted image 205 using the generative model M1.

[0047] When the generative model M1 is stored in a device other than the information processing device 1A, the image generation unit 14A, for example, inputs the input information to the generative model M1 by transmitting the input information to the device storing the generative model M1 via the communication unit 30A. In this case, the image generation unit 14A receives the predicted image 205 output by the generative model M1 from the device via the communication unit 30A.

[0048] (Generating multiple predicted images) Furthermore, the image generating unit 14A may generate a plurality of predicted images 205. For example, the image generating unit 14A may generate a plurality of predicted images 205 using a plurality of different generation models. Furthermore, the image generating unit 14A may generate a predicted image 205 by inputting a plurality of different pieces of input information to the generation model M1. In this case, for example, the plurality of pieces of input information may contain different medical information 201 in part or in whole. Furthermore, when the discrimination unit 13A generates a discrimination result 204 of a plurality of lesion candidates, the image generating unit 14A may generate a predicted image 205 for each of the discrimination results 204 of the plurality of lesion candidates generated by the discrimination unit 13A. Furthermore, when the image generating unit 14A generates a plurality of predicted images 205, the image generating unit 14A may generate an accuracy for each of the plurality of predicted images 205.

[0049] (Output control section) The output control unit 15A outputs one or more predicted images 205. As an example, the output control unit 15A may output the predicted image 205 by writing the predicted image 205 to a storage destination (which may be a storage device within the information processing device 1A or a storage device external to the information processing device 1A) designated by the user of the information processing device 1A. Furthermore, the output control unit 15A may output the predicted image 205 by transmitting the predicted image 205 via the communication unit 30A, or may output the predicted image 205 to an output device such as a display. Furthermore, the output control unit 15A may output multiple predicted images 205 and the accuracy of each predicted image 205.

[0050] 5 and 6 are diagrams showing specific examples of predicted image 205 output by output control unit 15A. As an example, as shown in FIG. 5, output control unit 15A may cause a display device to display lesion candidate information 203, lesion candidate discrimination results 204, and predicted image 205. In the example of FIG. 5, the output information output by output control unit 15A includes lesion candidate information 203, which is a medical image on which the location of a lesion is marked, and lesion candidate discrimination results 204. The output information also includes predicted images of the progression of the lesion one year later, two years later, three years later, and so on.

[0051] 6, the output information output by the output control unit 15A includes predicted images 205-1 and 205-2 corresponding to the differentiation results of different lesion candidates. More specifically, in the example of FIG. 6, predicted image 205-1 for "adenoma" and its probability, and predicted image 205-2 for "carcinoma" and its probability are displayed as "possible neoplastic lesion candidates in your intestines." Medical professionals and others who treat patients can use predicted image 205 output by the output control unit 15A as information for deciding, for example, a treatment plan.

[0052] (Image generation method flow) FIG. 7 is a flow diagram showing an example of the flow of an image generation method S1A executed by information processing device 1A. The steps included in the flow diagram of FIG. 7 may be executed in parallel or in a different order. In step S101, acquisition unit 11A acquires medical image 202. In step S102, detection unit 12A detects lesion candidates from medical image 202. In step S103, differentiation unit 13A uses lesion candidate information 203 to generate lesion candidate differentiation result 204 related to the patient's medical condition. In step S104, image generation unit 14A generates predicted image 205 representing the state of the lesion candidate after the passage of time.

[0053] In step S105, the image generating unit 14A determines whether the examination has ended. This determination may be made, for example, by determining whether an instruction to end the examination has been input to the information processing device 1A by a medical professional or the like. If the examination has ended (YES in step S105), the image generating unit 14A proceeds to the process of step S106. In step S106, the output control unit 15A outputs the predicted image 205 generated in step S104. On the other hand, if the examination is to be continued (NO in step S105), the image generating unit 14A proceeds to the process of step S101. In this way, the information processing device 1A repeatedly executes the processes of steps S101 to S104 until the examination is ended.

[0054] (Effects of information processing devices) As described above, information processing device 1A employs a configuration including acquisition unit 11A that acquires medical image 202 of a patient, detection unit 12A that detects lesion candidates from medical image 202, classification unit 13A that generates lesion candidate classification results 204 related to the patient's condition from lesion candidate information 203, image generation unit 14A that generates predicted image 205 that represents the state of the lesion candidate after the passage of time based on lesion candidate information 203 and lesion candidate classification results 204, and output control unit 15A that outputs predicted image 205. By generating predicted image 205 using not only lesion candidate information 203 but also lesion candidate classification results 204, information processing device 1A achieves the effect of being able to more accurately predict changes in lesions over time.

[0055] Furthermore, information processing device 1A employs a configuration in which image generation unit 14A generates multiple predicted images 205 that represent the state of a lesion candidate after the passage of time, based on lesion candidate information 203 and lesion candidate discrimination result 204. Therefore, information processing device 1A has the effect of being able to predict multiple future states of a lesion from medical image 202 of a patient.

[0056] Furthermore, the information processing device 1A employs a configuration in which the image generation unit 14A generates the accuracy of each of the multiple predicted images 205, and the output control unit 15A outputs the multiple predicted images 205 and the accuracy of each predicted image 205. Therefore, the information processing device 1A provides an effect in which a medical professional or the like who treats a patient can grasp multiple states as predicted future states of the lesion and the accuracy of each of the multiple states by checking the output predicted image 205 and the accuracy of each predicted image 205.

[0057] Furthermore, information processing device 1A employs a configuration in which classification unit 13A generates classification results 204 of a plurality of lesion candidates from lesion candidate information 203, and image generation unit 14A generates predicted images 205 for each of the plurality of lesion candidate classification results 204. Therefore, information processing device 1A has the effect of enabling a medical professional or the like who treats a patient to grasp predicted images 205 corresponding to each of the plurality of lesion candidate classification results 204.

[0058] Furthermore, information processing device 1A employs a configuration in which image generation unit 14A generates predicted image 205 based on lesion candidate information 203 and lesion candidate discrimination result 204, using generation model M1 that has been machine-learned to generate a predicted image representing the state of the lesion after time has passed, using the time-lapse result of the lesion as correct answer data. Therefore, information processing device 1A has the effect of being able to more accurately predict changes in the lesion over time.

[0059] Furthermore, in information processing device 1A, acquisition unit 11A acquires patient medical information 201 in addition to medical image 202, and image generation unit 14A inputs lesion candidate information 203, lesion candidate discrimination results 204, and medical information 201 into generative model M1 to generate predicted image 205. Therefore, information processing device 1A has the advantage of being able to more accurately predict changes in lesions over time by using medical information 201 in addition to lesion candidate information 203 and lesion candidate discrimination results 204.

[0060] Furthermore, the information processing device 1A employs a configuration in which the medical information 201 includes at least one of the patient's personal information, the patient's medical findings, and the patient's medical history information. Therefore, the information processing device 1A can obtain an effect of being able to more accurately predict changes in a lesion over time by using at least one of the patient's personal information, the patient's medical findings, and the patient's medical history information.

[0061] Furthermore, the information processing device 1A employs a configuration in which the predicted image 205 is used for decision-making by medical professionals who provide medical care. Therefore, the information processing device 1A allows medical professionals who provide medical care to make more appropriate decisions.

[0062] [Software implementation example] Some or all of the functions of the image generating device 1 and the information processing device 1A (hereinafter also referred to as "the above devices") may be realized by hardware such as an integrated circuit (IC chip), or by software.

[0063] In the latter case, each of the above devices is realized by, for example, a computer that executes instructions of a program, which is software that realizes each function. An example of such a computer (hereinafter referred to as computer C) is shown in Figure 8. Figure 8 is a block diagram showing the hardware configuration of computer C that functions as each of the above devices.

[0064] The computer C includes at least one processor C1 and at least one memory C2. The memory C2 stores a program P for causing the computer C to operate as each of the above-mentioned devices. In the computer C, the processor C1 reads and executes the program P from the memory C2, thereby realizing the functions of each of the above-mentioned devices.

[0065] The processor C1 may be, for example, a central processing unit (CPU), a graphic processing unit (GPU), a digital signal processor (DSP), a micro processing unit (MPU), a floating point number processing unit (FPU), a physics processing unit (PPU), a tensor processing unit (TPU), a quantum processor, a microcontroller, or a combination thereof. The memory C2 may be, for example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), or a combination thereof.

[0066] The computer C may further include a RAM (Random Access Memory) for expanding the program P during execution and for temporarily storing various data. The computer C may also include a communication interface for transmitting and receiving data to and from other devices. The computer C may also include an input / output interface for connecting input / output devices such as a keyboard, mouse, display, and printer.

[0067] Furthermore, the program P can be recorded on a non-transitory tangible recording medium M that can be read by the computer C. Such a recording medium M can be, for example, a tape, a disk, a card, a semiconductor memory, or a programmable logic circuit. The computer C can acquire the program P via such a recording medium M. The program P can also be transmitted via a transmission medium. Such a transmission medium can be, for example, a communication network or broadcast waves. The computer C can also acquire the program P via such a transmission medium.

[0068] Furthermore, the functions of each of the devices may be realized by a single processor provided in a single computer, by multiple processors provided in a single computer working in cooperation, or by multiple processors provided in each of multiple computers working in cooperation. Furthermore, the programs for causing each of the devices to realize the functions may be stored in a single memory provided in a single computer, or may be distributed and stored in multiple memories provided in a single computer, or may be distributed and stored in multiple memories provided in each of multiple computers.

[0069] [Appendix 1] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0070] [Appendix A] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0071] (Appendix A1) an acquisition means for acquiring a medical image of a patient; a detection means for detecting a lesion candidate from the medical image; a classification means for generating a classification result of the lesion candidate related to the patient's condition from the information representing the lesion candidate; an image generating means for generating a predicted image representing a state of the lesion candidate after a time has elapsed based on the information representing the lesion candidate and a result of differentiation of the lesion candidate; an output means for outputting the predicted image; An image generating device comprising:

[0072] (Appendix A2) the image generating means generates a plurality of predicted images representing the state of the lesion candidate after the passage of time based on the information representing the lesion candidate and the differentiation result of the lesion candidate; 10. The image generating device according to claim 1.

[0073] (Appendix A3) the image generating means generates a likelihood of each of the plurality of predicted images; the output means outputs the plurality of predicted images and the accuracy of each predicted image. 10. The image generating device according to claim A2.

[0074] (Appendix A4) the discrimination means generates discrimination results for a plurality of lesion candidates from the information representing the lesion candidates; the image generating means generates a predicted image for each of the differentiation results of the plurality of lesion candidates; 10. The image generating device according to claim 9, wherein the image generating device is a

[0075] (Appendix A5) the image generation means generates the predicted image based on the information representing the lesion candidate and the differentiation result of the lesion candidate, using a generation model that has been machine-learned to generate a predicted image representing the state of the lesion after the passage of time, using the time-lapse result of the lesion as correct data; 10. The image generating device according to any one of appendices A1 to A4.

[0076] (Appendix A6) the acquiring means acquires medical information of the patient in addition to the medical image; the image generation means generates the predicted image by inputting information representing the lesion candidate, a differentiation result of the lesion candidate, and the medical information into the generation model; 10. The image generating device according to claim A5.

[0077] (Appendix A7) The medical information includes at least one of personal information of the patient, medical examination findings information of the patient, and medical history information of the patient. 10. The image generating device according to claim 6.

[0078] (Appendix A8) The predictive image is used for decision-making by medical personnel who will be performing the treatment. 10. The image generating device according to any one of appendices A1 to A7.

[0079] [Appendix B] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0080] (Appendix B1) an acquisition process in which at least one processor acquires medical images of a patient; a detection process by the at least one processor for detecting lesion candidates from the medical image; a classification process in which the at least one processor generates a classification result of the lesion candidate related to the patient's medical condition from the information representing the lesion candidate; an image generation process in which the at least one processor generates a predicted image representing a state of the lesion candidate after time has elapsed based on the information representing the lesion candidate and a differentiation result of the lesion candidate; an output process in which the at least one processor outputs the predicted image; An image generation method comprising:

[0081] (Appendix B2) In the image generation process, the at least one processor generates a plurality of predicted images representing the state of the lesion candidate after the passage of time based on the information representing the lesion candidate and the differentiation result of the lesion candidate. 10. The image generation method according to claim 8, wherein the image generation method is performed in a manner similar to that described in claim 1.

[0082] (Appendix B3) In the image generation process, the at least one processor generates a likelihood of each of the plurality of predicted images; In the output process, the at least one processor outputs the plurality of predicted images and the accuracy of each predicted image. 10. The image generation method according to claim B2.

[0083] (Appendix B4) In the differentiation process, the at least one processor generates a differentiation result of a plurality of lesion candidates from the information representing the lesion candidates; In the image generation process, the at least one processor generates a predicted image for each of the discrimination results of the plurality of lesion candidates. 10. The image generating method according to claim B2 or B3.

[0084] (Appendix B5) In the image generation process, the at least one processor generates the predicted image based on information representing the lesion candidate and a differentiation result of the lesion candidate, using a generation model that has been machine-learned to generate a predicted image representing the state of the lesion after the passage of time, using the time-lapse result of the lesion as correct data. 10. The image generating method according to any one of appendices B1 to B4.

[0085] (Appendix B6) In the acquiring process, the at least one processor acquires medical information of the patient in addition to the medical image; In the image generation process, the at least one processor generates the predicted image by inputting information representing the lesion candidate, a differentiation result of the lesion candidate, and the medical information into the generation model. 10. The image generation method according to claim 8, wherein the image generation method is a method for generating an image according to claim 9

[0086] (Appendix B7) The medical information includes at least one of personal information of the patient, medical examination findings information of the patient, and medical history information of the patient. 10. The image generating method according to claim 6.

[0087] (Appendix B8) The predictive image is used for decision-making by medical personnel who will be performing the treatment. 10. The image generating method according to any one of appendices B1 to B7.

[0088] [Appendix C] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0089] (Appendix C1) A program for causing a computer to function as an image generating device, The computer an acquisition means for acquiring a medical image of a patient; a detection means for detecting a lesion candidate from the medical image; a classification means for generating a classification result of the lesion candidate related to the patient's condition from the information representing the lesion candidate; an image generating means for generating a predicted image representing a state of the lesion candidate after a time has elapsed based on the information representing the lesion candidate and a result of differentiation of the lesion candidate; an output means for outputting the predicted image; An image generation program to function as a.

[0090] (Appendix C2) the image generating means generates a plurality of predicted images representing the state of the lesion candidate after the passage of time based on the information representing the lesion candidate and the differentiation result of the lesion candidate; 10. The image generation program according to claim 1.

[0091] (Appendix C3) the image generating means generates a likelihood of each of the plurality of predicted images; the output means outputs the plurality of predicted images and the accuracy of each predicted image. 10. The image generation program according to claim 9, wherein the image generation program is a program for generating an image according to claim 10.

[0092] (Appendix C4) the discrimination means generates discrimination results for a plurality of lesion candidates from the information representing the lesion candidates; the image generating means generates a predicted image for each of the differentiation results of the plurality of lesion candidates; 10. The image generating program according to claim 9, wherein the image generating program is a program for generating an image according to claim 1.

[0093] (Appendix C5) the image generation means generates the predicted image based on the information representing the lesion candidate and the differentiation result of the lesion candidate, using a generation model that has been machine-learned to generate a predicted image representing the state of the lesion after the passage of time, using the time-lapse result of the lesion as correct data; 10. The image generation program of claim 9, wherein the image generation program is a program for generating an image.

[0094] (Appendix C6) the acquiring means acquires medical information of the patient in addition to the medical image; the image generation means generates the predicted image by inputting information representing the lesion candidate, a differentiation result of the lesion candidate, and the medical information into the generation model; 10. The image generating program according to claim 9, wherein said image generating program is a program for generating an image according to claim 10.

[0095] (Appendix C7) The medical information includes at least one of personal information of the patient, medical examination findings information of the patient, and medical history information of the patient. 10. The image generating program according to claim 6.

[0096] (Appendix C8) The predictive image is used for decision-making by medical personnel who will be performing the treatment. 10. The image generation program of claim 9, wherein the image generation program is a program for generating an image.

[0097] [Appendix D] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0098] (Appendix D1) at least one processor, an acquisition process for acquiring medical images of a patient; a detection process for detecting lesion candidates from the medical image; a differentiation process for generating a differentiation result of the lesion candidate related to the patient's condition from the information representing the lesion candidate; an image generation process for generating a predicted image representing a state of the lesion candidate after a time has elapsed based on the information representing the lesion candidate and a differentiation result of the lesion candidate; an output process for outputting the predicted image; An image generating device that performs the above.

[0099] The image generating device may further include a memory, and the memory may store a program for causing the at least one processor to execute each of the processes.

[0100] (Appendix D2) In the image generation process, the at least one processor generates a plurality of predicted images representing the state of the lesion candidate after the passage of time based on the information representing the lesion candidate and the differentiation result of the lesion candidate. 10. The image generating device according to claim D1.

[0101] (Appendix D3) In the image generation process, the at least one processor generates a likelihood of each of the plurality of predicted images; In the output process, the at least one processor outputs the plurality of predicted images and the accuracy of each predicted image. 10. The image generating device of claim D2.

[0102] (Appendix D4) In the differentiation process, the at least one processor generates a differentiation result of a plurality of lesion candidates from the information representing the lesion candidates; In the image generation process, the at least one processor generates a predicted image for each of the discrimination results of the plurality of lesion candidates. 10. The image generating device according to claim D2 or D3.

[0103] (Appendix D5) In the image generation process, the at least one processor generates the predicted image based on information representing the lesion candidate and a differentiation result of the lesion candidate, using a generation model that has been machine-learned to generate a predicted image representing the state of the lesion after the passage of time, using the time-lapse result of the lesion as correct data. 10. The image generating device according to any one of appendices D1 to D4.

[0104] (Appendix D6) In the acquiring process, the at least one processor acquires medical information of the patient in addition to the medical image; In the image generation process, the at least one processor generates the predicted image by inputting information representing the lesion candidate, a differentiation result of the lesion candidate, and the medical information into the generation model. 10. The image generating device according to claim D5.

[0105] (Appendix D7) The medical information includes at least one of personal information of the patient, medical examination findings information of the patient, and medical history information of the patient. 10. The image generating device according to claim D6.

[0106] (Appendix D8) The predictive image is used for decision-making by medical personnel who will be performing the treatment. 10. The image generating device of claim 9, wherein the image generating device comprises:

[0107] [Appendix E] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0108] (Appendix E1) A program that causes a computer to function as an image generating device, The computer, an acquisition process for acquiring medical images of a patient; a detection process for detecting lesion candidates from the medical image; a differentiation process for generating a differentiation result of the lesion candidate related to the patient's condition from the information representing the lesion candidate; an image generation process for generating a predicted image representing a state of the lesion candidate after a time has elapsed based on the information representing the lesion candidate and a differentiation result of the lesion candidate; an output process for outputting the predicted image; A non-transitory recording medium on which an image generation program for executing the above is recorded. [Explanation of symbols]

[0109] 1. Image generation device 1A Information processing equipment 11, 11A Acquisition Department 12, 12A Detector 13, 13A Identification Department 14, 14A Image generation unit 15, 50A output section 15A Output Control Unit

Claims

1. an acquisition means for acquiring a medical image of a patient; a detection means for detecting a lesion candidate from the medical image; a classification means for generating a classification result of the lesion candidate related to the patient's condition from the information representing the lesion candidate; an image generating means for generating a predicted image representing a state of the lesion candidate after a time has elapsed based on the information representing the lesion candidate and a result of differentiation of the lesion candidate; an output means for outputting the predicted image; An image generating device comprising:

2. the image generating means generates a plurality of predicted images representing the state of the lesion candidate after the passage of time based on the information representing the lesion candidate and the differentiation result of the lesion candidate; The image generating device of claim 1 .

3. the image generating means generates a likelihood of each of the plurality of predicted images; the output means outputs the plurality of predicted images and the accuracy of each predicted image. The image generating device of claim 2 .

4. the discrimination means generates discrimination results for a plurality of lesion candidates from the information representing the lesion candidates; the image generating means generates a predicted image for each of the differentiation results of the plurality of lesion candidates; 4. The image generating device according to claim 2 or 3.

5. the image generation means generates the predicted image based on the information representing the lesion candidate and the differentiation result of the lesion candidate, using a generation model that has been machine-learned to generate a predicted image representing the state of the lesion after the passage of time, using the time-lapse result of the lesion as correct data; 3. The image generating device according to claim 1 or 2.

6. the acquiring means acquires medical information of the patient in addition to the medical image; the image generation means generates the predicted image by inputting information representing the lesion candidate, a differentiation result of the lesion candidate, and the medical information into the generation model; The image generating device according to claim 5 .

7. The medical information includes at least one of personal information of the patient, medical examination findings information of the patient, and medical history information of the patient.

7. The image generating device of claim 6.

8. The predictive image is used for decision-making by medical personnel who will be performing the treatment.

3. The image generating device according to claim 1 or 2.

9. an acquisition process in which at least one processor acquires medical images of a patient; a detection process in which the at least one processor detects lesion candidates from the medical image; a classification process in which the at least one processor generates a classification result of the lesion candidate related to the patient's medical condition from the information representing the lesion candidate; an image generation process in which the at least one processor generates a predicted image representing a state of the lesion candidate after it has changed over time, based on the information representing the lesion candidate and a result of differentiation of the lesion candidate; an output process in which the at least one processor outputs the predicted image; An image generation method comprising:

10. A program for causing a computer to function as an image generating device, the program comprising: an acquisition means for acquiring a medical image of a patient; a detection means for detecting a lesion candidate from the medical image; a classification means for generating a classification result of the lesion candidate related to the patient's condition from the information representing the lesion candidate; an image generating means for generating a predicted image representing a state of the lesion candidate after it has changed over time, based on the information representing the lesion candidate and the result of differentiation of the lesion candidate; an output means for outputting the predicted image; An image generation program to function as a

Images