Information processing method, learning model generation method, program, and information processing device.

Abstract

Provided are an information processing method and the like, that reduces the burden on a subject and that can infer information relating to cervical lesions.　This information processing method causes a computer to execute processing for: acquiring a medical image of the cervix; acquiring, on the basis of the acquired medical image, information relating to a lesion in the depth direction from a tissue surface of the cervix; and displaying the medical image and the information relating to the lesion.

Description

Technical Field

[0005]

[0001] The present invention relates to an information processing method, a learning model generation method, a program, and an information processing apparatus.

Background Art

[0002] Currently, the number of patients diagnosed with cervical cancer in Japan is said to be about 11,000 per year. Cervical cancer originates from the cervix at the entrance of the uterus and often occurs near the entrance of the uterus, so it is easy to detect early through regular examinations. If it can be detected early, it is relatively easy to treat and has a good prognosis. However, since it becomes difficult to treat as the symptoms progress, early detection is said to be extremely important.

[0003] In the cervical cancer screening, conventionally, as a screening test, a long, tube-shaped instrument is inserted through the uterine opening to collect the mucosa on the surface of the endometrium, and a cytological examination is performed by observing it with a microscope or the like. If it is determined necessary from the results of the cytological examination, a detailed examination is performed. The detailed examination is performed by a histological examination in which a small amount of tissue from the cervix is collected by biopsy from the subject and observed with a microscope or the like. Also, colposcopy examination is performed to observe the tissue surface using a microscope that magnifies the surface of the cervix and vagina called a colposcope.

[0004] If it is determined necessary from the results of the detailed examination, conization is performed. Conization is a surgical method in which the cervix is excised in a conical shape to perform a pathological diagnosis using the excised section and at the same time perform treatment. Based on the pathological diagnosis, a definitive diagnosis is made, and it is determined whether treatment is completed or additional treatment such as an excision is required.

[0005] Pathological diagnoses based on tissue biopsy can vary depending on the skill of the examiner. Therefore, techniques to improve diagnostic accuracy have been developed. For example, Patent Document 1 discloses a method for more accurately and simply determining whether a sample contains abnormal cells originating from lesions of high-grade cervical dysplasia or higher, by detecting methylation of the genomic DNA of human papillomavirus contained in a sample taken from the cervix of a subject. [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] Re-tabled publication No. 2010 / 047211 [Overview of the project] [Problems that the invention aims to solve]

[0007] However, the method described in Patent Document 1 has the problem of being burdensome for the subject, as it requires taking tissue samples from the cervix of the subject in order to obtain information about cervical lesions.

[0008] The purpose of this disclosure is to provide an information processing method, etc., that can estimate information about cervical lesions while reducing the burden on the subject. [Means for solving the problem]

[0009] An information processing method according to one aspect of this disclosure involves obtaining a medical image of the cervix, obtaining information about lesions extending from the tissue surface to the depth direction of the cervix based on the obtained medical image, and causing a computer to perform a process of displaying the medical image and the information about the lesions.

[0010] A learning model generation method according to one aspect of the present disclosure acquires training data including a medical image of the cervix and information about lesions from the tissue surface to the depth direction of the cervix, and generates a learning model that is trained to output information about lesions from the tissue surface to the depth direction of the cervix when a medical image of the cervix is ​​input, based on the training data. [Effects of the Invention]

[0011] According to this disclosure, it is possible to estimate information about cervical lesions while reducing the burden on the subject. [Brief explanation of the drawing]

[0012] [Figure 1] This is a schematic diagram of the information processing system in the first embodiment. [Figure 2] This is a block diagram showing an example of the configuration of an information processing system. [Figure 3] This is an explanatory diagram showing an overview of the learning model. [Figure 4] This figure shows an example of the content of information stored in the training data database. [Figure 5] This flowchart shows an example of the procedure for generating training data. [Figure 6] This is a schematic diagram showing an example of a reception screen. [Figure 7] This is a flowchart illustrating an example of the process for generating a learning model. [Figure 8] This is a flowchart illustrating an example of the procedure for acquiring estimated information. [Figure 9] This is a schematic diagram showing an example of a screen displayed on a display device. [Figure 10] This is a schematic diagram showing an example of a screen displayed on a display device. [Figure 11] This is a flowchart showing an example of the retraining process for the learning model in the second embodiment. [Modes for carrying out the invention]

[0013] The present disclosure will be specifically described with reference to the drawings showing embodiments thereof.

[0014] (First Embodiment) FIG. 1 is a schematic diagram of an information processing system according to the first embodiment. The information processing system includes an information processing apparatus 1 and an image diagnostic apparatus 2. The information processing apparatus 1 and the image diagnostic apparatus 2 are communicatively connected to a network N such as a LAN (Local Area Network) or the Internet.

[0015] The image diagnostic apparatus 2 is a device unit for imaging a luminal organ of a subject. In the present embodiment, the luminal organ to be examined is the uterus, particularly the cervix (cervical canal). The image diagnostic apparatus 2 is a device unit for generating an ultrasonic tomogram (medical image) of the cervix of a subject using a detection device 21 and performing ultrasonic examination and diagnosis of the cervix. The image diagnostic apparatus 2 includes a detection device 21, an image processing device 22, a display device 23, and the like.

[0016] The detection device 21 is a device for obtaining an ultrasonic tomogram of the cervix of a subject. The detection device 21 includes an insertion tube 211 and a drive device 212. The insertion tube 211 is long and is a portion that is inserted from the vagina of the subject into the cervix. The insertion tube 211 has a probe portion 213 and a connector portion 214 disposed at an end of the probe portion 213. The probe portion 213 is connected to the drive device 212 via the connector portion 21 | 4. In the following description, the side far from the connector portion 214 of the insertion tube 211 is referred to as the distal end side.

[0017] A shaft 215 is inserted inside the probe portion 213. A sensor 216 is connected to the distal end side of the shaft 215. The sensor 216 is, for example, an ultrasonic transducer. The sensor 216 transmits ultrasonic waves based on a pulse signal inside the cervix and receives reflected waves reflected by the biological tissue of the cervix. The shaft 215 and the sensor 216 are configured to be able to advance and retreat in the longitudinal direction of the cervix while rotating in the circumferential direction of the cervix (cervical canal) inside the probe portion 213.

[0018] The insertion tube 211 can capture tomographic images that include not only the luminal wall of the cervix, but also reflectors present within the luminal wall tissue of luminal organs, such as cancer cells.

[0019] The detection device 21 may, for example, use a conventional imaging catheter and MDU (Motor Driving Unit), or it may be a dedicated detection device suitable for imaging diagnosis of the inside of the uterus. Furthermore, the detection device 21 is not limited to one that generates ultrasound tomography images using an ultrasound transducer. The detection device 21 may be a detection device for generating optical tomography images, such as for OCT (Optical Coherence Tomography) or OFDI (Optical Frequency Domain Imaging), which generates optical tomography images using near-infrared light. In this case, the sensor 216 is a transmitting and receiving unit that irradiates near-infrared light and receives reflected light. The detection device 21 may have both an ultrasound transducer and a transmitting and receiving unit 216 for OCT or OFDI, and may be used to generate medical images that include both ultrasound tomography images and optical tomography images.

[0020] The drive unit 212 is to which the probe unit 213 is detachably attached. The drive unit 212 controls the movement of the probe unit 213 inserted into the cervix by driving a built-in motor in response to user operation. The drive unit 212 rotates the shaft 215 and sensor 216 in the circumferential direction while moving them longitudinally from the tip to the base. The sensor 216 continuously scans the inside of the cervix at predetermined time intervals and outputs the detected ultrasonic reflected wave data to the image processing device 22.

[0021] The image processing device 22 is a processing device that generates an ultrasound tomographic image (medical image) of the cervix based on the reflected wave data output from the ultrasound probe of the detection device 21. The image processing device 22 generates one image for each rotation of the sensor 216. The generated image is a transverse layer image centered on the probe unit 213 and approximately perpendicular to the probe unit 213. The image processing device 22 displays the generated medical image, estimated information acquired from the information processing device 1, etc., on the display device 23.

[0022] The display device 23 is a liquid crystal display panel, an organic EL (Electro-Luminescence) display panel, etc. The display device 23 displays medical images generated by the image processing device 22, estimated information received from the information processing device 1, etc.

[0023] Information processing device 1 is an information processing device capable of various information processing and information transmission and reception, such as a server computer or a personal computer. Information processing device 1 may be a local server installed in the same facility (hospital, etc.) as the medical imaging device 2, or it may be a cloud server that is connected to the medical imaging device 2 via the internet or the like. Information processing device 1 functions as an estimation device that estimates information regarding lesions of the cervix (hereinafter also referred to as estimated information) using medical images of the subject's cervix generated by the medical imaging device 2. Information processing device 1 provides the estimated information to the medical imaging device 2.

[0024] Estimated information refers to information about lesions in the cervical tissue from the surface to the depth. In this embodiment, the lesions to be estimated include precancerous lesions, such as cervical cancer, cervical intraepithelial neoplasia (CIN), and cervical adenocarcinoma in situ (AIS). In this specification, the term "lesion" includes candidate lesions suspected to be lesions. Information regarding the lesion includes, for example, information about the location (extent) and type (symptoms) of the lesion. The depth direction refers to the radial direction of the cervix (cervical canal), that is, the direction from the tissue surface (epidermis) toward the interior. The length direction refers to the longitudinal direction of the cervix, that is, the direction from the vaginal side toward the uterine side.

[0025] Generally, in the diagnosis and treatment of cervical cancer, a screening test using cytology is performed first. If deemed necessary based on the cytology results, further examinations are conducted. If deemed necessary, a cone biopsy is performed, which simultaneously diagnoses and treats the condition, and a definitive diagnosis is made. The decision of whether or not to perform a cone biopsy is made based on the results of the further examinations. These further examinations include a histological examination, in which tissue from the cervix taken from the subject is examined under a microscope, and a colposcopy, in which the tissue surface is examined using a colposcope (microscope) to magnify the surface of the cervix and vagina. Since both of these are based on observation of the tissue surface, it is difficult to grasp the progression of the lesion in the depth direction of the cervix, which cannot be confirmed from the tissue surface, or the condition of the lesion inside the cervix (in the depth direction). For this reason, even in patients with mild cases who do not actually need treatment (cone biopsy), a cone biopsy may be performed to make a definitive diagnosis. Diagnosis and treatment by cone biopsy are burdensome for the patient and may affect pregnancy and childbirth. On the other hand, while non-invasive treatments that do not involve surgical removal reduce the burden on the patient, it is difficult to definitively diagnose the progression and extent of the lesion, making it challenging to fully guarantee that the treatment has been completed.

[0026] This information processing system provides physicians and other medical professionals with estimated information that accurately estimates the state of lesions in the depth direction based on medical images of the patient's cervix, using a learning model described later. Because physicians and other medical professionals can use this estimated information to make diagnoses without surgical intervention, the burden on patients can be reduced.

[0027] The configuration and detailed processing content of this information processing system are described below.

[0028] Figure 2 is a block diagram showing an example configuration of an information processing system. The information processing device 1 comprises a control unit 11, a main memory unit 12, an auxiliary memory unit 13, a communication unit 14, a display unit 15, and an operation unit 16. The information processing device 1 may be a multicomputer consisting of multiple computers, or it may be a virtual machine virtually constructed by software.

[0029] The control unit 11 is a computing device such as one or more CPUs (Central Processing Units) or GPUs (Graphics Processing Units). The control unit 11 reads and executes the program 13P stored in the auxiliary storage unit 13, thereby enabling the server computer to function as an information processing device that performs various processes related to the generation of support information.

[0030] The main memory unit 12 is a temporary storage area such as SRAM (Static Random Access Memory), DRAM (Dynamic Random Access Memory), or flash memory. The main memory unit 12 temporarily stores the program 13P read from the auxiliary memory unit 13 when the control unit 11 performs its calculations, or various data generated by the calculations of the control unit 11.

[0031] The auxiliary storage unit 13 is a non-volatile storage area such as a hard disk, EEPROM (Electrically Erasable Programmable ROM), or flash memory. The auxiliary storage unit 13 may also be an external storage device connected to the information processing device 1. The auxiliary storage unit 13 stores programs and data, including the program 13P necessary for the control unit 11 to execute processing.

[0032] Furthermore, the auxiliary storage unit 13 stores the learning model 131 and the training data DB (Data Base) 132. The learning model 131 is a machine learning model that has been trained on training data. The learning model 131 is intended to be used as a program module that constitutes artificial intelligence software. Details of the learning model 131 and the training data DB 132 will be described later.

[0033] The program 13P may be recorded on the recording medium 13A in a computer-readable manner. The auxiliary storage unit 13 stores the program 13P read from the recording medium 13A by a reading device (not shown). The recording medium 13A is a semiconductor memory such as flash memory, an optical disk, a magnetic disk, a magneto-optical disk, etc. Alternatively, the program 13P according to this embodiment may be downloaded from an external server (not shown) connected to a communication network and stored in the auxiliary storage unit 13. The program 13P can be deployed on a single computer or at a single site, or distributed across multiple sites and deployed to run on multiple computers interconnected by a communication network.

[0034] The communication unit 14 is a communication module for performing communication-related processing. The control unit 11 transmits and receives information with the image diagnostic device 2 via the communication unit 14.

[0035] The display unit 15 is an output device that outputs information such as medical images and estimated information. The output device is, for example, a liquid crystal display or an organic EL (Electro-Luminescence) display.

[0036] The operation unit 16 is an input device that receives user input. The input device is, for example, a pointing device such as a keyboard or a touch panel.

[0037] The image processing device 22 includes a control unit 221, a main memory unit 222, an auxiliary memory unit 223, a communication unit 224, an input / output unit 225, and a detection device control unit 226.

[0038] The control unit 221 is one or more arithmetic processing units such as CPUs and GPUs. The main memory unit 222 is a temporary storage area such as SRAM, DRAM, or flash memory. The control unit 221 performs various information processing by reading and executing programs stored in the auxiliary memory unit 223.

[0039] The main memory unit 222 temporarily stores programs read from the auxiliary memory unit 122 when the control unit 11 performs calculation processing, or various data generated by the calculation processing of the control unit 11.

[0040] The auxiliary storage unit 223 is a non-volatile storage area such as a hard disk, EEPROM, or flash memory. The auxiliary storage unit 223 stores the programs and data necessary for the control unit 11 to execute processing. The auxiliary storage unit 223 may also store the learning model 131.

[0041] The communication unit 224 is a communication module for performing communication-related processing. The control unit 221 transmits and receives information with the information processing device 1 via the communication unit 224 and acquires estimated information.

[0042] The input / output unit 225 is an input / output interface (I / F) for connecting external devices. The display device 23 and input device 24 are connected to the input / output unit 225. The display device 23 is, for example, a liquid crystal display or an organic EL display. The input device 24 is, for example, a pointing device such as a keyboard or a touch panel. The control unit 221 outputs medical images and estimated information to the display device 23 via the input / output unit 225. The control unit 221 also receives information input to the input device 24 via the input / output unit 225.

[0043] The detection device control unit 226 controls the drive unit 212, controls the sensor 216, and generates medical images based on signals received from the sensor 216. The functions and configuration of the detection device control unit 226 are the same as those of conventional ultrasound diagnostic devices, so a detailed explanation is omitted. Note that the control unit 221 may also perform the functions of the detection device control unit 226.

[0044] Figure 3 is an explanatory diagram illustrating the overview of the learning model 131. The learning model 131 is a machine learning model that takes medical images of the subject's cervix as input and outputs information about lesions from the tissue surface to the depth direction of the cervix.

[0045] The information processing device 1 pre-generates a learning model 131 by performing machine learning to learn predetermined training data. The information processing device 1 then inputs the patient's medical images acquired from the image diagnostic device 2 into the learning model 131 and outputs information about the lesion.

[0046] The learning model 131 uses image recognition technology, such as a semantic segmentation model (a type of Convolutional Neural Network), to recognize on a pixel-by-pixel basis whether each pixel in the input image corresponds to an object (lesion) region. The learning model 131 has an input layer into which a medical image is input, an intermediate layer that extracts and reconstructs image features, and an output layer that outputs information indicating the location and type of lesion contained in the medical image. The learning model 131 is, for example, a U-Net.

[0047] The input layer of the learning model 131 has multiple nodes that accept the pixel value of each pixel in the medical image as input, and passes the input pixel value to the intermediate layer. The intermediate layer has a convolutional layer (CONV layer) and a deconvolutional layer (DECONV layer). The convolutional layer is a layer that compresses the dimensions of the image data. Through dimensionality reduction, the features of the lesion are extracted. The deconvolutional layer performs deconvolution and restores the original dimensions. The restoration process in the deconvolutional layer generates a binarized label image that indicates whether each pixel in the medical image is a lesion or not. The output layer has one or more nodes that output the label image.

[0048] The medical images input to the learning model 131 include multiple frames in chronological order generated by a single pullback operation. These multiple medical images in chronological order are, for example, tomographic images observed over a predetermined range from the uterine side to the vaginal side in the depth direction of the cervix. The medical images are tomographic images that include a predetermined detection range in the depth direction of the cervix.

[0049] The label image output from the learning model 131 is an image that identifies the object class for each pixel of the medical image. Objects detected using the learning model 131 include, for example, lesions such as cancer, CIN, and AIS. CIN is further classified into three stages of dysplasia, CIN1 to CIN3, depending on the degree of appearance of atypical cells appearing in the epithelium. The label image is an image in which each pixel of the medical image has a pixel value corresponding to these object classes: cancer, CIN1 to CIN3, AIS, and other objects. The label image shows information indicating the location (range) and type (symptoms) of the lesion contained in the medical image for each pixel. This makes it possible to visually recognize the state of the lesion in the depth direction.

[0050] The learning model 131 may also detect other cellular tissues as objects in addition to the lesions described above. These other cellular tissues include cellular tissues other than lesions in the cervix, and include normal cells. By adding cellular tissues within the cervix to the output data, the positional relationship between cellular tissues and lesions within the cervix can be recognized in more detail, and the location of lesions in the depth direction from the tissue surface (inner surface) can be accurately determined.

[0051] The output data produced by the learning model 131 is not limited to labeled images. The learning model 131 may, for example, output bounding boxes indicating the area and type of lesion included in medical images. The learning model 131 may also output information indicating the location of the lesion, for example, by direction. The learning model 131 may also output information indicating the degree of the lesion (normal, requires observation, excision, etc.).

[0052] The above example describes a learning model 131 that is a CNN, but the configuration of the learning model 131 is not limited; it just needs to be able to identify the location and type of lesions contained in medical images. Furthermore, the learning model 131 may include a first model that detects lesions and a second model that detects other cell tissues.

[0053] Figure 4 shows an example of the information stored in the training data DB132. The training data DB132 stores the training data in association with an ID that identifies the training data, medical images, and lesion data (information about lesions). The medical image column records medical images acquired using the image diagnostic device 2. The lesion data column records images in which each pixel constituting the medical image has a pixel value corresponding to an object class.

[0054] The information processing device 1 generates a learning model 131 using training data during the learning phase, which is a preliminary stage to the operational phase in which lesion estimation is performed, and stores the generated learning model 131. Then, during the operational phase, it generates estimation information using the stored learning model 131.

[0055] Figure 5 is a flowchart showing an example of the training data generation process. The following processes are executed by the control unit 11 in the learning phase according to the program 13P stored in the auxiliary storage unit 13 of the information processing device 1.

[0056] The control unit 11 of the information processing device 1 acquires a medical image from the image diagnostic device 2 (step S11). The medical image includes multiple frames in chronological order generated by a single pullback operation.

[0057] The control unit 11 acquires section images of previously performed cone biopsy sections, associated with the medical image (step S12). The section images are image data of cone biopsy sections taken from the same patient as the subject observed in the medical image. The observation range of the medical image includes the area of ​​the cone biopsy procedure. The control unit 11 may also acquire medical record information such as the definitive diagnosis result, associated with the image.

[0058] The control unit 11 displays a reception screen 151 on the display unit 15 for receiving information about lesions for each frame of the medical image (step S13). The control unit 11 receives information about lesions using the reception screen 151 (step S14).

[0059] Figure 6 is a schematic diagram showing an example of the reception screen 151. The reception screen 151 includes a medical image display unit 152, a section image display unit 153, and a lesion input unit 154. The medical image display unit 152 displays each frame of the medical image. The section image display unit 153 displays a section image associated with the medical image. The lesion input unit 154 displays an input field for receiving information about lesions in the medical image. The control unit 11 receives information about lesions (objects) entered into the input field when a doctor or other medical professional operates the operation unit 16. The information about the lesion includes the coordinate range corresponding to the lesion area and the type of lesion. The doctor or other medical professional inputs the lesion area using the medical image displayed on the medical image display unit 152, for example, with a mouse, and also inputs the type of lesion for that lesion area. This information about lesions is input based on the judgment of a specialist doctor or other medical professional with advanced knowledge of cervical cancer.

[0060] Medical images display acoustic and optical shadows corresponding to normal and diseased cells. Doctors input the location and type of lesions in the medical image by comparing the section image with each frame of the medical image. In this case, the control unit 11 may superimpose information indicating the correspondence between the orientation of the section image and the orientation of the medical image, depending on the doctor's selection. For example, based on the orientation input by the doctor, a line object indicating the orientation in the section image (in the example of Figure 6, the orientation shown by "what time") may be superimposed and displayed on the medical image. If the control unit 11 receives orientation information for any one frame of the medical image, it may apply the received orientation to all other frames in the same medical image and display them accordingly. The control unit 11 may rotate the medical image based on the orientation input by the doctor and display it on the medical image display unit 152. The control unit 11 may also superimpose information indicating the frame position of the medical image onto the section image. The frame position relative to the section image is calculated based on the correspondence between the observation range of the medical image and the surgical range of the cone biopsy.

[0061] Returning to Figure 5, the explanation continues. The control unit 11 generates training data, which is a dataset in which information about lesions is labeled as the correct value for medical images (step S15). More specifically, the control unit 11 generates training data in which the medical images are assigned labels (metadata) representing the coordinate range corresponding to the object region and the type of object.

[0062] The control unit 11 stores the generated training data in the training data DB 132 (step S16) and terminates the series of processes. The control unit 11 collects a large amount of medical images and information about lesions, and stores multiple sets of information generated based on the collected data as training data in the training data DB 132.

[0063] In the process described above, the control unit 11 may receive information about the lesion without acquiring section images. For example, if a physician or other medical professional can identify information about the lesion based solely on medical images, information about the lesion may be input without using section images.

[0064] Figure 7 is a flowchart showing an example of the process for generating the learning model 131. The following processes are executed by the control unit 11 according to the program 13P stored in the auxiliary storage unit 13 of the information processing device 1, for example, after the completion of the process shown in Figure 5 during the learning phase.

[0065] The control unit 11 of the information processing device 1 acquires a set of training data extracted from the information group based on the information stored in the training data DB 132 (step S21). Using the acquired training data, the control unit 11 generates a learning model 131 that outputs information about lesions from the tissue surface to the depth direction of the cervix when a medical image of the cervix is ​​input (step S22).

[0066] Specifically, the control unit 11 inputs each frame of medical images included in the training data as input data to the learning model 131, and obtains the coordinate range and type of the object output from the learning model 131. The control unit 11 calculates the error between the coordinate range and type of the output object and the coordinate range and type of the correct object using a predetermined loss function. The control unit 11 adjusts parameters such as the weights between nodes, for example, using backpropagation, to optimize (minimize or maximize) the loss function. Before learning begins, the definition information describing the learning model 131 is assumed to be given initial settings. When learning is completed because the error and the number of learning iterations meet predetermined criteria, optimized parameters are obtained.

[0067] Once learning is complete, the control unit 11 stores the definition information of the learned learning model 131 in the auxiliary storage unit 13 as the learned learning model 131 (step S23), and terminates the process according to this flowchart. Through the above process, a learned model 131 can be constructed that is capable of appropriately estimating information about lesions from the tissue surface to the depth direction of the cervix in medical images of the cervix.

[0068] Figures 5 and 7 above illustrate an example in which the control unit 11 of the information processing device 1 executes a series of processes, but the processing entity for each process is not limited. Some or all of the above processes may be executed by the control unit 31 of the image diagnostic device 2. The information processing device 1 and the image diagnostic device 2 may cooperate to perform a series of processes, for example, by performing inter-process communication. The learning model 131 may be generated by the information processing device 1 and learned by the image diagnostic device 2.

[0069] Using the learning model 131 generated as described above, the information processing system provides estimated information about the subject's medical images. The following describes the processing procedures performed by the information processing system during the operational phase.

[0070] Figure 8 is a flowchart showing an example of the procedure for acquiring estimated information. The following processes are executed by the control unit 11 according to the program 13P stored in the auxiliary storage unit 13 of the information processing device 1. The control unit 11 may perform the following processes in real time, for example, each time a medical image is transmitted from the image diagnostic device 2, or it may perform the processes retrospectively at any time based on recorded medical images.

[0071] The control unit 11 of the information processing device 1 acquires a medical image by receiving a medical image transmitted from the image diagnostic device 2 (step S31). The medical image is a tomographic image consisting of multiple frames in chronological order, including multiple frames that are continuous from the uterine side to the vaginal side along the depth direction of the cervix. The medical image may also be continuous from the vaginal side to the uterine side.

[0072] The control unit 11 inputs the acquired medical image as input data to the learning model 131 (step S32). The control unit 11 acquires the estimated information output from the learning model 131 (step S33). The estimated information is output as a labeled image having pixel values ​​corresponding to the location and type of lesions and other cells, for example.

[0073] The control unit 11 analyzes the label image to determine whether there is a discrepancy between it and adjacent label images (step S34). The method for determining the discrepancy is not limited, but as an example, the control unit 11 determines the presence or absence of a discrepancy by comparing the pixel values ​​of the target label image with the pixel values ​​of the preceding and succeeding label images. As described above, the label image is an image consisting of multiple consecutive frames extending in the depth direction of the cervix. Normally, since lesions have a certain length in the depth direction of the cervix, it is presumed that the pixel values ​​indicating the lesion are shown consecutively in the corresponding pixels of consecutive frames. Therefore, if the target frame shows a different pixel value from the preceding and succeeding frames, there is a high possibility that the estimation result shown by that pixel value is incorrect. The control unit 11 determines such a discrepancy in pixel values.

[0074] The control unit 11 determines whether there is a discrepancy by determining whether at least one of the pixel values ​​of the target label image matches the pixel values ​​of the preceding and succeeding label images. If the pixel value of the target label image matches the pixel value of the preceding label image and at least one of the pixel values ​​of the succeeding label image, the control unit 11 determines that there is no discrepancy. If the pixel value of the target label image does not match the pixel value of both the preceding label image and the succeeding label image, the control unit 11 determines that there is a discrepancy.

[0075] If the pixel value of the target label image matches the pixel value of both the previous label image and the subsequent label image, it is determined that the same lesion state has been detected in three consecutive frames, and the estimation results do not deviate. If the pixel value of the target label image matches either the pixel value of the previous label image or the pixel value of the subsequent label image, it indicates that the lesion appeared or disappeared before or after the target frame, and the estimation results do not deviate. On the other hand, if the pixel value of the target label image does not match the pixel value of both the previous label image and the subsequent label image, it indicates that only the pixel value of the target label image has changed, and the estimation results are determined to deviate. The control unit 11 performs the above determination process for each pixel of all label images.

[0076] If a discrepancy is determined (step S34: NO), the control unit 11 recognizes the pixels containing the discrepancy as noise (step S35). The control unit 11 performs a predetermined noise reduction process and returns to step S32. The noise reduction method is not limited, but for example, image processing is performed on the pixels of the original medical image corresponding to the pixels containing the discrepancy, by applying frame interpolation based on the pixel values ​​of the preceding and succeeding frames. The control unit 11 inputs the noise-reduced medical image into the learning model 131 and obtains the estimated information output from the learning model 131 to re-estimate information about the lesion. The control unit 11 may also return to step S32 without performing any noise reduction process.

[0077] If it is determined that there is no discrepancy (Step S34: YES), the control unit 11 estimates information suggesting the necessity of treatment such as conization based on the acquired estimated lesion information (Step S36), and derives the estimation result. The control unit 11 derives the estimated result of the necessity of treatment based on a table (not shown) that associates the type and location (size) of the lesion with the necessity of treatment. The control unit 11 may also estimate the necessity of treatment using machine learning methods, using a learning model that outputs information suggesting the necessity of treatment when estimated lesion information is input. Note that the processing in Step S36 is not mandatory.

[0078] The control unit 11 generates screen information including estimated information (step S37). The control unit 11 transmits the generated screen information to the image processing device 22, and the image processing device 22 displays the screen 231 including the estimated information on the display device 23 (step S38). The control unit 11 then completes the series of processes.

[0079] The above describes an example in which the control unit 11 of the information processing device 1 executes a series of processes, but the processing entity for each process is not limited. The process in Figure 8 may be partially or entirely executed by the control unit 31 of the image diagnostic device 2. The control unit 31 of the image diagnostic device 2 may store the learning model 131 acquired from the information processing device 1 in the auxiliary storage unit 33 and execute the process of acquiring estimated information based on the learning model 131. Furthermore, the estimated information is not limited to what is displayed on the display device 23 via the image processing device 22. The control unit 11 may output the estimated information to a device other than the image processing device 22 (for example, a personal computer) and have it displayed.

[0080] Figures 9 and 10 are schematic diagrams showing examples of screens 231 displayed on the display device 23. Figure 9 is an example of a screen 231 including a two-dimensional image, and Figure 10 is an example of a screen 231 including a three-dimensional image. The control unit 221 of the image processing device 22 receives screen information including estimated information transmitted from the information processing device 1, and based on the received screen information, displays the screen 231 including the estimated information on the display device 23 as shown in Figure 9 or Figure 10.

[0081] The screen 231, which includes a two-dimensional image, includes, for example, a medical image display unit 232, a two-dimensional image display unit 233, a label display unit 234, a treatment necessity display unit 235, and a display switching button 236. The medical image display unit 232 displays a medical image received from the image diagnostic device 2. The two-dimensional image display unit 233 displays a two-dimensional image in which estimated information, indicated by labels corresponding to lesions, is superimposed on the medical image. In the example shown in Figure 9, the two-dimensional image display unit 233 includes multiple two-dimensional images corresponding to multiple frames contained in the medical image, and an enlarged display of a two-dimensional image selected by the user from among the multiple two-dimensional images. The medical image display unit 232 may display a medical image corresponding to the enlarged two-dimensional image. The label display unit 234 displays the type of lesion indicated by the label and the display mode of the label in association. The treatment necessity display unit 235 displays the estimated result of whether treatment is necessary in text or the like. If the processing in step S36 is omitted, the estimated result of whether treatment is necessary will not be displayed, and doctors and other medical professionals may make a judgment on whether treatment is necessary based on other information.

[0082] The control unit 11 of the information processing device 1 processes the label image output from the learning model 131 into a semi-transparent mask for each frame of a series of medical images along the depth direction of the cervix, and generates a guide image superimposed on the original frame. In this case, the control unit 11 changes the display color of the mask according to the type of lesion, and otherwise varies the display manner of each lesion area according to the type of lesion. Furthermore, if the learning model 131 detects other tissue cells, the control unit 11 masks only the boundary portion of the pixels representing other cell tissues with other pixels, and omits the display of areas other than the boundary portion. In other words, only the tissue surface (inner surface) is masked and the inside of the tissue is omitted from the mask display. This makes it easy to understand the relationship between the tissue surface and the lesion. The control unit 11 displays the medical image and the 2D image in correspondence.

[0083] In addition to outputting screen information, the control unit 11 may also notify the user of information regarding lesions through warning sounds, synthesized speech, screen flashing, etc., depending on the type of lesion. If a lesion is included in the medical image, the information can be reliably notified by outputting a warning sound, etc.

[0084] When the control unit 221 receives a tap operation on the display switching button 236 while the screen 231 shown in Figure 9 is displayed, it displays the screen 231 including the three-dimensional image shown in Figure 10 on the display device 23. The screen 231 including the three-dimensional image includes, for example, a three-dimensional image display unit 237, a label display unit 234, a treatment necessity display unit 235, and the display switching button 236. The three-dimensional image display unit 237 overlays estimated information, indicated by labels corresponding to lesions, onto a three-dimensional image of the cervix. The configuration other than the three-dimensional image display unit 237 is the same as in Figure 9.

[0085] The control unit 11 of the information processing device 1 generates a three-dimensional image of the cervix including the lesion by stacking a plurality of label images (slice data) that are continuous along the depth direction. The three-dimensional image can be generated, for example, by the voxel method. The three-dimensional image is represented by volume data, which consists of the coordinate values ​​of voxels in a predetermined coordinate system and voxel values ​​indicating the type of lesion. The data format of the three-dimensional image is not particularly limited and may be polygon data or point cloud data. Other tissue cells in the three-dimensional image may only be shown on the tissue surface, similar to the two-dimensional image.

[0086] The control unit 11 of the information processing device 1 generates information for a screen 231 containing either a two-dimensional image or a three-dimensional image, in response to the operation of a display switching button 236 acquired via the image processing device 22, and outputs it to the image processing device 22. The control unit 11 may output information for both two-dimensional and three-dimensional images on the screen 231, and the image processing device 22 may switch the display. Furthermore, the screen 231 may include both a two-dimensional image display unit 233 and a three-dimensional image display unit 236, displaying the two-dimensional and three-dimensional images in parallel. Based on the screen information, physicians can grasp the state of the lesion in the depth direction, estimate whether treatment is necessary, and make a diagnosis. For subjects judged to have high-grade dysplasia, etc., a decision is made regarding treatment such as surgical excision.

[0087] According to this embodiment, information regarding lesions, including precancerous lesions, is accurately estimated using a learning model 131 based on the subject's medical images. The estimation results are displayed in a visually easily recognizable manner using two-dimensional and three-dimensional images, effectively supporting the diagnosis of physicians and other medical professionals. By using tomographic medical images, information regarding lesions in the depth direction, which cannot be obtained from tissue surface alone, can be estimated without taking tissue samples from the subject. In other words, the extent and progression of lesions can be estimated more accurately without examining tissue sections obtained by cone biopsy, thereby reducing the burden on the subject.

[0088] (Second Embodiment) In the second embodiment, the learning model 131 is retrained. Below, the differences from the first embodiment will be mainly described, and components common to the first embodiment will be denoted by the same reference numerals and their detailed descriptions will be omitted.

[0089] Figure 11 is a flowchart showing an example of the retraining process for the learning model in the second embodiment. The control unit 11 of the information processing device 1 acquires estimated information output from the learning model 131 (step S41). The control unit 11 acquires correction information for the estimated information (step S42). The control unit 11 may acquire the correction information by receiving input of correction information from a doctor or the like via the image processing device 22. For example, the control unit 221 of the image processing device 22 accepts correction input to correct the position or type of each object displayed on the screen 231 illustrated in Figure 9, and transmits the received correction information to the information processing device 1.

[0090] The control unit 11 retrains the learning model 131 using the correction information for the estimated information (step S43). Specifically, the control unit 11 retrains the learning model 131 using the medical images input to the learning model 131 and the correction information for the estimated information as training data, and updates the learning model 131. In other words, the control unit 11 optimizes parameters such as weights between nodes so that the estimated information output from the learning model 131 approximates the corrected estimated information, and regenerates the learning model 131.

[0091] According to this embodiment, the learning model 131 can be further optimized through the operation of this information processing system.

[0092] The embodiments disclosed above are illustrative in all respects and not restrictive. The scope of the invention is indicated by the claims, and all modifications within the meaning and scope of the claims are intended to be included. Furthermore, at least some of the embodiments described above may be combined as desired. [Explanation of symbols]

[0093] 1. Information Processing Device 11 Control Unit 12 Main memory 13 Auxiliary storage 14 Communications Department 15 Display 16 Control section 13P Program 131 Learning Models 132 Training Data Database 13A Recording medium 2. Diagnostic imaging equipment 21 Detection device 211 Insertion tube 212 Drive unit 213 Probe section 214 Connector section 215 Shaft 216 sensors 22 Image Processing Devices 221 Control Unit 222 Main memory 223 Auxiliary storage 224 Communications Department 225 Input / output section 226 Detection device control unit 23 Display device 24 Input devices

Claims

1. Medical images of the cervix are obtained, Based on the acquired medical images, information regarding lesions in the cervical tissue from the surface to the depth is obtained. Display the aforementioned medical image and information regarding the lesion. It is a process, A series of medical images along the depth direction of the cervix are acquired. Determine whether there is a discrepancy in the information regarding the lesion between any one of the target medical images and medical images adjacent to the target medical image. If a discrepancy is detected, noise reduction processing is performed on the target medical image based on the frames before and after the target medical image from among the multiple medical images acquired. Based on the aforementioned medical image after noise reduction, information regarding the lesion is reacquired. An information processing method that involves having a computer perform a task.

2. The medical image of the cervix is ​​input to a learning model that has been trained to output information about lesions from the tissue surface to the depth direction of the cervix, and the acquired medical image is input to obtain the information about the lesions output from the learning model. The information processing method according to claim 1.

3. A learning model, trained to output an image showing information about lesions from the tissue surface to the depth direction of the cervix when a medical image of the cervix is ​​input, is used to obtain an image showing information about lesions output by the learning model. The information processing method according to claim 1 or claim 2.

4. The information relating to the lesion includes the location and type of the lesion. The information processing method according to any one of claims 1 to 3.

5. A series of medical images along the depth direction of the cervix are acquired. For each of the acquired medical images, information regarding the lesion is estimated. The information processing method according to any one of claims 1 to 4.

6. A series of medical images along the depth direction of the cervix are acquired. Based on the acquired medical images, a three-dimensional image of the cervix containing information about the lesion is generated. The information processing method according to any one of claims 1 to 5.

7. The image showing information about a lesion output from the learning model is a binarized label image indicating whether or not each pixel of the acquired medical image is a lesion, By comparing the pixel values ​​of the target label image output from the learning model based on the target medical image with the pixel values ​​of the preceding and succeeding label images output from the learning model based on the preceding and succeeding frames of the target medical image, the presence or absence of a discrepancy is determined. If a discrepancy is detected, the noise reduction process involves performing image processing on the target medical image, which includes frame interpolation based on the pixel values ​​of the preceding and succeeding frames. The information processing method according to claim 3.

8. Based on information regarding lesions extending from the surface to the depth of the cervical tissue, the necessity of treatment is estimated. The information processing method according to any one of claims 1 to 7.

9. Medical images of the cervix are obtained, Based on the acquired medical images, information regarding lesions in the cervical tissue from the surface to the depth is obtained. Output the aforementioned medical image and information relating to the lesion. It is a process, A series of medical images along the depth direction of the cervix are acquired. Determine whether there is a discrepancy in the information regarding the lesion between any one of the target medical images and medical images adjacent to the target medical image. If a discrepancy is detected, noise reduction processing is performed on the target medical image based on the frames before and after the target medical image from among the multiple medical images acquired. Based on the aforementioned medical image after noise reduction, information regarding the lesion is reacquired. A program that causes a computer to perform a process.

10. A first acquisition unit for acquiring medical images of the cervix, A second acquisition unit acquires information regarding lesions in the cervical tissue from the surface to the depth direction based on the medical image acquired by the first acquisition unit, The system includes an output unit that outputs the aforementioned medical image and information regarding the lesion, A series of medical images along the depth direction of the cervix are acquired. Determine whether there is a discrepancy in the information regarding the lesion between any one of the target medical images and medical images adjacent to the target medical image. If a discrepancy is detected, noise reduction processing is performed on the target medical image based on the frames before and after the target medical image from among the multiple medical images acquired. Based on the aforementioned medical image after noise reduction, information regarding the lesion is reacquired. Information processing device.

Images