Information processing apparatus, information processing method, and information processing program

Abstract

To obtain an information processing apparatus, an information processing method, and an information processing program that can make efficient the collection, management, and operation of learning data.SOLUTION: An information processing apparatus 20 comprises at least one processor, and the processor converts an image added with additional information into structural data, provides annotation information to the structural data, and trains a learning model using the structural data as input and the annotation information as output, by using the combination of the structural data and annotation information.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, an information processing method, and an information processing program. [Background technology]

[0002] Research has been conducted on diagnostic support for medical images using machine learning. Generally, to achieve a high-performance learning model using supervised learning, a large amount of annotations defined to correspond to medical images must be prepared as training data. Annotations indicate, for example, the location of lesions and structures in medical images, as well as classification information related to their characteristics and diagnosis. Furthermore, annotations are not limited to information within medical images; information such as the subject's age and gender, as well as other test results, may also be added.

[0003] For example, Patent Document 1 discloses that information is managed in a hierarchical structure, such as a hierarchy of patient information, a hierarchy of medical images, and a hierarchy of findings information, and annotation information is associated with each hierarchy, thereby supplying learning data with appropriately constructed hierarchies and relationships.

[0004] Furthermore, for example, Patent Document 2 discloses that a large amount of learning data can be easily collected by classifying endoscopic images into groups based on the linked finding diagnostic information on a screen for creating learning data by linking endoscopic images with finding diagnostic information. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] International Publication No. 2020 / 153493 [Patent Document 2] International Publication No. 2019 / 230302 Summary of the Invention [Problem to be solved by the invention]

[0006] In recent years, there has been a demand for the construction of learning models that enable diagnostic support for each medical image taken by various imaging devices (modalities), such as CT (Computed Tomography) devices, MRI (Magnetic Resonance Imaging) devices, and endoscopes. Medical images taken by various imaging devices may have different formats depending on the type of imaging device or the settings of the imaging device, which makes the collection, management, and operation of learning data complicated.

[0007] For example, when using video captured by an endoscope as learning data, some frames are extracted from the video as still images, and annotations are added to the extracted still images. In this case, it is necessary to manage the video captured by the endoscope and the still images and annotations for the learning data separately. Therefore, even if information that can be used as annotations, such as information about the subject and other test results, is associated with the video, it can be time-consuming to refer to the information from the still images.

[0008] The present disclosure provides an information processing device, an information processing method, and an information processing program that improve the efficiency of collection, management, and operation of learning data. [Means for solving the problem]

[0009] A first aspect of the present disclosure is an information processing device comprising at least one processor, which converts an image accompanied by additional information into structured data, adds annotation information to the structured data, and uses a combination of the structured data and the annotation information to train a learning model whose input is structured data and whose output is annotation information.

[0010] In the above aspect, the processor may extract at least one image from a group of images accompanied by accompanying information, and convert the extracted image into structured data.

[0011] In the above aspect, the series of images may be a moving image that includes a plurality of images in chronological order.

[0012] In the above aspect, the processor may extract, from the series of images, images that include a region of a predetermined structure.

[0013] In the above aspect, the structured data may be data that complies with DICOM (Digital Imaging and Communications in Medicine).

[0014] In the above aspect, the processor may accept input of annotation information by a user.

[0015] In the above aspect, the processor may identify a region of interest in an image included in the structured data, and generate annotation information based on the identified region of interest.

[0016] In the above aspect, the processor may generate annotation information based on additional information included in the structured data.

[0017] In the above aspect, the processor may perform at least one of modifying, deleting, and adding to the auxiliary information, and then convert the auxiliary information into structured data.

[0018] In the above aspect, the processor may generate new annotation information for the new structured data by inputting new structured data that is different from the structured data used to train the learning model into the trained learning model.

[0019] In the above aspect, the processor may retrain the learning model using a combination of the new structured data and the generated new annotation information.

[0020] In the above aspect, the processor may assign different types of first annotation information and second annotation information to the structured data, and use the combination of the structured data and the first annotation information to train a first learning model that has the structured data as input and the first annotation information as output, and use the combination of the first annotation information and the second annotation information to train a second learning model that has the first annotation information as input and the second annotation information as output.

[0021] In the above aspect, the supplementary information may indicate at least one of information relating to the subject in the image and information relating to the imaging of the image.

[0022] In the above aspect, the image may be a medical image.

[0023] In the above aspect, the annotation information may indicate at least one of the position and the size of a region of interest included in the image.

[0024] In the above aspect, the annotation information may indicate findings of the image.

[0025] A second aspect of the present disclosure is an information processing method in which a computer executes a process of converting an image accompanied by additional information into structured data, adding annotation information to the structured data, and using a combination of the structured data and the annotation information to train a learning model whose input is structured data and whose output is annotation information.

[0026] A third aspect of the present disclosure is an information processing program for causing a computer to execute a process of converting an image accompanied by additional information into structured data, adding annotation information to the structured data, and using a combination of the structured data and the annotation information to train a learning model whose input is structured data and whose output is annotation information. [Effects of the Invention]

[0027] According to the above aspects, the information processing device, information processing method, and information processing program of the present disclosure can improve the efficiency of collecting, managing, and using learning data. [Brief explanation of the drawings]

[0028] [Figure 1] FIG. 1 is a diagram illustrating an example of a schematic configuration of an information processing system. [Figure 2] FIG. 2 is a block diagram illustrating an example of a hardware configuration of an information processing device. [Figure 3] FIG. 2 is a block diagram illustrating an example of a functional configuration of an information processing device. [Figure 4] FIG. 10 is a diagram illustrating an example of a screen displayed on a display. [Figure 5] FIG. 10 is a diagram for explaining a diagnostic model. [Figure 6] 10 is a flowchart illustrating an example of information processing. DETAILED DESCRIPTION OF THE INVENTION

[0029] An example of an embodiment of the disclosed technology will be described below with reference to the drawings. In each drawing, identical or equivalent components and parts are designated by the same reference numerals, and duplicate descriptions will be omitted. Also, the dimensional proportions in the drawings are exaggerated for the sake of explanation and may differ from the actual proportions.

[0030] First, an information processing system 1 to which an information processing device 20 according to this embodiment is applied will be described with reference to Fig. 1. Fig. 1 is a diagram showing a schematic configuration of the information processing system 1. The information processing system 1 is a system for collecting learning data for training a diagnostic model M based on information obtained by various devices actually operating in a medical institution or the like (so-called production environment), and for managing and operating the learning data.

[0031] Specifically, the information processing system 1 includes a medical institution system 10 already established in a medical institution such as a hospital, and an information processing device 20. The medical institution system 10 and the information processing device 20 are connected to each other in a communicable state via a wired or wireless network 9. The network 9 is, for example, a network such as a LAN (Local Area Network) or a WAN (Wide Area Network).

[0032] The medical institution system 10 photographs the subject's examination area and stores the photographed medical images based on examination orders from doctors in the medical departments using a known ordering system. It also enables radiologists to interpret the medical images and create radiology reports, and allows doctors in the requesting medical departments to view the radiology reports.

[0033] The medical institution system 10 includes an imaging device 11, an interpretation WS (Workstation) 12 which is an interpretation terminal, a medical treatment WS 13, an image server 14, and a report server 16. The medical institution system 10 may also include known information systems such as a RIS (Radiology Information System) 18 and an HIS (Hospital Information System) 19. The imaging device 11, the interpretation WS 12, the medical treatment WS 13, the image server 14, the report server 16, the RIS 18, and the HIS 19 are connected to each other via a wired or wireless network 9 in a state where they can communicate with each other.

[0034] Each device is a computer installed with an application program that causes the device to function as a component of the medical institution system 10. The application program may be recorded on a recording medium such as a DVD-ROM (Digital Versatile Disc Read Only Memory) or a CD-ROM (Compact Disc Read Only Memory) and distributed, and then installed on the computer from the recording medium. Alternatively, the application program may be stored in a storage device or network storage of a server computer connected to the network 9 in an externally accessible state, and downloaded to the computer upon request and installed.

[0035] The imaging device 11 is a device (modality) that captures an image of a diagnostic target region of a subject, thereby generating a medical image representing the diagnostic target region. Examples of the imaging device 11 include a plain X-ray imaging device, a CT (Computed Tomography) device, an MRI (Magnetic Resonance Imaging) device, a PET (Positron Emission Tomography) device, an ultrasound diagnostic device, an endoscope, and a fundus camera. The medical image generated by the imaging device 11 is transmitted to the image server 14.

[0036] The image interpretation workstation 12 is a computer used by medical professionals, such as radiologists, to interpret medical images and prepare image interpretation reports, and is configured to include a processing device, a display device such as a monitor, and an input device such as a keyboard and a mouse. The image interpretation workstation 12 issues requests for viewing medical images to the image server 14, performs various image processing on medical images received from the image server 14, displays the medical images, and accepts input of text related to the medical images. The image interpretation workstation 12 also performs analysis processing on medical images, supports the preparation of image interpretation reports based on the analysis results, requests the report server 16 to register and view image interpretation reports, and displays image interpretation reports received from the report server 16. These processes are performed by the image interpretation workstation 12 executing software programs for each process.

[0037] The clinical workstation 13 is a computer used by medical personnel such as doctors in a clinical department to perform detailed observations of medical images, view interpretation reports, and create electronic medical records, and is configured to include a processing device, a display device such as a monitor, and input devices such as a keyboard and a mouse. The clinical workstation 13 requests the image server 14 to view medical images, displays medical images received from the image server 14, requests the report server 16 to view interpretation reports, and displays interpretation reports received from the report server 16. These processes are performed by the clinical workstation 13 executing software programs for each process.

[0038] The image server 14 is a general-purpose computer installed with a software program that provides the functions of a database management system (DBMS). The image server 14 is connected to an image DB (DataBase) 15. When the image server 14 receives a request to register a medical image from the imaging device 11, it converts the medical image into a database format and registers it in the image DB 15. When the image server 14 receives a request to view from the image interpretation workstation 12 or the medical treatment workstation 13, it searches for medical images registered in the image DB 15 and transmits the searched medical image to the image interpretation workstation 12 or the medical treatment workstation 13 that sent the view request.

[0039] The image DB 15 is realized by a storage medium such as a hard disk drive (HDD), a solid state drive (SSD), or a flash memory. Medical images acquired by the imaging device 11 and metadata related to the medical images are registered in the image DB 15 in association with each other. The connection between the image server 14 and the image DB 15 is not particularly limited, and they may be connected by a data bus or via a network such as a network attached storage (NAS) or a storage area network (SAN).

[0040] The metadata may include, for example, identification information such as an image ID (identification) for identifying a medical image, a cross-sectional ID assigned to each cross-sectional image included in the medical image, a subject ID for identifying the subject, and an examination ID for identifying the examination. The metadata may also include, for example, information related to the medical image capture, such as the imaging method, imaging conditions, imaging purpose, and imaging date and time. Examples of the "imaging method" and "imaging conditions" include the type of imaging device 11, the manufacturer, the imaging region, the imaging protocol, the imaging sequence, the imaging technique, whether or not a contrast agent was used, and the slice thickness in tomography. The metadata may also include information related to the subject, such as the subject's name, date of birth, age, and gender. The metadata may be obtained, for example, from the RIS 18 and the HIS 19.

[0041] The report server 16 is a general-purpose computer installed with a software program that provides the functions of a database management system. The report server 16 is connected to a report DB 17. When the report server 16 receives a request to register an interpretation report from the image interpretation workstation 12, it formats the interpretation report in a database format and registers it in the report DB 17. When the report server 16 receives a request to view an interpretation report from the image interpretation workstation 12 or the medical care workstation 13, it searches for the interpretation report registered in the report DB 17 and transmits the found interpretation report to the image interpretation workstation 12 or the medical care workstation 13 that sent the view request.

[0042] The report DB 17 is realized by a storage medium such as an HDD, SSD, or flash memory. The report DB 17 registers the interpretation reports created by the interpretation WS 12. The connection between the report server 16 and the report DB 17 is not particularly limited, and they may be connected by a data bus or via a network such as a NAS or a SAN.

[0043] The devices included in the medical institution system 10 may be located in the same facility (e.g., a hospital) or in different facilities. The number of devices included in the medical institution system 10 is not particularly limited, and each device may be composed of multiple devices each having the same function.

[0044] The information processing device 20 uses various information such as medical images, metadata, and radiology reports obtained in the medical institution system 10 as learning data to train a diagnostic model M for making diagnoses related to medical images. That is, the information processing device 20 collects learning data for the diagnostic model M and manages and operates the learning data. The information processing device 20 is connected to a learning data DB 29.

[0045] The learning data DB 29 is realized by a storage medium such as an HDD, an SSD, or a flash memory. Learning data collected by the information processing device 20 is registered in the learning data DB 29. The connection between the information processing device 20 and the learning data DB 29 is not particularly limited, and they may be connected by a data bus or via a network such as a NAS or a SAN.

[0046] Here, it is required to construct a diagnostic model M that can accommodate each of the medical images captured by various imaging devices 11, such as a CT device, an MRI device, and an endoscope. However, the medical images captured by the various imaging devices 11 may have different formats depending on the type of imaging device 11 or the settings of the imaging device 11, which complicates the collection, management, and operation of the learning data.

[0047] For example, when using video images captured by an endoscope as learning data, it is conceivable to extract some frames from the video images as still images and add annotations to the extracted still images. In this case, the video images captured by the endoscope are managed by the image server 14, and the still images and annotations for the learning data are managed by the information processing device 20. When video images and still images are managed separately in this way, even if information that could be used as annotations (e.g., metadata and radiology reports) is associated with the video images, it may be time-consuming to refer to the information from the still images.

[0048] Therefore, the information processing device 20 converts various information such as medical images, metadata, and radiology reports obtained in the medical institution system 10 into structured data that conforms to a predetermined standard, thereby improving the efficiency of collection, management, and operation of learning data. The information processing device 20 will be described in detail below.

[0049] First, an example of the hardware configuration of the information processing device 20 will be described with reference to Fig. 2. The information processing device 20 includes a CPU (Central Processing Unit) 21, a non-volatile storage unit 22, and a memory 23 as a temporary storage area. The information processing device 20 also includes a display 24 such as a liquid crystal display, an input unit 25 such as a keyboard and a mouse, and a network I / F (Interface) 26. The network I / F 26 is connected to the network 9 and performs wired and / or wireless communication. The CPU 21, the storage unit 22, the memory 23, the display 24, the input unit 25, and the network I / F 26 are connected via a bus 28 such as a system bus and a control bus so as to be able to exchange various information with one another.

[0050] The storage unit 22 is realized by a storage medium such as an HDD, an SSD, or a flash memory. The storage unit 22 stores an information processing program 27 for the information processing device 20. The CPU 21 reads the information processing program 27 from the storage unit 22, loads it into the memory 23, and executes the loaded information processing program 27. The CPU 21 is an example of a processor of the present disclosure. The storage unit 22 also stores a diagnostic model M for making a diagnosis regarding medical images (details will be described later). As the information processing device 20, for example, a personal computer, a server computer, a smartphone, a tablet terminal, a wearable terminal, etc. can be appropriately applied.

[0051] Next, an example of the functional configuration of the information processing device 20 will be described with reference to Fig. 3. The information processing device 20 includes an acquisition unit 30, a processing unit 32, a conversion unit 34, an assignment unit 36, a learning unit 38, and a control unit 40. When the CPU 21 executes the information processing program 27, the CPU 21 functions as each of the functional units of the acquisition unit 30, the processing unit 32, the conversion unit 34, the assignment unit 36, the learning unit 38, and the control unit 40.

[0052] The acquisition unit 30 acquires medical images accompanied by additional information. The additional information indicates at least one of information about the subject in the medical image and information about the imaging of the medical image. For example, the acquisition unit 30 may acquire medical images from the imaging device 11 and / or the image server 14, and may acquire metadata and interpretation reports about the medical images as additional information from the image server 14, the report server 16, the RIS 18, and / or the HIS 19.

[0053] The medical image acquired by the acquisition unit 30 may be at least one image, or may be a series of images accompanied by additional information. For example, it may be a still image, or a moving image including multiple medical images in chronological order. It may also be a group of images consisting of multiple still images captured multiple times at time intervals. A medical image is an example of an image disclosed herein.

[0054] An example of a medical image and accompanying information acquired by the acquisition unit 30 is shown in a speech bubble for the acquisition unit 30 in Fig. 3. The medical image illustrated in Fig. 3 is moving picture data in MPEG (Moving Picture Experts Group) format (illustrated as "ES001.mp4"). One piece of accompanying information attached to the moving picture data is metadata in CSV (Comma Separated Values) format (illustrated as "A001.csv"). Another piece of accompanying information attached to the moving picture data is text data representing an interpretation report (illustrated as "B001.txt").

[0055] The processing unit 32 performs various processing operations on the medical images and the accompanying information acquired by the acquisition unit 30. Specifically, the processing unit 32 may extract at least one image from a series of images to which accompanying information is attached. For example, the processing unit 32 may extract an image that includes an area of a predetermined structure from the series of images. For example, when video data captured by an endoscope includes a frame capturing an image of the esophagus and a frame capturing an image of the stomach, the processing unit 32 may clip only the frame capturing the image of the stomach. Furthermore, the processing unit 32 may perform at least one of changing, deleting, and adding the accompanying information to the accompanying information.

[0056] An example of processing of medical images and incidental information by the processing unit 32 is shown in the speech bubble for the processing unit 32 in FIG. 3. In FIG. 3, the processing unit 32 extracts some frames from the video data acquired by the acquisition unit 30 to obtain still image data in JPEG (Joint Photographic Experts Group) format (illustrated as "IMG001.jpg," "IMG002.jpg," ...). The processing unit 32 also anonymizes the metadata acquired by the acquisition unit 30 by deleting the subject's name information (illustrated as "Name: Fuji Taro") from the metadata. The processing unit 32 also changes the information that was collectively listed under the "Examination Date and Time" heading in the metadata to separate it into "Examination Date" and "Examination Time" headings. The processing unit 32 also extracts the finding information "tumor" from the text data representing the radiology report and adds (integrates) it into the metadata.

[0057] Note that a known method can be appropriately applied as a method for extracting finding information related to a medical image from a radiology report. For example, a named entity extraction method using a natural language processing model such as BERT (Bidirectional Encoder Representations from Transformers) can be applied to extract named entities (words) from a finding sentence, and the finding information corresponding to the extracted named entities can be identified.

[0058] Finding information is information that indicates at least one of various findings, such as the type (name), characteristics, position, measurement values and number of areas of interest contained in a medical image, words or phrases that describe changes in the areas of interest, and the suspected disease name (diagnosis result).

[0059] Examples of types (names) of regions of interest include names of structures such as "lung" and "liver" and names of lesions such as "nodule" and "tumor." The properties of a region of interest primarily refer to the characteristics of the lesion. For example, in the case of nodules, these include absorption values such as "solid," "low absorption," and "high absorption," margin shapes such as "clear / indistinct," "smooth / irregular," "spicules," "lobulated," and "serrated," and findings indicating the overall shape such as "near-circular" and "irregular." Other examples include the relationship with surrounding tissues such as "protrusion," and findings regarding the presence or absence of contrast enhancement and washout.

[0060] The location of a region of interest refers to its anatomical location, its location in a medical image, and its relative positional relationship with other regions of interest, such as "inside," "edge," and "surroundings." The anatomical location may be expressed by organ names, such as "lung" and "liver," or by subdivisions of the lungs, such as "right lung," "upper lobe," and apical segment ("S1").

[0061] The measurement value of a region of interest is a value that can be quantitatively measured from a medical image, such as at least one of the size and signal value of the region of interest. The size is expressed, for example, by the long and short diameters, area, and volume of the region of interest. The signal value is expressed, for example, by the pixel value of the region of interest and the CT value in units of HU. The finding information indicating the measurement value may be divided into predetermined classes, such as "0 mm or more but less than 5 mm," "5 mm or more but less than 10 mm," or "10 mm or more but less than 15 mm." In this case, for example, if the radiology report contains a description of "12 mm," the finding information of "10 mm or more but less than 15 mm" is identified. The finding information indicating the measurement value may also simply indicate whether or not the measurement value is described in the radiology report. This is because measurement values may vary widely or may be omitted. The number of regions of interest may be expressed as a specific number, such as one or two, or in relative terms, such as "single / multiple" and "few / many."

[0062] The term describing a change in the region of interest is a term describing the change over time in the characteristics, location, measurement values, number, etc., when observing the region of interest over time, and examples of such terms include "appearance / disappearance," "increase / reduction," "worsening / improvement," and "metastasis." The estimated disease name is an evaluation result estimated based on the lesion contained in the medical image, and examples include disease names such as "cancer" and "inflammation," as well as evaluation results regarding the disease name and characteristics such as "negative / positive," "benign / malignant," and "mild / severe."

[0063] The conversion unit 34 converts the image with the accompanying information into structured data. Specifically, the conversion unit 34 converts the image (e.g., an image extracted from a series of images) processed by the processing unit 32 into structured data. The conversion unit 34 also converts the accompanying information processed by the processing unit 32 into structured data.

[0064] Structured data is data in which multiple attribute items, including at least each item and image included in the supplementary information, are associated with the values of each attribute item. In other words, structured data is data that conforms to a predetermined standard for what attribute items are included and how the values of each attribute item are represented (encoding format).

[0065] An example of such a standard is DICOM (Digital Imaging and Communications in Medicine), which is used as an international standard for management and communication methods of medical images. That is, the structured data may be data that conforms to DICOM. An example of DICOM-format structured data converted from the medical image and accompanying information processed by the processing unit 32 is shown in the speech bubble for the conversion unit 34 in FIG. 3 (illustrated as "TD001.dcm"). In DICOM-format structured data (hereinafter referred to as "DICOM data"), images are coded as "pixel information." In DICOM data, attribute items are also called tags.

[0066] The standard for structured data is not limited to DICOM, and any standard that can associate supplementary information (text data) with an image can be used, such as PDF (Portable Document Format).

[0067] The annotation unit 36 adds annotation information to the structured data. The annotation information may indicate, for example, at least one of the position and size of a region of interest (lesion and structure) included in the medical image. Alternatively, for example, the annotation information may indicate findings in the medical image. The findings in the medical image are the same as the above-described finding information. Alternatively, for example, the annotation information is not limited to information in the medical image, but may also include information about the subject not included in the incidental information, information indicating other test results, and the like.

[0068] Specifically, the assigning unit 36 may accept input of annotation information by the user. Fig. 4 shows an example of a screen D1 for accepting input of annotation information, which is displayed on the display 24 by the control unit 40. The screen D1 includes a medical image B1, an area 91 for inputting annotation information A1 indicating the position and size of a region of interest in the medical image B1, and an area 92 for inputting annotation information A2 indicating findings in the medical image B1.

[0069] For example, the user generates a bounding box that surrounds a region of interest by operating a pointer 90 via the input unit 25 (mouse). The assigning unit 36 generates annotation information A1 based on coordinate information of the bounding box in the medical image B1. Alternatively, for example, the user may input annotation information A2 in text format via the input unit 25 (keyboard). In the example of FIG. 4, annotation information A2 of "malignant" is input for the incidental information of "tumor."

[0070] Furthermore, since DICOM is a widely used standard, there are many existing software and hardware (viewers) that support viewing of DICOM data. Therefore, the annotation unit 36 may display the screen D1 on the display of any terminal device that supports viewing of DICOM data, and may accept input of annotation information on the terminal device.

[0071] The assigning unit 36 may also identify a region of interest in a medical image included in the structured data and generate annotation information based on the identified region of interest. For example, the assigning unit 36 may extract a region of interest from a medical image using a learning model such as a convolutional neural network (CNN) that receives the medical image as input and is trained to extract and output a region of interest included in the medical image. The assigning unit 36 may use information indicating the position and size of the extracted region of interest as annotation information.

[0072] Furthermore, the annotation unit 36 may generate findings for the region of interest using a learning model such as CNN that is pre-trained to input the extracted region of interest and output findings for the region of interest. The annotation unit 36 may use the generated findings as annotation information. The annotation unit 36 may also generate annotation information based on incidental information included in the structured data.

[0073] The annotation unit 36 may apply the technology described in International Publication No. 2020 / 153493 when adding annotation information to structured data. That is, the annotation information may be managed in a hierarchical structure such as by patient, image, or finding.

[0074] The attachment unit 36 stores the combination of annotation information and structured data generated as described above in the training data DB 29. In this case, the attachment unit 36 may attach, to the training data, metadata such as information about the user who generated the annotation information, such as an identification ID of the user, and the date and time when the annotation information was generated.

[0075] The learning unit 38 trains the diagnostic model M using a combination of structured data and annotation information stored in the learning data DB 29. As shown in FIG. 5, the diagnostic model M is a learning model that uses structured data as input and annotation information as output. The diagnostic model M is a machine learning model using supervised learning, including a neural network such as a CNN and an RNN (Recurrent Neural Network). The diagnostic model M is an example of a learning model of the present disclosure.

[0076] That is, new structured data ("TD100.dcm") different from the structured data ("TD001.dcm") used to train the diagnostic model M is input to the trained diagnostic model M, thereby generating new annotation information for the new structured data. The trained diagnostic model M can be used, for example, when interpreting medical images in the interpretation WS 12.

[0077] Furthermore, the learning unit 38 may re-learn the diagnostic model M using a combination of the new structured data and the new annotation information generated by the diagnostic model M. In this case, the learning unit 38 may accept user corrections to the new annotation information.

[0078] The diagnostic model M may be configured by combining multiple learning models. For example, a first-stage learning model may be a region-of-interest extraction model that receives structured data as input and is trained to extract and output a region of interest included in a medical image included in the structured data. A second-stage learning model may be a finding information generation model that receives a region of interest extracted by the region-of-interest extraction model as input and is trained to output finding information about the region of interest.

[0079] That is, the assigning unit 36 may assign different types of first annotation information (e.g., the position of a region of interest) and second annotation information (e.g., findings) to the structured data. The learning unit 38 may use a combination of the structured data and the first annotation information to train a first learning model in which the structured data is input and the first annotation information is output. Furthermore, the learning unit 38 may use a combination of the first annotation information and the second annotation information to train a second learning model in which the first annotation information is input and the second annotation information is output.

[0080] Next, the operation of the information processing device 20 will be described with reference to Fig. 6. In the information processing device 20, the CPU 21 executes the information processing program 27, thereby executing the information processing shown in Fig. 6. The information processing is executed, for example, when a user issues an instruction to start execution via the input unit 25.

[0081] In step S10, the acquisition unit 30 acquires a medical image accompanied by additional information. In step S12, the processing unit 32 performs various processing operations on the medical image and additional information acquired in step S10. In step S14, the conversion unit 34 converts the image accompanied by additional information, which has been processed in step S12, into structured data.

[0082] In step S16, the assigning unit 36 assigns annotation information to the structured data converted in step S14. In step S18, the learning unit 38 trains the diagnostic model M using the structured data to which the annotation information has been assigned in step S16, i.e., the combination of the structured data and the annotation information, and then ends this information processing.

[0083] As described above, the information processing device 20 according to one embodiment of the present disclosure includes at least one processor, which converts an image with accompanying information into structured data, adds annotation information to the structured data, and uses a combination of the structured data and the annotation information to train a learning model whose input is structured data and whose output is annotation information.

[0084] In other words, by converting images and incidental information into structured data, the collection, management, and operation of training data can be made more efficient than when various medical images and incidental information in non-standard formats are individually collected, managed, and operated. Furthermore, centralized management of training data in a standardized format is advantageous for ensuring security.

[0085] Furthermore, using a widely used standard such as DICOM as the structured data standard ensures compatibility with existing software and hardware (viewers), improving convenience. For example, by using attribute items in DICOM data, DICOM data can be classified by image type, subject, or series, making it easy to manage the relationships between DICOM data (medical images). In other words, it is easy for users or computers to refer to the information needed to add annotation information, enabling efficient annotation, which is advantageous for improving the efficiency of collecting, managing, and operating training data.

[0086] In the above embodiment, the information processing device 20 has been described as having the functional units of the acquisition unit 30, the processing unit 32, the conversion unit 34, the attachment unit 36, the learning unit 38, and the control unit 40, but is not limited to this. For example, a computer having the function of the processing unit 32 may upload the processed medical images and accompanying information to an external DB server, and another computer having the function of the conversion unit 34 may download the medical images and accompanying information from the DB server and convert them into structured data.

[0087] In the above embodiment, the processing unit 32 processes the medical images and the incidental information before converting them into structured data, but this is not limiting. For example, the structured data converted by the conversion unit 34 may be processed to extract the medical images and to change, delete, or add the incidental information.

[0088] Furthermore, in the above embodiment, diagnostic support for medical images using machine learning has been described, but the present disclosure is not limited thereto. The technology of the present disclosure can be applied to various images obtained by photographing an object and including a region of interest. For example, in non-destructive testing such as radiographic testing and ultrasonic flaw detection testing, images obtained from equipment, buildings, pipes, welds, and the like as objects may be applied. In this case, the region of interest may indicate, for example, cracks, flaws, bubbles, foreign objects, and the like.

[0089] Furthermore, in the above embodiment, the following various processors can be used as the hardware structure of processing units that perform various processes, such as the acquisition unit 30, the processing unit 32, the conversion unit 34, the assignment unit 36, the learning unit 38, and the control unit 40. As described above, the various processors include a CPU, which is a general-purpose processor that executes software (programs) and functions as various processing units, as well as dedicated electrical circuits that are processors having a circuit configuration specifically designed to perform specific processes, such as a programmable logic device (PLD), a processor whose circuit configuration can be changed after manufacture, such as an FPGA (Field Programmable Gate Array), and an ASIC (Application Specific Integrated Circuit).

[0090] A single processing unit may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, multiple processing units may be configured with a single processor.

[0091] Examples of configuring multiple processing units with a single processor include, first, a form in which one processor is configured with a combination of one or more CPUs and software, and this processor functions as multiple processing units, as typified by computers such as client and server. Second, a form in which a processor is used to realize the functions of the entire system including multiple processing units with a single IC (Integrated Circuit) chip, as typified by systems on chips (SoCs). In this way, various processing units are configured using one or more of the above-mentioned various processors as a hardware structure.

[0092] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements.

[0093] In the above embodiment, the information processing program 27 is pre-stored (installed) in the storage unit 22, but this is not limiting. The information processing program 27 may be provided in a form recorded on a recording medium such as a CD-ROM (Compact Disc Read Only Memory), a DVD-ROM (Digital Versatile Disc Read Only Memory), or a USB (Universal Serial Bus) memory. The information processing program 27 may also be downloaded from an external device via a network. Furthermore, the technology disclosed herein extends to not only information processing programs but also storage media that non-temporarily store information processing programs.

[0094] The technology of the present disclosure can also be appropriately combined with the above-described exemplary embodiments and examples. The above-described description and illustrations are detailed descriptions of the parts related to the technology of the present disclosure and are merely examples of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or new elements may be replaced with other parts from the description and illustrations shown above, within the scope of the gist of the technology of the present disclosure.

[0095] The following additional notes are further disclosed regarding the above embodiment. [Appendix 1] at least one processor; The processor: Images with accompanying information are converted into structured data, Adding annotation information to the structured data; Using a combination of the structured data and the annotation information, a learning model is trained, with the structured data as input and the annotation information as output. Information processing device. [Appendix 2] The processor: extracting at least one image from the series of images to which the supplementary information is attached; Converting the extracted image into the structured data 2. The information processing device according to claim 1. [Appendix 3] The series of images is a moving image that includes a plurality of images in a time series. 3. The information processing device according to claim 2. [Appendix 4] The processor: extracting, from the series of images, images that include a region of a predetermined structure; 4. The information processing device according to claim 2 or 3. [Appendix 5] The structured data is DICOM (Digital Imaging and Communications in Medicine) compliant data. 5. An information processing device according to any one of claims 1 to 4. [Appendix 6] The processor: Accepting input of the annotation information by a user 6. An information processing device according to any one of Supplementary Note 1 to Supplementary Note 5. [Appendix 7] The processor: identifying a region of interest of the image contained in the structured data; generating the annotation information based on the identified region of interest; 7. An information processing device according to any one of claims 1 to 6. [Appendix 8] The processor: The annotation information is generated based on the additional information included in the structured data. 8. An information processing device according to any one of claims 1 to 7. [Appendix 9] The processor: The additional information is converted into the structured data after at least one of changing, deleting, and adding the additional information. 9. An information processing device according to any one of Supplementary Note 1 to Supplementary Note 8. [Appendix 10] The processor: New structured data different from the structured data used for training the learning model is input to the trained learning model, thereby generating new annotation information for the new structured data. 10. An information processing device according to any one of Supplementary Note 1 to Supplementary Note 9. [Appendix 11] The processor: The learning model is retrained using a combination of the new structured data and the generated new annotation information. 11. The information processing device according to claim 10. [Appendix 12] The processor: assigning first annotation information and second annotation information of different types to the structured data; using a combination of the structured data and the first annotation information, training a first learning model that uses the structured data as input and the first annotation information as output; Using a combination of the first annotation information and the second annotation information, a second learning model is trained, the second learning model having the first annotation information as input and the second annotation information as output. 12. An information processing device according to any one of claims 1 to 11. [Appendix 13] The supplementary information indicates at least one of information about the subject of the image and information about capturing the image. 13. An information processing device according to any one of claims 1 to 12. [Appendix 14] The image is a medical image. 14. An information processing device according to any one of claims 1 to 13. [Appendix 15] The annotation information indicates at least one of the position and the size of a region of interest included in the image. 15. An information processing device according to any one of claims 1 to 14. [Appendix 16] The annotation information indicates findings of the image. 16. An information processing device according to any one of claims 1 to 15. [Appendix 17] extracting at least one image and the accompanying information from a series of images to which the accompanying information is attached; converting the extracted image and the extracted additional information into structured data; Adding annotation information to the structured data; Using a combination of the structured data and the annotation information, a learning model is trained, with the structured data as input and the annotation information as output. An information processing method in which processing is performed by a computer. [Appendix 18] extracting at least one image and the accompanying information from a series of images to which the accompanying information is attached; converting the extracted image and the extracted additional information into structured data; Adding annotation information to the structured data; Using a combination of the structured data and the annotation information, a learning model is trained, with the structured data as input and the annotation information as output. An information processing program that allows a computer to execute processing. [Explanation of symbols]

[0096] 1. Information Processing Systems 9 Network 10 Medical Institution System 11 Imaging equipment 12 Image Reading Workshop 13 Medical Workshop 14 Image Server 15 Image DB 16 Report Server 17 Report DB 18 RIS 19 HIS 20 Information processing equipment 21 CPU 22 Memory section 23 Memory 24 displays 25 Input section 26 Network I / F 27 Information Processing Program 28 Bus 29 Learning Data DB 30 Acquisition Department 32 Processing Department 34 Conversion unit 36 Granting Department 38 Learning Department 40 Control Unit 90 Pointer 91, 92 area A1 and A2 annotation information B1 Medical Imaging D1 screen M diagnostic model

Claims

1. at least one processor; The processor: Images with accompanying information are converted into structured data, Adding annotation information to the structured data; Using a combination of the structured data and the annotation information, a learning model is trained, with the structured data as input and the annotation information as output. Information processing device.

2. The processor: extracting at least one image from the series of images to which the supplementary information is attached; Converting the extracted image into the structured data The information processing device according to claim 1 .

3. The series of images is a moving image that includes a plurality of images in a time series. The information processing device according to claim 2 .

4. The processor: extracting, from the series of images, images that include a region of a predetermined structure; The information processing device according to claim 2 .

5. The structured data is DICOM (Digital Imaging and Communications in Medicine) compliant data. The information processing device according to claim 1 .

6. The processor: Accepting input of the annotation information by a user The information processing device according to claim 1 .

7. The processor: identifying a region of interest of the image contained in the structured data; generating the annotation information based on the identified region of interest; The information processing device according to claim 1 .

8. The processor: The annotation information is generated based on the additional information included in the structured data. The information processing device according to claim 1 .

9. The processor: The additional information is converted into the structured data after at least one of changing, deleting, and adding the additional information. The information processing device according to claim 1 .

10. The processor: New structured data different from the structured data used for training the learning model is input to the trained learning model, thereby generating new annotation information for the new structured data. The information processing device according to claim 1 .

11. The processor: The learning model is retrained using a combination of the new structured data and the generated new annotation information. The information processing device according to claim 10.

12. The processor: assigning first annotation information and second annotation information of different types to the structured data; using a combination of the structured data and the first annotation information, training a first learning model that uses the structured data as input and the first annotation information as output; Using a combination of the first annotation information and the second annotation information, a second learning model is trained, the second learning model having the first annotation information as input and the second annotation information as output. The information processing device according to claim 1 .

13. The supplementary information indicates at least one of information about the subject of the image and information about capturing the image. The information processing device according to claim 1 .

14. The image is a medical image. The information processing device according to claim 1 .

15. The annotation information indicates at least one of the position and the size of a region of interest included in the image. The information processing device according to claim 1 .

16. The annotation information indicates findings of the image. The information processing device according to claim 1 .

17. Images with accompanying information are converted into structured data, Adding annotation information to the structured data; Using a combination of the structured data and the annotation information, a learning model is trained, with the structured data as input and the annotation information as output. An information processing method in which processing is performed by a computer.

18. Images with accompanying information are converted into structured data, Adding annotation information to the structured data; Using a combination of the structured data and the annotation information, a learning model is trained, with the structured data as input and the annotation information as output. An information processing program that allows a computer to execute processing.

Images