Information processing device, information processing method, and information processing program

Abstract

To provide an information processing device, an information processing method, and an information processing program that are capable of supporting interpretation of medical images.SOLUTION: An information processing device 10 includes at least one processor. The processor obtains an image; displays, on a display, a graphic indicating a first area of interest included in the image so as to have it overlapped on the image; receives a correction instruction for at least a part of the graphic; and specifies a second area of interest at least a part of which overlaps the first area of interest, based on an image feature of the image and the correction instruction.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus, an information processing method, and an information processing program.

Background Art

[0002] Conventionally, image diagnosis has been performed using medical images obtained by imaging apparatuses such as CT (Computed Tomography) apparatuses and MRI (Magnetic Resonance Imaging) apparatuses. Further, medical images are analyzed by CAD (Computer Aided Detection / Diagnosis) using a discriminator trained by deep learning or the like to detect and / or diagnose regions of interest including structures, lesions, and the like contained in the medical images. The medical images and the analysis results by CAD are transmitted to terminals of medical personnel such as interpreting physicians who interpret medical images. Medical personnel such as interpreting physicians refer to the medical images and analysis results using their own terminals to interpret the medical images and create interpretation reports.

[0003] Various methods have also been proposed to support medical image interpretation. For example, Patent Document 1 discloses a technique for creating an interpretation report based on keywords input by an interpreting physician and analysis results of a medical image. In the technique described in Patent Document 1, sentences to be included in an interpretation report are created using a recurrent neural network that has been trained to generate sentences from input characters.

[0004] Further, for example, Patent Document 2 discloses a technique for correcting the shape of a region of interest in a medical image, in which multi-segment line segments or point sequences are arranged at equal intervals, and by dragging one of them, other adjacent multi-segment line segments or point sequences are moved following according to a predetermined tension.

Prior Art Literature

Patent Literature

[0005]

Patent Literature 1

[0006] Conventionally, when detecting regions of interest from medical images using CAD, detection was sometimes inaccurate due to the inclusion of extraneous surrounding areas or the failure to detect the peripheral areas. When using the detected region of interest (for example, when highlighting the region of interest in a medical image, or when generating a report on the region of interest), the user is required to manually correct it to the correct region of interest, and there is a need for a technology that can reduce the effort involved in this process. In the technology described in Patent Document 2, other points adjacent to the dragged point move according to a predetermined tension, making it difficult to accurately correct the region of interest to match the correct region of interest.

[0007] This disclosure provides an information processing device, an information processing method, and an information processing program that can assist in the interpretation of images. [Means for solving the problem]

[0008] A first aspect of the present disclosure is an information processing apparatus comprising at least one processor, the processor acquires an image, overlays a figure indicating a first region of interest contained in the image onto the image and displays it on a display, receives a modification instruction for at least a portion of the figure, and identifies a second region of interest that overlaps with at least a portion of the first region of interest based on the image features of the image and the modification instruction.

[0009] A second aspect of this disclosure is that, in the first aspect described above, the processor may overlay a figure representing the second region of interest onto the image and display it on the display.

[0010] A third aspect of this disclosure is that, in the first or second aspect described above, the processor may accept a modification instruction for a modification of at least one of the points forming the figure.

[0011] A fourth aspect of this disclosure is that, in the third aspect described above, the processor may identify a second region of interest among the regions of interest included in the image, in which at least one point that forms the modified figure is located within a predetermined range from the outer edge of the region of interest.

[0012] A fifth aspect of this disclosure is that, in any one of the first to fourth aspects described above, the processor may accept, as a modification instruction, a language instruction representing a change in the shape of a graphic.

[0013] A sixth aspect of this disclosure is that, in any one of the first to fifth aspects described above, the processor may identify the second region of interest using a first learning model that is pre-trained to take an image, a first region of interest, and a modification instruction as inputs and output a second region of interest.

[0014] A seventh aspect of this disclosure is that, in any one of the first to fifth aspects described above, the processor may generate a feature map of an image using a second learning model that is pre-trained to take an image as input and output a feature map of the input image, and identify a second region of interest using a third learning model that is pre-trained to take a feature map, a first region of interest and a correction instruction as input and output a second region of interest.

[0015] An eighth aspect of this disclosure is that, in the seventh aspect described above, the processor may identify a first region of interest based on a feature map.

[0016] A ninth aspect of this disclosure is that in any one of the first to eighth aspects described above, the figure may be at least one of a bounding box, a mask, and a mesh.

[0017] According to a tenth aspect of the present disclosure, in any one of the first to ninth aspects described above, the image may be a medical image, and the region of interest may be at least one of a region of a structure included in the medical image and a region of a lesion included in the medical image.

[0018] An eleventh aspect of the present disclosure is an information processing method, comprising: acquiring an image; causing a graphic indicating a first region of interest included in the image to be superimposed on the image and displayed on a display; receiving a correction instruction for at least a part of the graphic; and specifying a second region of interest at least partially overlapping the first region of interest based on image features of the image and the correction instruction.

[0019] A twelfth aspect of the present disclosure is an information processing program for causing a computer to execute processing comprising: acquiring an image; causing a graphic indicating a first region of interest included in the image to be superimposed on the image and displayed on a display; receiving a correction instruction for at least a part of the graphic; and specifying a second region of interest at least partially overlapping the first region of interest based on image features of the image and the correction instruction. Effects of the Invention

[0020] According to the above aspects, the information processing apparatus, information processing method, and information processing program of the present disclosure can support image interpretation. Brief Description of the Drawings

[0021] [Figure 1] It is a diagram showing an example of a schematic configuration of an information processing system. [Figure 2] It is a diagram showing an example of a medical image. [Figure 3] It is a diagram showing an example of a medical image. [Figure 4] It is a block diagram showing an example of a hardware configuration of an information processing apparatus. [Figure 5] It is a block diagram showing an example of a functional configuration of an information processing apparatus. [Figure 6] It is a diagram showing an example of a screen displayed on a display. [Figure 7]This figure shows an example of a screen displayed on a display. [Figure 8] This is a diagram illustrating how to identify the second area of ​​interest. [Figure 9] This figure shows an example of a screen displayed on a display. [Figure 10] This is a flowchart illustrating an example of information processing. [Figure 11] This is a diagram illustrating how to identify the second area of ​​interest. [Modes for carrying out the invention]

[0022] Embodiments of this disclosure will be described below with reference to the drawings. First, the configuration of the information processing system 1 to which the information processing device 10 of this disclosure is applied will be described. Figure 1 is a diagram showing the schematic configuration of the information processing system 1. The information processing system 1 shown in Figure 1 performs imaging of the area to be examined of a subject and stores the medical images obtained by imaging, based on examination orders from physicians in clinical departments using a known ordering system. It also performs medical image interpretation work by radiologists and the creation of interpretation reports, and allows physicians in the requesting clinical departments to view the interpretation reports.

[0023] As shown in Figure 1, the information processing system 1 includes an imaging device 2, an image interpretation terminal (Image Interpretation WS (WorkStation) 3, a medical examination WS 4, an image server 5, an image database (DB) 6, a report server 7, and a report database 8. The imaging device 2, image interpretation WS 3, medical examination WS 4, image server 5, image database 6, report server 7, and report database 8 are connected to each other via a wired or wireless network 9, enabling them to communicate with one another.

[0024] Each device is a computer on which an application program is installed to function as a component of the information processing system 1. The application program may be recorded and distributed on recording media such as DVD-ROM (Digital Versatile Disc Read Only Memory) and CD-ROM (Compact Disc Read Only Memory), and installed on the computer from that recording media. Alternatively, it may be stored in a storage device or network storage of a server computer connected to network 9 in an externally accessible state, and downloaded and installed on the computer upon request.

[0025] The imaging device 2 is a modality that generates medical images representing the area to be diagnosed by imaging that area of ​​the subject. Examples of imaging devices 2 include plain X-ray machines, CT (Computed Tomography) machines, MRI (Magnetic Resonance Imaging) machines, PET (Positron Emission Tomography) machines, ultrasound diagnostic machines, endoscopes, and fundus cameras. The medical images generated by imaging device 2 are transmitted to the image server 5 and stored in the image database 6.

[0026] Figure 2 is a schematic diagram showing an example of a medical image acquired by imaging device 2. The medical image T shown in Figure 2 is a CT image consisting of multiple tomographic images T1 to Tm (where m is 2 or more), each representing a cross-sectional plane from the head to the waist of a single subject (human body).

[0027] Figure 3 schematically shows an example of a tomographic image Tx from among multiple tomographic images T1 to Tm. The tomographic image Tx shown in Figure 3 represents a tomographic plane including the lung. Each tomographic image T1 to Tm may include regions SA of structures showing various organs and tissues of the human body (e.g., lungs and liver, etc.) and various tissues that constitute these organs and tissues (e.g., blood vessels, nerves and muscles, etc.). In addition, each tomographic image may include regions AA of lesions such as nodules, tumors, injuries, defects and inflammation. In the tomographic image Tx shown in Figure 3, the lung region is the region SA of structures, and the nodule region is the region AA of lesions. Note that a single tomographic image may contain multiple regions SA of structures and / or regions AA of lesions. Hereinafter, at least one of the regions SA of structures and regions AA of lesions included in a medical image will be referred to as the "region of interest".

[0028] The Image Interpretation WS3 is a computer used by medical professionals, such as radiologists in the radiology department, for interpreting medical images and creating interpretation reports, and it incorporates the information processing device 10 according to this embodiment. The Image Interpretation WS3 handles requests to view medical images from the image server 5, various image processing on medical images received from the image server 5, display of medical images, and acceptance of input of text related to medical images. The Image Interpretation WS3 also performs analysis processing on medical images, assists in creating interpretation reports based on the analysis results, requests registration and viewing of interpretation reports from the report server 7, and displays interpretation reports received from the report server 7. These processes are performed by the Image Interpretation WS3 executing software programs for each process.

[0029] The Medical WS4 is a computer used by medical professionals, such as physicians in a clinical department, for tasks such as detailed observation of medical images, viewing of image interpretation reports, and creation of electronic medical records. It consists of a processing unit, display devices such as a display, and input devices such as a keyboard and mouse. The Medical WS4 performs tasks such as requesting the viewing of medical images from the image server 5, displaying medical images received from the image server 5, requesting the viewing of image interpretation reports from the report server 7, and displaying image interpretation reports received from the report server 7. These processes are carried out by the Medical WS4 executing software programs for each process.

[0030] Image Server 5 is a general-purpose computer with a software program installed that provides the functionality of a Database Management System (DBMS). Image Server 5 is connected to Image DB 6. The connection method between Image Server 5 and Image DB 6 is not particularly limited; it may be connected via a data bus, or via a network such as NAS (Network Attached Storage) or SAN (Storage Area Network).

[0031] The image database 6 is implemented using storage media such as an HDD (Hard Disk Drive), SSD (Solid State Drive), and flash memory. The image database 6 stores medical images acquired by the imaging device 2, along with associated information attached to those medical images.

[0032] The supplementary information may include identification information such as an image ID (identification) for identifying medical images, a tomographic ID assigned to each tomographic image contained in the medical image, a subject ID for identifying the subject, and an examination ID for identifying the examination. The supplementary information may also include information related to the acquisition of medical images, such as the acquisition method, acquisition conditions, acquisition purpose, and acquisition date and time. "Acquisition method" and "acquisition conditions" refer to, for example, the type of imaging device 2, the acquisition site, acquisition protocol, acquisition sequence, imaging technique, whether or not contrast agent was used, and slice thickness in tomography. The supplementary information may also include information related to the subject, such as the subject's name, date of birth, age, and sex.

[0033] Furthermore, when the image server 5 receives a request to register a medical image from the imaging device 2, it formats the medical image into a database format and registers it in the image DB 6. Also, when the image server 5 receives a viewing request from the image interpretation WS3 and the medical treatment WS4, it searches for the medical image registered in the image DB 6 and sends the retrieved medical image to the image interpretation WS3 and the medical treatment WS4 that made the viewing request.

[0034] Report Server 7 is a general-purpose computer with software programs installed that provide the functionality of a database management system. Report Server 7 is connected to Report DB 8. The connection method between Report Server 7 and Report DB 8 is not particularly limited; it may be connected via a data bus, or via a network such as a NAS or SAN.

[0035] The report database (DB8) is implemented using storage media such as HDDs, SSDs, and flash memory. The report database (DB8) stores the image interpretation reports created in the image interpretation workstation (WS3). The report database (DB8) may also store findings information related to medical images. Finding information includes, for example, information obtained by the image interpretation workstation (WS3) through image analysis using CAD (Computer-Aided Detection / Diagnosis) technology and AI (Artificial Intelligence) technology, as well as information entered by the user after interpreting the medical images.

[0036] Findings information includes various findings such as the type (name), characteristics, location, size, and estimated disease name of the region of interest contained in the medical image. Examples of types (names) include the type of structure, such as "lung" and "liver," and the type of lesion, such as "nodule" and "tumor." Characteristics mainly refer to the features of the lesion. For example, in the case of a lung nodule, findings include attenuation values ​​such as "solid" and "ground-glass opacity," marginal shape such as "clear / indistinct," "smooth / irregular," "spicula," "lobulated" and "serrated," and overall shape such as "round" and "irregular." Other findings include the relationship with surrounding tissues, such as "pleural contact" and "pleural invagination," and findings regarding the presence or absence of contrast and washout.

[0037] Location refers to the anatomical location, the location in the medical image, and the relative positional relationship with other regions of interest such as "internal," "peripheral," and "surrounding." The anatomical location may be indicated by organ names such as "lung" and "liver," or by subdivisions of the lung such as "right lung," "upper lobe," and apical segment ("S1"). Measured values ​​are values ​​that can be quantitatively measured from the medical image, such as at least one of the size and signal value of the region of interest. Size is expressed, for example, by the major axis, minor axis, area, and volume of the region of interest. Signal value is expressed, for example, by the pixel value of the region of interest and the CT value with HU as the unit. Estimated disease name is an evaluation result estimated based on the lesion, such as disease names such as "cancer" and "inflammation," and evaluation results such as "negative / positive," "benign / malignant," and "mild / severe" regarding the disease name and characteristics.

[0038] Furthermore, when the report server 7 receives a request to register an image interpretation report from the image interpretation WS3, it formats the image interpretation report into a database format and registers it in the report DB8. Also, when the report server 7 receives a request to view an image interpretation report from the image interpretation WS3 and the medical WS4, it searches the report DB8 for the image interpretation report registered therein and sends the retrieved image interpretation report to the image interpretation WS3 and medical WS4 that made the viewing request.

[0039] Network 9 is, for example, a LAN (Local Area Network) and a WAN (Wide Area Network). The imaging device 2, image interpretation WS3, medical examination WS4, image server 5, image DB6, report server 7, and report DB8 included in the information processing system 1 may be located in the same medical institution or in different medical institutions. Furthermore, the number of each device, imaging device 2, image interpretation WS3, medical examination WS4, image server 5, image DB6, report server 7, and report DB8, is not limited to the number shown in Figure 1, and each device may consist of multiple devices with similar functions.

[0040] Incidentally, when detecting regions of interest from medical images using CAD, there were instances where the detection was incorrect due to the inclusion of extraneous surrounding areas or the failure to detect the peripheral areas. When using the detected regions of interest (for example, when highlighting regions of interest in medical images or when generating findings statements related to regions of interest), users are required to manually correct them to the correct regions of interest.

[0041] Therefore, the information processing device 10 according to this embodiment has a function that reduces the effort required for correction and supports the interpretation of medical images by assisting in the correction of regions of interest detected from medical images. The information processing device 10 will be described below. As described above, the information processing device 10 is included in the image interpretation WS3.

[0042] First, an example of the hardware configuration of the information processing device 10 according to this embodiment will be described with reference to Figure 4. As shown in Figure 4, the information processing device 10 includes a CPU (Central Processing Unit) 21, a non-volatile storage unit 22, and a memory 23 as a temporary storage area. The information processing device 10 also includes a display 24 such as a liquid crystal display, an input unit 25 such as a keyboard and mouse, and a network interface 26. The network interface 26 is connected to a network 9 and performs wired and / or wireless communication. The CPU 21, storage unit 22, memory 23, display 24, input unit 25, and network interface 26 are connected to each other via a bus 28 such as a system bus and a control bus, enabling the exchange of various types of information.

[0043] The storage unit 22 is implemented by a storage medium such as an HDD, SSD, or flash memory. The information processing program 27 of the information processing device 10 is stored in the storage unit 22. The CPU 21 reads the information processing program 27 from the storage unit 22, expands it into memory 23, and executes the expanded information processing program 27. The CPU 21 is an example of the processor of this disclosure. The information processing device 10 can be appropriately applied to, for example, a personal computer, a server computer, a smartphone, a tablet terminal, or a wearable terminal.

[0044] Next, an example of the functional configuration of the information processing device 10 according to this embodiment will be described with reference to Figures 5 to 9. As shown in Figure 5, the information processing device 10 includes an acquisition unit 30, a specification unit 32, a reception unit 34, and a control unit 36. By executing the information processing program 27, the CPU 21 functions as each of the functional units of the acquisition unit 30, specification unit 32, reception unit 34, and control unit 36.

[0045] The acquisition unit 30 acquires medical images obtained by photographing the subject from the image server 5.

[0046] The identification unit 32 identifies the first region of interest A1 contained in the medical image acquired by the acquisition unit 30. As a method for identifying the first region of interest A1, known CAD and AI technologies can be appropriately applied. For example, the identification unit 32 may identify the first region of interest A1 from the medical image using a learning model such as a CNN (Convolutional Neural Network) that has been trained to take the medical image as input and identify and output the region of interest contained in the medical image.

[0047] Furthermore, the identification unit 32 may generate a report on the identified first region of interest A1. Methods for generating the report can appropriately apply known CAD and AI technologies. For example, the identification unit 32 may use a pre-trained CNN or similar model to take the region of interest identified from the medical image as input and output the report on the region of interest, thereby generating the report on the first region of interest A1. Subsequently, the identification unit 32 may generate a report containing the generated report information. For example, the identification unit 32 may generate the report using a machine learning method such as the recurrent neural network described in Patent Document 1. Alternatively, the identification unit 32 may generate the report by embedding the report information into a predetermined template.

[0048] The control unit 36 ​​controls the display 24 to overlay a figure representing the first region of interest A1, identified by the identification unit 32, onto the medical image acquired by the acquisition unit 30. Figure 6 shows an example of screen D1 displayed on the display 24 by the control unit 36. Screen D1 includes a medical image T10, in which the first region of interest A1 is highlighted by being enclosed in a bounding box B1. The bounding box B1 is an example of a figure representing the first region of interest A1 in this disclosure.

[0049] Note that while Figure 6 shows an example where the bounding box B1 is a rectangle, the bounding box B1 is not limited to this and may be a polygon other than a rectangle. Furthermore, the figure representing the first region of interest A1 is not limited to the bounding box B1, but may be, for example, at least one of a mask that displays different colors, etc., between the first region of interest A1 and other regions, and a mesh that represents the geometric shape of the first region of interest A1.

[0050] Furthermore, the control unit 36 ​​may control the display 24 to display the observation statement regarding the first region of interest A1 generated by the identification unit 32. The screen D1 in Figure 6 contains the observation statement 92 regarding the first region of interest A1.

[0051] Suppose the user wishes to change the region of interest (correct region of interest) in the medical image T10 on screen D1 from the first region of interest A1 to the second region of interest A2. In this case, the user selects the correction button 96 on screen D1 via the input unit 25. When the correction button 96 is selected, the control unit 36 ​​transitions to screen D2 shown in Figure 7. Screen D2 is a screen for accepting corrections to the figure representing the first region of interest A1.

[0052] The reception unit 34 receives modification instructions for at least a portion of the figure (bounding box B1) representing the first region of interest A1. Specifically, the reception unit 34 may accept modification instructions for at least one point among the points forming the figure representing the first region of interest A1. The user operates the mouse pointer 90 via the input unit 25 to modify at least one point (the upper left point in Figure 7) among the points forming the bounding box B1 displayed on screen D2 to align with the second region of interest A2. On screen D2, the bounding box B1 before modification is shown with a dotted line, and the bounding box BB after modification is shown with a solid line.

[0053] The identification unit 32 identifies a second region of interest A2 that overlaps with the first region of interest A1 in at least part, based on the image features of the medical image acquired by the acquisition unit 30 and the correction instructions received by the reception unit 34. In other words, the second region of interest A2 only needs to overlap with the first region of interest A1 in at least part, and does not need to overlap in any part. Also, the second region of interest A2 may be smaller or larger than the first region of interest A1. On the other hand, a region of interest that does not overlap with the first region of interest A1 at all will not be identified as the second region of interest A2.

[0054] Specifically, the identification unit 32 identifies a region of interest in the medical image where at least one point forming the modified shape (bounding box BB) is located within a predetermined range from the outer edge of that region of interest, as the second region of interest A2. In other words, the identification unit 32 searches within a predetermined range from the modified shape (bounding box BB) in the medical image and identifies a region of interest where at least a part of the outer edge is located within that range as the second region of interest A2. This allows the second region of interest A2 to be appropriately identified regardless of whether part or all of the second region of interest A2 that the user wishes to identify is contained within or protrudes from the modified shape (bounding box BB).

[0055] For example, as shown in Figure 8, the identification unit 32 may identify the second region of interest A2 using a first learning model 40 that has been pre-trained to take a medical image, a first region of interest A1, and a correction instruction (bounding box BB) as input and output a second region of interest A2. The first learning model 40 is a learning model that includes a CNN or the like, which has been trained using a combination of a medical image, a first region of interest A1, a correction instruction, and a second region of interest A2 as training data.

[0056] Furthermore, the specific unit 32 may generate an observation statement concerning the second area of ​​interest A2, similar to the generation of an observation statement concerning the first area of ​​interest A1 described above.

[0057] The control unit 36 ​​may also control the display 24 to overlay a figure representing the second region of interest A2, identified by the identification unit 32, onto the medical image acquired by the acquisition unit 30. Figure 9 shows an example of a screen D3 displayed on the display 24 by the control unit 36. Screen D3 includes a medical image T10, in which the second region of interest A2 is highlighted by being enclosed in a bounding box B2. The bounding box B2 is an example of a figure representing the second region of interest A2 in this disclosure.

[0058] Note that while Figure 9 shows an example where the bounding box B2 is a rectangle, the bounding box B2 is not limited to this and may be a polygon other than a rectangle. Furthermore, the figure representing the second region of interest A2 is not limited to the bounding box B2, but may be, for example, at least one of a mask that displays different colors, etc., between the second region of interest A2 and other regions, and a mesh that represents the geometric shape of the second region of interest A2.

[0059] Furthermore, the control unit 36 ​​may also control the display 24 to display the observation statement regarding the second region of interest A2 generated by the identification unit 32. In screen D3 of Figure 9, the observation statement 93 regarding the second region of interest A2 is included instead of the observation statement 92 regarding the first region of interest A1 in screen D1.

[0060] Next, the operation of the information processing device 10 according to this embodiment will be described with reference to Figure 10. In the information processing device 10, the CPU 21 executes the information processing program 27, thereby executing the information processing shown in Figure 10. The information processing is executed, for example, when the user gives an instruction to start execution via the input unit 25.

[0061] In step S10, the acquisition unit 30 acquires a medical image obtained by photographing a subject from the image server 5. In step S12, the identification unit 32 identifies the first region of interest included in the medical image acquired in step S10. In step S14, the control unit 36 ​​controls the display 24 to display a figure representing the first region of interest identified in step S12, superimposed on the medical image acquired in step S10.

[0062] In step S16, the reception unit 34 receives a modification instruction for at least a portion of the figure representing the first region of interest displayed on the display 24 in step S14. In step S18, the identification unit 32 identifies a second region of interest that overlaps with the first region of interest in at least a portion, based on the image features of the medical image acquired in step S10 and the modification instruction received in step S16. In step S20, the control unit 36 ​​controls the display 24 to display the figure representing the second region of interest identified in step S18, superimposed on the medical image acquired in step S10, and terminates this information processing.

[0063] As described above, an information processing device 10 according to one aspect of the present disclosure comprises at least one processor, which acquires an image, overlays a figure indicating a first region of interest contained in the image onto the image and displays it on a display, receives a modification instruction for at least a portion of the figure, and identifies a second region of interest that overlaps with the first region of interest in at least a portion based on the image features of the image and the modification instruction.

[0064] In other words, according to the information processing device 10 of this embodiment, when a user modifies the first region of interest detected from an image, the modified second region of interest can be identified based on image features in addition to the modification instructions. Therefore, the second region of interest can be identified with high accuracy while reducing the effort required for modification, thereby supporting image interpretation.

[0065] In the above embodiment, a configuration was described in which the modification instructions were for modifying the figure representing the first region of interest, but the system is not limited to this. For example, the receiving unit 34 may accept linguistic instructions that represent a change in the shape of the figure representing the first region of interest as a modification instruction. For example, the receiving unit 34 may accept inputs such as "make it bigger," "make it smaller," or "make the edges finer" as words or phrases that represent a change in the shape of the figure representing the first region of interest.

[0066] In this case, for example, the identification unit 32 may identify the second region of interest using a pre-trained learning model that takes a medical image, a first region of interest, and a linguistic correction instruction as input and outputs a second region of interest. This learning model includes a CNN or the like that has been trained using a combination of a medical image, a first region of interest, a linguistic correction instruction, and the second region of interest as training data. As a result, for example, for a correction instruction such as "make larger," the model is trained so that the figure representing the second region of interest is larger than the figure representing the first region of interest. Also, for example, for a correction instruction such as "fine edges," the model is trained so that the number of vertices in the figure representing the second region of interest is greater than the number of vertices in the figure representing the first region of interest.

[0067] For example, the identification unit 32 may pre-determine conditions for how the figure representing the first region of interest changes for each linguistic modification instruction, and identify the second region of interest based on the conditions corresponding to the received modification instruction. For example, for a modification instruction of "larger," the condition may be pre-determined so that at least one of the vertical and horizontal dimensions of the figure representing the first region of interest becomes larger. For example, for a modification instruction of "fine edges," the condition may be pre-determined so that the number of vertices of the figure representing the first region of interest increases. When a modification instruction is received by the receiving unit 34, the identification unit 32 refers to the pre-determined conditions and modifies the figure representing the first region of interest according to the modification instruction. Subsequently, as described in the above embodiment, the identification unit 32 may identify as the second region of interest a region within the medical image in which at least one point forming the modified figure is located within a predetermined range from the outer edge of the region of interest.

[0068] Furthermore, although the above embodiment describes a configuration in which a first learning model 40 that takes a medical image, a first region of interest, and a correction instruction as inputs is used to identify the second region of interest, the embodiment is not limited to this. For example, as shown in Figure 11, the identification unit 32 may identify the second region of interest by combining two learning models (a second learning model 42 and a third learning model 44). Specifically, the identification unit 32 may first use a second learning model 42 that has been pre-trained to take a medical image as input and output feature maps C1 to Cn of the input medical image. The second learning model 42 is, for example, a CNN or the like that performs convolutional processing using various kernels on the input image and outputs feature maps C1 to Cn consisting of feature data obtained by the convolutional processing. n corresponds to the number of various kernels.

[0069] Next, the identification unit 32 may identify the second region of interest using a third learning model 44 that has been pre-trained to take the feature maps C1 to Cn, the first region of interest, and the correction instructions generated by the second learning model 42 as input and output the second region of interest. The third learning model 44 is, for example, a learning model including a CNN that has been trained using a combination of information from the first region of interest and information from the second region of interest including the correction instructions.

[0070] Here, the "information of the first region of interest" used in training the third learning model 44 may be, for example, a combination of feature maps C1 to Cn and a figure representing the first region of interest (e.g., bounding box B1) or the coordinates of the first region of interest. Alternatively, for example, a partial feature map obtained by editing each of the feature maps C1 to Cn based on this combination, and then cutting out the portion of each of the feature maps C1 to Cn that represents the first region of interest, may be used.

[0071] Furthermore, the "information of the second domain of interest" used in training the third learning model 44 may be, for example, a combination of feature maps C1 to Cn and modification instructions. Alternatively, for example, a partial feature map may be used, which is obtained by editing each of the feature maps C1 to Cn based on this combination, and then cutting out the part of each of the feature maps C1 to Cn corresponding to the modification instructions.

[0072] Furthermore, the identification unit 32 may also use this second learning model 42 to identify the first region of interest. For example, the identification unit 32 may use the second learning model 42 to generate feature maps C1 to Cn from a medical image, and then identify the first region of interest based on the generated feature maps.

[0073] Furthermore, while the above embodiments describe a configuration intended for the interpretation of medical images, the invention is not limited to this. The information processing device 10 of this disclosure is applicable to various images obtained by photographing a subject and including regions of interest. For example, the information processing device 10 may be applied to images acquired in non-destructive testing such as radiographic inspection and ultrasonic testing, where equipment, buildings, pipes, and welds are the subjects. In this case, for example, the region of interest may represent cracks, scratches, bubbles, and foreign matter.

[0074] Furthermore, in the above embodiment, the hardware structure of the processing unit that executes various processes, such as the acquisition unit 30, the identification unit 32, the receiving unit 34, and the control unit 36, can be the various processors shown below. As mentioned above, these various processors include a CPU, which is a general-purpose processor that executes software (programs) and functions as various processing units, as well as programmable logic devices (PLDs), such as FPGAs (Field Programmable Gate Arrays), which are processors whose circuit configuration can be changed after manufacturing, and dedicated electrical circuits, such as ASICs (Application Specific Integrated Circuits), which are processors with circuit configurations specifically designed to execute specific processes.

[0075] A single processing unit may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, multiple processing units may be composed of a single processor.

[0076] Examples of configuring multiple processing units with a single processor include, firstly, a configuration where one or more CPUs and software combine to form a single processor, which then functions as multiple processing units, as exemplified by client and server computers. Secondly, a configuration using a processor that realizes the functions of the entire system, including multiple processing units, on a single IC (Integrated Circuit) chip, as exemplified by System on Chip (SoC). Thus, various processing units are configured, in terms of hardware structure, using one or more of the above-mentioned processors.

[0077] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits, which are combinations of circuit elements such as semiconductor devices.

[0078] Furthermore, although the above embodiment describes an embodiment in which the information processing program 27 is pre-stored (installed) in the storage unit 22, the invention is not limited to this. The information processing program 27 may be provided in the form of a recording medium such as a CD-ROM (Compact Disc Read Only Memory), DVD-ROM (Digital Versatile Disc Read Only Memory), and USB (Universal Serial Bus) memory. Alternatively, the information processing program 27 may be provided in the form of a download from an external device via a network. Moreover, the technology of this disclosure extends not only to the information processing program but also to storage media for non-temporarily storing the information processing program.

[0079] The technology of this disclosure can also be appropriately combined from the above-described embodiments and examples. The descriptions and illustrations shown above are detailed explanations of the parts relating to the technology of this disclosure and are merely examples of the technology of this disclosure. For example, the above-described explanation of the configuration, function, operation, and effect is an explanation of an example of the configuration, function, operation, and effect of the parts relating to the technology of this disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements added, or replaced from the descriptions and illustrations shown above, as long as they do not depart from the spirit of the technology of this disclosure. [Explanation of Symbols]

[0080] 1. Information Processing System 2. Imaging device 3 Image Interpretation Workshop 4. Medical Workshop 5 Image Server 6 Image Database 7. Report Server 8 Report Database 9 Network 10 Information Processing Devices 21 CPU 22 Memory section 23 memory 24 displays 25 Input section 26 Network Interface 27 Information Processing Programs 28 buses 30 Acquisition Department 32 Specific part 34 Reception Department 36 Control Unit 40. First Learning Model 42. Second Learning Model 44. Third Learning Model 90 Mouse pointers 92, 93 Observations 96 Edit button A1 First area of ​​interest A2 Second area of ​​interest Area of ​​AA lesion B1, B2, BB Bounding Box C1~Cn Feature Map D1-D3 screens Area of ​​SA structures T, T10 medical images T1-Tm, Tx tomographic images

Claims

1. Equipped with at least one processor, The aforementioned processor, Get the image, A figure representing one of the multiple regions of interest contained in the aforementioned image is superimposed on the image and displayed on the display. As a modification instruction for at least a part of the aforementioned figure, a modification of at least one point among the points forming the figure is accepted. Based on the image features of the aforementioned image and the modification instructions, a second region of interest that overlaps with the first region of interest in at least part is identified. Among the multiple regions of interest included in the aforementioned image, the region of interest in which at least one point forming the modified figure is located within a predetermined range from the outer edge of the aforementioned region of interest is identified as the second region of interest. Information processing device.

2. The aforementioned processor, The figure representing the second region of interest is superimposed on the image and displayed on the display. The information processing apparatus according to claim 1.

3. The aforementioned processor, The aforementioned modification instructions accept linguistic instructions that describe changes in the shape of the figure. The information processing apparatus according to claim 1.

4. The aforementioned processor, Using a first learning model that has been pre-trained to take the aforementioned image, the first region of interest, and the correction instructions as inputs and output the second region of interest, the second region of interest is identified. The information processing apparatus according to claim 1.

5. The aforementioned processor, Using a second learning model that has been pre-trained to take an image as input and output a feature map of the input image, the feature map of the image is generated. The second region of interest is identified using a third learning model that has been pre-trained to take the feature map, the first region of interest, and the modification instructions as inputs and output the second region of interest. The information processing apparatus according to claim 1.

6. The aforementioned processor, Based on the feature map, identify the first region of interest. The information processing apparatus according to claim 5.

7. The aforementioned figure is at least one of a bounding box, a mask, and a mesh. The information processing apparatus according to claim 1.

8. The aforementioned image is a medical image. The first region of interest and the second region of interest are at least one of the regions of structures included in the medical image and the regions of lesions included in the medical image. The information processing apparatus according to claim 1.

9. Get the image, A figure representing one of the multiple regions of interest contained in the aforementioned image is superimposed on the image and displayed on the display. As a modification instruction for at least a part of the aforementioned figure, a modification of at least one point among the points forming the figure is accepted. Based on the image features of the aforementioned image and the modification instructions, a second region of interest that overlaps with the first region of interest in at least part is identified. Among the multiple regions of interest included in the aforementioned image, the region of interest in which at least one point forming the modified figure is located within a predetermined range from the outer edge of the aforementioned region of interest is identified as the second region of interest. An information processing method in which a computer performs the processing.

10. Get the image, A figure representing one of the multiple regions of interest contained in the aforementioned image is superimposed on the image and displayed on the display. As a modification instruction for at least a part of the aforementioned figure, a modification of at least one point among the points forming the figure is accepted. Based on the image features of the aforementioned image and the modification instructions, a second region of interest that overlaps with the first region of interest in at least part is identified. Among the multiple regions of interest included in the aforementioned image, the region of interest in which at least one point forming the modified figure is located within a predetermined range from the outer edge of the aforementioned region of interest is identified as the second region of interest. An information processing program that causes a computer to perform a task.

Images