Medical image processing apparatus and method of operation thereof

Abstract

The purpose of this invention is to provide a medical image processing apparatus and its operating method, which can reduce the workload of doctors in selecting key images recorded in reports when the key images are automatically selected. A medical image processing apparatus (10) includes: an image acquisition unit (20) for acquiring a plurality of medical images (12); an image classification unit (30) for classifying the medical images (12) into a plurality of categories; an image display unit (50) for displaying at least one of the medical images (12) as an automatically selected image (54) based on the classification results; an input receiving unit (70) for receiving an input from a user to select an image (55) that needs to be reselected as a key image (13) from the displayed automatically selected images (54); and a display control unit (51) for switching and displaying non-automatically selected images (57) other than the automatically selected images (54) from the plurality of medical images (12) classified by category as switching images (56) instead of displaying the selection image (55).

Description

Technical Field

[0001] This invention relates to a medical image processing apparatus and its working method for acquiring key graphic images for reports. Background Technology

[0002] In the medical field, during endoscopic or ultrasound examinations, doctors diagnose patients while operating the equipment. After the diagnosis, key images, including areas of interest, are extracted from the acquired images and recorded in the report. The criteria or rules for recording key images in the report vary depending on the facility or doctor; therefore, doctors select key images from the large number of images taken during the diagnosis.

[0003] Reviewing each acquired examination image and identifying key images containing lesions or other areas of interest requires time and effort, becoming a burden for doctors. Therefore, technologies are being developed to facilitate the acquisition of key images.

[0004] Specifically, Patent Document 1 describes an automatic highlighting display of frame images of bleeding areas and a display showing the before and after of the image frame selected by the user in a summary display.

[0005] Previous technical documents

[0006] Patent documents

[0007] Patent Document 1: International Publication No. 2009 / 008125 Summary of the Invention

[0008] The technical problem to be solved by the invention

[0009] In Patent Document 1, by automatically highlighting frames of areas such as bleeding, users can easily determine key image candidates. Furthermore, by displaying images before and after the user-selected image frames, it is easy to identify similar images that are more suitable for the key image than the user-selected key image candidate. However, this approach sometimes fails to alleviate the burden when the automatically displayed images do not contain images with the features required by the user.

[0010] Imagine a scenario where, after a doctor completes an examination, multiple images are automatically selected from a set of images captured during the examination for the report. While the system learns from diagnostic usefulness and report styles to automatically select key image candidates, problems arise such as selecting inappropriate images due to reasoning errors, failing to reflect the doctor's preferences, and being unable to handle special examinations. Therefore, the selection process requires doctor confirmation, but if this task is burdensome, the result fails to alleviate the workload.

[0011] The purpose of this invention is to provide a medical image processing device and its operating method, which can reduce the workload of doctors in selecting results when key images recorded in the report are automatically selected.

[0012] means for solving technical problems

[0013] The medical image processing apparatus of the present invention includes a processor that performs the following processing: acquiring multiple medical images, classifying the medical images into at least one of multiple categories, displaying at least one of the multiple medical images as an automatically selected image based on the classification result, accepting a user's input of selecting an image that needs to be reselected as a key image from the displayed automatically selected images as input for selecting an image instead of displaying the selected image, and switching and displaying non-automatically selected images other than the automatically selected images from the multiple medical images classified into categories as switching images.

[0014] A preferred feature is that the processor switches between selected images and displayed switching images based on multiple categories.

[0015] A preferred feature is that the processor switches to display switching images that are categorized into the same category as the selected image.

[0016] The preferred processor displays multiple candidate switching images as candidates for switching images and accepts user input to determine the switching image from the multiple displayed candidate switching images.

[0017] A preferred feature is that the switching candidate images include images of the same category as the selected image.

[0018] A preferred feature is that when the processor selects and switches images from multiple medical images based on the automatically selected image category, it takes into account the photographic conditions of the medical images.

[0019] A preferred feature is that, when the processor selects and switches images from multiple medical images based on the automatically selected image category, it considers the diagnostic usefulness of the medical images.

[0020] The preferred processor assigns character or label information corresponding to the category to each image for display.

[0021] The preferred processor changes the display method of each image according to its category.

[0022] The preferred processor displays switching images and automatically selected images in different ways.

[0023] The preferred processor accepts input that recognizes each image displayed on the screen as a key image, and displays them in different ways depending on whether they have been recognized.

[0024] The preferred processor classifies images based on a learned CNN model obtained by learning from a group of images including multiple medical images.

[0025] The preferred category is information related to the parts of the subject being photographed.

[0026] Preferred categories are information related to whether the image contains a region of interest.

[0027] Preferred categories are information related to whether the image contains treatment equipment or the type of treatment equipment contained therein.

[0028] The preferred category is information related to the distribution state of pigments or dyes in the image.

[0029] The working method of the medical image processing device of the present invention includes the following steps: acquiring multiple medical images; classifying the medical images into at least one of multiple categories; displaying at least one of the multiple medical images as an automatically selected image based on the classification result; accepting the user's input of selecting an image that needs to be reselected as a key image from the displayed automatically selected images as the selection image; and switching and displaying non-automatically selected images other than the automatically selected images from the multiple medical images classified into categories as switching images instead of displaying the selection image.

[0030] Invention Effects

[0031] This invention reduces the burden on users when they select images for reports from a large number of medical images and assists in report creation. Attached Figure Description

[0032] Figure 1 This is a schematic diagram showing a device connected to a medical image processing apparatus.

[0033] Figure 2 This is a block diagram representing the functions of a medical image processing device.

[0034] Figure 3 This is an illustration of how to automatically select an image and display it on the monitor.

[0035] Figure 4 This is an illustration of selecting the image that needs to be reselected as the key image and switching the displayed image.

[0036] Figure 5 This is a conceptual diagram of switching images on a monitor.

[0037] Figure 6 It is an explanatory diagram that displays information about the category and usefulness of the displayed image.

[0038] Figure 7This is an explanatory diagram illustrating how to distinguish the display of approved images when key images are approved separately.

[0039] Figure 8 It is an explanatory diagram for the report production screen that uses key images.

[0040] Figure 9 This is an explanatory diagram using the switching candidate image display bar in the second embodiment.

[0041] Figure 10 This is an illustration showing the page transitions when switching the candidate image display bar.

[0042] Figure 11 It is a flowchart representing a series of processes that switch the selected image to a different display. Detailed Implementation

[0043] [First Implementation]

[0044] In the medical image processing apparatus 10, programs related to various processes are assembled in a program memory (not shown). The medical image processing apparatus 10 is equipped with a central control unit (not shown) consisting of a processor. By executing the programs in the program memory, the central control unit performs the functions of the image acquisition unit 20, image classification unit 30, usefulness determination unit 40, image display unit 50, display control unit 51, input receiving unit 70, key image storage memory 80, and report generation device 90.

[0045] The image acquisition unit 20 acquires multiple medical images 12 from the endoscope system 11 and sends them to the image classification unit 30. From the medical images 12, key images 13 for report preparation are identified. Key images 13 are those that clearly convey the examination results in the report, preferably with clearly defined features such as lesions.

[0046] The image classification unit 30 acquires image information of the medical image 12 sent from the image acquisition unit 20, and performs category information acquisition and image quality information calculation. Based on the acquired category information, the medical image 12 is classified into various categories. The classification categories are any one or a combination of the following: imaging site, presence and type of lesion, usage status of treatment instruments, and distribution status of pigments. In the image quality information, information such as brightness, noise, and blur level is acquired and appended to the medical image 12. Image information is extracted by learning a CNN (Convolutional Neural Network) model. The learned CNN model is obtained by learning from an image group including at least a plurality of medical images 12. Preferably, the category information and image quality information acquired during imaging and appended to the medical image 12 in the endoscope system 11 are preferred over the information extracted by the learned CNN model. The classified medical image 12 is sent to the usefulness determination unit 40.

[0047] In the usefulness determination unit 40, the diagnostic usefulness of the medical image 12 is determined based on the classification results and image quality information obtained by the image classification unit 30. The higher the diagnostic usefulness, the higher the likelihood that it is suitable for the key image 13. The usefulness evaluation is expressed, for example, by a grading evaluation. A medical image 12 that has items such as lesions in the classification results and has low blur in the image quality evaluation can be evaluated as having high usefulness. In the case of no category information, high blur, excessive brightness, or excessive brightness, the usefulness decreases. The medical image 12 whose usefulness determination has been completed is sent to the image display unit 50. The threshold for evaluating usefulness can be arbitrarily set by the user.

[0048] In the image display unit 50, the medical image 12 sent from the usefulness determination unit 40 by the display control unit 51 is displayed on the display 52 according to the classification category.

[0049] The input receiving unit 70 accepts user input such as image selection for switching displays, page changes, switching implementation methods, switching category displays, layout changes, and saving key images.

[0050] The key image 13 used for the report is saved in the key image storage memory 80.

[0051] In the report generation apparatus 90, a report on the inspection results is generated using the key image 13. Preferably, the report is generated automatically after the key image 13 is acquired.

[0052] like Figure 1As shown, the endoscope system 11 acquires a large number of medical images 12 using a light source device 11a, an endoscope 11b, and a processor device 11c. The endoscope system 11 is connected to a medical image processing device 10 and sends the acquired medical images 12 to the medical image processing device 10.

[0053] like Figure 2 As shown, medical images 12 transmitted from the endoscope system 11 are received by the image acquisition unit 20 and transmitted to the image display unit 50 via the image classification unit 30 and the usefulness determination unit 40. The image display unit 50 includes a display control unit 51 and a display 52. ​​In the image display unit 50, at least one image from the medical images 12 classified by category by the image classification unit 30 is selected as the automatically selected image 54. The automatically selected image 54 is an image selected based on the results of category classification and usefulness determination, and is a candidate image that can become the key image 13 to be published in the report. The automatically selected image 54 is displayed on the display 52. ​​The displayed automatically selected image 54 is approved as the key image 13 by user input received by the input receiving unit 70 and stored in the key image storage memory 80.

[0054] like Figure 3 As shown, the acquired medical images 12 are displayed as automatically selected images 54 according to the category categorized by the display 52. ​​An image display bar 53 is provided on the display 52, in which the automatically selected images 54 are displayed in a single view. Preferably, the automatically selected images 54 are selected to include all predefined categories, and preferably, the automatically selected images 54 displayed in a single view are evaluated to have at least a specified level of usefulness, and are in the chronological order of image acquisition. The display 52 has a cursor 71 reflecting user operations such as mouse operation, a display toggle button 73, and an approval button 74.

[0055] like Figure 4 As shown, in the first embodiment, if there is an image that needs to be reselected as the key image 13 among the automatically selected images 54 displayed in the image display bar 53 on the display 52, it is selected as the key image 55 via the cursor 71 through mouse operation or the like. The selected image 55 switches the display of images that were not automatically selected from the previously displayed image group to a switching image 56 based on category information. Furthermore, the image that needs to be reselected as the key image 13 is one that is temporarily not displayed and is retained as the key image 13 based on the user's judgment. Also, there are cases where the original image is restored after switching to the switching image 56.

[0056] Examples of images selected as switching images 56 include "images of the same photographic area as selected image 55", "images of areas photographed close to selected image 55", "images without processing equipment if processing equipment is being used in selected image 55", "images without pigment dispersion if selected image 55 is in a state of pigment dispersion", and "images with the same light source mode as selected image 55". Furthermore, the selection of switching images 56 can also consider the image's photographing time, photographic mode (magnification or non-magnification, etc.), or information useful for diagnosis based on image quality evaluation. Specifically, this includes "images with similar photographing times", "images with low blur levels", and "images with appropriate brightness". In the image display bar 53, switching images 56 are preferably highlighted with a box, making it easy to identify that a switching display has been made. Additionally, by selecting the approval button 74, the displayed images are also approved as key images 13. When a switching display is not required, all the initially displayed automatically selected images 54 are directly approved as key images 13.

[0057] Figure 5 This is a conceptual diagram illustrating the display switching between selected image 55 and switching image 56. When selected image 55 is selected by the user, the display control unit 51 detects the switching image 56 to be displayed based on the category of selected image 55 or associated image quality information. Switching image 56 is detected from the non-automatically selected images 57 (excluding the automatically selected image 54) in the medical images 12 categorized by the image classification unit 30. Figure 5 In the process, the image group 58 after category classification consists of an automatically selected image group 59 and a non-automatically selected image group 60. The automatically selected image group 59 includes the automatically selected image 54 obtained through automatic selection, while the non-automatically selected image group 60 is not selected as the automatically selected image 54. An image 56 is automatically selected and switched from the non-automatically selected image group 60.

[0058] If both the automatically selected image 54 and the switched image 56 displayed in the image display bar 53 of the display 52 are recognized as key images 13, select the approval button 74. When the approval button 74 is selected, the displayed images are also saved as key images 13 to the key image storage memory 80 and sent to the report generation device 90. Each key image 13 is used to generate a report.

[0059] exist Figure 6The display methods of the automatically selected image 54 and the switched image 56 displayed in the image display bar 53 will be explained below. The display control unit 51 visualizes the category information and displays it along with each image. Furthermore, the display can be turned on / off using the display switching button 73, and the displayed information can also be switched. For example, if the normal display is set to "Display a", then in "Display b", the image information display bar 61 expands below each image displayed in the image display bar 53, and the category of each image is displayed in characters. Furthermore, in "Display c", the usefulness evaluation calculated by the usefulness determination unit 40 is displayed in the image information display bar 61. By utilizing the above display methods, a more appropriate key image 13 can be selected. Furthermore, a display method combining the above display methods can also be set.

[0060] exist Figure 7 In this section, other modes of approval processing based on the approval button 74 will be explained. While the images 54 in the image display bar 53 are all processed together with the key image 13, it is also possible to process each displayed image individually. In this case, it is necessary to display the images in a way that distinguishes between approved and unapproved images. For example, for images that have undergone approval processing, display modes such as shadow display, surrounding the image, or marking can be used to make it immediately clear that approval processing has been performed.

[0061] like Figure 8 As shown, the key image 13, which has undergone approval processing, is sent to the report generation device 90 for the creation of report 91. After the approval processing is completed, the display screen 52 switches from the image display bar 53 to report 91, allowing the user to generate report 91 immediately after approving the key image 13. Pre-entered patient information, user information, etc., are automatically entered into report 91. The user enters diagnostic opinions in the diagnostic opinion input bar 92, etc., to generate the report.

[0062] [Second Implementation]

[0063] In the second embodiment, the switching from the selected image 55 to the switching image 56 is not automatic. Instead, when the selected image 55 is selected, multiple candidate switching images 63 are displayed, from which the user selects any one of the candidate switching images 56. Furthermore, content shared with the first embodiment is omitted.

[0064] like Figure 9As shown, in the second embodiment, similarly to the first embodiment, the user selects the image 55 that needs to be reselected while the image is categorized. When the user selects the image 55, the switching image 56 is not switched; instead, a switching candidate image display bar 62 that displays a plurality of switching candidate images 63 is expanded on the display 52, and the user selects the switching image 56 from the plurality of switching candidate images 63. When the switching image 56 is selected, the expansion of the switching candidate image display bar 62 ends, and the selected image 55 and the switching image 56 switch. Thus, compared to the first embodiment, it is possible to select key images 13 that further reflect the user's preferences for reporting. The switching candidate images 63 are a subset of images selected according to the category of the selected image 55, and not all medical images 12 that are not displayed in the overview are shown.

[0065] The image groups displayed in the switching candidate image display bar 62 for the switching candidate image 63 can include "image groups of the same imaging area as the selected image 55", "image groups of imaging areas close to the selected image 55", "image groups of images where no treatment equipment is being used in the selected image 55", and "image groups of images where no pigment dispersion is occurring in the selected image 55", etc. Furthermore, the image groups displayed as switching candidate images 63 can also be configured to not display all medical images 12 that meet the above conditions, but instead display a limited number of images. This limits the number of images that need to be checked during selection, thus further reducing the workload. Moreover, the limited number of images can be selected considering the image's imaging time, imaging mode, or diagnostic usefulness based on image quality evaluation.

[0066] like Figure 10 As shown, the candidate images 63 displayed at a glance can also be further displayed by operating the page switching button 75 located on the candidate image display bar 62. The page switching button 75 has two functions: "forward" and "back," which the user can use to switch pages. Furthermore, it is preferable to display the page number in the center of the page switching button. Preferably, the display is initiated by images of the same category as the candidate images 63 and deemed to be of high usefulness.

[0067] Figure 11 This is a flowchart illustrating a series of processes that switch the selected image to a different display. The medical image processing apparatus 10 acquires medical images 12 captured by the endoscope system 11 using the image acquisition unit 20. The acquired medical images 12 are sent to the image classification unit 30 for classification.

[0068] In the image classification unit 30, image information of the medical image 12 is extracted and added to the original medical image 12 as category information and image quality information. Based on the category information, each medical image 12 is classified into a specified category. Image information extraction is performed by a CNN (Convolutional Neural Network) model that has been trained. The classified medical images 12 are then sent to the usefulness determination unit 40.

[0069] In the usefulness determination unit 40, usefulness is determined based on the category information and image quality information of the medical image 12, and the determination result is appended to the original medical image 12. The usefulness-determined medical image 12 is then sent to the image display unit 50.

[0070] In the image display unit 50, based on the results of category classification and usefulness determination, the medical images 12 are displayed on the monitor 52 as a list of automatically selected images 54. The images displayed in the list are preferably selected to include all prescribed categories and have received high evaluation in the usefulness determination.

[0071] The user can check from the list of automatically selected images 54 whether there is an image that needs to be reselected as the key image 13. If it exists, the user selects the image and switches the display to another image. If it does not exist, the user selects the "Accept" button 74.

[0072] The image corresponding to the category or usefulness of the selected image 55 is selected as the switching image 56 for switching display, and the one with high usefulness is preferred.

[0073] Repeat the process of reselecting the switching image 56 until the switching image 56 can be recognized as the key image 13. If no image is reselected in the list of images, select the approve button 74.

[0074] After the approval button is selected, the key image 13 is sent to the key image storage memory 80 and the report generation device 90.

[0075] In the above embodiments, the hardware structure of the processing units that perform various processes, such as the image acquisition unit 20, image classification unit 30, usefulness determination unit 40, image display unit 50, display control unit 51, input receiving unit 70, key image storage memory 80, and report generation device 90, is various processors as described below. These processors include general-purpose processors that execute software (programs) and function as various processing units, such as CPUs (Central Processing Units), GPUs (Graphical Processing Units), and FPGAs (Field Programmable Gate Arrays); processors whose circuit structure can be changed after manufacturing, i.e., Programmable Logic Devices (PLDs); and processors with circuit structures specifically designed for performing various processes, i.e., dedicated circuits.

[0076] A processing unit can be composed of one of these various processors, or it can be composed of a combination of two or more processors of the same or different types (e.g., multiple FPGAs, a combination of CPUs and FPGAs, or a combination of CPUs and GPUs, etc.). Furthermore, a single processor can also constitute multiple processing units. As examples of a single processor constituting multiple processing units, firstly, there is a method where, as exemplified by a client or server computer, a combination of one or more CPUs and software constitutes a single processor, which functions as multiple processing units. Secondly, there is a method where, as exemplified by a System-on-a-Chip (SoC), a processor is used to implement the overall system functionality including multiple processing units using a single IC (Integrated Circuit) chip. Thus, various processing units are constructed using one or more of the aforementioned processors as hardware structures.

[0077] More specifically, the hardware structure of these various processors is a circuit composed of semiconductor components and other circuit elements. Furthermore, the hardware structure of the storage section is a storage device such as an HDD (hard disk drive) or an SSD (solid state drive).

[0078] Symbol Explanation

[0079] 10-Medical image processing device, 11-Endoscope system, 11a-Light source device, 11b-Endoscope, 11c-Processor device, 12-Medical image, 13-Key image, 20-Image acquisition unit, 30-Image classification unit, 40-Usefulness determination unit, 50-Image display unit, 51-Display control unit, 52-Display, 53-Image display bar, 54-Automatically selected image, 55-Selected image, 56-Switching image, 57-Non-automatically selected image, 58-Classified image group, 59-Automatically selected image group, 60-Non-automatically selected image group, 61-Image information display bar, 62-Switching candidate image display bar, 63-Switching candidate image, 70-Input receiving unit, 71-Cursor, 73-Display switching button, 74-Approval button, 75-Page switching button, 80-Key image storage memory, 90-Report generation device, 91-Report, 92-Diagnostic opinion input bar.

Claims

1. A medical image processing device, comprising a processor, The processor performs the following processing: Acquire multiple medical images, The medical images are classified into at least one of several categories. Based on the classification results and image quality information, the diagnostic usefulness of the medical images is determined. Based on the classification results and the usefulness determination results, at least one of the multiple medical images is automatically selected for display. The system accepts the user's input from the displayed automatically selected images to be reselected as the key images. Instead of displaying the selected image, non-automatically selected images (excluding the automatically selected image) from among the multiple medical images categorized into the stated category are displayed as switching images. Accept input that recognizes the automatically selected image and the switched image displayed on the screen as the key image. A report on the inspection results was generated using the key images.

2. The medical image processing device according to claim 1, characterized in that, The processor switches the display of the selected image based on multiple categories.

3. The medical image processing device according to claim 1, characterized in that, The processor switching display shows the switching image that is categorized into the same category as the selected image.

4. The medical image processing device according to claim 2, characterized in that, The processor switching display shows the switching image that is categorized into the same category as the selected image.

5. The medical image processing apparatus according to claim 1, wherein, The processor performs the following processing: Multiple candidate switching images, which are candidates for the switching image, are displayed on the screen. Accept user input to determine the switching image from the multiple displayed switching candidate images.

6. The medical image processing apparatus according to claim 5, characterized in that, The switching candidate images include images of the same category as the selected image.

7. The medical image processing apparatus according to claim 1, characterized in that, When the processor selects the switching image from a plurality of medical images according to the category of the automatically selected image, it takes into account the photographic conditions of the medical image.

8. The medical image processing apparatus according to claim 1, characterized in that, When the processor selects the switching image from a plurality of medical images according to the category of the automatically selected image, it considers the diagnostic usefulness of the medical image.

9. The medical image processing apparatus according to any one of claims 1 to 4, 6 to 8, wherein, The processor assigns characters corresponding to the category to each image for display.

10. The medical image processing apparatus according to any one of claims 1 to 4, 6 to 8, wherein, The processor assigns label information corresponding to the category to each image for display.

11. The medical image processing apparatus according to any one of claims 1 to 4, 6 to 8, wherein, The processor changes the display method of each image according to the category.

12. The medical image processing apparatus according to any one of claims 1, 7, or 8, wherein, The processor displays the switching image and the automatically selected image in different ways.

13. The medical image processing apparatus according to claim 1, wherein, The processor performs the following processing: Depending on whether it has been accepted or approved, it will be displayed in different ways.

14. The medical image processing apparatus according to claim 1, wherein, The processor classifies images based on a learned CNN model obtained by learning from a group of images including multiple medical images.

15. The medical image processing apparatus according to any one of claims 1 to 4, 6 to 8, wherein, The category refers to information related to the parts of the subject being photographed.

16. The medical image processing apparatus according to any one of claims 1 to 4, 6 to 8, wherein, The category refers to information related to whether the image contains a region of interest.

17. The medical image processing apparatus according to any one of claims 1 to 4, 6 to 8, wherein, The category is information relating to whether the image contains treatment equipment or the type of treatment equipment contained therein.

18. The medical image processing apparatus according to any one of claims 1 to 4, 6 to 8, wherein, The category refers to information related to the distribution state of pigments or dyes in an image.

19. A method for operating a medical image processing device, comprising the following steps: Acquire multiple medical images; The medical images are classified into at least one of several categories; Based on the classification results and image quality information, the diagnostic usefulness of the medical image is determined. Based on the classification results and the usefulness determination results, at least one of the multiple medical images is automatically selected as the image to be displayed on the screen. The system accepts the user's input of selecting an image from the displayed automatically selected images that needs to be reselected as the key image; Instead of displaying the selected image, non-automatically selected images (excluding the automatically selected image) from among the multiple medical images classified into the category are displayed as switching images. Accept input that recognizes the automatically selected image and the switched image displayed on the screen as the key image; and A report on the inspection results was generated using the key images.

Images