Medical image processing apparatus and endoscope system

The medical image processing device with a classifier and multiple recognizers for body parts allows real-time, accurate medical image analysis by selecting optimal models for each part, addressing the challenge of varying image characteristics and processing speed in medical imaging technologies.

JP2024164943APending Publication Date: 2024-11-28FUJIFILM CORP

Patent Information

Application Number
JP2023080690
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-16
Publication Date
2024-11-28

AI Technical Summary

Technical Problem

Existing medical imaging technologies face challenges in achieving optimal performance and real-time processing for different body parts during examinations using machine learning, particularly with modalities like endoscopes and ultrasound diagnostic equipment, as they require different image characteristics and high-speed processing.

Method used

A medical image processing device equipped with a classifier for part classification and multiple recognizers for region of interest, enabling parallel processing and selection of optimal models based on classification results, using deep learning models trained on specific body parts to support real-time lesion detection and differentiation.

Benefits of technology

Enables simultaneous selection of the optimal examination support model for each body part and real-time processing, improving accuracy and efficiency in medical image analysis.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024164943000001_ABST
    Figure 2024164943000001_ABST
Patent Text Reader

Abstract

To provide a medical image processing apparatus and endoscope system capable of achieving both optimal portion-specific inspection support model selection for a medical image and real-time processing.SOLUTION: A medical image processing apparatus includes: a processor; a classifier that classifies an imaging portion in a medical image; and multiple recognizers that recognize a region of interest in the medical image and correspond to each imaging portion. The processor acquires multiple medical images, including a first medical image and a second medical image obtained at a later time than the first medical image, acquires a classification result of the imaging portion in the first medical image from the classifier, selects the recognizer corresponding to the classification result of the first medical image from the multiple recognizers, and acquires the recognition result of the region of interest in the second medical image from the selected recognizer.SELECTED DRAWING: Figure 7
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a medical image processing apparatus and an endoscope system. [Background technology]

[0002] Endoscopes and ultrasound diagnostic devices observe multiple parts of the body in a single examination. When implementing machine learning-based diagnostic support functions such as lesion detection and lesion differentiation for such medical devices, there is a problem that the desired performance cannot be achieved with a single machine learning model because image features differ depending on the part of the body. In such cases, a possible configuration would be to install a trained model optimized for each part of the body and use different trained models to provide examination support depending on the part classification results during the examination.

[0003] Since the user is required to perform complicated procedures during the examination, a configuration that automatically selects and switches depending on the classification result is preferable. In other words, a trained model of part classification that classifies the imaged part from medical images is prepared in addition to the examination support model for each part, and the trained examination support is selected depending on the result. Specifically, in Patent Document 1, lesion position information is obtained from CT (Computed Tomography) images and MRI (Magnetic Resonance Imaging) images, and the identity of the lesion is determined depending on the position. In Patent Document 2, the part is identified from the image data of a capsule endoscope, and an analysis according to the part is performed. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2013-123528 A [Patent Document 2] JP 2012-249956 A Summary of the Invention [Problem to be solved by the invention]

[0005] On the other hand, the subject of Patent Document 1 is a still image and processing of moving images is not anticipated, and in Patent Document 2, high-speed processing is difficult because part identification and analysis are performed on the same image. However, when providing support through machine learning or the like to modalities that perform examinations on moving images, such as endoscopes and ultrasound diagnostic equipment, it is required to output the processing results in real time during the examination.

[0006] An object of the present invention is to provide a medical image processing apparatus and an endoscope system that enable both selection of an optimal examination support model for each part of a medical image and real-time processing. [Means for solving the problem]

[0007] The medical image processing device of the present invention comprises a processor, a classifier that classifies imaged areas in medical images, and a recognizer that recognizes areas of interest in the medical images, with multiple recognizers corresponding to each imaged area, and the processor acquires multiple medical images having a first medical image and a second medical image acquired at a later time than the first medical image, acquires a classification result of the imaged areas in the first medical image from the classifier, selects a recognizer from the multiple recognizers that corresponds to the classification result of the first medical image, and acquires a recognition result of the area of ​​interest in the second medical image from the selected recognizer.

[0008] It is preferable that the classification of the imaged portion in the second medical image by the classifier and the recognition of the region of interest in the second medical image by the recognizer are processed in parallel.

[0009] It is preferable to notify the recognition result.

[0010] It is preferable to create a display image using the recognition result and notify the user by displaying the display image on a screen.

[0011] It is preferable to process in parallel the classification of the imaged area in a third medical image acquired at a later time than the second medical image by a classifier, the recognition of the area of ​​interest in the third medical image by a recognizer based on the imaged area in the second medical image, and the creation of a display image of the recognition result in the second medical image.

[0012] It is preferable to create an image for display by superimposing the recognition result on the third medical image.

[0013] In addition to the recognition result, it is preferable to notify the type of recognizer used to recognize the region of interest.

[0014] Preferably, the classification result is obtained using the first medical image and a plurality of medical images in the temporal vicinity of the first medical image that are acquired at an earlier time than the second medical image.

[0015] It is preferable to select a recognizer depending on the classification result and the selection history of recognizers at previous times.

[0016] Preferably, classification results of multiple medical images in the temporal vicinity are weighted based on the selection history, and the weighted classification results are used to select a recognizer.

[0017] It is preferable to obtain a classification result in which a certainty factor is calculated for each imaging part from the classifier, and if the certainty factor is less than a specific reference value, to continue using the recognizer from the previous time based on the selection history.

[0018] The multiple recognizers are preferably trained deep learning models, each trained on a different dataset.

[0019] When the classification result is the first site, it is preferable to select a recognizer trained on a dataset that includes more images captured at the first site than images captured at sites other than the first site.

[0020] It is preferable that the multiple recognizers each have the function of recognizing a different region of interest, and that a selected recognizer performs at least one of position detection, type determination, and region measurement for the region of interest in the medical image to obtain a recognition result.

[0021] It is preferable that the device is equipped with a classifier and a discriminator that integrates multiple recognizers, which calculates intermediate features in a medical image, classifies the imaged area from the intermediate features, and recognizes a region of interest from the intermediate features and the imaged area, obtains a first intermediate feature calculated from a first medical image and a second intermediate feature calculated from a second medical image, obtains a classification result based on the first intermediate features, selects a deep learning model from the multiple deep learning models that corresponds to the classification result for the first intermediate features, and obtains a recognition result for the region of interest in the second intermediate features from the selected deep learning model.

[0022] Preferably, the system is an endoscope system having a medical image processing device and an endoscope for acquiring medical images. Effect of the Invention

[0023] According to the present invention, it is possible to achieve both selection of an optimal examination support model for each part of a medical image and real-time processing. [Brief description of the drawings]

[0024] [Figure 1] FIG. 2 is an explanatory diagram showing connected devices of the medical image processing apparatus. [Diagram 2] 1 is an explanatory diagram of a group of time-series medical images acquired by a medical image processing apparatus; [Diagram 3] FIG. 2 is a block diagram showing the functions of the medical image processing apparatus. [Figure 4] FIG. 2 is an explanatory diagram of a medical image for which region classification is performed. [Diagram 5] FIG. 1 is an explanatory diagram of a medical image in which region of interest recognition is performed using classification results. [Figure 6] 1A is an explanatory diagram showing a case where serial processing is performed and a case where parallel processing is performed in part classification and attention area recognition. FIG. [Figure 7] 11 is an explanatory diagram of a parallel process of part classification and attention area recognition using a classification result at a previous time point. FIG. [Figure 8] FIG. 11 is an explanatory diagram of a method for selecting a recognizer in a second selection mode. [Figure 9]FIG. 13 is an explanatory diagram of a method for selecting a recognizer in a third selection mode. [Figure 10] FIG. 13 is an explanatory diagram of a method for selecting a recognizer in a fourth selection mode. [Figure 11] FIG. 11 is an explanatory diagram of parallel processing of part classification, attention area recognition using the classification result from the previous time, and image creation using the recognition result from the previous time. [Figure 12] FIG. 1 is an explanatory diagram illustrating parallel processing of part classification, attention area detection, and image creation for each frame. [Figure 13] FIG. 13 is an image diagram showing a display image. [Figure 14] FIG. 13 is an explanatory diagram showing a schematic diagram displayed on a screen. [Figure 15] 2 is a flowchart showing a series of processing steps according to the present invention. [Figure 16] FIG. 11 is a block diagram showing the functions of a medical image processing apparatus according to a second embodiment. [Figure 17] FIG. 11 is an explanatory diagram of parallel processing for performing part classification, attention area detection, and display image creation in the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0025] [First embodiment] 1 is a diagram showing connected devices of a medical image processing device 11 in a medical image processing system 10 according to an embodiment of the present invention. The medical image processing system 10 has a medical image processing device 11, a database 12, an endoscope system 13 including an endoscope 13a, and a user interface (UI) 14. The medical image processing device 11 is electrically connected to the database 12, the endoscope system 13, and the user interface 14. Alternatively, instead of the medical image processing system 10, the endoscope system 13 may be configured to include the medical image processing device 11.

[0026] The database 12 is a device that stores acquired images and can transmit and receive data to and from the medical image processing device 11, and may be a recording medium such as a USB (Universal Serial Bus) or a HDD (Hard Disc Drive).

[0027] The user interface 14 may be a device in which a notification means such as a touch panel and an input means are integrated, or a device in which the notification means and the input means are separate, such as a display and a keyboard or mouse. It may also be equipped with all of these devices. The input means in the user interface 14 may be, in addition to or instead of a keyboard or mouse, input means provided in the medical device, such as a foot pedal, a voice recognizer, or a switch on the endoscope 13a.

[0028] As shown in FIG. 2, the medical image processing device 11 acquires a group of medical images 17 captured during a medical examination using an endoscope system 13 or other medical equipment. The group of medical images 17 is a time series of images made up of a plurality of medical images 18 that are consecutive in time. The medical images 18 are frames that make up a video or still images. Unless otherwise specified in the photographing of the medical examination, white light is used as the illumination light, a video signal of 60 frames per second (60 fps (frames per second)) is acquired, and the photographing time is recorded. Furthermore, when the video signal is 60 fps, it is preferable to measure the time in units of 1 / 100th of a second.

[0029] 3, in the medical image processing device 11, a central control unit (not shown) constituted by a processor operates a program in a program memory to realize and control the functions of an input receiving unit (not shown), an image acquiring unit 20, a part classifier 21, a classification result deciding unit 22, an attention area recognizing unit 24, a recognition result acquiring unit 29, an image creating unit 30, and an output control unit 31. The attention area recognizing unit 24 also has a classification result storage unit 25, a first recognizer 26, a second recognizer 27, and a third recognizer 28.

[0030] Since lesions and organs vary greatly depending on the imaging site, the attention area recognition unit 24 has a trained model optimized for each site. By switching the trained model for each site according to the imaging site, support can always be provided by a high-performance trained model. For this reason, the medical image processing device 11 includes a site classifier 21 that classifies the imaging site in the medical image 18, and a recognizer that recognizes the attention area R in the medical image 18, and includes multiple recognizers corresponding to each imaging site.

[0031] The imaging parts classified by the part classifier 21 correspond to the types of imaging parts learned by each recognizer. The correspondence may be a coincidence, or a correspondence may be such that a recognizer learned with "stomach" is selected when the classification result is "fundus of stomach" or "body of stomach". This allows examination support such as detection, differentiation, and measurement of lesions for more accurate recognition of the region of interest, compared to covering multiple parts during the same examination with the same learned model.

[0032] The input receiving unit receives an input from a user via the user interface 14. In the medical image processing apparatus 11, programs relating to processes such as image processing are stored in a program memory (not shown).

[0033] The image acquisition unit 20 acquires medical images 18, which are frame images, from the endoscope system 13 or the like. Each time a medical image 18 is acquired, it is input to the part classifier 21, the area of ​​interest recognition unit 24, and the image creation unit 30, and part classification, area of ​​interest recognition, and image editing are performed.

[0034] The part classifier 21 has a function of a trained model required for classification processing of parts such as organs and tissues of a living body captured in the medical image 18, and executes part classification processing for each image. The part classifier 21 is equipped with a CNN (Convolutional neural network), which is a computer algorithm consisting of a neural network that performs machine learning, and learns in advance with data covering all parts that may be photographed in an upper endoscopy examination, such as the esophagus, stomach, duodenum, nasal cavity, oral cavity, and pharynx, and performs inference of part classification in the input medical image 18 and calculation of the certainty of the classified part. The inferred part and its certainty are regarded as the classification result.

[0035] As shown in Fig. 4, when a medical image 18 is input to the part classifier 21, a classification process of the part to be classified that corresponds to the medical image 18 is performed by inference using machine learning, and the imaging part is inferred. For example, when observing the upper digestive tract with an endoscope, the inferred part is the esophagus, stomach, duodenum, etc. of the medical image 18. In addition, a confidence level corresponding to the inferred part is calculated, and a classification result having the imaging part and the confidence level is transmitted to the classification result determination unit 22.

[0036] The confidence level is an index showing the reliability of the classification result calculated by inference from the learning model using the match rate with previously learned learning content, the false positive rate for each classification target, image quality information, etc., and is expressed, for example, as a percentage (%).

[0037] The classification result determination unit 22 obtains the classification result in which the confidence level is calculated for each imaging part from the part classifier 21, and determines whether the confidence level is equal to or higher than a specific reference value, such as a preset threshold. The confidence level is calculated from the matching rate with the learned data set in the classification by the part classifier 21. If the confidence level of the classification result is equal to or higher than the specific reference value, it is sent to the attention area recognition unit 24 and can be used to select a recognizer, but if the confidence level is less than the specific reference value, the classification result is changed to "unclassifiable" and output to the attention area recognition unit 24.

[0038] The attention area recognition unit 24 determines a recognizer to be used for the attention area recognition process from the first recognizer 26, the second recognizer 27, and the third recognizer 28 based on the classification result output from the classification result determination unit 22, and executes attention area recognition for the input medical image 18 using the recognizer. The classification result of the previous time stored in the classification result storage unit 25 may be used to determine the recognizer. Also, a selection history of recognizers for a certain period of time may be stored, and the past selection history may be used to select a recognizer.

[0039] The classification result storage unit 25 temporarily stores the classification result acquired by the attention area recognition unit 24 from the part classifier 21. The stored classification result is used to select a recognizer according to a selection mode described later. The temporary storage is preferably performed in a storage memory (not shown) realized in the medical image processing device. Also, instead of the storage memory, the database 12 may have that function.

[0040] Each recognizer is equipped with a CNN (Convolutional neural network), a computer algorithm consisting of a neural network that performs machine learning, and recognizes a specific subject captured in the input medical image 18 according to the learning content of each part of the living body that has been previously performed. Diagnostic support by recognizing the region of interest includes detection of lesions, differentiation of lesions, detection of organs, measurement of lesions, etc. Each recognizer has the function of a trained deep learning model trained with a different data set.

[0041] Each recognizer is trained using a data set that includes more images captured at the corresponding imaging site than images captured at other imaging sites to be classified by site classifier 21. For example, a recognizer that performs region of interest recognition for a medical image of the "esophagus" is a recognizer trained with a data set that includes more images captured at the "esophagus" than images captured at the "stomach" or "duodenum," which are sites different from the "esophagus." Each recognizer is preferably trained with a data set made up of images captured at a site to be classified by site classifier 21.

[0042] As shown in Fig. 5, the attention area recognition unit 24 receives the classification result from the classification result determination unit 22 and the input of the medical image 18 from the image acquisition unit 20, and selects a recognizer to be used for recognition processing of the medical image 18 according to the classification result. In attention area recognition, the presence or absence of an attention area R, such as a specific object, is determined for each input medical image 18. In addition to determining the presence or absence, each recognizer has a different attention area recognition function, and it is preferable that the selected recognizer obtains the recognition result by performing at least one of position detection, type determination, and size measurement of the attention area R. The specific object recognized as the attention area R is an object that is highly likely to be a disease, a treatment scar, etc.

[0043] Specifically, each recognizer has at least one of a position detection function, a type determination function, and a size measurement function according to the corresponding imaging part. The position detection function detects the position within an organ or image relative to the attention region R, the type determination function determines whether the attention region R is a lesion or the type such as a tumor or bleeding, and the size measurement function measures the size, such as the total length, of the attention region R. For example, a recognizer corresponding to the "esophagus" operates the position detection function of the attention region R, and a recognizer corresponding to the "stomach" also operates the type determination function of the attention region.

[0044] Four or more types of recognizers may be provided, and in that case, classification is performed so that the classification results by the site classifier 21 correspond to each recognizer. For example, there may be recognizers that are trained on the "nasal cavity", "oral cavity", and "pharynx". Alternatively, instead of the "stomach", different recognizers may be provided for the "fundus", "body", and "vestibule", and the site classifier 21 may be trained to classify the "fundus", "body", and "vestibule" instead of the "stomach".

[0045] The recognition result acquisition unit 29 acquires the recognition result of each recognition, for example, information such as the position, type, or size of the region of interest R. The recognition result includes at least the determination result of the presence or absence of the region of interest R in the medical image 18 and the type of recognizer used.

[0046] The image creation unit 30 obtains the recognition results and classification results, and creates a display image. The display image is created by superimposing the recognition results and classification results on the medical image 18 and a schematic diagram. The created display image is sent to the output control unit 31. The classification result is for the medical image 18 at the timing used to select the recognizer.

[0047] The output control unit 31 controls the display of the classification results and the recognition results, such as the display of the display image created by the image creation unit 30. Apart from the image display, audio notification may be performed. When the display image is displayed, the medical image 18 may be displayed on the screen at the same time, or only the display image may be displayed. When audio notification is performed without displaying the display image, the medical image group 17 is displayed.

[0048] The medical image group 17 acquired by the image acquisition unit 20 from the database 12 or the endoscope system 13 is subjected to part classification processing and recognition processing one by one as a medical image 18. By using a learnt recognizer for the classified imaging parts, it is possible to recognize the region of interest with high accuracy.

[0049] On the other hand, in order to shorten the processing time for image classification and recognition using a trained model, the model size needs to be reduced, resulting in a decrease in accuracy. The medical image processing device 11 performs real-time site classification and area of ​​interest recognition for the medical image group 17, and therefore needs to secure processing time to maintain sufficient accuracy for both.

[0050] 6A, in serial processing of performing region-of-interest recognition processing on medical image 18 that has undergone region classification processing, if a certain level of accuracy is maintained, the sum of the processing time by region classifier 21 and the processing time by the recognizer does not fall within the time available for real-time processing, and it may not be possible to achieve both accuracy and real-time processing. Therefore, as shown in FIG6B, by selecting a recognizer corresponding to the classification result of the previous frame, region classification in medical image 18 by region classifier 21 and region-of-interest recognition in medical image 18 by one of the recognizers are processed in parallel.

[0051] In the case of endoscopic and ultrasound diagnosis, examinations are performed continuously, so switching of parts on a frame-by-frame basis does not occur frequently. Therefore, it can be said that there is almost no adverse effect even if the part recognition results of the previous frame are applied to the current frame. Note that each recognizer may require different processing times when arbitrarily executing functions such as position detection, size measurement, and type discrimination for the area of ​​interest in the recognition process.

[0052] By performing parallel processing of part classification and area of ​​interest recognition, the processing time for part classification and area of ​​interest recognition is secured separately, so that a high-performance trained model can be used for part classification and area of ​​interest recognition.

[0053] The recognition result is preferably notified by using a screen display, a voice, etc. In real-time observation, if the notification is to distinguish only the presence or absence of an attention area without creating an image, such as by issuing an alert when an attention area is recognized, the notification may be performed consecutively with the attention area recognition. On the other hand, when a specific recognition result of the attention area is to be notified, in addition to the part classification by the part classifier 21 and the attention area recognition by the attention area recognition unit 24, parallel processing of image editing by the image creation unit 30 is performed.

[0054] Regarding examples of this embodiment, a first example describes a parallel process that notifies only the presence or absence of a region of interest in the recognition result, and a second example describes a parallel process that displays an image of the recognition result. In the first and second examples, the first recognizer 26 is a recognizer that corresponds to the classification result of "esophagus", the second recognizer 27 is a recognizer that corresponds to the classification result of "stomach", and the third recognizer 28 is a recognizer that corresponds to the classification result of "duodenum".

[0055] A first embodiment will be described. In the first embodiment, in real-time observation, only the presence or absence of a region of interest is output as the recognition result. The recognition result by the recognizer is notified to the user within the real-time processing time. The notification method is, for example, a sound such as a beep that requires a short processing time, or a screen display of a fixed symbol or character.

[0056] As shown in FIG. 7, in parallel processing performed at any time T1, in order to use the classification result performed on the medical image 18 at time T0 at time T1, the image acquisition unit 20 acquires a medical image group 17, which is a time series of medical images having the medical image 18 at time T0 and the medical image 18 at time T1 acquired at a time later than the medical image 18 at time T0. From time T0 to T1, the attention area recognition unit 24 acquires the classification result of the imaging area in the medical image 18 at time T0 from the area classifier 21, selects a recognizer corresponding to the classification result of the medical image 18 at time T0 from the first recognizer 26, the second recognizer 27, or the third recognizer 28, and performs attention area recognition. The recognition result of the attention area in the medical image 18 at time T1 is acquired from the selected recognizer. The acquired recognition result is transmitted to the recognition result acquisition unit 29.

[0057] For example, when the classification result determination unit 22 obtains a classification result of "esophagus" in the part classification at time T0, the first recognizer 26 is selected for attention area recognition at time T1, which is the timing for performing parallel processing next to time T0, and performs attention area recognition on the medical image 18 at time T1. Also, the part classification of the medical image 18 at time T1 is processed in parallel.

[0058] When the classification result determination unit 22 obtains a classification result of "stomach" in the part classification at time T1, the second recognizer 27 is selected as the recognizer at time T2, which is the timing for performing parallel processing next to time T1. When the same classification result is continuously obtained such that the part classification at time T2 also results in a classification result of "stomach", the attention area recognition unit 24 may continue the selection state rather than resetting the selection state and selecting again.

[0059] The classification result determination unit 22 obtains the classification result in which the certainty factor is calculated for each imaging part from the part classifier 21, and determines whether the certainty factor is equal to or greater than a specific reference value such as a preset threshold. If the certainty factor is less than the specific reference value, it transmits a classification result of "unclassifiable" instead of the inferred imaging part to the attention area recognition unit 24. If the attention area recognition unit 24 obtains a classification result of "unclassifiable", it continues to use the recognizer selected at the previous time from the recognizer selection history to recognize the attention area.

[0060] Regarding the selection of a recognizer, not only the classification result obtained in the previous parallel processing is used, but also classification results from multiple previous times may be used to select a recognizer. For example, the classification result determination unit 22 has first to fourth selection modes, and each selection mode selects a recognizer according to a different rule. It is preferable that the user can arbitrarily switch between each mode. Also, the contents of each selection mode may be combined and executed.

[0061] In the first selection mode, as described above, a recognizer is selected using the classification result in which the confidence level was determined for the medical image 18 in the immediately preceding parallel processing.

[0062] In the second selection mode, the classification result is determined using the medical image 18 that has been processed in parallel and multiple medical images 18 that are temporally close to the medical image 18. When parallel processing is performed at time T1, the classification results of the medical image 18 that has been processed in parallel at time T0 and the medical images 18 that have been temporarily stored and acquired within a certain range from time T0 are integrated. Note that the multiple medical images 18 used to select the recognizer are all acquired at a timing prior to the medical image 18 at time T1 that is used for parallel processing.

[0063] As shown in FIG. 8, when the latest five classification results are integrated, at time T5, the classification result storage unit 25 stores the classification results from times T0 to T4, and the attention area recognition unit 24 uses the most popular classification result as the integrated result to select a recognizer. Even if there is a classification result of "esophagus" at the immediately preceding time T4, if "stomach" is more popular by majority vote, the second recognizer 27 is selected. At least three classification results that allow a majority vote are used for the temporally adjacent classification results. The integrated result may be obtained by averaging the CNN results or removing outliers instead of a majority vote based on multiple classification results. By integrating and using the results of multiple frames, more accurate classification can be achieved.

[0064] In the third selection mode, a recognizer is selected according to the classification result of the acquired medical image 18 and the recognizer selection history at the previous time. The recognizer selection history in the previous parallel processing is compared with the imaged part of the acquired classification result, and a classification result that is likely to be misclassified is identified by determining whether the transition of the classification result of the imaged part is natural or unnatural in terms of the human body structure. In the upper digestive tract, at least the order of the human body structures of "esophagus," "stomach," and "duodenum" is stored.

[0065] As shown in FIG. 9, for example, if "esophagus" is continuously detected in the parallel processing up to time T1, the transition state is determined to be normal in the parallel processing up to time T2, and the first recognizer 26 corresponding to the imaging site of "esophagus" is selected. If the classification result of "duodenum" is obtained for the parallel processing at time T2, it is determined to be an abnormal state with a high possibility of misclassification, and the classification result of "duodenum" is not used in selecting the recognizer to be used at time T3, and the second recognizer 27, which is the previous selection result, is continuously used. It is possible to prevent the selection of an incorrect recognizer, and the deterioration of the accuracy of the attention area recognition and the occurrence of unexpected behavior due to an incorrect recognizer. By excluding classification results with a high possibility of misclassification, an appropriate recognizer is selected and highly accurate attention area recognition is performed.

[0066] In the fourth selection mode, the classification results of multiple temporally nearby medical images 18 are weighted based on the recognizer selection history at the previous time, and a recognizer is selected from the integrated result obtained by integrating the weighted classification results. For example, a score is calculated by multiplying the number of times each classification result is detected in a certain period of time by a weighting coefficient. For weighting purposes, only two temporally nearby medical images 18 may be used.

[0067] As shown in FIG. 10, if the previous selection result is the second recognizer 27, the classification results of the parts not corresponding to "esophagus" are weighted at the same weight, and the parts corresponding to "esophagus" are weighted at 1.5 times to calculate the score. The classification result of "unclassifiable" is not used in the score calculation. The part with the highest total value of the calculated scores is used as the integrated result, and the recognizer corresponding to the integrated result is selected. If the classification results are "esophagus" three times, "duodenum" once, and "unclassifiable" once at times T0 to T4, the score is 4.5 for the esophagus and 1 for the duodenum, and the second recognizer 27 is selected for the parallel processing at time T5.

[0068] A second embodiment will be described. In the second embodiment, a display image is created in real time and is displayed on the screen to notify the user. In an endoscopic examination, a display image that can support real-time diagnosis and treatment is created, making it possible to determine the necessity of treatment during the same examination.

[0069] 11, when part classification, area of ​​interest recognition, and display image creation are performed in parallel, the classification result from the previous timing is used for area of ​​interest recognition, and the recognition result from the previous timing based on the classification result from two timings ago is used for display image creation. Specifically, in the parallel processing at time T2, the part classifier 21 classifies the imaged part in the medical image 18 at time T2, the recognizer recognizes the area of ​​interest in the medical image 18 at time T2 based on the imaged part in the medical image 18 at time T1, and the recognition result for the medical image at time T1 is superimposed on the medical image 18 at time T2 to create the display image 40.

[0070] As shown in Fig. 12, when parallel processing is performed on a frame by frame basis on a plurality of medical images 18 in a medical image group 17, an image is created using a recognition result for the previous frame using a classification result for the frame two frames before. The parallel processing for acquiring medical images 18 of the nth frame at time T0, the n+1th frame at time T1, and the n+2th frame at time T2 will be described. In the parallel processing at time T2, the part classifier 21 performs part classification for the n+2th frame, a recognizer based on the classification result for the n+1th frame performs attention area recognition for the n+2th frame, and the image creator 30 edits the n+2th frame using the recognition result for the n+1th frame to create an image for display. The created image for display is displayed on the screen in real time.

[0071] 13, for image display, a display image 40 created by the image creation unit 30 based on the recognition result is used. In the display image 40, whether to display a marker 41, a type display field 42, and a size display field 43 is appropriately determined according to the pre-settings and the functions realized by the recognizer. A used recognizer display field 44 displays the selection result of the recognizer used for recognizing the region of interest.

[0072] A marker 41 that highlights the position of the region of interest R is superimposed on the medical image 18, and a type display field 42 displays whether or not the region of interest R is a lesion when the recognizer identifies it, and if so, the type of lesion. A size display field 43 displays the measurement results obtained when the recognizer measures the region of interest R. A recognizer used display field 44 displays the type of recognizer used to recognize the region of interest R.

[0073] The type display field 42, size display field 43, and used recognizer display field 44 may be displayed superimposed on the medical image 18, or may be displayed without superimposition, such as next to the area of ​​the medical image 18.

[0074] As shown in Fig. 14, a schematic diagram 45 in which information on the region of interest R is superimposed may be created and displayed on the screen together with the display image 40. The schematic diagram 45 is a schematic diagram of the range of the living body of the subject, for example, displaying the range from the esophagus to the duodenum. The schematic diagram 45 can notify the position in the body of the subject at which the region of interest R was detected. Furthermore, when multiple regions of interest R are detected, the positions of the respective regions of interest R may be displayed simultaneously.

[0075] A series of flows of medical image processing of this embodiment will be described with reference to the flowchart shown in Fig. 15. The medical image processing device 11 acquires a medical image 18 at time T0 constituting a medical image group 17 captured by a medical examination (step ST100). The acquired medical image 18 is input to the part classifier 21, which performs part classification (step ST110). The acquired classification result is sent to the attention area recognition unit 24, which selects a recognizer corresponding to the classification result (step ST120).

[0076] A medical image 18 at time T1, which is the timing next to time T0 input to the selected recognition, is acquired (step ST130). The selected recognizer is used to recognize the area of ​​interest in the medical image 18 at time T1 (step ST140). The acquired recognition result is transmitted to the image creation unit 30 (step ST150).

[0077] The image creation unit 30 acquires the medical image 18 at time T2, which is the timing following time T1, to be used for creating the display image 40 of the recognition result (step ST160). The display image 40 is created using the medical image 18 at time T2 and the recognition result (step ST170). The created display image 40 is displayed on the screen to notify the recognition result (step ST180).

[0078] As a result of the above, by performing parallel processing of part classification and regions of interest, it is possible to achieve both real-time processing and the selection of a recognizer, which is the optimal inspection support model for each part.

[0079] [Second embodiment] In the first embodiment, the selection of the recognizer is controlled using the part classification result obtained by the part classifier 21 for each medical image 18 in parallel processing. In contrast, a description will be given of a form in which a determiner is provided that integrates the functions of the part classifier 21 and multiple recognizers corresponding to each imaging part, and the calculation of intermediate feature amounts is performed as a common process. Note that a description of the contents common to the other embodiments will be omitted.

[0080] 16, the medical image processing device 50 in the second embodiment has the functions of an image acquisition unit 20, an integrated determiner 60, a recognition result acquisition unit 29, an image creation unit 30, and an output control unit 31. The integrated determiner 60 performs parallel processing of part classification and area of ​​interest recognition for a medical image 18, in place of the part classifier 21, the classification result decision unit 22, and the area of ​​interest recognition unit 24 of the first embodiment.

[0081] The integrated determinator 60 has a function that integrates an intermediate feature calculation unit 61, a part classification unit 62, a classification result storage unit 25, a first recognition unit 64, a second recognition unit 65, and a third recognition unit 66. The integrated determinator 60 calculates intermediate features for the medical image 18 transmitted from the image acquisition unit 20 in the intermediate feature calculation unit 61, and transmits the calculated intermediate features to the part classification unit 62 and one of the selected recognition units.

[0082] The intermediate feature calculation unit 61 calculates intermediate features of the medical image 18, which are common to the part classification and the area of ​​interest recognition. The calculated intermediate features are transmitted to the part classification unit 62 and to one of the recognition units selected by the classification result acquisition unit 63.

[0083] Part classification unit 62 performs part classification processing on the acquired intermediate feature amounts. Part classification unit 62 preferably has the functions of part classifier 21 of the first embodiment, and receives input of intermediate feature amounts instead of medical image 18 to perform part classification.

[0084] The classification result acquisition unit 63 acquires the classification result by the part classification unit 62, and selects a recognition unit that performs attention area recognition corresponding to the classification result from one of the first recognition unit 64, the second recognition unit 65, and the third recognition unit 66. The intermediate feature amount is transmitted to the selected recognition unit. In addition, the classification result acquisition unit 63 may have the functions of the classification result determination unit 22 and the classification result storage unit 25 in the first embodiment, and may switch between the first to fourth selection modes.

[0085] When selected by the classification result acquisition unit 63, the first recognition unit 64 performs area of ​​interest recognition in the transmitted intermediate feature amount. The first recognition unit 64 preferably has the function of the first recognizer 26 of the first embodiment, and receives input of the intermediate feature amount instead of the medical image 18 to perform area of ​​interest recognition.

[0086] Like the first recognition unit 64, the second recognition unit 65 has the function of the second recognizer 27, and the third recognition unit 66 has the function of the third recognizer 28, and it is preferable that they perform region of interest recognition by receiving input of intermediate features instead of the medical image 18. Note that each recognition unit may be provided with a deep learning model that corresponds to each body part and performs recognition processing from the intermediate features.

[0087] In the parallel processing, the integrated determiner 60 obtains the classification result based on the intermediate feature amount in the previous parallel processing from the part classification unit 62, and uses it to recognize the region of interest of the intermediate feature amount in the obtained medical image 18. From each recognition unit, a recognition unit corresponding to the classification result in the previous parallel processing is selected, and the recognition result of the region of interest in the intermediate feature amount is obtained from the selected recognition unit.

[0088] As shown in FIG. 17, in the parallel processing at time T1, the classification result of the medical image 18 at time T0 and the intermediate feature amount calculated from the medical image 18 at time T1 are used to perform site classification and attention area recognition. The recognition result by the attention area recognition is transmitted to the recognition result acquisition unit 29. For example, the integrated determiner 60 acquires the classification result of "esophagus" for the intermediate feature amount at time T0 by the time of calculating the intermediate feature amount at time T1, and selects the first recognition unit 64, which is the recognition unit corresponding to "esophagus", which is the classification result of the intermediate feature amount at time T0, from among the multiple recognition units. The intermediate feature amount at time T1 is transmitted to the site classification unit 62 and the first recognition unit 64, which is the recognition unit corresponding to "esophagus", and site classification and attention area recognition are performed to acquire the classification result and recognition result of "stomach".

[0089] Similarly, in the parallel processing at time T2, the classification result of "stomach" classified at time T1 is transmitted to the classification result acquisition unit 63 by the time of calculating the intermediate features of the medical image 18 at time T2, the second recognition unit 65 corresponding to "stomach" is selected, and parallel processing is performed.

[0090] As described above, even before the acquisition of the part classification result, a part of the processing of the part classification and the region of interest recognition of the medical image 18 at each time can be started, so that the parallel processing, especially the time for the region of interest recognition, can be secured longer than in the configuration of the first embodiment. Also, since the time from the acquisition of the classification result to the end of the region of interest recognition can be shortened, real-time processing of the medical image group 17 at a frame rate higher than that of the first embodiment is possible.

[0091] It is also possible to realize a medical image processing system 10 that includes a medical image processing device that has the functions of the part classifier 21, classification result determination unit 22, and area of ​​interest recognition unit, as well as the function of the integrated judger 60, and that selectively uses different functions as needed.

[0092] In the above embodiment, an example has been described in which the medical image processing device 11 performs parallel processing of medical images captured by the endoscope 13a. However, the present invention is not limited to this, and parallel processing may also be performed on each medical image 18 of the medical image group 17 acquired by other medical examination devices, such as an ultrasound imaging device or a radiography device.

[0093] In the above embodiment, the hardware structure of the processing units that execute various processes, such as the central control unit, image acquisition unit 20, part classifier 21, classification result determination unit 22, attention area recognition unit 24, recognition result acquisition unit 29, image creation unit 30, output control unit 31, and integrated determiner 60, is various processors as shown below. The various processors include a CPU (Central Processing Unit), which is a general-purpose processor that executes software (programs) and functions as various processing units, a programmable logic device (PLD), which is a processor whose circuit configuration can be changed after manufacture, such as an FPGA (Field Programmable Gate Array), and a dedicated electric circuit, which is a processor having a circuit configuration designed specifically for executing various processes.

[0094] One processing unit may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same type or different types (for example, multiple FPGAs, or a combination of a CPU and an FPGA). Also, multiple processing units may be configured with one processor. As an example of configuring multiple processing units with one processor, first, as represented by a computer such as a client or server, one processor is configured with a combination of one or more CPUs and software, and this processor functions as multiple processing units. Second, as represented by a system on chip (SoC), there is a form in which a processor is used that realizes the functions of the entire system including multiple processing units with one IC (Integrated Circuit) chip. In this way, the various processing units are configured using one or more of the above various processors as a hardware structure.

[0095] Furthermore, the hardware structure of these various processors is, more specifically, an electric circuit (circuitry) in the form of a combination of circuit elements such as semiconductor elements, and the hardware structure of the memory unit is a storage device such as a hard disk drive (HDD) or a solid state drive (SSD).

[0096] Moreover, from the above description, the medical image processing apparatuses described in Supplementary Notes 1 to 15 below and the endoscope system described in Supplementary Note 16 can be understood.

[0097] [Appendix 1] A processor; A classifier for classifying an imaging region in a medical image; a recognizer for recognizing an area of ​​interest in the medical image, the recognizer including a plurality of recognizers corresponding to each of the imaging parts; The processor, acquiring a plurality of medical images including a first medical image and a second medical image acquired at a later time than the first medical image; obtaining a classification result of the imaging region in the first medical image from the classifier; selecting the recognizer from the plurality of recognizers corresponding to the classification result of the first medical image; A medical image processing device that obtains a recognition result of the region of interest in the second medical image from the selected recognizer. [Appendix 2] The processor, classifying the imaged region in the second medical image by the classifier; and 2. A medical image processing device as described in claim 1, wherein the recognition of the region of interest in the second medical image by the recognizer is processed in parallel. [Appendix 3] The processor, 3. The medical image processing device according to claim 1, further comprising: [Appendix 4] The processor, Creating a display image using the recognition result; 4. The medical image processing device according to claim 3, wherein the display image is displayed on a screen to notify the user. [Appendix 5] The processor, a classification of the imaging site in a third medical image acquired at a later time than the second medical image by the classifier; Recognition of the region of interest in the third medical image by the recognizer based on the imaging part in the second medical image; 5. A medical image processing device as claimed in claim 4, further comprising: a processor for processing the display image of the recognition result in the second medical image in parallel; [Appendix 6] The processor, 6. A medical image processing device as described in claim 5, which creates the display image by superimposing the recognition result on the third medical image. [Appendix 7] The processor, 7. The medical image processing device according to claim 3, further comprising: a notification of the type of the recognizer used to recognize the region of interest in addition to the recognition result. [Appendix 8] The processor, A medical image processing device as described in any one of appendices 1 to 7, which obtains the classification result using the first medical image and a plurality of medical images in the temporal vicinity of the first medical image that were acquired at a time earlier than the second medical image. [Appendix 9] The processor, 9. The medical image processing device according to claim 8, wherein the recognizer is selected according to the classification result and a selection history of the recognizer at the previous time. [Appendix 10] The processor, weighting the classification results of the plurality of medical images in the temporal vicinity based on the selection history; 10. The medical image processing device of claim 9, wherein the recognizer is selected using the weighted classification result. [Appendix 11] The processor, obtaining the classification result in which a confidence level is calculated for each of the imaging parts from the classifier; 10. The medical image processing device of claim 9, wherein if the confidence level is less than a specific reference value, the use of the recognizer at the previous time from the selection history is continued. [Appendix 12] 12. The medical image processing device according to claim 1, wherein the plurality of recognizers are trained deep learning models trained on different data sets. [Appendix 13] The processor, 13. The medical image processing device of claim 12, wherein when the classification result is a first part, the medical image processing device selects the recognizer trained with a dataset including more images captured at the first part than images captured at parts other than the first part. [Appendix 14] The plurality of recognizers each have a function of recognizing a different one of the attention regions, 14. A medical image processing device according to any one of claims 1 to 13, wherein the selected recognizer performs at least one of position detection, type determination, and area measurement for the area of ​​interest in the medical image to obtain the recognition result. [Appendix 15] A discriminator that integrates the classifier and a plurality of the recognizers, a classifier that calculates intermediate features in the medical image, classifies the imaging region from the intermediate features, and recognizes the region of interest from the intermediate features and the imaging region; The processor, obtaining a first intermediate feature calculated from the first medical image and a second intermediate feature calculated from the second medical image; obtaining the classification result based on the first intermediate feature; selecting a deep learning model corresponding to the classification result for the first intermediate feature from a plurality of deep learning models; A medical image processing device as described in Appendix 1, which obtains a recognition result of the region of interest in the second intermediate feature from the selected deep learning model. [Appendix 16] A medical image processing device according to any one of claims 1 to 15; and an endoscope for acquiring the medical image. [Explanation of symbols]

[0098] 10 Medical image processing system 11 Medical image processing equipment 12 Database 13 Endoscope System 13a Endoscope 14 User Interface 17 Medical Images 18 Medical Imaging 20 Image acquisition section 21 Part classifier 22 Classification result determination section 24 Attention area recognition unit 25 Classification result storage unit 26 1st recognizer 27 Second recognizer 28 Third recognizer 29 Recognition result acquisition section 30 Image Creation Department 31 Output control section 40 Images for display 41 Marker 42 Type display column 43 Size display column 44 Use recognition device display field 45 Schematic diagram 50 Medical image processing device 60 Integrated judger 61 Intermediate feature calculation unit 62 Part classification section 63 Classification result acquisition part 64 1st recognition part 65 2nd recognition part 66 Third recognition part R Area of ​​interest ST100~180 steps T0~T5 time

Claims

1. A processor; A classifier for classifying an imaging region in a medical image; a recognizer for recognizing an area of ​​interest in the medical image, the recognizer including a plurality of recognizers corresponding to each of the imaging parts; The processor, acquiring a plurality of medical images including a first medical image and a second medical image acquired at a later time than the first medical image; obtaining a classification result of the imaging region in the first medical image from the classifier; selecting the recognizer corresponding to the classification result of the first medical image from the plurality of recognizers; A medical image processing device that obtains a recognition result of the region of interest in the second medical image from the selected recognizer.

2. The processor, classifying the imaging site in the second medical image by the classifier; The medical image processing apparatus according to claim 1 , wherein the recognition of the regions of interest in the second medical image by the recognizer is performed in parallel.

3. The processor, The medical image processing apparatus according to claim 1 , further comprising: a notification unit configured to notify the recognition result.

4. The processor, Creating a display image using the recognition result; 4. The medical image processing apparatus according to claim 3, wherein the display image is displayed on a screen to notify the user.

5. The processor, classifying the imaging site in a third medical image acquired at a later time than the second medical image by the classifier; Recognition of the region of interest in the third medical image by the recognizer based on the imaging part in the second medical image; 5. The medical image processing apparatus according to claim 4, wherein the display images of the recognition results in the second medical images are generated in parallel.

6. The processor, 6. The medical image processing apparatus according to claim 5, wherein the display image is generated by superimposing the recognition result on the third medical image.

7. The processor, The medical image processing apparatus according to claim 3 , further comprising: a notification of the type of the recognizer used to recognize the region of interest in addition to the recognition result.

8. The processor, The medical image processing apparatus according to claim 1 , wherein the classification result is obtained using the first medical image and a plurality of medical images temporally adjacent to the first medical image that have been acquired at a time earlier than the second medical image.

9. The processor, The medical image processing apparatus according to claim 8 , wherein the recognizer is selected in accordance with the classification result and a selection history of a recognizer at a previous time.

10. The processor, weighting the classification results of the plurality of medical images in the temporal vicinity based on the selection history; The medical image processing apparatus of claim 9 , wherein the weighted classification results are used to select the recognizer.

11. The processor, obtaining the classification result in which a confidence level is calculated for each of the imaging parts from the classifier; The medical image processing apparatus according to claim 9 , wherein, if the confidence level is less than a specific reference value, the use of the recognizer at the previous time from the selection history is continued.

12. The medical image processing device according to claim 1 , wherein the plurality of recognizers are deep learning models trained on different data sets.

13. The processor, The medical image processing device according to claim 12 , wherein when the classification result is a first part, the recognizer trained with a dataset including more images captured at the first part than images captured at parts other than the first part is selected.

14. The plurality of recognizers each have a function of recognizing a different one of the attention regions, The medical image processing apparatus according to claim 1 , wherein the selected recognizer acquires the recognition result by performing at least one of position detection, type determination, and area measurement for the region of interest in the medical image.

15. A discriminator that integrates the classifier and a plurality of the recognizers, a classifier that calculates intermediate features in the medical image, classifies the imaging region from the intermediate features, and recognizes the region of interest from the intermediate features and the imaging region; The processor, obtaining a first intermediate feature calculated from the first medical image and a second intermediate feature calculated from the second medical image; obtaining the classification result based on the first intermediate feature; selecting a deep learning model corresponding to the classification result for the first intermediate feature from a plurality of deep learning models; The medical image processing device according to claim 1 , further comprising: acquiring a recognition result of the region of interest in the second intermediate feature from the selected deep learning model.

16. A medical image processing apparatus according to any one of claims 1 to 15, and an endoscope for acquiring the medical image.

Citation Information

Patent Citations

  • Capsule endoscope image processing apparatus and capsule endoscope system

    JP2012249956A

  • Image diagnosis support device and image diagnosis support method

    JP2013123528A

Cited By

  • Endoscopic image diagnosis assistance processor, operation method for endoscopic image diagnosis assistance processor, and program for endoscopic image diagnosis assistance processor

    WO2026083521A1