System, device, and method for providing medical examination information and feedback by utilizing image processing and artificial intelligence technology

Abstract

The present invention relates to a system, a device, and a method for providing medical examination information and feedback by utilizing image processing and artificial intelligence technology. The method may comprise the steps of: acquiring a medical image by photographing a part of the body of a subject to be examined by a medical examination device; extracting, from the medical image, medical information including first information related to usability of the medical examination device and second information related to a standardized examination index; identifying whether the medical information satisfies a specified rule; and providing feedback information notifying that the rule fails to be satisfied when at least a part of the medical information fails to satisfy the rule. The present invention can improve usability (or performance capability) of a medical examination device (e.g., endoscope) and / or a quality of a medical examination (e.g., the accuracy and / or efficiency of diagnosis). In addition, the present invention can improve the accuracy and / or completeness of an interpretation.

Description

Systems, devices, and methods for providing medical examination information and feedback using image processing and artificial intelligence technologies.

[0001] The present invention relates to a system, device and method for providing medical examination information and feedback using image processing and artificial intelligence technologies.

[0002]

[0003] The number of cancer patients has been increasing recently. Cancer is a disease in which cells proliferate abnormally and disrupt normal cell function. Representative cancers include lung cancer, gastric cancer (GC), breast cancer (BRC), and colorectal cancer (CRC), but it can develop in virtually any tissue. Some of these cancers (e.g., gastric cancer, colon cancer, etc.) can be prevented through endoscopic examinations. For example, early detection and removal of polyps (also known as gastric fundic gland polyps or adenomatous polyps) in the stomach or colon can prevent cancer.

[0004] The above endoscopic examination relies almost entirely on the subjective judgment of the examiner (e.g., physician) performing (or performing) the endoscopic examination. Therefore, the diagnosis rate of polyps (or adenomas) can vary significantly depending on the examiner's skill level. For example, the risk of developing polyps into cancer varies depending on their size, so accurately measuring their size is very important. However, the accuracy of size measurement can vary depending on the examiner's skill level and / or the location of the polyp. In other words, the current endoscopic examination has the problem that the diagnosis can vary depending on the examiner's skill and / or experience, and an accurate diagnosis cannot be made if the examiner's skill level is low. Furthermore, even if the captured image is unclear due to light reflection issues that are commonly encountered during endoscopic examinations (e.g., light reflection from the mucus inside the colon), interpretation errors can occur, leading to misdiagnosis.

[0005] Meanwhile, currently, skilled examiners (hereinafter, "experts") observe endoscopic examinations of less skilled examiners (e.g., beginners) and provide training in endoscope operation and observation techniques. The current training method can vary depending on the expert, and training gaps can exist. Furthermore, the current training method is inefficient because the expert cannot always observe. Furthermore, even experienced examiners with excellent observation skills and knowledge may not be able to demonstrate high-level endoscopic observation and operation skills due to workload and personal circumstances. Therefore, a method is needed that provides real-time feedback to examiners on whether the endoscopic examination is being performed appropriately.

[0006] The present invention can provide a system, device and method for providing medical examination information and feedback using image processing and artificial intelligence technologies that can improve the usability of medical examination devices.

[0007] In addition, the present invention can provide a system, device, and method for providing medical examination information and feedback using image processing and artificial intelligence technologies that can improve the accuracy and / or efficiency of diagnosis.

[0008] In addition, the present invention can provide a system, device, and method for providing medical examination information and feedback using image processing and artificial intelligence technologies that can provide standardization and / or quality uniformity of medical examinations.

[0009] A method for providing medical examination information and feedback using image processing and artificial intelligence technology according to one embodiment of the present invention may include the steps of: obtaining a medical image by photographing a part of a body of a subject for examination using a medical examination device; extracting medical information including first information related to the ability to use the medical examination device and second information related to a standardized examination index from the medical image; confirming whether the medical information satisfies a specified regulation; and providing feedback information notifying that the regulation is not satisfied when at least a part of the medical information does not satisfy the regulation.

[0010] According to one embodiment, the method may further include a step of pre-processing the acquired medical image.

[0011] According to one embodiment, the step of extracting the medical information may include a step of extracting the medical information from the medical image through at least one of deep learning-based image processing and non-deep learning-based image processing.

[0012] According to one embodiment, the method may further include a step of providing guide information for guiding the operation of the medical examination device to satisfy the regulation based on the feedback information.

[0013] According to one embodiment, the method may further include the step of providing the medical information via a display.

[0014] According to one embodiment, the method may further include the step of editing the provided medical information in response to a request from the examiner; and the step of generating a reading statement based on the edited medical information.

[0015] According to one embodiment, the step of generating the reading text may include a step of generating the reading text based on a standardized template; or a step of converting the generated reading text based on the standardized template.

[0016] According to one embodiment, the method may further include the step of transmitting the readout to a designated server via a communication module.

[0017] According to one embodiment of the present invention, a device for providing medical examination information and feedback using image processing and artificial intelligence technology may include: a camera module for photographing a body part of a subject for examination to obtain a medical image; a processing module for extracting medical information including first information related to the usability of a medical examination device and second information related to a standardized examination index from the medical image, checking whether the medical information satisfies a specified regulation, and generating feedback information for notifying that at least a portion of the medical information does not satisfy the regulation when at least a portion of the medical information does not satisfy the regulation; and a display for providing the feedback information.

[0018] According to one embodiment, the processing module can extract the medical information from the medical image through at least one of deep learning-based image processing and non-deep learning-based image processing.

[0019] According to one embodiment, the processing module may generate guide information for guiding the operation of the medical examination device so as to satisfy the regulation based on the feedback information, and provide the guide information through the display.

[0020] According to one embodiment, the processing module can generate a reading text based on the medical information.

[0021] According to one embodiment, the processing module can generate the reading text based on a standardized template or convert the generated reading text based on the standardized template.

[0022] According to one embodiment, the device may further include a communication module that transmits the readout to a designated server.

[0023] A system for providing medical examination information and feedback using image processing and artificial intelligence technology according to an embodiment of the present invention may include: a medical examination device that photographs a part of a body of a subject for examination to obtain a medical image, extracts medical information including first information related to the usability of a medical examination device and second information related to a standardized examination index from the medical image, determines whether the medical information satisfies a specified regulation, generates feedback information notifying that at least a part of the medical information does not satisfy the regulation when at least a part of the medical information does not satisfy the regulation, provides the generated feedback information, and generates a reading statement based on the medical information; and a server that receives and stores the reading statement from the medical examination device.

[0024] Various embodiments of the present invention can improve the usability (or performance) of medical examination devices (e.g., endoscopes). For example, the present invention can improve an examiner's ability to use a medical examination device by providing (or providing in real time) feedback information on whether the medical examination device is being used appropriately.

[0025] Additionally, the present invention can improve the quality of medical tests (e.g., diagnostic accuracy and / or efficiency). For example, the present invention can maintain or improve the quality of medical tests by providing (or providing in real time) feedback information on whether a test performed using a medical test device (hereinafter, "medical test") satisfies standardized test indicators (e.g., quality indicators).

[0026] Furthermore, the present invention can improve the accuracy and / or completeness of the medical report. For example, the present invention can extract at least a portion of the information to be included in the medical report from a medical image (or test result) and automatically input the extracted information into the medical report, thereby preventing errors (e.g., omissions, misrepresentations, subjectivity, etc.) by the examiner. As a result, the present invention can improve the accuracy and / or completeness of the medical report. Furthermore, the present invention can standardize the medical report, facilitating communication among medical professionals (e.g., enabling standardized interpretation through recording objective information rather than subjective information).

[0027] Furthermore, the present invention can rapidly analyze and / or interpret medical images and automatically record at least a portion of the medical report. This can save time, manpower, and / or costs for the examiner. Furthermore, the present invention can more accurately assess and monitor the patient's condition, thereby improving the efficiency of patient management.

[0028] FIG. 1 is a diagram illustrating a system that provides medical examination information and feedback using image processing and artificial intelligence technology according to one embodiment of the present invention.

[0029] FIG. 2 is a schematic drawing of a medical examination device of a system according to one embodiment of the present invention.

[0030] FIG. 3 is a flowchart illustrating a method for providing medical examination information and feedback using image processing and artificial intelligence technology of a system according to one embodiment of the present invention.

[0031] FIG. 4a is a diagram illustrating an example of estimating location information of a dragon according to one embodiment of the present invention.

[0032] FIG. 4b is a diagram illustrating an example of detecting a polyp and segmenting a polyp area according to one embodiment of the present invention.

[0033] FIG. 5 is a diagram illustrating an example of quantitative characteristics of a dragon according to one embodiment of the present invention.

[0034] FIG. 6a and FIG. 6b are exemplary diagrams showing an example of a screen providing medical information according to one embodiment of the present invention.

[0035] Figure 7 is a block diagram illustrating the configuration of a medical examination device according to one embodiment of the present invention.

[0036] Figure 8 is a flowchart illustrating an operation method of a medical examination device according to one embodiment of the present invention.

[0037] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the attached drawings. The advantages and features of the present invention, and methods for achieving them, will become clear with reference to the embodiments described in detail below together with the attached drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms. These embodiments are provided only to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention, and the present invention is defined only by the scope of the claims. Hereinafter, like reference numerals refer to like elements.

[0038] Although the terms first, second, etc. are used to describe various elements, components, and / or sections, these elements, components, and / or sections are not limited by these terms. These terms are only used to distinguish one element, component, or section from other elements, components, or sections. Accordingly, it should be understood that a first element, a first component, or a first section referred to below may also be a second element, a second component, or a second section within the technical spirit of the present invention.

[0039] The terminology used herein is for the purpose of describing embodiments only and is not intended to be limiting of the present invention. In this specification, the singular also includes the plural unless the context clearly dictates otherwise. As used herein, the terms "comprises" and / or "made of" do not exclude the presence or addition of one or more other components, steps, operations, and / or elements.

[0040] Unless otherwise defined, all terms (including technical and scientific terms) used herein may be used in their common sense to those of ordinary skill in the art to which the present invention pertains. Furthermore, terms defined in commonly used dictionaries are not to be interpreted ideally or excessively unless explicitly and specifically defined otherwise.

[0041] Hereinafter, the configuration of the present invention will be described in detail with reference to the attached drawings.

[0042]

[0043] FIG. 1 is a drawing illustrating a system that provides medical examination information and feedback using image processing and artificial intelligence technology according to an embodiment of the present invention, and FIG. 2 is a drawing schematically illustrating a medical examination device of the system according to an embodiment of the present invention.

[0044] Referring to FIGS. 1 and 2, a system (1000) for providing information related to a medical examination according to one embodiment of the present invention may include a medical examination device (100) and a server (200).

[0045] The medical examination device (100) can provide medical examinations on the inside of the body (e.g., inside the stomach, inside the large intestine, etc.) of an examination target (e.g., a human or an animal) (hereinafter, a subject). For example, the medical examination device (100) can be an endoscope, as illustrated in FIG. 2. The medical examination device (100) can include a camera module (110) for photographing the inside of the body of the subject, an operation module (120) for adjusting the camera module (110), a processing module (130) for processing medical images (e.g., lesion detection, segmentation, diagnosis, etc.) and generating medical information and a reading based on the processing results, and a display (140) for providing medical information. A detailed description of each component (110, 120, 130, 140) of the medical examination device (100) will be described later with reference to FIG. 7.

[0046] The above medical examination device (100) can capture the inside of the subject's body to obtain a medical image, analyze (or diagnose) the medical image (e.g., detect a lesion (e.g., polyp, adenoma, etc.), determine the status of the lesion (e.g., negative, benign, or malignant)), and provide the analysis result (hereinafter, test result).

[0047] According to one embodiment, the medical examination device (100) can provide feedback information on whether an examiner (e.g., a doctor) is using the medical examination device (100) appropriately. In addition, the medical examination device (100) can provide feedback information on whether the examination result satisfies (or meets) standardized examination indicators (e.g., quality indicators). In addition, the medical examination device (100) can extract at least a portion of the information required for the reading report from the examination result and automatically generate the reading report based on the extracted information. Through this, the present invention can prevent the problem of an inexperienced person who is not skilled in using the medical examination device (100) omitting or incorrectly entering information that should be recorded in the reading report. In addition, the present invention can improve the objectivity of the reading report by automatically recording at least a portion of the reading report, thereby eliminating subjective judgment as much as possible.

[0048] The server (200) can store various medical information obtained through the medical examination device (100). For example, the server (200) can receive and store a medical report from the medical examination device (100). The server (200) can store medical information (e.g., a medical report) in a database (DB). For example, the server (200) can store a medical report based on a quantified template. Through this, the system (100) of the present invention can standardize various types of medical reports, improve objectivity, and facilitate communication among medical professionals.

[0049] Meanwhile, a detailed description of the operation of the above system (1000) will be described later with reference to FIGS. 3 to 8.

[0050]

[0051] FIG. 3 is a flowchart illustrating a method for providing medical examination information and feedback using image processing and artificial intelligence technology of a system according to an embodiment of the present invention, FIG. 4a is a diagram illustrating an example of estimating location information of a polyp according to an embodiment of the present invention, FIG. 4b is a diagram illustrating an example of detecting a polyp and segmenting a polyp area according to an embodiment of the present invention, FIG. 5 is a diagram illustrating an example of quantitative characteristics of a polyp according to an embodiment of the present invention, and FIGS. 6a and 6b are exemplary diagrams illustrating an example of a screen providing medical information according to an embodiment of the present invention.

[0052] Referring to FIGS. 3 to 6B, a method for providing information related to a medical examination of a system according to an embodiment of the present invention may include a step (S301) of acquiring a medical image. For example, the medical examination device (100) may acquire a medical image by photographing the inside of the subject's body through a camera module (110). The medical examination device (100) may acquire the medical image based on a specified frame rate (e.g., 30 FPS (frames per second)).

[0053] The above method may include a step (S303) of preprocessing a medical image. For example, the medical examination device (100) may preprocess a medical image to make it suitable for image analysis. The preprocessing may include sampling, which converts the frame rate of the medical image to a specified frame rate (e.g., 15 FPS). In addition, the preprocessing may include image processing such as cropping, which cuts each frame into a specified size (e.g., 512 pixels * 512 pixels), resizing, which converts it into a specified size, and intensity normalization.

[0054] The above method may include a step (S305) of extracting medical information. For example, the medical examination device (100) may analyze a pre-processed medical image to extract medical information. The medical information may be extracted through image processing based on artificial intelligence (e.g., deep learning) and image processing based on non-artificial intelligence. The image processing based on artificial intelligence (e.g., deep learning) may train an artificial neural network and use the trained artificial neural network to detect lesions (e.g., polyps, adenomas, etc.) in the medical image, segment the lesion area, and / or diagnose the lesion (e.g., benign, negative, malignant, etc.), thereby extracting medical information from the medical image. The artificial neural network may include, but is not limited to, a convolution neural network (CNN), a vision transformer (ViT), etc. Meanwhile, the non-AI-based image processing can extract meaningful medical information from medical images through various image processing techniques (e.g., image enhancement, image restoration, image analysis, image compression, image synthesis, etc.). For example, the meaningful medical information obtained through the non-AI-based image processing can include characteristics related to texture information such as compactness, skewness, and coarseness.

[0055] According to one embodiment, the present invention can extract medical information using at least one of artificial intelligence-based image processing and non-AI-based image processing. The artificial intelligence-based image processing can learn complex patterns and relationships and increase accuracy, but requires a large amount of data and computational power. On the other hand, the non-AI-based image processing can be performed with relatively little data and has a fast processing speed, but may have relatively low accuracy. The present invention can extract medical information from medical images using at least one of artificial intelligence-based image processing and non-AI-based image processing depending on the characteristics of the medical image. As a result, the present invention can analyze (or interpret) medical images more effectively (or efficiently).

[0056] Meanwhile, the medical information may include first information related to the ability to use a medical examination device and second information related to a standardized examination index (e.g., quality index) for evaluating the quality of a medical examination. The first information is information for evaluating the appropriateness of the examiner's technique for observing (or examining) the internal organs (or tissues) of a subject using an endoscope. For example, a colonoscopy involves inserting an endoscope through the anus to the cecum, inflating the colon through appropriate air injection to minimize wrinkles, and then retracting the endoscope from the cecum to the anus to observe each structure and surface of the colon. At this time, a good examiner can observe each structure and surface of the colon at an appropriate speed while looking at the screen of the colonoscope, and can confirm the presence or absence of polyps (or adenomas) by washing foreign substances from the intestinal surface. An excellent examiner with a high polyp detection rate can manipulate the endoscope so that all quadrants of the cylindrical endoscope structure are included in the screen without missing any of them, and can move the endoscope delicately and at an appropriate speed to implement a clear image quality without blurring the frames of the image on the screen. In addition, an excellent examiner can learn various types of polyp recognition techniques through experience and learning, and can find polyps that are included in the screen without missing any of them. The medical examination device (100) can evaluate the examiner's ability to use the medical examination device (100) by checking whether all quadrants of the endoscopic structure are included in the screen, whether the movement speed of the endoscope is appropriate, or whether the image quality of the medical image is appropriate. The medical examination device (100) can provide the examiner with feedback information on the use of the medical examination device (100) based on the evaluation results.

[0057] Meanwhile, the second information is information for assessing compliance (or satisfaction) with standardized examination indicators (e.g., quality indicators) to compensate for qualitative differences in endoscopic examinations due to the examiner's skill level. For example, the second information may include polyp characteristics, insertion location (e.g., cecal insertion rate), examination time (or observation time), bowel cleansing, and follow-up monitoring intervals.

[0058] The above-mentioned adenoma characteristics may include, but are not limited to, the number of adenomas, the adenoma detection rate (ADR), and the advanced adenoma detection rate (AADR). For example, the medical examination device (100) may detect adenomas using an artificial intelligence model (e.g., a CNN-based detection model) and calculate the total number of adenomas based on the number of times adenomas are detected during an endoscopic examination. Meanwhile, the medical examination device (100) according to an embodiment of the present invention may prevent the same adenoma from being detected more than twice. In detail, the present invention trains a detection model (e.g., an auto-encoder) after applying the same line type to input data and output data, estimates unique features of each line type based on the trained detection model, and processes the same line type when the similarity (e.g., cosine similarity) of the estimated unique features is greater than a specified reference value (e.g., 0.85), thereby preventing the number of detected line types from being over-counted (e.g., the same line type being counted repeatedly).

[0059] Meanwhile, the medical examination device (100) can calculate the adenoma detection rate (ADR) by dividing the number of adenomas found in the entire examination by the total number of examinations. Similarly, the medical examination device (100) can calculate the high-risk adenoma detection rate (AADR) by dividing the number of high-risk adenomas found in the entire examination (e.g., adenomas larger than a specified size (e.g., 1 cm), villous adenoma, or high-grade dysplasia adenoma, etc.) by the total number of examinations.

[0060] The medical examination device (100) may provide feedback information based on respective set criteria for the number of adenomas, the adenoma detection rate, and the high-risk adenoma detection rate. For example, the medical examination device (100) may determine that the test indicator (e.g., quality indicator) is not satisfied and provide feedback information if at least one of the number of adenomas, the adenoma detection rate, and the high-risk adenoma detection rate is lower than the respective set criteria. The respective criteria for the number of adenomas, the adenoma detection rate, and the high-risk adenoma detection rate may vary based on the patient's age, health status, medical history, etc.

[0061] Additionally, the medical examination device (100) can recognize the location of a tumor (or polyp) through a pre-learned artificial intelligence model (e.g., a location estimation model). The above location estimation model may include a zone estimation model (420) that estimates a zone in which a polyp is located among a plurality of segmented zones, and a pose estimation model (430) that estimates the pose of an endoscope and / or a polyp (e.g., 6 degrees of freedom (6DOF) of an endoscope) when photographing a polyp. For example, the zone estimation model (420) and the pose estimation model (430) may be learned based on learning data (410) obtained from each zone by dividing the colon into a specified number (e.g., 3) of zones (Zone 1, Zone 2, Zone 3), as illustrated in FIG. 4A. The learning data (410) may include a medical image photographing a polyp (hereinafter, “adenoma image”) and labeling information. The labeling information may include location information of a zone in which a polyp image is photographed and pose information of an endoscope and / or a polyp when photographing the polyp image. For example, the pose information may be obtained by a camera included in an endoscope device. It may include 6-degree-of-freedom information for identifying the angle and / or position of the module (110). In addition, the posture information may further include information (e.g., 3-dimensional information) for supplementing the 2-dimensional information acquired from the 2-dimensional medical image. For example, the posture information may further include 6-degree-of-freedom information of the polyp estimated based on the angle and / or position of the camera module (110), depth information of the polyp, and rotation information of the polyp.

[0062] The above insertion position (e.g., cecal intubation rate) is one of the key pieces of information that must be provided in the reading report. The medical examination device (100) can estimate the position of the endoscope through the position estimation model of FIG. 4A and automatically record the final insertion position (e.g., the position just before the endoscope is withdrawn) in the reading report. The medical examination device (100) can divide the number of examinations in which the endoscope reached the cecum by the total number of examinations and express the cecal intubation rate as a percentage. Specifically, the medical examination device (100) can check whether the cecum (or appendiceal opening, ileocecal valve) is recognized from the medical image, and if the cecum is not recognized, it can be determined that the examination index is not satisfied and provide feedback information. For example, the medical examination device (100) can change the color of the screen to a specified color (e.g., red).

[0063] The examination time (or observation time) refers to the time from the start of the examination to its completion. For example, the medical examination device (100) can calculate the examination time by comparing the time when the examination starts and the time when the examination ends, and record the calculated examination time in the reading report. If the calculated examination time is shorter than the reference time, the medical examination device (100) can determine that the examination indicator (e.g., quality indicator) is not satisfied and provide feedback information.

[0064] Meanwhile, if only the endoscope is removed from the subject without a command to terminate the examination (e.g., without turning off the camera module), the examination time may be mismeasured (e.g., over-estimated). To prevent this, the medical examination device (100) may determine that the examination is terminated when the endoscope is exposed to the outside of the body. Specifically, the medical examination device (100) may extract a feature map for the medical image, compare the extracted feature maps, and determine that the endoscope has come out of the body from inside the body if the similarity difference exceeds a specified standard value, thereby terminating the examination.

[0065] Bowel cleanliness refers to the cleanliness of the colonic mucosa. Bowel cleanliness can be classified into three or five levels according to guidelines. The medical examination device (100) may include a classification model (e.g., a multi-class classification model, a regression-based CNN model, etc.) that analyzes medical images to classify bowel cleanliness. If the bowel cleanliness is lower than a reference level, the medical examination device (100) may determine that the test indicator (e.g., a quality indicator) is not satisfied and provide feedback information.

[0066] The follow-up monitoring interval refers to the period for monitoring the condition of the examinee. The examiner can set the follow-up monitoring interval based on the examination results. For example, the examiner can set a long follow-up monitoring interval if the examination results are good, and can set a short follow-up monitoring interval if the examination results are poor. The medical examination device (100) can recommend a follow-up monitoring interval based on the analysis results and / or interpretation results of the medical image. For example, the medical examination device (100) can recommend a follow-up monitoring interval based on information (e.g., a database) about follow-up monitoring intervals previously provided in similar cases. Meanwhile, the medical examination device (100) can determine that the follow-up monitoring interval set by the examiner is smaller or larger than the recommended follow-up monitoring interval by a specified amount or more and provide feedback information, thereby failing to satisfy the examination index (e.g., quality index). The examiner can finally determine the follow-up monitoring interval for the examinee with reference to the recommended follow-up monitoring interval. Through this, the present invention can prevent unnecessary medical expenses and examination risks by performing examinations at short intervals, and can prevent the problem of not detecting cancer at an early stage by performing examinations at too long intervals.

[0067] Meanwhile, the second information may further include quantitative characteristics of the polyp. The quantitative characteristics of the polyp may include, but are not limited to, size, shape, texture, and color.

[0068] The medical examination device (100), as illustrated in FIG. 4B, can detect (451, 453) a polyp from an original medical image (441, 443) using a detection model, segment (461, 463) a polyp area using a segmentation model, and recognize the size of the polyp based on the segmented polyp area. Specifically, the medical examination device (100) includes a distance sensor module (e.g., a depth camera or a laser sensor) for measuring a distance, and can calculate the distance between the camera and the polyp through the distance sensor module. Thereafter, the medical examination device (100) can calculate and quantify the size of the polyp based on the number of pixels of the segmented polyp area and the calculated distance.

[0069] In addition, the medical examination device (100) can identify and quantify the shape (e.g., circular, oval, etc.) of the segmented polyp area through morphological calculation of the segmentation map. Meanwhile, the shape of the polyp may include a region and a contour, as illustrated in FIG. 5.

[0070] Additionally, the medical examination device (100) can identify and quantify surface characteristics (e.g., roughness, presence of repetitive patterns) based on radiomics features of the segmented polyp area. Meanwhile, the surface characteristics of the polyp may include spatial and spectral characteristics, as illustrated in FIG. 5 .

[0071] Additionally, the medical examination device (100) can identify and quantify color information based on characteristic information related to the color of the polyp. Meanwhile, the color of the polyp may include a color histogram, a color coherence vector, color moments, and a color correlogram, as illustrated in FIG. 5.

[0072] The above method may include a step (S307) of providing medical information. For example, the medical examination device (100) may provide (e.g., display) the extracted medical information through a display (140). Specifically, as illustrated in FIG. 6A, the medical examination device (100) may provide medical images (601), analysis information (602), quarterly indicator information (603), recent / annual average information (604) of major indicators (e.g., polyp detection rate (PDR), adenoma detection rate (ADR)) among the quarterly indicator information (603), and detailed information (605) on the polyp detection rate (PDR) through the display (140). The analysis information (602) may include examination results (e.g., detected location (zone) information and / or the probability of polyp (HP) or adenoma (AD). Quarterly indicator information (603) may include evaluation scores and reference scores for polyp detection rate (PDR), adenoma detection rate (ADR), small polyp detection rate (Cecal Small PDR: CSPDR), time required for examination (WT), cancer detection rate (CDR), and bowel preparation quality (BBPS). The examiner can easily check the level of his or her endoscopic examination (e.g., similarity to an examination by an experienced examiner) by comparing the evaluation scores and reference scores.

[0073] Meanwhile, the medical examination device (700) can provide various calculated values ​​that are helpful for medical information to the examiner during the endoscopic examination through the display (140). For example, as illustrated in FIG. 6b, the medical examination device (700) can provide the examiner with a medical image (611), a reference image (612), and other information (613) acquired in real time during the endoscopic examination through the display (140). The reference image (612) can include at least one corrected image (612a) that is a correction (e.g., inpainting, heatmap) of a polyp image detected from the medical image (611), and an image (612b) (and a diagnosis result) of at least one polyp that is most similar to the polyp detected from the medical image (611) among previously stored polyps. The other information (613) can include texture characteristic information, blood vessel characteristic information, color characteristic information, and morphological characteristic information. The above figures 6a and 6b are only examples and do not limit the present invention.

[0074] The above method may include a step (S309) of editing medical information. For example, the medical examination device (100) may edit medical information according to a request from the examiner obtained through the input module. In other words, the examiner may review the medical information provided on the display (140) and, based on their own opinions, correct incorrect medical information, add missing medical information, or delete unnecessary medical information.

[0075] The method may include a step (S311) of generating a reading statement. For example, the medical examination device (100) may generate a reading statement based on at least a portion of the extracted medical information. In one embodiment, the medical examination device (100) may generate a reading statement based on a standardized template.

[0076] The above method may include a step (S313) of transmitting a reading text. For example, the medical examination device (100) may transmit the reading text to a designated server (200) via a communication module (not shown).

[0077] The above method may include a step (S315) of storing the read text. For example, the server (200) may store the received read text. The server (200) may store the read text in a database. According to some embodiments, the server (200) may check the format of the read text, and if it differs from a specified format, convert the read text to correspond to a specified template and store it. In addition, the server (200) may convert non-standard terms recorded in the read text to standard terms, correct typos, and then store the read text.

[0078] The present invention described above can automatically provide information related to standardized examination indicators (e.g., quality indicators) to compensate for qualitative differences in endoscopic examinations between examiners in real time. In other words, the present invention can automatically provide real-time information on whether the examination quality indicators are met based on medical images. In contrast, the conventional method of writing an endoscopic report involves the examiner evaluating the endoscopic performance process and expressing the diagnostic results in writing. This conventional method may lack objectivity and / or accuracy due to its dependence on the examiner's subjective judgment and experience. Furthermore, while the conventional method aims to include all information identified in the medical image in the report, there are limitations in describing all information in reality. Consequently, the conventional method may only include key information or omit important information.

[0079]

[0080] Figure 7 is a block diagram illustrating the configuration of a medical examination device according to one embodiment of the present invention.

[0081] Referring to FIG. 7, a medical examination device (700) according to an embodiment of the present invention may include a camera module (710), an operation module (720), a processing module (730), a display (740), a communication module (750), and an input module (760). The medical examination device (700) may be the medical examination device (100) of FIG. 1. The camera module (710), the operation module (720), the processing module (730), and the display (740) of the medical examination device (700) may correspond to the camera module (110), the operation module (120), the processing module (130), and the display (140) of the medical examination device (100) of FIG. 1.

[0082] The camera module (710) can capture images of the inside of the subject's body. For example, the camera module (710) can be inserted into the body of the subject (e.g., the large intestine) to capture images of the inside of the body. The camera module (710) can include a light source (711), an image sensor (712), and a driving module (713). The light source (711) can be turned on by the operation module (720) or the processing module (730) to obtain a clear medical image of the inside of the body. The light source (711) can include a light-emitting diode including a light-emitting diode (LED), an organic light-emitting diode (OLED), or a laser diode, or a filament lamp. The image sensor (712) may include a CMOS (Complementary Metal Oxide Semiconductor) or CCD (Charge-Coupled Device) image sensor. The driving module (713) may adjust the camera module (710) in response to an operation signal of the operation module (720). For example, the driving module (713) may change the direction and / or angle of the camera module (710) or rotate it in response to an operation signal of the operation module (720).

[0083] The manipulation module (720) is located at the rear end of the camera module (710) and can receive manipulation commands from the examiner. The manipulation module (720) is a part that is held by the examiner and can manipulate the direction, angle, and / or rotation of the camera module (710) inserted into the body of the examinee. The manipulation module (720) may include a number of buttons for manipulating the photographing module (710).

[0084] The processing module (730) can control the overall operation of the medical examination device (700). The processing module (730) can receive commands or instructions from a memory (not shown) and control each component according to the received commands or instructions to perform various functions. The processing module (730) can be formed of a central processing unit (CPU), a micro control unit (MCU), a micro processor unit (MPU), or the like.

[0085] According to one embodiment, the processing module (730) may include an image processing module (731), a feedback provision module (732), and a reading text generation module (733).

[0086] The image processing module (731) can extract medical information from a medical image, including first information related to the usability of the medical examination device (700) and second information related to standardized examination indicators. According to one embodiment, the image processing module (731) can extract medical information from the medical image through at least one of deep learning-based image processing and non-deep learning-based image processing. Meanwhile, the image processing module (731) can perform pre-processing on the medical image.

[0087] The feedback provision module (732) may generate and provide feedback information regarding whether a medical examination via the medical examination device (700) is being performed appropriately. For example, the feedback provision module (732) may verify whether the extracted medical information satisfies specified regulations, and generate and provide feedback information notifying the examiner if at least some of the extracted medical information does not satisfy the regulations. In addition, the feedback provision module (732) may generate and provide guide information that guides the operation of the medical examination device (700) so that the regulations can be satisfied based on the feedback information.

[0088] The reading text generation module (733) can generate a reading text based on medical information. For example, the reading text generation module (733) can extract information to be recorded in the reading text from the medical information and automatically record the extracted information in the reading text. Meanwhile, the reading text generation module (733) can generate a reading text based on a standardized template. Alternatively, the reading text generation module (733) can convert a previously generated reading text (e.g., a non-standardized reading text) based on the standardized template.

[0089] The display (740) can provide (e.g., display) various medical information acquired through the medical examination device (700). According to one embodiment, the display (740) can provide feedback information (e.g., outputting a warning message) indicating that the medical information does not meet specified requirements. In addition, the display (740) can provide guidance information that guides the operation of the medical examination device so that the medical information can meet specified requirements.

[0090] The communication module (750) can perform wired or wireless communication with a designated external device (e.g., server (200). For example, the communication module (750) can transmit a read text generated by the read text generation module (733) to the external device.

[0091] The input module (760) may include a plurality of input devices (e.g., buttons, dials, keyboards, mice, etc.) for operating the medical examination device (700). In one embodiment, the examiner may edit medical information and / or a diagnosis text through the input module (760).

[0092] Meanwhile, the medical examination device (700) may further include, although not shown in FIG. 7, an air supply module that supplies air to a part inserted into the body of the subject (e.g., the area around the camera module (710)), a water supply unit that supplies water, a biopsy module that removes (or collects) a portion of tissue, and a suction module that sucks in air or foreign substances.

[0093]

[0094] Figure 8 is a flowchart illustrating an operation method of a medical examination device according to one embodiment of the present invention.

[0095] Referring to FIG. 8, the operating method of a medical examination device according to an embodiment of the present invention may include a step (S801) of acquiring a medical image. For example, the medical examination device (700) may acquire a medical image by photographing the inside of the subject's body through a camera module (710). In addition, although not illustrated, the method may further include a step of pre-processing the medical image to make it suitable for image analysis.

[0096] The method may include a step (S803) of extracting medical information from a medical image. For example, the medical examination device (700) may extract medical information from a medical image (or a pre-processed medical image). The medical information may be extracted through image processing based on artificial intelligence (e.g., deep learning) and image processing based on non-artificial intelligence, as described in FIG. 3. The medical information may include first information related to the ability to use the medical examination device and second information related to a standardized examination index (e.g., a quality index) for evaluating the quality of a medical examination. The first information may include, but is not limited to, information for evaluating the appropriateness of the examiner's technique, and may include information on whether each quadrant of the endoscopic structure is fully included in the screen, information on the movement speed of the endoscope, and / or information on the image quality of the medical image. The second information is information for evaluating compliance (or satisfaction) with the standardized examination index (e.g., a quality index) for compensating for qualitative differences in endoscopic examinations depending on the examiner's skill level. For example, the second information may include polyp characteristics, insertion location (e.g., cecal insertion rate), examination time (or observation time), bowel cleanliness, and follow-up monitoring intervals. The second information may further include quantitative characteristics of the polyp.

[0097] The above method may include a step (S805) of verifying whether the medical information satisfies the specified regulations. For example, the medical examination device (700) may verify whether the first information and the second information satisfies the specified regulations. Specifically, the medical examination device (700) may verify whether each quadrant of the endoscopic structure is fully included in the screen, whether the movement speed of the endoscope is appropriate (e.g., whether the movement speed of the endoscope is within a specified speed range), or whether the image quality of the medical image is appropriate (e.g., whether the image quality of the medical image is above a specified sharpness).

[0098] In addition, the medical examination device (700) can check whether the polyp characteristics (e.g., number of polyps, polyp detection rate, high-risk polyp detection rate) meet the criteria. As another example, the medical examination device (700) can check whether the shape of the cecum, the appendix opening, and the ileocecal valve are recognized. As another example, the medical examination device (700) can check whether the observation time (e.g., the time for observing the colonic mucosa purely excluding the polyp removal time) is less than a specified criteria time. As another example, the medical examination device (700) can check whether the cleanliness of the colonic mucosa is less than a criteria. As another example, the medical examination device (700) can check whether the follow-up monitoring interval is greater than or less than a specified size than the recommended follow-up monitoring interval.

[0099] If the medical information does not satisfy the specified regulations as a result of the verification in step S805, the method may proceed to step S807 of providing feedback information. For example, the medical examination device (700) may provide feedback information that visually, tactilely, and / or audibly notifies that the medical information does not satisfy the specified regulations. The medical examination device (700) may further provide guide information on how to operate the medical examination device (700) (e.g., the camera module (710)) so that the first information or the second information satisfies the regulations. For example, the medical examination device (700) may provide guide information (e.g., a message) requesting to reduce the movement speed, guide information requesting to insert more of the endoscope, etc.

[0100] The above method may include a step (S809) of adjusting the medical examination device. For example, the medical examination device (700) may adjust the position, posture, direction, and / or movement speed of the camera module (710) in response to an operator's operation command via the operation module (720).

[0101] Meanwhile, if the medical information satisfies the specified regulations as a result of the verification in step S805, the method may proceed to step S811 of providing medical information. For example, the medical examination device (700) may provide medical information on the display (140), as illustrated in FIG. 6A. Meanwhile, the medical examination device (700) may provide the examiner with various calculated values ​​helpful for providing medical information during an endoscopic examination via the display (140), as illustrated in FIG. 6B.

[0102] The above method may include a step (S813) of determining whether editing of medical information is requested. For example, the medical examination device (700) may determine whether the examiner requests modification, deletion, or addition of medical information.

[0103] If, as a result of the verification in step S813, editing the medical information is requested, the method may proceed to step S815 of editing the medical information. For example, the medical examination device (700) may modify, delete, or add medical information in response to a request from the examiner.

[0104] Meanwhile, if the verification result of step S813 indicates that editing of the medical information is not requested, the method may proceed to step S817 for determining whether the examination is terminated. For example, the medical examination device (700) may determine that the examination is terminated if the end (e.g., rectum or anus) of the examination tissue (e.g., colon) is detected, or if the endoscope is detected to be protruding from the body. Alternatively, the medical examination device (700) may determine that the examination is terminated if a designated button (e.g., end button) is pressed.

[0105] If the test is not completed as a result of the verification in step S817, the method may return to step S801 and repeat the above-described process. On the other hand, if the test is completed as a result of the verification in step S817, the method may proceed to the step of generating a reading statement (S819). For example, the medical examination device (700) may extract information to be recorded in the reading statement from the extracted medical information (or test results) and generate the reading statement based on the extracted information.

[0106] Meanwhile, although not shown, the method may further include a step of transmitting the reading to a designated server. For example, the medical examination device (700) may transmit the reading to a designated server via the communication module (737).

[0107] Meanwhile, the above description exemplifies a colonoscope as the medical examination device. However, the present invention can also be applied to other endoscopes (e.g., gastroscopes, laryngoscopes, bronchoscopes, earscopes, laparoscopes, cystoscopes, etc.).

[0108]

[0109] While the present invention has been described with reference to the illustrated embodiments as described above, these are merely exemplary, and those skilled in the art to which the present invention pertains will readily appreciate that various modifications, changes, and equivalent variations of the present invention are possible without departing from the spirit and scope of the invention. Therefore, the true technical protection scope of the present invention should be determined by the technical spirit of the appended claims.

Claims

1. A method for providing medical examination information and feedback using image processing and artificial intelligence technology. A step of obtaining a medical image by taking a picture of a body part of a subject for examination using a medical examination device; A step of extracting medical information including first information related to the usability of the medical examination device and second information related to standardized examination indicators from the medical image; A step of checking whether the above medical information satisfies the specified regulations; and A method comprising the step of providing feedback information notifying that at least some of the medical information does not satisfy the above provisions.

2. In paragraph 1, A method further comprising a step of pre-processing the acquired medical image.

3. In paragraph 1, The steps for extracting the above medical information are A method comprising a step of extracting medical information from the medical image through at least one of deep learning-based image processing and non-deep learning-based image processing.

4. In paragraph 1, A method further comprising the step of providing guide information for guiding the operation of the medical examination device to satisfy the regulation based on the feedback information.

5. In paragraph 1, A method further comprising the step of providing said medical information through a display.

6. In paragraph 5, A step of editing the provided medical information in response to the examiner's request; and A method further comprising a step of generating a reading statement based on the above-mentioned edited medical information.

7. In paragraph 6, The steps for generating the above reading text are a step of generating the above-mentioned reading text based on a standardized template; or A method comprising a step of converting the generated reading text based on the standardized template.

8. In paragraph 6, A method further comprising the step of transmitting the reading text to a designated server via a communication module.

9. In a device that provides medical examination information and feedback using image processing and artificial intelligence technology, A camera module that acquires medical images by photographing parts of the body of the subject; A processing module for extracting medical information including first information related to the usability of a medical examination device and second information related to a standardized examination index from the medical image, checking whether the medical information satisfies a specified regulation, and generating feedback information notifying that at least a portion of the medical information does not satisfy the regulation if at least a portion of the medical information does not satisfy the regulation; and A device comprising a display providing the above feedback information.

10. In paragraph 9, The above processing module A device for extracting medical information from a medical image through at least one of deep learning-based image processing and non-deep learning-based image processing.

11. In paragraph 9, The above processing module A device that generates guide information for guiding the operation of the medical examination device so as to satisfy the above regulations based on the above feedback information and provides the information through the display.

12. In paragraph 9, The above processing module A device that generates a reading statement based on the above medical information.

13. In paragraph 12, The above processing module A device for generating the above-mentioned reading text based on a standardized template or converting the generated reading text based on the standardized template.

14. In paragraph 12, A device further comprising a communication module for transmitting the above-mentioned reading text to a designated server.

15. In a system that provides medical examination information and feedback using image processing and artificial intelligence technology, A medical examination device that acquires a medical image by photographing a part of a body of a subject for examination, extracts medical information including first information related to the ability to use a medical examination device and second information related to a standardized examination index from the medical image, verifies whether the medical information satisfies a specified regulation, and generates feedback information notifying that at least a part of the medical information does not satisfy the regulation if at least a part of the medical information does not satisfy the regulation, provides the generated feedback information, and generates a reading statement based on the medical information; and A system including a server that receives and stores the reading from the medical examination device.

Images