Diagnostic assistance system and report writing of some medical findings with artificial intelligence

Abstract

This invention introduces a program within the fields of radiology and pathology, leveraging artificial intelligence as a user-friendly diagnostic assistant. Programmed primarily in Python, users can easily interact with the system, choosing among four distinct modules: chest X-ray, liver, Pap smear, and bone fracture diagnostics. Upon uploading the relevant image and submitting it, YOLO (you only look once) version. 8 efficiently detects and analyzes the input images. The integration includes prompt engineering after receiving the required information of the patient and doctor, then it establishes a connection with OpenAI APIs for seamless information exchange and present it to the users. Then there is this chance of allowing users to pose additional questions before receiving the diagnosis report in a printable format which is a Microsoft word file. The patent's main purpose is to elevate accuracy levels in medical diagnosis, real-time imaging responses, and accessibility for users without specialized knowledge.

Description

Diagnostic Assistance System and Report Writing of Some Medical Findings with Artificial Intelligence

[0001] Radiology & pathology are crucial fields in medicine and often rely heavily on manual diagnosis, which can be subjective. This design aims to introduce a consistent, AI-driven approach to improve diagnostic accuracy and develop an AI system capable of analyzing medical images for pathology diagnosis, generating comprehensive reports, and serving as a reliable pathologist consultation tool.

[0002] Technologies and Models Used in this patent are Python for programming, OpenAI for AI-driven analysis and report generation, YOLO v8 for object detection, and fine-tuned to detect specific classes of radiology and pathology, and also System Architecture which is a modular system integrating advanced object detection with AI-driven data synthesis for generating diagnostic reports.

[0003] This innovation has the potential to revolutionize diagnostic accuracy and efficiency across various fields of medicine, leading to better patient outcomes and supporting pathologists in decision-making. It is ready to cover a broader spectrum of diagnostic medicine and establish a universal AI-assisted diagnostic tool.

[0004] This method can be extended not only to all branches of pathology and radiology but also to all fields of diagnostic medicine. It just needs more datasets.

[0005] G05D 101 / 10 - G16H 30 / 40

[0006] After searching through WIPO Patent scope, Free Patent, and Google Patent, we did not find any exact matches for our invention - the Diagnostic Assistance System and Report Writing of Some Medical Findings with Artificial Intelligence. This is because our design employs the YOLO (You Only Look Once) detection, which was released in 2020 and we used the 2023 version (version 8) in our invention. In addition, we incorporated the OpenAI 3.5 model of Chat GPT, which was introduced in 2022. Although we did not find any identical matches, we did come across some relatively similar items:

[0007] US20210397888

[0008] Diagnostic assistance system and method therefor

[0009] The present invention makes an efficient diagnosis of an object. A general server 3 of a diagnostic assistance system 1 provides data (a large volume of photographic images) concerning an object managed by each client, to said client and a plurality of analysts and experts, and allows such diagnosers to diagnose the object and enables sharing of diagnosis results among the diagnosers. The plurality of analysts each partially contribute in viewing the large volume of photographic images so as to find an abnormal site in the object. The client and the experts then conduct a more detailed diagnosis on the abnormal site discovered by the analysts. An AI server 5 of the diagnostic assistance system 1 creates training data from the diagnosis results provided by the plurality of diagnosers, performs machine learning on the diagnosis results, and carries out automated diagnosis using the learned method.

[0010] While there are similarities between the mentioned patent and our project, there are also notable differences. The mentioned patent seems to focus on a diagnostic assistance system that efficiently diagnoses an object by involving a general server, analysts, experts, and an AI server. On the other hand, our project focuses on medical imaging analysis, involving Python programming, YOLO-based object detection models, a chatbot, and OpenAI APIs. Its main goal is to provide a user-friendly diagnostic assistant for medical conditions such as chest X-rays, liver diseases, Pap smears, and bone fractures. In our project, the emphasis is on an integrated system with a unified framework that seamlessly combines various medical imaging processes. The mentioned patent also involves an AI server creating training data from diagnosis results and performing machine learning for automated diagnosis, our project primarily leverages pre-trained object detection models like YOLO for real-time object detection, with the integration of a chatbot and OpenAI APIs for information exchange.

[0011] US20240046468

[0012] COMPUTER-ASSISTED MEDICAL DIAGNOSIS SYSTEM AND METHOD

[0013] A computer-implemented diagnostic-assistance system for medical applications, comprises: an artificial intelligence neural network configured to classify images of an obtained image dataset according to a set of classes; a confidence module configured to generate a confidence measure associated with each of the classified images; a tagging module configured to generate, for the patient, a diagnostic signal based on the generated confidence measures associated with the classified images, wherein the diagnostic signal for the patient is tagged as conclusive if a processed combination of the confidence measures fulfills a condition and tagged as inconclusive if the processed combination of the confidence measures does not fulfill the condition; and an output interface configured to output the diagnostic signal, wherein if the diagnostic signal is conclusive the classification result is released, and if the diagnostic signal is inconclusive an additional diagnostic analysis is triggered.

[0014] The mentioned patent focuses on a computer-implemented diagnostic assistance system specifically designed for medical applications while our project is a broader diagnostic assistant system that encompasses medical imaging analysis, including areas such as chest X-rays, liver diseases, Pap smears, and bone fractures. The mentioned patent utilizes an artificial intelligence neural network to classify images in a dataset based on a set of classes but in our project, the YOLO (You Only Look Once) version 8 is employed for object detection in medical images, which may include identifying abnormalities in Pap smears, evaluating liver health, detecting bone fractures, and identifying chest X-ray abnormalities. The mentioned patent abstract introduces a diagnostic signal for the patient, tagged as conclusive or inconclusive based on the processed combination of confidence measures. On the other hand, our project includes a comprehensive diagnostic report generated after the completion of the diagnostic analysis. Users can provide feedback, contributing to system improvement and user satisfaction.

[0015] CN116092645

[0016] Medical assistance AI intelligent management system and method based on big data

[0017] The invention relates to the technical field of big data, in particular to a medical auxiliary AI intelligent management system and method based on big data, and the system comprises an auxiliary information collection module, a database, an image annotation analysis module, a training sample screening module, and a medical image AI annotation module. Medical image labeling information and training sample information used for labeling are collected through an auxiliary information collection module, all collected data are stored through a database, the accuracy rate of medical image labeling is analyzed through an image labeling analysis module, whether screening processing needs to be carried out on training samples or not is selected, and the accuracy rate of medical image labeling is improved. Objects needing to be screened out are screened out through the training sample screening module, screening processing is carried out on training samples, medical image AI labeling is carried out through the medical image AI labeling module, and the manual labeling cost is reduced on the premise that the medical image labeling accuracy and labeling efficiency are improved.

[0018] The mentioned patent is related to the technical field of big data and specifically focuses on a medical auxiliary AI intelligent management system and method based on big data whereas our project is centered around a diagnostic assistance system for medical imaging analysis, with features like object detection, chatbot interaction, and report generation, targeting a broader scope of medical diagnostic processes. The mentioned patent includes components like an auxiliary information collection module, a database, an image annotation analysis module, a training sample screening module, and a medical image AI annotation module. On the other hand, our project features components such as object detection models (e.g., YOLO), a chatbot, a feedback mechanism, and a user-friendly interface, providing a more diverse set of functionalities. The mentioned patent aims to reduce manual labeling costs in medical image AI annotation by improving accuracy and efficiency through big data-based techniques. our project focuses on enhancing diagnostic accuracy, providing real-time object detection, and offering a user-friendly interface for a comprehensive diagnostic assistant system.

[0019] KR1020210152048

[0020] METHOD FOR AUTOMATICALLY READING X-RAY IMAGES IN REAL-TIME BASED ON ARTIFICIAL INTELLIGENCE AND SYSTEM THEREOF

[0021] Provided are a method for reading X-ray images based on artificial intelligence (AI) and a system thereof. According to one embodiment of the present invention, an automatic real-time X-ray image reading system comprises: an X-ray image acquisition unit acquiring an X-ray image of a patient; an image console viewer unit converting the X-ray image received from the image acquisition unit into an image file of a predetermined format by performing a preprocessing process; an image reading unit reading an image on the basis of AI by receiving the image file from the image console viewer unit, generating tag information including an image reading result, and adds the tag information to the image file; an image management unit storing the image file to which the tag information is added; and an image viewer unit receiving the image file to which the tag information stored in the image management unit is added and displaying the image file on a screen. Accordingly, as soon as the patient's X-ray image is acquired, a result of an AXIR (image reading unit) reading an AI-based automatic X-ray image reading result by using a remote image console viewer, that is, information such as a screening score, a bounding box, a contour map, a heatmap, and the like can be generated as reading assistance data to be provided to medical staff. Through this, the medical staff can efficiently and effectively read a digital video image based on reading assistance information.

[0022] The mentioned patent is specifically designed for automatically reading X-ray images in real-time based on artificial intelligence (AI). It involves an X-ray image acquisition unit, image console viewer unit, image reading unit, image management unit, and image viewer unit. The system is dedicated to real-time processing of X-ray images immediately upon acquisition and utilizes AI for reading X-ray images and generates tag information, including screening score, bounding box, contour map, heatmap, etc. uses a remote image console viewer for AI-based automatic X-ray image reading results. our project involves a broader scope, focusing on diagnostic assistance for various medical imaging analyses, including object detection, chatbot interaction, and report generation. It integrates advanced image processing and AI models, such as YOLO, for real-time object detection. Covers multiple medical imaging domains beyond X-rays, including Pap smears, liver scans, and bone X-rays, generates detailed diagnostic reports summarizing findings, potential health risks, and recommended next steps. Emphasizes accessibility for users without specialized knowledge and aims to enhance accuracy in medical diagnosis. While the mentioned patent is specifically tailored for real-time AI-based X-ray image reading, our project has a more comprehensive approach, addressing diagnostic assistance for various medical imaging types with a focus on user interaction and accessibility.

[0023] United States Patent 10970365

[0024] System and method for medical image interpretation

[0025] An artificial intelligence findings system includes a findings engine that receives medical image data and generates findings based on the medical image data and image interpretation algorithms. An adjustment engine allows the user to adjust the findings to produce a report. A tracking module tracks findings and adjustments made to the findings by the user when producing the report. The tracking module produces tracking information. A machine learning engine receives the tracking information.

[0026] The mentioned patent is focused on a system and method for medical image interpretation using artificial intelligence. It highlights a findings engine, adjustment engine, tracking module, and machine learning engine for generating and adjusting findings based on medical image data. Our project centers on a diagnostic assistance system for medical imaging analysis. It involves object detection, chatbot interaction, and report generation based on uploaded medical images, covering a broader range of diagnostic processes. The mentioned patent involves components like a findings engine, adjustment engine, tracking module, and machine learning engine, emphasizing the generation and adjustment of findings based on medical image data. Our project includes components like object detection models, chatbot interaction, and user-friendly interfaces, offering a more comprehensive set of functionalities beyond findings adjustment. The mentioned patent includes a machine learning engine that receives tracking information, suggesting a learning mechanism likely aimed at improving the findings generation and adjustment process over time. Our project does not explicitly mention a dedicated machine-learning engine but focuses on features like object detection and chatbot interaction for diagnostic assistance.

[0027] CN111292821B

[0028] Medical diagnosis and treatment system

[0029] The embodiment of the application discloses a medical diagnosis and treatment system. The system comprises: the diagnosis and treatment data integration module is used for acquiring diagnosis and treatment data related to patients from one or more medical information sources and integrating the diagnosis and treatment data; the diagnosis and treatment data at least comprise medical image data and medical text data; the artificial intelligence image analysis module is used for analyzing the medical image data based on an artificial intelligence image analysis technology and generating an image analysis report; and the artificial intelligence text analysis module is used for extracting information from the medical text data and / or the image analysis report based on an artificial intelligence information extraction technology to obtain structured diagnosis and treatment data related to diseases.

[0030] The mentioned patent focuses on a medical diagnosis and treatment system that integrates data from multiple medical sources, including medical image data and medical text data. Emphasizes the use of artificial intelligence image analysis and text analysis modules for generating an image analysis report and extracting information from medical text data. It involves a diagnosis and treatment data integration module to acquire data from multiple medical information sources and integrate it and utilizes an artificial intelligence image analysis module to analyze medical image data and generate image analysis reports, also employs an artificial intelligence text analysis module for extracting information from medical text data and image analysis reports, resulting in structured diagnosis and treatment data. On the other hand, our project is a diagnostic assistance system for medical imaging analysis, focusing on object detection, chatbot interaction, and report generation based on user-uploaded medical images.

[0031] Encompasses a broader set of functionalities related to diagnostic assistance, including object detection models, chatbot interaction, and user-friendly interfaces. Involves object detection models (YOLO) for real-time object detection within uploaded medical images. Our claimed invention has a specific emphasis on automating diagnostic processes related to Pap smears, liver diseases, bone fractures, and chest X-rays. In summary, while the mentioned patent focuses on a broader medical diagnosis and treatment system with an emphasis on data integration and artificial intelligence analysis modules, our project specifically addresses diagnostic assistance for medical imaging, including object detection, chatbot interaction, and report generation.

[0032] AU2020357886A1

[0033] AI-assisted medical image interpretation and report generation

[0034] Disclosed herein are systems, methods, and software for providing a platform for AI-assisted medical image interpretation and report generation. One or more subsystems allow for the capturing of user input such as eye gaze and dictation for automated generation of clinical findings. Additional features include quality metric tracking and feedback, and worklist management system and communications queueing.

[0035] The mentioned patent focuses on AI-assisted medical image interpretation with a specific emphasis on capturing user input for automated clinical findings, our invention encompasses a broader set of functionalities related to diagnostic assistance for medical imaging, including object detection, user interaction through a chatbot, and detailed report generation.

[0036] This invention is a diagnostic assistance system for medical imaging analysis. It integrates various medical imaging processes effortlessly. The software starts with four options, including Pap smear analysis, liver disease detection, bone fracture identification, and chest X-ray analysis. After choosing an option and submitting the related radiology images, the system uses Yolo version 8 to detect objects in the images. The patient's information is then inputted, and a request is sent to the Open AI Chabot for a diagnosis report. The user can save the report as a Word file or send it with additional questions.

[0037] Traditional diagnostic procedures for PAP smears, liver diseases, bone fractures, and chest X-rays heavily rely on manual examination by healthcare professionals, introducing several intricate challenges that necessitate advanced solutions. The complexities associated with these diagnostic processes include:

[0038] 1. Manual Examination Bottlenecks:

[0039] Traditional methods involve manual scrutiny of medical images, resulting in time-consuming processes and potential backlogs. The labor-intensive nature of manual examinations contributes to diagnosis delays, limiting the efficiency of healthcare workflows.

[0040] 2. Diversity in Medical Image Characteristics:

[0041] Each diagnostic domain exhibits unique image characteristics, including PAP smears, liver diseases, bone fractures, and chest X-rays. The diverse nature of these medical images poses a significant challenge in developing a unified solution that can adapt to and accurately interpret the intricacies of each modality.

[0042] 3. Communication Gaps in Results:

[0043] The manual diagnostic approach often leads to communication gaps between healthcare professionals, causing delays in conveying critical information. Automating these procedures requires precise detection of abnormalities and clear and understandable communication of results to ensure swift and informed decision-making.

[0044] 4. Inherent Subjectivity in Manual Diagnoses:

[0045] Manual examinations can be subjective, varying between healthcare practitioners and introducing the potential for human error. The technical problem involves addressing this subjectivity by implementing standardized, objective, and consistent diagnostic criteria across different modalities.

[0046] 5. Integration of Diverse Diagnostic Tasks:

[0047] Automation is further complicated by integrating various diagnostic tasks, such as Pap smear analysis, liver disease detection, bone fracture identification, and chest X-ray assessments. Ensuring seamless collaboration between different AI models for these tasks is a technical challenge that demands an integrated and cohesive solution.

[0048] 6. Interpretability for Healthcare Professionals:

[0049] Healthcare professionals require accurate diagnostic results and interpretable insights into the AI-driven decision-making process. Bridging the gap between complex algorithmic outputs and the understanding of medical practitioners is crucial for building trust and confidence in automated diagnostic systems.

[0050] 7. Resource Constraints and Scalability:

[0051] Traditional diagnostic procedures face resource constraints, including a shortage of skilled professionals. The technical problem involves creating an automated system that optimizes available resources and scales efficiently to handle increasing volumes of medical data.

[0052] 8. Ethical Considerations and Patient Privacy:

[0053] The automation of diagnostic procedures raises ethical considerations regarding patient privacy and consent. Designing a system that adheres to ethical standards and ensures patient data privacy throughout the diagnostic process is a critical technical challenge.

[0054] Addressing these detailed challenges is pivotal in developing a sophisticated and effective solution that automates traditional diagnostic procedures, enhances diagnostic accuracy, and ensures timely and secure communication of results across diverse medical imaging domains.

[0055] The programming language of this project is Python, it serves as the backbone of the diagnostic system, providing a versatile, readable, and powerful foundation for the seamless integration of diverse components. Its role in object detection, dataset management, user interaction, and adaptability solidifies Python's significance in delivering an innovative and effective solution for multi-modal medical imaging analysis.

[0056] The rest of this patent is based on the integration of advanced image processing and AI Models:

[0057] 1. Unified Framework:

[0058] The proposed system adopts a unified framework that seamlessly integrates various medical imaging processes. It combines Pap smear analysis, liver disease detection, bone fracture identification, and chest X-ray analysis into a singular pipeline.

[0059] 2. Object Detection Models:

[0060] Utilizing Roboflow, YOLO (You Only Look Once), and Ultralytics, the system employs advanced object detection models. YOLO, in particular, enables real-time object detection in images, facilitating rapid analysis.

[0061] 3. Efficient Training and Deployment:

[0062] Roboflow is utilized for efficient dataset management, ensuring diverse and comprehensive datasets for model training. YOLO serves as the backbone for object detection, and Ultralytics streamlines the training and deployment processes.

[0063] 4. Prompt Engineering Integration:

[0064] After the completion of object detection, the result, along with patient information and detailed summaries, is sent to OpenAI as a single prompt. The prompt requests a standardized report, leveraging OpenAI’s language model capabilities for an informative and coherent final analysis. This innovative approach streamlines the reporting process, ensuring consistency and efficiency in generating comprehensive diagnostic reports

[0065] Workflow of the Integrated System:

[0066] 1. Image Input:

[0067] Users upload medical images encompassing Pap smears, liver scans, bone X-rays, and chest X-rays. The system is designed to handle a variety of image formats.

[0068] 2. Object Detection:

[0069] The integrated system employs YOLO-based models for real-time object detection within the uploaded images. Different YOLO configurations are utilized for each medical imaging category, optimizing detection accuracy.

[0070] 3. Diagnostic Analysis:

[0071] Once objects are detected, the system conducts a diagnostic analysis specific to each medical domain. For instance:

[0072] Pap smear: Identifies cellular abnormalities indicative of cervical health.

[0073] Liver Disease: Evaluates liver health based on features extracted from medical images.

[0074] Bone Fracture: Detects and analyzes fractures in bone X-rays.

[0075] Chest X-ray: Identifies abnormalities in chest X-ray images, aiding in respiratory diagnostics.

[0076] 4. Patient Information Input:

[0077] To enhance diagnostic accuracy, users provide relevant patient information. This step ensures a more personalized and precise diagnosis tailored to individual health contexts.

[0078] 5. Chatbot Interaction:

[0079] The system incorporates a chatbot that conversationally engages users. Users can seek additional information, ask questions about the diagnosis, and receive detailed explanations from the chatbot.

[0080] 6. Report Generation:

[0081] Upon completing the diagnostic analysis, the system generates comprehensive reports summarizing the findings. These reports include detailed information about the diagnostic results, potential health risks, and recommended next steps.

[0082] User Interaction and Feedback:

[0083] The system offers an intuitive interface, allowing seamless navigation and interaction for users with varying levels of technical expertise.

[0084] 2. Chatbot Guidance:

[0085] The chatbot serves as a virtual assistant, guiding users through the diagnostic process, offering explanations, and addressing user queries. It enhances user engagement and understanding.

[0086] 3. Feedback Mechanism:

[0087] Users have the option to provide feedback on the diagnostic results and the overall usability of the system.

[0088] The advantages of this invention across multiple medical domains include:

[0089] Efficiency: Automated image analysis reduces the time and effort required for diagnosing PAP smears, liver diseases, bone fractures, and chest X-rays.

[0090] Accessibility: The user-friendly chatbot interface enhances accessibility for healthcare professionals and patients, providing quick and understandable results across different diagnostic areas.

[0091] Consistency: Automation reduces the potential for human error, ensuring consistent and reliable analysis in various medical contexts.Brief Description of Drawing

[0092] declares a general process of the system.

[0093] declares the process of the system which begins by running the Python code, then four options show up including the intended body parts, after choosing one of them, the related images must be selected and uploaded. Here the YOLO v8 detects the images and in the next step, the patient’s information gets uploaded and by prompt engineering, the chatbot helps to write a diagnosis report based on the information and the detected images, finally, the diagnosis report is presented, and there is an option to ask extra question and another option for saving as a word file and printing it.Examples

[0094] Users receive a holistic experience, gaining valuable insights into their medical condition through a combination of advanced image analysis, chatbot interaction, and detailed diagnostic reporting. The steps are:

[0095] Image Submission:

[0096] The user starts by providing input to the system, typically in the form of medical images. This can include Pap smear images, X-rays for chest analysis, or images related to liver disease or bone fractures.

[0097] Object Detection:

[0098] The system processes the provided images using object detection techniques. In the case of Pap smears, it identifies and analyzes cell structures. For liver disease, bone fractures, and chest X-rays, it recognizes and analyzes relevant anatomical features.

[0099] Diagnostic Analysis:

[0100] The system performs a diagnostic analysis based on the detected objects. For Pap smears, it may identify abnormalities or patterns associated with different conditions. For liver disease, bone fractures, and chest X-rays, it provides insights into potential health issues.

[0101] Patient Information Input:

[0102] The user is prompted to input additional patient information. This includes details such as patient name, date of birth, contact information, and other relevant data.

[0103] Chatbot Interaction:

[0104] The system incorporates a chatbot that engages with the user. The chatbot provides information about the diagnostic results, answers user queries, and potentially offers additional medical advice or context.

[0105] Patient Report Generation:

[0106] A final diagnosis report is generated, combining the diagnostic analysis, patient information, and chatbot interactions. This report serves as a comprehensive document for the user and healthcare professionals.

[0107] User Interaction and Feedback:

[0108] The user may interact further with the system, asking questions or seeking clarification on the diagnosis. The chatbot responds to user queries and ensures a clear understanding of the diagnostic results.

[0109] Saving the Report:

[0110] The user has the option to save the generated diagnosis report. The system allows the user to choose a location for the saved report.

[0111] This invention holds significant potential for the medical industry, impacting pathology, radiology, and diagnostics. The automated analysis of diverse medical images contributes to streamlined diagnostic processes, improving overall efficiency in healthcare settings.

Claims

A diagnostic assistance system for medical imaging analysis, comprising a unified framework that seamlessly integrates various medical imaging processes.According to claim 1, the software initiates by executing the code, presenting the user with the first initial page offering four options for Pap smear analysis, liver disease detection, bone fracture identification, and chest X-ray analysis.According to claim 2, upon selecting one of the options, a second page emerges, facilitating the input of relevant radiology or pathology images through drag-and-drop functionality.According to the claim 3, Yolo (You only look once) version. 8, YOLO (You Only Look Once) version 8 commences object detection within the uploaded images.According to claim 4, a third page prompts the user to input patient information, including name, medical background, doctor’s name, and date.According to claim 5, prompt engineering initiates the transmission of three files—patient information, object detection results, and a detailed summary—to the OpenAI Chatbot, accompanied by a request for generating a diagnosis report.According to claim 6, a fourth page presents three options: view the diagnosis report, save the report, and send.According to claim 7, selecting the diagnosis report option provides a comprehensive report, including patient and doctor details, date, a signature space, and most importantly, the final diagnosis.According to claim 8, by choosing the save option, the report will be saved as a word file and it will be ready to be printed.According to claim 9, if the report needs to be more specific, the option send has a place above it to insert questions and get its answers.

Images