Method for training high-resolution image analysis model, and method and system for providing diagnostic answers based on high-resolution image analysis

Abstract

Provided according to an embodiment is a method for training a high-resolution image analysis model, the method comprising the steps of: generating an integrated vision embedding from a high-resolution image by using a vision encoding module; concatenating the integrated vision embedding with a captioning text embedding corresponding to captioning information for the high-resolution image, so as to generate a first vision-language embedding; carrying out first training involving training a large language model (LLM) to output the captioning information by using the first vision-language embedding as an input; generating a question-answer dataset in which question text and answer text related to the high-resolution image are matched; concatenating the integrated vision embedding with a question embedding corresponding to the question text, so as to generate a second vision-language embedding; and carrying out second training involving training the LLM to output the answer text by using the second vision-language embedding as an input.

Description

Method for learning a high-resolution image analysis model, method and system for providing diagnostic responses based on high-resolution image analysis

[0001] The present disclosure relates to a system for analyzing high-resolution images and providing a diagnostic response based thereon by utilizing artificial intelligence, particularly large-scale language models (LLMs) and multimodal learning techniques, and more particularly, to a method for learning a high-resolution image analysis model for integrating a high-resolution image and captioning information related thereto to provide a diagnostic response suitable for the high-resolution image, and a method and system for providing a diagnostic response based on high-resolution image analysis.

[0002] Understanding and interpreting high-resolution images is a crucial process for problem diagnosis and analysis in various fields. However, due to their complexity and vast amount of information, high-resolution images require expert analysis and extensive resources. Furthermore, diagnostic guidelines and analysis criteria are subject to continuous change due to the emergence of new technologies and evolving requirements.

[0003] With recent advances in digitalization, various computational methodologies have been introduced for automated image analysis. Deep learning, in particular, has proven effective in understanding complex and atypical images. However, existing deep learning models tend to operate independently, without interaction between experts and algorithms, due to limited interpretability and the ability to infer from derived results.

[0004] Large Language Models (LLMs) have revolutionized the field of Natural Language Processing (NLP), significantly impacting various fields by aiding in the interpretation of complex data. LLMs outperform existing NLP models thanks to extensive training on large datasets, a transformer architecture, and enhanced contextual understanding. While LLMs have demonstrated their utility in image analysis, their inherent limitation of only understanding textual information limits their contribution and application in real-world scenarios.

[0005] To overcome these limitations, multimodal LLM (MLLM) emerged and demonstrated the effectiveness of combining visual and textual information. MLLM combines different modalities into a unified representational space, enabling deep learning models to better understand the interactions between various modalities. As a related technology, a multimodal dialogue system is known that can assist experts by answering various questions about a given image.

[0006] However, existing MLLMs are mainly trained using high-resolution images at the patch level, and these patches contain only partial information of the high-resolution image, making it difficult to identify the entire image structure and correlations between features within the image, which are essential for diagnosis. Accordingly, there are limitations in fully interpreting the relevant context provided with the entire high-resolution image.

[0007] Furthermore, securing high-quality, high-resolution clinical or industrial image datasets is a significant challenge. Public datasets often contain patch-based, unstructured data, which can lead to a lack of in-depth contextual understanding of the entire image.

[0008] In particular, understanding and interpreting tissue pathology images is a crucial process in cancer diagnosis. However, due to the complexity of tissue pathology images, image examination for cancer diagnosis is difficult and resource-intensive, and pathologists require extensive training to understand the medical context. Furthermore, cancer diagnostic guidelines are constantly evolving due to the discovery of new biomarkers and the emergence of new cancer types.

[0009] However, existing MLLMs are trained primarily using patch-level histopathology images, which contain only partial information. Therefore, they struggle to identify correlations between overall tissue structures and lesions, which are essential for pathological diagnosis. Consequently, they hinder the full interpretation of the clinical context provided with whole-slide images (WSI). Furthermore, although large-scale, high-quality clinical datasets are essential for developing clinically applicable MLLMs in the medical field, securing them remains challenging.

[0010] According to various embodiments of the present disclosure, there is provided a method for learning a high-resolution image analysis model capable of processing multimodal input including questions related to high-resolution images to generate accurate and context-appropriate responses based on a given context, and a method and system for providing diagnostic responses based on high-resolution image analysis.

[0011] According to various embodiments of the present disclosure, a method is provided for efficiently building a large-scale, high-quality, multimodal training dataset for a high-resolution image analysis model. Pre-training a high-resolution image analysis model requires securing a high-resolution image and its associated rich text dataset, but building such datasets on a large scale is challenging. Considering this, the present invention provides a document analysis-based data generation pipeline that can efficiently build a high-quality question-answer dataset based on documents associated with high-resolution images.

[0012] According to various embodiments of the present disclosure, an AI-based evaluation protocol is provided for comprehensively assessing and evaluating the contextual understanding of a high-resolution image analysis-based response provision system. This evaluation protocol utilizes Chain-of-Thought (CoT) prompts to enable an LLM-based evaluation module to provide detailed reasoning during the evaluation process, thereby enhancing the reliability and transparency of the evaluation.

[0013] However, the technical problems that the various embodiments of the present disclosure seek to solve are not limited to the technical problems described above, and other technical problems may exist.

[0014] One example is,

[0015] A method for learning a high-resolution image analysis model performed by a computing device is provided, comprising: a step of generating an integrated vision embedding from a high-resolution image using a vision encoding module; a step of generating a first vision-language embedding by connecting the integrated vision embedding and a captioning text embedding corresponding to captioning information for the high-resolution image; a first learning step of training a large language model (LLM) to output the captioning information using the first vision-language embedding as an input; a step of generating a question-response dataset in which question text and response text related to the high-resolution image are matched; a step of generating a second vision-language embedding by connecting the integrated vision embedding and a question embedding corresponding to the question text; and a second learning step of training the large language model to output the response text using the second vision-language embedding as an input.

[0016] In another aspect, in the first learning step, when the vision encoding module and the giant language model are combined, the parameters of the giant language model may be fixed and the parameters of the vision encoding module may be updated.

[0017] In another aspect, in the second learning step, when the vision encoding module and the giant language model are combined, both the parameters of the giant language model and the parameters of the vision encoding module can be updated.

[0018] In another aspect, in the second learning step, the parameters of the giant language model can be updated in the LoRA (Low-Rank Adaptation) manner.

[0019] In another aspect, the step of generating the integrated vision embedding may include the steps of dividing the high-resolution image into a plurality of patches, generating a plurality of patch embeddings from the plurality of patches, integrating at least some of the plurality of patch embeddings to generate a single high-resolution image embedding, and adjusting a dimension of the single high-resolution image embedding to generate the integrated vision embedding.

[0020] In another aspect, the step of generating the single high-resolution image embedding may include the step of calculating a plurality of weights according to the importance of each of the plurality of patch embeddings through an attention operation on the plurality of patch embeddings, and the step of calculating a sum of weights for at least some of the plurality of patch embeddings based on the plurality of weights to generate the single high-resolution image embedding.

[0021] In another aspect, the high resolution image may include a tissue pathology image.

[0022] In another aspect, the step of generating the question-response dataset may include the steps of extracting a plurality of texts related to the high-resolution image from a document related to the high-resolution image, constructing a text vector database based on the plurality of texts related to the high-resolution image, calculating a similarity between a question text related to the high-resolution image and a plurality of texts in the text vector database, generating the response text using a large language model based on at least one text among the plurality of texts having a similarity to the question text greater than or equal to a predetermined value, and generating the question-response dataset by matching the question text and the response text to generate a question-response pair.

[0023] In another aspect, the step of generating the response text using the large language model may include the step of providing the large language model with a system prompt for generating the response text, and the step of generating the response text using the large language model based on the question text, the at least one text, and the system prompt.

[0024] One example is,

[0025] A method for analyzing a high-resolution image using a large language model performed by a computing device, the method comprising: a first learning step in which a large language model (LLM) is trained to output captioning information by inputting a first vision-language embedding that connects an integrated vision embedding generated from a high-resolution image by a vision encoding module and a captioning text embedding corresponding to captioning information for the high-resolution image; a second learning step in which the large language model is trained to output a response text for the question text by inputting a second vision-language embedding that connects the integrated vision embedding and a question embedding corresponding to a question text related to the high-resolution image; a step of receiving a high-resolution image to be analyzed according to a user input; a step of receiving a question text related to the high-resolution image to be analyzed; and a step of generating a diagnostic response based on the high-resolution image to be analyzed and the question text by using an integrated high-resolution image analysis module in which the vision encoding module and the large language model are combined, in which the first and second learnings are completed.

[0026] In another aspect, the step of generating the diagnostic response includes the step of generating a plurality of diagnostic responses for the question text using the integrated high-resolution image analysis module, and the method for providing a diagnostic response based on high-resolution image analysis may further include the step of calculating an evaluation score for each of the plurality of diagnostic responses using a quality evaluation module, and the step of selecting a diagnostic response having the highest evaluation score among the plurality of diagnostic responses.

[0027] In another aspect, the step of calculating an evaluation score for each of the plurality of diagnostic responses may include the step of calculating a first similarity for the question text of each of the plurality of diagnostic response text embeddings corresponding to the plurality of diagnostic responses, the step of calculating a second similarity for the integrated vision embedding of each of the plurality of diagnostic response text embeddings, and the step of calculating an evaluation score for each of the plurality of diagnostic responses based on the first similarity and the second similarity.

[0028] On the other side,

[0029] The method for providing a diagnostic response based on the above-described high-resolution image analysis may further include a step of providing a Chain-of-Thought (CoT) prompt to a large language model-based evaluation module to perform inference to determine whether to provide the selected diagnostic response to the user, and a step of obtaining information regarding whether to provide the selected diagnostic response and the basis therefor output from the large language model-based evaluation module.

[0030] In another aspect, the CoT prompt may be configured to include at least one evaluation criterion information for determining whether to provide the selected diagnostic response to the user, request to perform inference based on the at least one evaluation criterion, and request to determine whether to provide the selected diagnostic response based on the performed inference result.

[0031] One example is,

[0032] At least one memory, and at least one processor for executing a high-resolution image analysis method by reading at least one command stored in the memory, wherein the at least one processor generates an integrated vision embedding from a high-resolution image using a vision encoding module, connects the integrated vision embedding and a captioning text embedding corresponding to captioning information for the high-resolution image to generate a first vision-language embedding, and performs first learning to train a large language model (LLM) to output the captioning information using the first vision-language embedding as an input, generates a question-response dataset in which question text and response text related to the high-resolution image are matched, connects the integrated vision embedding and a question embedding corresponding to the question text to generate a second vision-language embedding, and performs second learning to train the large language model to output the response text using the second vision-language embedding as an input, and receives a high-resolution image to be analyzed according to a user input, receives a question text related to the high-resolution image to be analyzed, and performs the first and second learning. A high-resolution image analysis-based diagnostic response providing system is provided, which generates a diagnostic response based on the high-resolution image to be analyzed and the question text using an integrated high-resolution image analysis module in which the vision encoding module and the large language model are combined.

[0033] In another aspect, a system for providing a diagnostic response based on high-resolution image analysis may further include a plurality of neurons configured as an array including at least one register, at least one programmable logic, and at least one input interface, a plurality of synaptic circuits storing synaptic weights that adjust connection strengths between the plurality of neurons, and at least one routing network that controls data flow between the plurality of neurons, wherein each of the plurality of neurons is connected to at least one other neuron through the routing network to set a transmission path of the weights, and may further include a field programmable gate array (FPGA) implementation for a predetermined artificial neural network.

[0034] In another aspect, a high-resolution image analysis-based diagnostic response providing system comprises a plurality of neurons organized in an array including at least one register, at least one microprocessor and at least one input, and a plurality of synaptic circuits storing synaptic weights that adjust connection strengths between the plurality of neurons, wherein each of the plurality of neurons may further include an application specific integrated circuit (ASIC) for a predetermined artificial neural network connected to at least one other neuron through one of the plurality of synaptic circuits.

[0035] According to various embodiments of the present disclosure, improved contextual understanding and accurate response generation at the high-resolution image level are possible. Existing multimodal large-scale language models (MLLMs) are primarily trained using high-resolution images at the patch level. These patches contain only partial information of the high-resolution image, making it difficult to grasp the overall image structure and correlations between features within the image, which are essential for diagnosis. Consequently, there are limitations in fully interpreting the relevant context provided with the entire high-resolution image.

[0036] According to various embodiments of the present disclosure, a vision encoding module generates an integrated vision embedding from a high-resolution image and aligns it to the text embedding space of a large language model (LLM), enabling the LLM to process comprehensive visual and textual information from the entire high-resolution image together, thereby better understanding the correlation between complex features within the image and textual descriptions, and generating accurate and context-sensitive responses based on a given context. This can contribute to effectively understanding a wide range of contexts in various types of high-resolution images.

[0037] According to various embodiments of the present disclosure, efficient construction and increased accuracy of a large-scale, high-quality multimodal learning dataset are possible. Pre-training a high-resolution image analysis model requires securing a high-resolution image and associated rich text dataset, but directly building such datasets on a large scale presents challenges.

[0038] Considering these points, various embodiments of the present disclosure provide a document analysis-based data generation pipeline capable of efficiently constructing a high-quality question-answer dataset based on documents associated with high-resolution images. The document analysis-based data generation pipeline converts original documents (e.g., diagnostic reports) associated with high-resolution images into text, builds a text vector database, and retrieves text chunks with high similarity to the question from this text vector database, which are then used by a large language model (LLM) to generate responses.

[0039] This approach addresses the inherent heterogeneity of information inherent in the original documents due to their irregularity and information diversity. Furthermore, by selectively retrieving relevant information, the risk of hallucination in LLMs can be mitigated, resulting in a more accurate and robust training dataset. By intelligently leveraging limited original data to efficiently generate large-scale question-answering datasets, the challenges of securing high-quality multimodal training datasets can be addressed.

[0040] Various embodiments of the present disclosure provide an AI-based evaluation protocol for comprehensively evaluating the contextual understanding of a high-resolution image analysis-based response provision system. The evaluation protocol allows an LLM-based evaluation module to select an optimal response from among multiple responses generated by a model, determine whether the selected response meets quality criteria, and provide a detailed reasoning process during the evaluation process using Chain-of-Thought (CoT) prompts, thereby enhancing the reliability and transparency of the evaluation. This CoT prompt-based reasoning induction process not only enhances the user's (expert's) confidence in the judgment of the AI-based evaluation system by providing a clear basis for the evaluation results, but also helps an AI evaluation module without specialized knowledge accurately capture important evaluation factors of the image analysis results.

[0041] However, the effects that can be obtained through various embodiments of the present disclosure are not limited to the effects mentioned above, and other effects that are not mentioned can be clearly understood from the description below.

[0042] FIG. 1 illustrates an example block diagram of a computing system that implements a high-resolution image analysis-based diagnostic response provision service according to one embodiment.

[0043] FIG. 2 is a schematic diagram illustrating the structure of a neuromorphic circuit that a processor may include according to one embodiment.

[0044] FIG. 3 is a block diagram of a computing device that implements a high-resolution image analysis-based diagnostic response provision service according to one embodiment.

[0045] FIG. 4 is a block diagram of a computing device that implements a high-resolution image analysis-based diagnostic response provision service according to another embodiment.

[0046] FIG. 5 is a block diagram of a computing device that implements a high-resolution image analysis-based diagnostic response provision service according to another embodiment.

[0047] FIG. 6 is for explaining a first learning process for a high-resolution image analysis model according to one embodiment.

[0048] FIG. 7 is for explaining a second learning process for a high-resolution image analysis model according to one embodiment.

[0049] FIG. 8 is a flowchart illustrating a process for generating a question-answer dataset for second learning of a high-resolution image analysis model according to one embodiment.

[0050] FIG. 9 is a diagram illustrating a method for evaluating the performance of a high-resolution image analysis model according to one embodiment.

[0051] FIG. 10 is a method for evaluating the performance of a high-resolution image analysis model according to another embodiment.

[0052] Fig. 11 is a flowchart of a high-resolution image analysis model learning method according to one embodiment.

[0053] FIG. 12 is a flowchart illustrating steps that may be included in generating the integrated vision embedding of FIG. 11.

[0054] Fig. 13 is a flowchart of a method for providing a diagnostic response based on high-resolution image analysis according to one embodiment.

[0055] FIG. 14 is a flowchart illustrating steps that may be further included in the method for providing a high-resolution image analysis-based diagnostic response of FIG. 13.

[0056] Various modifications can be made to the various embodiments of the present disclosure, and thus specific embodiments are illustrated in the drawings and described in detail in the detailed description. The effects and features of the various embodiments of the present disclosure, as well as the methods for achieving them, will become clear with reference to the embodiments described in detail below together with the drawings. However, the various embodiments of the present disclosure are not limited to the embodiments disclosed below and may be implemented in various forms. In the following embodiments, terms such as "first" and "second" are not used in a limiting sense but are used for the purpose of distinguishing one component from another. Furthermore, the singular expression includes the plural expression unless the context clearly indicates otherwise. Furthermore, terms such as "include" or "have" indicate the presence of a feature or component described in the specification, and do not preemptively exclude the possibility that one or more other features or components may be added. Furthermore, the sizes of components in the drawings may be exaggerated or reduced for convenience of explanation. For example, the size and thickness of each component shown in the drawings are arbitrarily shown for convenience of explanation, and thus the various embodiments of the present disclosure are not necessarily limited to those illustrated.

[0057] Hereinafter, various embodiments of the present disclosure will be described in detail with reference to the attached drawings. When describing with reference to the drawings, identical or corresponding components are given the same drawing reference numerals, and redundant descriptions thereof will be omitted.

[0058]

[0059] - A system (1000) that provides a diagnostic response service based on high-resolution image analysis.

[0060] A system (1000) according to one embodiment generates an integrated vision embedding from a high-resolution image, and connects the vision-language embedding with a captioning text embedding corresponding to captioning information for the high-resolution image. Based on this, a learned large language model generates an accurate diagnostic response for the high-resolution image and a related query, and evaluates and optimizes the quality of the generated response, thereby providing a highly reliable diagnostic response to a user.

[0061] In this case, the system (1000) can solve the existing data acquisition and hallucination problems and secure the ability to generate more accurate and robust diagnostic responses by performing training of a large language model using a high-quality question-answer dataset built through search-based augmentation from original documents related to high-resolution images.

[0062] In addition, the system (1000) can evaluate multiple diagnostic response candidates using a large language model-based evaluation module, select an optimal response, and provide a detailed reasoning process through a Chain-of-Thought (CoT) prompt, thereby increasing the reliability and transparency of the evaluation and providing assurance to the user about the quality of the diagnostic response provided.

[0063] FIG. 1 illustrates an example of a block diagram of a computing system (1000) that implements a high-resolution image analysis-based diagnostic response provision service according to one embodiment.

[0064] Referring to FIG. 1, a computing system (1000) implementing a high-resolution image analysis-based diagnostic response provision service according to one embodiment includes a user computing device (110), a server computing system (130), and a training computing system (150), and the devices are capable of communicating via a network (170).

[0065] A method for providing a high-resolution image analysis-based diagnostic response according to one embodiment may be implemented and provided locally by a user computing device (110), implemented and provided in the form of a web service by a server computing system (130) communicating with the user computing device (110), or implemented and provided by a user computing device (110) and a server computing system (130) in connection with each other.

[0066] At this time, in the embodiment, the user computing device (110) and / or the server computing system (130) can train the machine learning model (120 and / or 140) through interaction with a training computing system (150) communicatively connected via a network (170). The training computing system (150) may be separate from the server computing system (130) or may be part of the server computing system (130).

[0067] And at this time, the artificial intelligence model can be 1) trained directly locally by the user computing device (110), 2) trained by the server computing system (130) and the user computing device (110) interacting with each other through a network (170), and 3) trained by a separate training computing system (150) using various training techniques and learning techniques. And the artificial intelligence model trained by the training computing system (150) can be implemented in a manner of being provided / updated by transmitting the model to the user computing device (110) and / or the server computing system (130) through a network (170).

[0068] In some embodiments, the training computing system (150) may be part of a server computing system (130) or part of a user computing device (110).

[0069]

[0070] -User Computing Device (110)

[0071] The user computing device (110) may include any type of computing device, such as a smart phone, a mobile phone, a digital broadcasting device, a personal digital assistant (PDA), a portable multimedia player (PMP), a desktop, a wearable device, an embedded computing device, a tablet PC, an augmented reality (VR) device, and / or a virtual reality (AR) device.

[0072] Such a user computing device (110) may include at least one processor (111) and memory (112). Here, the processor (111) may be composed of at least one or a plurality of electrically connected processors among a central processing unit (CPU), a graphics processing unit (GPU), application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors, and / or other electrical units for performing functions.

[0073] In particular, depending on the embodiment, such a processor (111) may be configured based on a field programmable gate array (FPGA) implementation, which is a hardware technology for implementing a predetermined digital circuit, and / or an application specific integrated circuit (ASIC).

[0074] Here, a field programmable gate array (FPGA) can mean a flexible digital circuit that can be programmed according to user needs.

[0075] In an embodiment, the field programmable gate array implementation may include a register that temporarily stores data and controls the flow and timing of signals to maintain intermediate results of operations or state information to support synchronized operation of the FPGA; programmable logic that programs operations within the FPGA to perform specific functions or operations as logic circuits that can be configured according to user needs; and an input interface that receives signals from an external device or sensor as a path for receiving data from outside the FPGA and transmits them to the internal circuit.

[0076] By combining the above components, field programmable gate array implementations can provide flexible and diverse forms of digital circuits.

[0077] Meanwhile, an application-specific integrated circuit (ASIC) can refer to a custom integrated circuit that is specifically designed to perform a specific purpose or function.

[0078] As an example, the application-specific integrated circuit may include a register, which is a small memory device that temporarily stores and manages data and stores intermediate results of calculations or status information to support rapid processing of the operation of the ASIC; a microprocessor, which is a central processing unit that performs control and operations within the ASIC and, if necessary, performs various operations or generates control signals to coordinate the operation of the entire system; and an input block, which is an interface that receives data from the outside and receives data to be processed by the ASIC and transmits it to the inside, and receives various input data through a connection with a sensor or an external device.

[0079] By combining the above components, application-specific integrated circuits can perform specific tasks in an optimized manner.

[0080] For example, ASICs may have a neuromorphic circuit structure in the form of an array containing multiple neuron circuits.

[0081] FIG. 2 is a schematic diagram illustrating the structure of a neuromorphic circuit (300) that may be included in a processor (111, 131, 151) according to one embodiment.

[0082] Referring to FIG. 2, for example, a neuromorphic circuit (300) may include a plurality of presynaptic neuron circuits (310), a plurality of presynaptic lines (311) extending laterally from the plurality of presynaptic neuron circuits (310), a plurality of post-synaptic neuron circuits (320), a plurality of post-synaptic lines (321) extending longitudinally from the plurality of post-synaptic neuron circuits (320), and a plurality of synaptic circuits (330) provided at intersections of the plurality of presynaptic lines (311) and the plurality of post-synaptic lines (321).

[0083] A plurality of presynaptic neuron circuits (310) can transmit signals input from the outside in the form of electrical signals to a plurality of synaptic circuits (330) through a plurality of presynaptic lines (311).

[0084] Additionally, a plurality of post-synaptic neuron circuits (320) can receive electrical signals from a plurality of synaptic circuits (330) through a plurality of post-synaptic lines (321).

[0085] Furthermore, a plurality of post-synaptic neuron circuits (320) may transmit electrical signals to a plurality of synaptic circuits (330) through a plurality of post-synaptic lines (321).

[0086] A plurality of synaptic circuits (330) can store weights included in layers constituting a neural network system implemented by a neuromorphic circuit (300) and perform a predetermined operation based on the weights and input data.

[0087] For example, each of the plurality of synaptic circuits (330) may include a resistive memory cell having a variable resistance. In this case, the plurality of synaptic circuits (330) may have a resistance value that changes according to a voltage applied through the plurality of presynaptic neuron circuits (310) or the plurality of postsynaptic neuron circuits (320), and may store weight data according to this resistance change.

[0088] The neuromorphic circuit (300) is formed by simulating the neuron and synapse structures, which are essential elements of the human brain. When a deep neural network (DNN) is implemented using the neuromorphic circuit (300), data processing speed can be improved and power consumption can be reduced compared to when the existing von Neumann structure is utilized.

[0089] The memory (112) may include one or more non-transitory / transitory computer-readable storage media such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, and combinations thereof, and may include web storage of a server that performs a memory storage function on the Internet. The memory (112) may store data (113) and commands (114) necessary for the at least one processor (111) to perform functional operations such as training an artificial intelligence model or performing high-resolution image analysis using an artificial intelligence model.

[0090] In one embodiment, the user computing device (110) may store at least one machine learning model (120).

[0091] For example, the machine learning model (120) may be various machine learning models such as multiple neural networks (e.g., deep neural networks) for performing a high-resolution image analysis-based diagnostic response providing method, or other types of machine learning models including nonlinear models and / or linear models, and may be composed of a combination thereof.

[0092] For example, the machine learning model may store a linear regression, decision tree, random forest, gradient boosting, pre-trained language model, or / and a deep learning model. The neural network may include at least one of feed-forward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks, or / and other types of neural networks.

[0093] Additionally, according to various embodiments, the user computing device (110) may store a model to be used in each process and a prompt template that serves as the basis for input to the model in order to perform at least part of the process performed for the method of providing a high-resolution image analysis-based diagnostic response through a large language model (LLM).

[0094] In one embodiment, a user computing device (110) may receive at least one machine learning model (120) from a server computing system (130) via a network (170), store the model in a memory (112), and then execute the stored machine learning model (120) via a processor (111) to perform operations for providing a high-resolution image analysis-based diagnostic response.

[0095] In another embodiment, the server computing system (130) may include at least one machine learning model (140) to perform operations through the machine learning model (140), and may communicate data related thereto with the user computing device (110) to provide a high-resolution image analysis-based diagnostic response provision service to the user.

[0096] For example, a user computing device (110) can perform a high-resolution image analysis-based diagnostic response provision service by having a server computing system (130) provide output for a user's input using a machine learning model (140) via the web.

[0097] Additionally, the artificial intelligence model may be implemented in such a way that at least some of the machine learning models (120 and / or 140) are executed on a user computing device (110) and the rest are executed on a server computing system (130).

[0098] In addition, the user computing device (110) may include at least one input component (121) that detects a user's input. For example, the user input component (121) may include a touch sensor (e.g., a touch screen and / or a touch pad, etc.) that detects a touch of a user's input medium (e.g., a finger or a stylus), an image sensor that detects a user's motion input, a microphone, a button, a mouse, and / or a keyboard that detects a user's voice input, etc. In addition, the user input component (121) may include an interface and an external controller when receiving an input to an external controller (e.g., a mouse and / or a keyboard, etc.) through an interface.

[0099]

[0100] -Server Computing System (130)

[0101] The server computing system (130) can perform a series of processes to provide a high-resolution image analysis-based diagnostic response provision service.

[0102] In detail, in an embodiment, the server computing system (130) can provide a high-resolution image analysis-based diagnostic response provision dialogue service by exchanging data necessary to enable a high-resolution image analysis-based diagnostic response provision service process to be driven in an external device, such as a user computing device (110), with the external device.

[0103] In more detail, in an embodiment, a server computing system (130) may provide an environment in which an application for providing a high-resolution image analysis-based diagnostic response provision service may operate on a user computing device (110).

[0104] To this end, the server computing system (130) may include application programs, data and / or commands for the application to operate, and may transmit and receive various data based thereon with the external device.

[0105] The server computing system (130) may include at least one processor (131) and memory (132). Here, the processor (131) may be composed of at least one or a plurality of electrically connected processors among a central processing unit (CPU), a graphics processing unit (GPU), application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors, and / or other electrical units for performing functions.

[0106] For example, ASICs may have a neuromorphic circuit structure in the form of an array containing multiple neuron circuits (see Figure 2).

[0107] And the memory (132) may include one or more non-transitory / transitory computer-readable storage media such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. This memory (132) may store data (133) and commands (134) necessary for the processor (131) to perform functional operations such as training an artificial intelligence model or executing a method for providing a high-resolution image analysis-based diagnostic response through an artificial intelligence model.

[0108] In one embodiment, the server computing system (130) may be implemented to include at least one computing device. For example, the server computing system (130) may be implemented to operate multiple computing devices according to a sequential computing architecture, a parallel computing architecture, or a combination thereof. Additionally, the server computing system (130) may include multiple computing devices connected via a network (170).

[0109] Additionally, the server computing system (130) may store at least one machine learning model (140). For example, the server computing system (130) may include a neural network and / or other multi-layer nonlinear model as the machine learning model (140). Exemplary neural networks may include feedforward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks.

[0110] In an embodiment, the server computing system (130) may further include a data store computing system (hereinafter, “data store”), which is a storage for continuously storing and managing raw data that is the basis of a high-resolution image analysis-based diagnostic response provision service.

[0111] These data stores can include various forms of data storage, ranging from file systems to cloud storage. For example, a data store can include at least one of the following: a relational database that uses a structured query language (SQL) to define and manipulate data; a NoSQL database designed for flexibility and scalability and to handle unstructured and semi-structured data; a data warehouse, which is a system used for reporting and data analysis, centralizes large amounts of data from multiple sources and is optimized for querying and analysis; a data warehouse that stores large amounts of raw data in its native formats of structured, semi-structured, and unstructured data; and a database, typically a local storage device or Network Attached Storage (NAS), that stores data in files in a format accessible by the computer operating system.

[0112]

[0113] -Training Computing System (150)

[0114] The training computing system (150) may include at least one processor (151) and memory (152). Here, the processor (151) may be composed of at least one or a plurality of electrically connected processors among a central processing unit (CPU), a graphics processing unit (GPU), application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors, and / or other electrical units for performing functions.

[0115] For example, ASICs may have a neuromorphic circuit structure in the form of an array containing multiple neuron circuits (see Figure 2).

[0116] And the memory (152) may include one or more non-transitory / transitory computer-readable storage media such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. This memory (152) may store data (153) and instructions (154) necessary for the processor (151) to perform learning of an artificial intelligence model, etc.

[0117] For example, the training computing system (150) may include a model trainer (160) that trains a machine learning model (120 and / or 140) stored in a user computing device (110) and / or a server computing system (130) using various training or learning techniques, such as backpropagation of errors.

[0118] For example, such a model trainer (160) may perform updates for one or more parameters of a machine learning model (120 and / or 140) for a high-resolution image analysis-based diagnostic response provision service in a backpropagation manner based on a defined loss function.

[0119] In some implementations, performing backward propagation of errors may include performing truncated backpropagation through time. The model trainer (160) may perform a number of generalization techniques (e.g., weight reduction, dropout, and / or knowledge distillation) to improve the generalization ability of the trained machine learning model (120 and / or 140).

[0120] For example, a model trainer (160) may train a machine learning model (120 and / or 140) based on a series of training data (161). Here, the training data (161) may include data in different formats, such as, for example, images, audio samples, and / or text.

[0121] Examples of image types that may be used may include video frames, LiDAR point clouds, X-ray images, computed tomography scans, hyperspectral images, and / or various other forms of images.

[0122] Such training data (161) may be provided by a user computing device (110) and / or a server computing system (130). When the training computing device trains a machine learning model (120 and / or 140) on specific data of the user computing device (110), the machine learning model (120 and / or 140) may be characterized as a personalized model.

[0123] And the model trainer (160) includes computer logic utilized to provide the desired function.

[0124] Additionally, the model trainer (160) may be implemented as hardware, firmware, and / or software that controls a general-purpose processor. In one implementation, the model trainer (160) includes a program file stored in a storage device, which may be loaded into a memory (152) and executed by one or more processors (151). In another implementation, the model trainer (160) includes one or more sets of computer-executable data (153) and instructions (154) stored in a tangible computer-readable storage medium, such as a RAM hard disk or an optical or magnetic medium.

[0125] The network (170) includes, but is not limited to, a 3rd Generation Partnership Project (3GPP) network, a Long Term Evolution (LTE) network, a World Interoperability for Microwave Access (WIMAX) network, the Internet, a Local Area Network (LAN), a Wireless Local Area Network (Wireless LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), a Bluetooth network, a satellite broadcasting network, an analog broadcasting network, and / or a Digital Multimedia Broadcasting (DMB) network.

[0126] In general, communication over a network (170) may be performed using any type of wired and / or wireless connection, using various communication protocols (e.g., TCP / IP, HTTP, SMTP, and / or FTP), encodings or formats (e.g., HTML and / or XML), and / or protection schemes (e.g., VPN, Secure HTTP, and / or SSL).

[0127] FIG. 3 is a block diagram of a computing device (100) that implements a high-resolution image analysis-based diagnostic response provision service according to one embodiment.

[0128] Referring to FIG. 3, the computing device (100) included in the user computing device (110), the server computing system (130), and the training computing system (150) includes a plurality of applications (e.g., Application 1 to Application N). Each application may include a machine learning library and one or more machine learning models. For example, the applications may include an image processing (e.g., Detection, Classification, and / or Segmentation, etc.) application, a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, and / or a chat-bot application.

[0129] In an embodiment, the computing device (100) may include a model trainer (160) for training an artificial intelligence model, and may store and operate the trained artificial intelligence model to provide output data according to predetermined input data (e.g., a conversation data set, etc.).

[0130] Each application of the computing device (100) may communicate with a number of other components of the computing device (100), such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In one embodiment, each application may communicate with each device component using an API (e.g., a public API). In one embodiment, the API used by each application may be specific to that application.

[0131] FIG. 4 is a block diagram of a computing device (200) that implements a high-resolution image analysis-based diagnostic response provision service according to another embodiment.

[0132] Referring to FIG. 4, a computing device (200) includes a plurality of applications (e.g., Application 1 to Application N). Each application may communicate with a central intelligence layer. For example, the applications may include an image processing application, a text messaging application, an email application, a dictation application, a virtual keyboard application, and / or a browser application. In one embodiment, each application may communicate with the central intelligence layer (and models stored therein) using an API (e.g., a common API across all applications).

[0133] The central intelligence layer may include multiple machine learning models. For example, as illustrated in FIG. 4, at least some of the machine learning models may be provided to each application and managed by the central intelligence layer. In other implementations, two or more applications may share a single machine learning model. For example, in some implementations, the central intelligence layer may provide a single model to all applications. In some implementations, the central intelligence layer may be included within the operating system of the computing device (200) or implemented differently.

[0134] The central intelligence layer may communicate with a central device data layer. The central device data layer may be a centralized data repository for the computing device (200). As illustrated in FIG. 5, the central device data layer may communicate with a number of other components of the computing device (200), such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, the central device data layer may communicate with each device component using an API (e.g., a private API).

[0135] The techniques described herein may refer to servers, databases, software applications, and other computer-based systems, as well as actions taken and information transmitted to or from such systems. It will be appreciated that the inherent flexibility of computer-based systems allows for a wide range of possible configurations, combinations, and division of labor and functionality between and among components. For example, the processes described herein may be implemented using a single device or component, or multiple devices or components operating in combination. Databases and applications may be implemented on a single system or in a distributed system across multiple systems. Distributed components may operate sequentially or in parallel.

[0136] FIG. 5 is a block diagram of a computing device (400) that implements a high-resolution image analysis-based diagnostic response provision service according to another embodiment. FIG. 6 is for explaining a first learning process for a high-resolution image analysis model according to one embodiment. FIG. 7 is for explaining a second learning process for a high-resolution image analysis model according to one embodiment. FIG. 8 is for explaining a process for generating a question-answer dataset for the second learning for a high-resolution image analysis model according to one embodiment.

[0137] Referring to FIG. 5, a computing device (400) included in a user computing device (110), a server computing system (130), and a training computing system (150) may include a vision encoding module (10), a text embedding module (20), a vision-language embedding generation module (30), a vision-language embedding processing module (40), and a response quality evaluation module (50).

[0138] The vision encoding module (10) can receive a high-resolution image (HRI) as input and convert it into an integrated vision embedding that can be processed by a large language model. The vision encoding module (10) can efficiently process the complexity and massive amount of information of the high-resolution image (HRI) and accurately encode key visual features within the high-resolution image (HRI).

[0139] Referring to FIG. 6, the vision encoding module (10) may largely include a patching layer (not shown), a patch encoder (11), a patch integration layer (12), and a vision projector (13).

[0140] An input high-resolution image (HRI) input to a computing device (400) can be divided into a plurality of small image areas, i.e., a plurality of patches, by being patched through a patching layer. Through this, the massive pixel information of the original high-resolution image (HRI) can be efficiently processed and adapted to the input format of a deep learning model. For example, the high-resolution image (HRI) may include a tissue pathology image, in which case the tissue pathology image may be a whole slide image (WSI) enlarged by 20x magnification, and may be divided into patches of 256x256 size having a resolution of pixels per 0.5 microns on an actual tissue.

[0141] Each segmented patch is provided as input to a patch encoder (11), which can generate a patch-level feature embedding. The patch encoder (11) learns complex image patterns and structures and expresses them in the form of compressed vectors, and may include, for example, a pre-trained robust patch-level foundation model based on a ViT (Vision Transformer) architecture. This patch encoder (11) functions as a patch-level feature extractor, and its parameters can be frozen during the learning process.

[0142] Multiple patch embeddings generated from the patch encoder (11) can be provided as input to the patch integration layer (12), and the patch integration layer (12) can integrate them to generate a single high-resolution image embedding. The patch integration layer (12) can be implemented as an aggregator based on CLAM (Clustering-constrained-attention Multiple-Instance Learning), and can calculate a weight according to the importance of each patch embedding by utilizing attention-based learning.

[0143] For example, a module such as the Gated Attention Network (GATN) can utilize two different attention modules: an attention module that uses the Tanh (Hyperbolic Tangent) activation function and an attention module that uses the Sigmoid activation function, to calculate weights based on the importance of each of the multiple patch embeddings.

[0144] For example, GATN first performs a linear transformation using a first attention module that uses a Tanh activation function for each patch embedding, and then performs a nonlinear activation operation based on the Tanh activation function (a function that outputs all elements of the embedding as values ​​between -1 and 1) on the linearly transformed embedding to determine the feature significance of the corresponding patch embedding.

[0145] Additionally, for example, GATN can perform a linear transformation using a second attention module that uses a sigmoid activation function for each patch embedding, and perform a gating operation based on the sigmoid activation function (a function that outputs all elements of the embedding as values ​​between 0 and 1) on the linearly transformed embedding to determine the degree of reflection or activation of the patch embedding for a single high-resolution image embedding.

[0146] The two outputs from these two attention modules are multiplied element-wise, thereby forming an integrated feature embedding (oi). Here, the output of the second attention module based on the sigmoid function acts as a 'gate' for the output of the first attention module based on the tanh function. Even if the first attention module highly evaluates the significance of a particular element of the patch embedding (if it outputs a value close to 1), if the second attention module determines that the element of the patch embedding will not pass (if it outputs a value close to 0), the final multiplication result of the two elements will be lowered, reducing the influence of the feature. Conversely, if the second attention module determines that the element will pass (if it outputs a value close to 1), the significance of the first attention module will be reflected almost as is. This allows GATN to capture robust aggregated features from patch embeddings by calculating importance that considers not only the presence or absence of a feature but also its reliability and the degree to which it reflects a single high-resolution image embedding.

[0147] The integrated feature embedding (oi) obtained through element-wise multiplication of the two outputs from the two attention modules can be input to a single linear layer. The output of this linear layer can be the final attention weight (αi) for the corresponding patch embedding.

[0148] This attention weight (αi) typically has a single scalar value and can be normalized by a softmax function, etc., so that the sum of the attention weights of all patches becomes 1. This normalized attention weight (αi) determines how much each patch will contribute to generating a single high-resolution image embedding that represents the meaning of the entire high-resolution image (HRI).

[0149] Finally, based on the multiple weights (αi) calculated for the multiple patch embeddings, a weighted sum of the multiple patch embeddings can be calculated to generate a single high-resolution image embedding. This single high-resolution image embedding is the result of efficiently integrating and compressing information determined to be important by GATN among the detailed features at the individual patch level extracted through the patch encoder (11).

[0150] Meanwhile, the patch integration layer (12) can perform weighted summation only by selecting patch embeddings with weights greater than a predetermined threshold, instead of calculating the weighted sum for all patch embeddings using all of the multiple weights (αi) produced by GATN. For example, this is a method in which the embedding of a specific patch is included in the integration process only if the weight assigned to that patch by GATN is greater than or equal to 0.1. This approach can induce the model to focus on key regions that are judged to play a more important role within the image, rather than considering the entire image.

[0151] Through this, patches with relatively low importance or background information can be naturally excluded during the integration process. Consequently, the integrated, single, high-resolution image embedding can more clearly and efficiently represent the image's core features. This not only increases computational efficiency but also prevents the model from being confused by unnecessary information, thereby improving overall performance.

[0152] Through this weight-based aggregation process, the patch integration layer (12) can selectively integrate the most essential and diagnostically relevant information from hundreds of thousands of massive patch embeddings, generating a single, concise and meaningful embedding (a single high-resolution image embedding) that represents the complex features of the entire high-resolution image. This can provide an essential foundation for the large-scale language model (41) in subsequent steps to accurately understand visual information and make inferences with textual information.

[0153] A single high-resolution image embedding, which is the output of the patch integration layer (12), can be provided as an input to the vision projector (13), so that an integrated vision embedding aligned to the text embedding space of the giant language model (41) can be generated.

[0154] The vision projector (13) may include an attention pooler, a normalization layer, and a linear projector (ex. Linear layer-GeLU-Linear layer structure).

[0155] The attention pooler weights features within a single high-resolution image embedding that are considered important when projected onto the text space. For example, the attention pooler uses a self-attention mechanism to weight each element within a single high-resolution image embedding, thereby highlighting key information within the image.

[0156] Afterwards, the normalization layer normalizes the output of the attention pooler to stabilize the scale and distribution of the embedding, making the input to the next stage, the linear projector, easier for model training.

[0157] Finally, a linear projector scales the normalized embedding to match the dimensionality of the language model's text embedding space, transforming visual information into a form better understood by the language model, allowing a single high-resolution image embedding to be smoothly projected onto the text space.

[0158] For example, the vision projector (13) may serve to adjust a single high-resolution image embedding to match the text embedding dimension of the language model (e.g., 4096 dimensions) in a specific dimension (e.g., 512 dimensions).

[0159] As a result, the vision encoding module (10) can process and compress the massive visual information of a high-resolution image (HRI) in stages to generate an integrated vision embedding that can be input into a large language model (15) together with text information.

[0160] The text embedding module (20) receives text (TXT) related to a high-resolution image (HRI) and converts it into a text embedding that can be processed by the vision-language embedding generation module (30). The text embedding module (20) can efficiently encode text-based features essential for learning and inference of a language model, just like visual information.

[0161] The text embedding module (20) may include a tokenizer (21) and an embedding layer. The input high-resolution image-related text (TXT) may refer to caption text (CXT) in the first learning step of FIG. 6, and may refer to question text (QXT) in the second learning step and inference step of FIG. 7.

[0162] Referring to FIGS. 6 and 7, the tokenizer (21) divides the input captioning text (CXT) or question text (QXT) into individual linguistic units, i.e., tokens. For example, it divides sentences into words or subwords, or separates punctuation marks, etc. to create the smallest units that the model can process. The tokenizer (21) assigns a unique ID to each token, which can be used as input to the embedding layer. This tokenization process enables the structuring and efficient processing of text information.

[0163] Token IDs generated by the tokenizer (21) are provided as input to the embedding layer, which generates text embeddings corresponding to each token. The embedding layer maps high-dimensional, sparse token IDs to a low-dimensional, dense, continuous vector space, and learns to embed tokens with similar meanings in this vector space in close proximity.

[0164] This text embedding provides the foundation necessary for a language model to understand the semantic, grammatical, and contextual information of the text. In the first learning stage, the text embedding module (20) receives caption text (CXT) as input and generates caption text embeddings. In the second learning stage and inference stage, it receives question text (QXT) as input and generates question text embeddings.

[0165] As a result, the text embedding module (20) can convert various forms of high-resolution image-related text (TXT) into a standardized text embedding form that the vision-language embedding generation module (30) can effectively process and understand.

[0166] The vision-language embedding generation module (30) performs the role of generating a vision-language embedding by connecting the integrated vision embedding generated from the vision encoding module (10) and the text embedding generated from the text embedding module (20). The vision-language embedding generation module (30) may be configured to form an integrated representation space so that the model can understand and process the relationship between visual information and text information.

[0167] The vision-language embedding generation module (30) can generate vision-language embeddings by concatenating integrated vision embeddings and text embeddings. This concatenation allows the large-scale language model (41) to understand the relationship between image embeddings and text embeddings. For example, the integrated vision embeddings can be expressed as a series of vision tokens, and the text embeddings can be expressed as a series of text tokens. The vision-language embedding generation module (30) can combine these two types of token sequences to generate a new integrated embedding sequence that can be used as input to the large-scale language model (41).

[0168] These vision-language embeddings are then provided as input to a large language model (41). The large language model (41) can simultaneously understand image and text information through this integrated vision-language embedding, and perform various tasks, such as generating answers to questions or generating captions based on that information.

[0169] As a result, the vision-language embedding generation module (30) can effectively fuse information from different modalities (vision and language) to provide a basis for a large language model (41) to comprehensively understand complex high-resolution images and related text contexts.

[0170] The vision language embedding processing module (40) can receive as input a multimodal embedding (i.e., vision-language embedding) generated from the vision-language embedding generation module (30) and provide it to a large language model (41).

[0171] The vision language embedding processing module (40) is a core part of the learned multimodal large-scale language model and is responsible for generating diagnostic response data (DR) based on integrated vision-language information.

[0172] A large language model (41) provided with a vision-language embedding from a vision-language embedding processing module (40) can generate a diagnostic response to a complex question related to a high-resolution image (HRI) through this vision-language embedding.

[0173] The vision-language embedding processing module (40) may include a self-built large-scale language model or may utilize a publicly available large-scale language model provided externally.

[0174] Referring to FIG. 6, the giant language model (41) is trained to output captioning text (CXT) based on the input vision-language embedding, and in this process, the parameters of the giant language model (41) are frozen, and the parameters of the vision encoding module (10) can be updated.

[0175] For example, the parameters updated within the vision encoding module (10) may be parameters of the patch integration layer (12) and the vision projector (13) excluding the patch encoder (11).

[0176] The patch encoder (11) is a pre-trained patch-level foundation model, and its parameters can be fixed because it acts as a robust feature extractor that extracts patch-level features from a high-resolution image (HRI) at this stage.

[0177] The patch integration layer (12) can be implemented as a CLAM-based patch aggregator and includes a Gated Attention Network (GATN). The patch integration layer (12) integrates multiple patch embeddings into a single high-resolution image embedding.

[0178] In the patch integration layer (12), the weight and bias parameters of the linear layer within the GATN can be updated. In addition, the internal linear layer weights and biases of the two attention modules using the Tanh (hyperbolic tangent) and sigmoid activation functions can also be updated.

[0179] The vision projector (13) projects a single high-resolution image embedding from the patch integration layer (12) into the text embedding space of the language model.

[0180] In the vision projector (13), linear layer weight and bias parameters (e.g., weights of the attention pooler, weights and bias of the linear projector) can be updated.

[0181] Based on the output of the fixed patch encoder (11) in this way, the parameters of the patch integration layer (12) and the vision projector (13) are updated so that visual information can be effectively understood by the large language model (41) and converted into a form that can be aligned with text information, and this process can be referred to as first learning.

[0182] Referring to FIG. 7, after the first learning is completed, the giant language model (41) is trained to output a response text (RXT) to a question text (QXT) based on the input vision-language embedding, and in this process, both the parameters of the giant language model (41) and the parameters of the vision encoding module (10) may be updated. This process is referred to as the second learning, and can enhance the model's ability to generate accurate and contextual responses to actual user queries based on the visual-language representations aligned through the first learning.

[0183] Secondary learning can be performed using a large, high-quality question-answering dataset. This dataset is built from original documents (e.g., pathology reports) associated with high-resolution images (HRI), and can be efficiently generated using a document analysis-based data generation pipeline.

[0184] For example, referring to Figure 8, the document analysis-based data generation pipeline first converts information contained in a source document, such as a pathology report, into plain text via OCR (Optical Character Recognition). The converted text can be stored and managed in a text vector database.

[0185] Thereafter, when a question text (QXT) is input, a response generation module (60) that can be included in a computing device (400) searches for at least one text chunk (TXT) among a plurality of text chunks (TXTs) in a text vector DB, wherein the semantic similarity (e.g., cosine distance) to the question text (QXT) is greater than a predetermined value. The response generation module (60) can utilize the text chunks (TXTs) searched for in this way as a context to cause a large language model (41) to generate a response text (RXT).

[0186] In this process, the response generation module (60) can additionally input a system prompt (SPT) into the large language model (41) to improve the accuracy and quality of response text (RXT) generation. Here, the system prompt (SPT) can provide detailed context and guidance to the large language model (41) to help generate an accurate answer based on the given searched text.

[0187] For example, a system prompt (SPT) can assign a specific role (e.g., pathology lab assistant) to the large language model (41) or present specific rules or constraints to follow when generating answers (e.g., extracting a diagnosis summary of 10 words or less, prohibiting mention of measurements or numbers, or basing answers solely on the provided context). By providing appropriate guidance to the large language model (41) in this way, the understanding and generation performance of the large language model (41) can be improved.

[0188] Question-response pairs, where question text (QXT) and generated response text (RXT) are paired, can be assembled to form a question-response dataset. This process addresses the heterogeneous information problem caused by the inherent irregularity and information diversity of existing documents. It also mitigates the risk of hallucination in large language models by selectively retrieving relevant information, thereby providing a more accurate and robust training dataset.

[0189] Meanwhile, the process of generating question text (QXT) for question-answer pairs can be accomplished in various ways. For example, the large language model (41) itself can be guided to understand the content of the original report text and generate questions based on that content. This means that the large language model (41) can be utilized not only to generate answers but also as a question generation tool for building datasets.

[0190] Additionally, it's possible to acquire a large number of consistent questions by leveraging predefined question templates or types tailored to specific tasks. Furthermore, domain experts can directly create actual clinical / industrial questions to guide the question generation capabilities of the large-scale language model (41), or they can leverage existing publicly available Q&A datasets to expand the question set.

[0191] In the second learning step, the integrated vision embedding generated from the vision encoding module (10) and the question embedding corresponding to the question text (QXT) generated from the text embedding module (20) are connected by the vision-language embedding generation module (30) to generate a second vision-language embedding. The second vision-language embedding is provided as an input to the large language model (41). At this time, the large language model (41) is trained to output a response text (RXT) for the question text (QXT) based on the input multimodal information.

[0192] In particular, parameter updates of the giant language model (41) can be fine-tuned using the Low-Rank Adaptation (LoRA) technique. LoRA improves computational efficiency and reduces the risk of overfitting by updating only parameters of specific parts of the model, such as linear projection layers (e.g., query, key, and output projection layers), rather than updating all parameters of the giant language model (41). This method effectively enhances the ability of the giant language model (41) to comprehensively understand the visual features of high-resolution images and the context of the question, and to generate accurate and context-appropriate diagnostic responses to various queries. The vision encoding module (10) can also have all its parameters updated at this stage, further optimizing the alignment of its visual information processing capabilities with the requirements of the giant language model (41).

[0193] Through this second learning process, the high-resolution image analysis model, which combines the vision encoding module (10) and the large language model (41), can acquire goal-oriented conversational abilities to respond to various task requests from actual users and perform the corresponding tasks based on the visual-language alignment ability acquired in the first learning.

[0194] After the first and second learning are completed, the high-resolution image analysis model combined with the vision encoding module (10) and the large language model (41) can perform inference on actual user input.

[0195] This inference is a process of utilizing a trained high-resolution image analysis model in a real-world environment to provide a diagnostic response to a user's query. During this inference process, a user can provide a high-resolution image (HRI) to be analyzed and a question text related thereto to a computing device (400). The computing device (400) receives these inputs and, through first and second learning, can generate a diagnostic response using an integrated high-resolution image analysis module (180) that combines a vision encoding module (10) and a large language model (41).

[0196] The integrated high-resolution image analysis module (180) can generate an integrated vision embedding using the vision encoding module (10) based on the received high-resolution image to be analyzed, and can generate a question embedding based on the received question text using the text embedding module (20). Thereafter, the vision-language embedding generation module (30) connects these embeddings to form a vision-language embedding, and inputs it into a large language model (41) through the vision-language embedding processing module (40). The large language model (41) can generate diagnostic response data (DR) for the query based on the vision-language embedding. This diagnostic response data (DR) can be utilized as final diagnostic response data (FDR) for the final diagnostic response that is ultimately provided to the user.

[0197] However, it is not limited to this, and diagnostic response data (DR) that has passed a predetermined standard by a response quality evaluation module (50) that performs a quality evaluation on diagnostic response data (DR) can be utilized as final diagnostic response data (FDR) for the final diagnostic response that is finally provided to the user.

[0198] Referring to Figures 9 and 10, the evaluation process can be applied differently in the model development and verification phase and the actual inference phase.

[0199] FIG. 9 is a diagram illustrating a method for evaluating the performance of a high-resolution image analysis model according to one embodiment.

[0200] Figure 9 illustrates how AI-based evaluation is performed during the model learning and validation stages. At this stage, correct data (Ground Truth Answers) for diagnostic responses are provided, and evaluation can be performed by comparing the correct data with the responses generated by the high-resolution image analysis model.

[0201] For example, the response quality evaluation module (50) can use an AI evaluation model (Evaluator) to evaluate the response generated by the high-resolution image analysis model based on seven evaluation criteria, such as how closely it matches the correct data (accuracy), whether it is relevant to the question (relevance), and whether it is comprehensive (completeness). This evaluation is intended to objectively measure the model's performance and derive future improvement directions.

[0202] FIG. 10 is a method for evaluating the performance of a high-resolution image analysis model according to another embodiment.

[0203] Figure 10 illustrates how the response quality assessment module (50) operates in the actual inference phase. In this phase, a high-resolution image analysis model receives a high-resolution image (HRI) and a question (Question) from the user and generates a diagnostic response. The high-resolution image analysis model trained in this process can generate multiple diagnostic responses (Answer 1 to Answer 10) for a single question text.

[0204] The response quality evaluation module (50) selects the optimal response from among the multiple diagnostic responses generated in this manner. This selection process can be performed by calculating an evaluation score for each response based on multimodal information, such as the first similarity between each diagnostic response and the question text, and the second similarity between each diagnostic response and the integrated vision embedding. The response quality evaluation module (50) can initially select the diagnostic response with the highest similarity score as the "optimal diagnostic response."

[0205] Thereafter, the response quality assessment module (50) may provide a Chain-of-Thought (CoT) prompt to the large language model-based assessment module to perform inference to determine whether the selected optimal diagnostic response will ultimately be provided to the user. This CoT prompt includes evaluation criteria information, requires the module to perform inference based on each evaluation criterion, explain the inference process, and, based on this inference process, make a final decision to either accept or reject the selected diagnostic response to ultimately be provided to the user.

[0206] The evaluation model based on the large language model then proceeds through this inference process to determine whether the optimal diagnostic response is acceptable and provide justification for it. This CoT prompt-based inference induction process not only provides a clear rationale for the evaluation results, enhancing user (expert) confidence in the judgments of the AI-based evaluation system. Furthermore, in the case of histopathological image analysis, it can help AI evaluation modules, lacking medical expertise, accurately capture critical evaluation factors in image analysis results. Ultimately, diagnostic responses that pass the quality assessment can be provided to users as final diagnostic response data (FDR).

[0207]

[0208] - High-resolution image analysis model learning method (S100)

[0209] The method for training a high-resolution image analysis model (S100) presents a comprehensive method for training a multimodal giant language model (LLM) that can perform in-depth analysis based on high-resolution images (HRI) and generate diagnostic responses.

[0210] Through the first learning stage (first learning), the model can acquire the ability to align the visual features of high-resolution images (HRI) with captioning information. This overcomes the limitations of existing patch-level MLLMs, and enables the vision encoding module (10) to generate an integrated visual embedding from the high-resolution image (HRI) and align it to the text embedding space of the large-scale language model (41), thereby enabling the large-scale language model (41) to process and understand comprehensive visual and textual information of the high-resolution image (HRI) together. This can significantly improve the contextual understanding at the high-resolution image level.

[0211] The second training phase (second training) enhances the model's ability to generate responses based on real questions. This phase utilizes a high-quality question-answering dataset, which pairs question and response texts with high-resolution human-reference (HRI) images. This addresses the inherent heterogeneity of information in the original documents associated with high-resolution images (HRI) and mitigates the risk of hallucination in large language models, providing a more accurate and robust training dataset. Command tuning based on the question-answering dataset builds on the visual-verbal alignment capabilities acquired in the first training phase, enabling the model to respond to diverse user requests and perform these tasks with goal-directed conversational skills, generating accurate and context-appropriate diagnostic responses.

[0212] Fig. 11 is a flowchart of a high-resolution image analysis model learning method (S100) according to one embodiment.

[0213] Referring to FIG. 11, a method for learning a high-resolution image analysis model (S100) according to one embodiment includes a step (S101) of generating an integrated vision embedding from a high-resolution image (HRI) using a vision encoding module (10), a step (S103) of generating a first vision-language embedding by connecting the integrated vision embedding and captioning text embedding corresponding to captioning information for the high-resolution image (HRI), a first learning step (S105) of training a large language model (41) to output captioning information using the first vision-language embedding as input, a step (S107) of generating a question-response dataset in which question text and response text related to the high-resolution image (HRI) are matched, a step (S109) of generating a second vision-language embedding by connecting the integrated vision embedding and the question embedding corresponding to the question text, and a second learning step of training a large language model (41) to output response text using the second vision-language embedding as input. It may include step (S111).

[0214] In one embodiment, the method (S100) may be performed by a processor (131) included in a server computing system (130). However, the present invention is not limited thereto, and at least a part of the method (S100) may be performed by a processor (111) of a user computing device (110) or a processor (151) of a training computing system (150), and another part may be performed by a processor (131) included in the server computing system (130).

[0215] For convenience of explanation, the method (S100) is described below as being performed by a processor (131) included in a server computing system (130).

[0216] In step (S101), the processor (131) can generate an integrated vision embedding from a high-resolution image (HRI) using the vision encoding module (10).

[0217] The vision encoding module (10) receives a high-resolution image (HRI) as input and converts it into an integrated vision embedding that can be processed by a language model. The vision encoding module (10) can efficiently process the complexity and massive amount of information of a high-resolution image and accurately encode key visual features within the image. The vision encoding module (10) can largely include a patching layer, a patch encoder (11), a patch integration layer (12), and a vision projector (13).

[0218] Referring to FIG. 12, the processor (131) may perform, in step (S101), a step of dividing a high-resolution image (HRI) into a plurality of patches using a vision encoding module (10), a step of generating a plurality of patch embeddings from the plurality of patches (S1013), a step of integrating at least a portion of the plurality of patch embeddings to generate a single high-resolution image embedding (S1015), and a step of adjusting a dimension of the single high-resolution image embedding to generate an integrated vision embedding (S1017).

[0219] The patching layer can split the input high-resolution image (HRI) into multiple patches (S1011). This is to efficiently process the vast pixel information of the original high-resolution image and to match the input format of the deep learning model.

[0220] A plurality of patch embeddings can be generated from a plurality of patches of a patch encoder (11) (S1013). The patch encoder (11) learns complex image patterns and structures and expresses them in the form of compressed vectors, and may include, for example, a pre-trained, robust patch-level foundation model based on a ViT (Vision Transformer) structure. This patch encoder (11) functions as a patch-level feature extractor, and its parameters can be frozen.

[0221] The patch integration layer (12) can integrate multiple patch embeddings generated from the patch encoder (11) to generate a single high-resolution image embedding (S1015). The patch integration layer (12) can be implemented as an aggregator based on CLAM (Clustering-constrained-attention Multiple-Instance Learning) and can calculate weights according to the importance of each patch embedding by utilizing attention-based learning. The patch integration layer (12) can generate a single high-resolution image embedding by calculating a weighted sum for multiple patch embeddings based on the calculated multiple weights.

[0222] In this case, instead of calculating the weighted sum for all of the multiple patch embeddings using all of the weights calculated for the multiple patch embeddings, the patch integration layer (12) can perform the weighted sum by selecting only patch embeddings with weights above a predetermined threshold. This approach can induce the model to focus on key regions within the image that are judged to play a more important role, rather than considering the entire image.

[0223] Through this weight-based aggregation process, the patch integration layer (12) can integrate the massive patch embeddings while giving greater importance to the most essential and diagnostically meaningful information from the massive patch embeddings, thereby generating a single concise and meaningful embedding that represents the complex features of the entire high-resolution image.

[0224] The vision projector (13) can generate an integrated vision embedding aligned to the text embedding space of the giant language model (41) based on a single high-resolution image embedding, which is the output of the patch integration layer (12) (S1017).

[0225] For example, the vision projector (13) can use self-attention to weight and stabilize important feature elements within a single high-resolution image embedding, and then expand and project them to fit the text embedding space dimension (e.g., 4096 dimensions) of the language model, so that visual information can be smoothly integrated into the text space.

[0226] In step (S103), the processor (131) can generate a first vision-language embedding by connecting the integrated vision embedding and the captioning text embedding corresponding to the captioning information for the high-resolution image (HRI).

[0227] The processor (131) performs step (S103) using the vision-language embedding generation module (30), and can form an integrated expression space so that the large language model (41) can understand and process the relationship between visual information and text information. The vision-language embedding generation module (30) can generate a vision-language embedding by concatenating the integrated vision embedding and the captioning text embedding.

[0228] In step (S105), the processor (131) can train a large language model (LLM) to output captioning information using the first vision-language embedding as input. This training process may be referred to as first training.

[0229] Vision-language alignment can be achieved through the first learning process. Through step (S105), the visual and textual representations generated by the vision encoding module (10) and the text embedding module (20) can be aligned so that the large language model (41) can understand them in a single, unified representation space.

[0230] The giant language model (41) is trained to generate captioning information in an auto-regressive manner based on this first vision-language embedding.

[0231] In the first learning stage, the parameters of the giant language model (41) are frozen while the vision encoding module (10) and the giant language model (41) are combined, and in contrast, the parameters of the vision encoding module (10) (e.g., weight and bias parameters in the patch integration layer (12) and vision projector (13)) can be updated.

[0232] In step (S107), the processor (131) can generate a question-response dataset in which question text and response text related to high-resolution images (HRI) are paired.

[0233] The processor (131) can build a question-answer dataset using a document analysis-based data generation pipeline. The document analysis-based data generation pipeline can convert an original document into OCR, build a text vector DB, and then search for text chunks (TXTs) with high similarity to the question text (QXT).

[0234] The processor (131) can cause the giant language model (41) to generate response text (RXT) by utilizing the searched text chunks (TXTs) as context. In this process, a system prompt (SPT) can be additionally input into the giant language model (41) to improve the accuracy and quality of response generation.

[0235] The question-answer pairs generated in this way address the heterogeneous information in existing documents and mitigate the risk of hallucination in large language models, thereby providing a more accurate and robust training dataset. Here, the question text (QXT) can be generated by the large language model (41) itself based on the original report text, or can be formed through a predefined question template or by a domain expert.

[0236] In step (S109), the processor (131) can generate a second vision-language embedding by connecting the integrated vision embedding and the question embedding corresponding to the question text (QXT).

[0237] The processor (131) can connect the integrated vision embedding generated from the high-resolution image (HRI) through the vision encoding module (10) and the question embedding for the question text (QXT) generated through the text embedding module (20). Through this, a vision-language embedding that allows the large language model (41) to simultaneously understand visual information and text information can be generated. The second vision-language embedding is then input into the large language model (41) and used to train the model to output a response text (RXT) for the question text (QXT).

[0238] In step (S111), the processor (131) can train the large language model (41) to output response text (RXT) using the second vision-language embedding as input. This learning process may be referred to as second learning.

[0239] Second learning can enhance the ability of the large language model (41) to generate accurate and contextual responses to actual user queries based on the visual-linguistic representations aligned through first learning.

[0240] In the second learning phase, the parameters of the giant language model (41) can be fine-tuned using the Low-Rank Adaptation (LoRA) method. Instead of updating all parameters of the giant language model, LoRA only updates parameters in specific parts of the model, such as linear projection layers (e.g., query, key, and output projection layers), thereby improving computational efficiency and reducing the risk of overfitting.

[0241] Through this method, the large language model (41) can effectively enhance its ability to comprehensively understand the visual features of high-resolution images and the context of questions, and generate accurate and context-appropriate diagnostic responses to various queries. The parameters of the vision encoding module (10) can also be updated in step (S111), further optimizing the alignment of visual information processing capabilities with the requirements of the large language model (41).

[0242]

[0243] -Method for providing diagnostic responses based on high-resolution image analysis (S200)

[0244] A method for providing a diagnostic response based on high-resolution image analysis (S200) proposes a method for providing a diagnostic response using a multimodal giant language model (LLM) that can perform in-depth analysis based on high-resolution images (HRI) and generate a diagnostic response therefor.

[0245] Method (S200) utilizes a pre-trained model built through the first and second learning processes. This model is capable of aligning visual features and textual information in high-resolution images (HRI) and generating question-based responses. The trained model receives a high-resolution image to be analyzed and a related question from the user, and generates a diagnostic response. During this process, the model can generate multiple diagnostic response candidates for a single question.

[0246] A response quality assessment module (50) can be utilized to evaluate the quality of the generated diagnostic responses and determine the optimal response. This module (50) selects the optimal response from multiple candidate responses, evaluates the quality of the response, and determines whether to provide the final response. The assessment process utilizes multimodal information, such as calculating an assessment score based on similarity between the question text and the integrated vision embedding, to enable objective and quantitative judgment. Finally, the large-scale language model-based assessment module can significantly enhance the reliability and transparency of the diagnostic responses provided to users by providing detailed reasoning processes and final decisions through Chain-of-Thought (CoT) prompts.

[0247] Fig. 13 is a flowchart of a method (S200) for providing a diagnostic response based on high-resolution image analysis according to one embodiment.

[0248] Referring to FIG. 13, a method for providing a high-resolution image analysis-based diagnostic response according to one embodiment (S200) includes a first learning step (S201) in which a vision encoding module (10) trains a large language model (41) to output captioning information by inputting a first vision-language embedding that connects an integrated vision embedding generated from a high-resolution image (HRI) and a captioning text embedding corresponding to captioning information for the high-resolution image (HRI), a second learning step (S203) in which a large language model (41) trains a second vision-language embedding that connects a question embedding corresponding to a question text related to the high-resolution image (HRI) and the integrated vision embedding generated from a high-resolution image (HRI) and a captioning text embedding corresponding to captioning information for the high-resolution image (HRI) by inputting a second vision-language embedding that connects an integrated vision embedding and a question embedding corresponding to a question text related to the high-resolution image (HRI), a step (S205) in which a high-resolution image to be analyzed is received according to a user input, a step (S207) in which a question text related to the high-resolution image to be analyzed is received, and an integrated high-resolution image in which the vision encoding module (10) and the large language model (41) are combined, in which the first and second learnings are completed. Using the analysis module (180), a step (S209) of generating a diagnostic response based on the analysis target high-resolution image and question text may be included.

[0249] In one embodiment, the method (S200) may be performed by a processor (131) included in a server computing system (130). However, the present invention is not limited thereto, and at least a part of the method (S200) may be performed by a processor (111) of a user computing device (110) or a processor (151) of a training computing system (150), and another part may be performed by a processor (131) included in the server computing system (130).

[0250] For convenience of explanation, the method (S200) is described below as being performed by a processor (131) included in a server computing system (130).

[0251] Steps (S201) and (S203) are substantially the same as steps (S101) to (S111) of FIG. 11, so descriptions of steps (S201) and (S203) are omitted here.

[0252] In step (S205), the processor (131) can receive a high-resolution image (HRI) to be analyzed according to user input.

[0253] For example, a user may provide a high-resolution image (HRI) requiring analysis and diagnosis to a server computing system (130) via a user computing device (110). This high-resolution image (HRI) may be a tissue image for pathological diagnosis, or may be a variety of high-resolution images, such as a wafer image for inspecting the quality of a product in an industrial setting.

[0254] In step (S207), the processor (131) can receive a question text related to the high-resolution image (HRI) to be analyzed.

[0255] In step (S207), the user can provide the system (130) with textual information asking questions or requesting analysis regarding the high-resolution image (HRI) received in step (S205). The textual inquiry can take various forms, such as a general description request for the image ("What does this image show?"), a request for identifying specific features ("Do you see any defects in this image?"), or a diagnosis request ("What type of cancer is this tissue?").

[0256] In step (S209), the processor (131) can generate a diagnostic response based on the high-resolution image to be analyzed and the question text using the integrated high-resolution image analysis module (180) in which the vision encoding module (10) and the large language model (41) are combined, in which the first and second learnings have been completed.

[0257] Step (S209) is a core inference process of the method (S200) for providing a diagnostic response based on high-resolution image analysis. The processor (131) inputs the analysis target high-resolution image (HRI) received in step (S205) and the question text received in step (S207) into the integrated high-resolution image analysis module (180).

[0258] The integrated high-resolution image analysis module (180) converts the high-resolution image (HRI) input through the vision encoding module (10) into a visual embedding, and converts the received question text into a question embedding through the text embedding module (20). Thereafter, the vision-language embedding generation module (30) connects these embeddings to form a vision-language embedding, which is then input into a large language model (41) through the vision-language embedding processing module (40).

[0259] The large-scale language model (41) can generate diagnostic response data (DR) for a given query by analyzing the input vision-language embedding based on its comprehensive understanding of high-resolution images acquired through the first learning (visual-language alignment) and second learning (command tuning), and its ability to generate responses to questions. This diagnostic response can be provided in a form that matches the user's query intent, such as pathological diagnosis, defect identification, or quality assessment.

[0260] Meanwhile, referring to FIG. 14, after the diagnostic response is output, a response quality evaluation may be performed, including a step of calculating an evaluation score for each of a plurality of diagnostic responses generated by the integrated high-resolution image analysis module using the response quality evaluation module (50) (S211), a step of selecting a diagnostic response with the highest evaluation score among the plurality of diagnostic responses (S213), a step of providing a chain of thought (CoT) prompt for determining whether to provide the selected diagnostic response to the user in the large language model-based evaluation module (S215), and a step of obtaining information on whether to provide the single diagnostic response output from the large language model-based evaluation module and the basis therefor (S217).

[0261] In step (S211), the processor (131) can calculate an evaluation score for each of a plurality of diagnostic responses generated by the integrated high-resolution image analysis model using the response quality evaluation module (50).

[0262] Step (S211) is a process of quantitatively evaluating the quality of multiple diagnostic response candidates (e.g., Answer 1 to Answer 10) generated by the integrated high-resolution image analysis model.

[0263] For example, the response quality evaluation module (50) can calculate a first similarity between each diagnostic response text embedding and a question text embedding, calculate a second similarity between each diagnostic response text embedding and an integrated vision embedding, and then comprehensively calculate an evaluation score for each of a plurality of diagnostic responses based on the first and second similarities.

[0264] In step (S213), the processor (131) may select the diagnostic response with the highest evaluation score from among multiple diagnostic responses. This is a process of comparing the evaluation scores of each diagnostic response calculated in step (S211) and selecting the response with the highest score as the optimal response candidate for the question.

[0265] In step (S215), the processor (131) may provide a Chain-of-Thought (CoT) prompt to cause the large language model-based evaluation model (Evaluator) to perform inference to determine whether to provide the selected diagnostic response to the user.

[0266] This CoT prompt may be configured to include at least one evaluation criterion information for determining whether to provide a selected diagnostic response to the user, request a large language model-based evaluation model (Evaluator) to perform inference based on the at least one evaluation criterion, and request the large language model-based evaluation model (Evaluator) to make a final decision on whether to provide the selected diagnostic response based on the performed inference result.

[0267] For example, a CoT prompt can be configured to present evaluation criteria (e.g., accuracy, relevance, completeness, clarity, appropriateness, consistency, and expression) to a large language model-based evaluation model, ask it to describe the detailed reasoning process for each evaluation criterion, and finally make a decision to accept or reject the optimal diagnostic response based on this reasoning process.

[0268] In step (S217), the processor (131) can obtain information regarding whether to provide (accept or reject) a diagnostic response output from a large language model-based evaluation model and the basis thereof. In step (S217), the evaluation model presents the inference process performed according to the CoT prompt and the final decision (accept or reject), and through this information, reliability of the quality of the corresponding diagnostic response can be secured and the final diagnostic response data (FDR) to be provided to the user can be determined.

[0269] The various embodiments of the present disclosure described above may be implemented in the form of program instructions that can be executed by various computer components and recorded on a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc., either singly or in combination. The program instructions recorded on the computer-readable recording medium may be specially designed and configured for the various embodiments of the present disclosure or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specifically configured to store and execute program instructions, such as ROMs, RAMs, and flash memories. Examples of program instructions include not only machine language codes generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc. Hardware devices may be modified into one or more software modules to perform processing according to the various embodiments of the present disclosure, and vice versa.

[0270] The specific implementations described in this disclosure are illustrative and do not limit the scope of various embodiments of the present disclosure in any way. For the sake of brevity of the specification, descriptions of conventional electronic components, control systems, software, and other functional aspects of the systems may be omitted. In addition, the lines connecting or connecting members between components depicted in the drawings are merely illustrative of functional connections and / or physical or circuit connections, and may be replaced or represented as various additional functional connections, physical connections, or circuit connections in an actual device. In addition, unless specifically mentioned as “essential,” “important,” etc., a component may not be absolutely necessary for the application of various embodiments of the present disclosure.

[0271] Although the detailed description of the present disclosure has been described with reference to preferred embodiments of the present disclosure, it will be understood by those skilled in the art or having ordinary knowledge in the art that various modifications and changes can be made to the various embodiments of the present disclosure without departing from the spirit and technical scope of the various embodiments of the present disclosure as set forth in the claims below. Accordingly, the technical scope of the various embodiments of the present disclosure should not be limited to the contents described in the detailed description of the specification, but should be defined by the claims.

[0272]

[0273]

[0274] The method and system for providing a diagnostic response based on high-resolution image analysis of the present disclosure have industrial applicability in that they generate an integrated vision embedding from a high-resolution image through a vision encoding module, align it to the text embedding space of a large language model (LLM), and enable the LLM to process comprehensive visual information and text information of the entire high-resolution image together, thereby improving contextual understanding at the high-resolution image level and generating an accurate diagnostic response for the high-resolution image based on the integrated vision embedding.

Claims

1. A method for learning a high-resolution image analysis model performed by a computing device, A step of generating an integrated vision embedding from a high-resolution image using a vision encoding module; A step of generating a first vision-language embedding by connecting the integrated vision embedding and the captioning text embedding corresponding to the captioning information for the high-resolution image; A first learning step for training a large language model (LLM) to output the captioning information using the first vision-language embedding as input; A step of generating a question-response dataset in which question text and response text related to the above high-resolution image are matched; A step of generating a second vision-language embedding by connecting the integrated vision embedding and the question embedding corresponding to the question text; and A method for training a high-resolution image analysis model, comprising: a second training step for training the large language model to output the response text using the second vision-language embedding as input; 2. In paragraph 1, In the first learning step above, A method for learning a high-resolution image analysis model, wherein the parameters of the giant language model are fixed and the parameters of the vision encoding module are updated, while the vision encoding module and the giant language model are combined.

3. In paragraph 1, In the second learning step above, A method for learning a high-resolution image analysis model, wherein the parameters of the giant language model and the parameters of the vision encoding module are both updated while the vision encoding module and the giant language model are combined.

4. In paragraph 3, In the second learning step above, A method for learning a high-resolution image analysis model, in which the parameters of the above-mentioned large language model are updated using the LoRA (Low-Rank Adaptation) method.

5. In paragraph 1, The steps of generating the above integrated vision embedding are: A step of dividing the above high-resolution image into multiple patches; A step of generating multiple patch embeddings from the multiple patches; generating a single high-resolution image embedding by integrating at least some of the plurality of patch embeddings; and A method for learning a high-resolution image analysis model, comprising: a step of adjusting the dimension of the single high-resolution image embedding to generate the integrated vision embedding; 6. In paragraph 5, The step of generating the above single high-resolution image embedding is: A step of calculating multiple weights according to the importance of each of the multiple patch embeddings through an attention operation on the multiple patch embeddings; and A method for learning a high-resolution image analysis model, comprising: generating a single high-resolution image embedding by calculating a weighted sum for at least some of the plurality of patch embeddings based on the plurality of weights; 7. In paragraph 1, A method for learning a high-resolution image analysis model, wherein the high-resolution image includes a tissue pathology image.

8. In paragraph 1, The steps for creating the above question-answer dataset are: A step of extracting a plurality of texts related to the high-resolution image from a document related to the high-resolution image; A step of constructing a text vector database based on a plurality of texts related to the above high-resolution image; A step of calculating the similarity between a question text related to the high-resolution image and a plurality of texts in the text vector database; A step of generating the response text using a large language model based on at least one text among the plurality of texts having a similarity to the question text greater than a predetermined value; and A method for learning a high-resolution image analysis model, comprising: a step of generating the question-response dataset by matching the question text and the response text to generate question-response pairs; 9. In paragraph 8, The step of generating the response text using the above large language model is: providing a system prompt for generating the response text to the large language model; and A method for training a high-resolution image analysis model, comprising: generating the response text using the large language model based on the question text, the at least one text, and the system prompt.

10. A high-resolution image analysis method using a large language model performed by a computing device, A first learning step in which a large language model (LLM) is trained to output captioning information by inputting a first vision-language embedding that connects a unified vision embedding generated from a high-resolution image by a vision encoding module and a captioning text embedding corresponding to captioning information for the high-resolution image; A second learning step for training the large language model to output a response text for the question text by inputting a second vision-language embedding that connects the integrated vision embedding and the question embedding corresponding to the question text related to the high-resolution image; A step of receiving a high-resolution image to be analyzed according to user input; A step of receiving a question text related to the high-resolution image to be analyzed; and A method for providing a diagnostic response based on high-resolution image analysis, comprising: a step of generating a diagnostic response based on the high-resolution image to be analyzed and the question text using an integrated high-resolution image analysis module in which the vision encoding module and the large language model are combined, after the first and second learnings have been completed; 11. In paragraph 10, The step of generating the above diagnostic response is: A step of generating multiple diagnostic responses to the question text using the integrated high-resolution image analysis module; A step of calculating an evaluation score for each of the plurality of diagnostic responses using a quality evaluation module; and A method for providing a high-resolution image analysis-based diagnostic response, further comprising: selecting a diagnostic response having the highest evaluation score among the plurality of diagnostic responses.

12. In paragraph 11, The step of calculating an evaluation score for each of the above multiple diagnostic responses is: A step of calculating a first similarity for each of the plurality of diagnostic response text embeddings corresponding to the plurality of diagnostic responses to the question text; A step of calculating a second similarity for the integrated vision embedding of each of the plurality of diagnostic response text embeddings; and A method for providing a high-resolution image analysis-based diagnostic response, comprising: a step of calculating an evaluation score for each of the plurality of diagnostic responses based on the first similarity and the second similarity; 13. In paragraph 11, A step of providing a Chain-of-Thought (CoT) prompt to a large language model-based evaluation module to perform inference to determine whether to provide the selected diagnostic response to the user; and A method for providing a high-resolution image analysis-based diagnostic response, further comprising: a step of obtaining information regarding whether the diagnostic response output from the large language model-based evaluation module is provided and the basis therefor; 14. In paragraph 13, The above CoT prompt is, A method for providing a high-resolution image analysis-based diagnostic response, comprising at least one evaluation criterion information for determining whether to provide the selected diagnostic response to the user, requesting to perform inference based on the at least one evaluation criterion, and requesting to determine whether to provide the selected diagnostic response based on the performed inference result.

15. At least one memory; and At least one processor for executing a high-resolution image analysis method by reading at least one command stored in the memory; At least one processor, Generate integrated vision embeddings from high-resolution images using the vision encoding module, Generating a first vision-language embedding by concatenating the integrated vision embedding and the captioning text embedding corresponding to the captioning information for the high-resolution image, Perform a first learning to train a large language model (LLM) to output the captioning information using the first vision-language embedding as input, Create a question-answer dataset with matching question text and response text related to the above high-resolution image, A second vision-language embedding is generated by concatenating the integrated vision embedding and the question embedding corresponding to the question text, Perform a second learning to train the large language model to output the response text using the second vision-language embedding as input, Receive high-resolution images to be analyzed based on user input, Receive a question text related to the high-resolution image to be analyzed above, A high-resolution image analysis-based diagnostic response providing system that generates a diagnostic response based on the high-resolution image to be analyzed and the question text by using an integrated high-resolution image analysis module in which the vision encoding module and the large language model are combined, after the first and second learnings have been completed.

16. In paragraph 15, A plurality of neurons comprising an array including at least one register, at least one programmable logic, and at least one input interface; a plurality of synaptic circuits storing synaptic weights that adjust connection strengths between the plurality of neurons; and at least one routing network that controls data flow between the plurality of neurons; A high-resolution image analysis-based diagnostic response providing system, wherein each of the plurality of neurons further includes a field programmable gate array (FPGA) implementation for a predetermined artificial neural network that is connected to at least one other neuron through the routing network to set a transmission path of the weight.

17. In paragraph 15, A plurality of neurons organized in an array including at least one register, at least one microprocessor, and at least one input; and a plurality of synaptic circuits storing synaptic weights that adjust connection strengths between the plurality of neurons; A high-resolution image analysis-based diagnostic response providing system, wherein each of the plurality of neurons further includes an application-specific integrated circuit (ASIC) for a predetermined artificial neural network connected to at least one other neuron through one of the plurality of synaptic circuits.

Images