Method for training high-resolution image analysis model on basis of patch selection using prototype, and method and system for providing diagnostic response on basis of high-resolution image analysis

Abstract

One embodiment provides a method for training an image analysis model, comprising the steps of: generating a plurality of patch embeddings from an image; calculating similarity between a plurality of prototype embeddings and the plurality of patch embeddings; selecting a portion of the plurality of patch embeddings on the basis of the similarity for each prototype embedding; calculating a prediction value on the basis of the plurality of selected patch embeddings; and updating parameters of an image analysis model so that a loss based on the prediction value and correct answer data is minimized.

Description

A method for learning a high-resolution image analysis model based on patch selection by prototype, and a method and system for providing diagnostic responses based on high-resolution image analysis.

[0001] The present disclosure relates to a method for learning a high-resolution image analysis model based on patch selection by artificial intelligence and prototypes, and a method and system for providing a diagnostic response based on high-resolution image analysis, and more particularly, to a method for learning a high-resolution image analysis model based on patch selection by prototypes, and a method and system for providing a diagnostic response based on high-resolution image analysis, which can simultaneously improve the accuracy of the response and the computational efficiency by defining a plurality of prototypes corresponding to various features of a high-resolution image and selecting important patches necessary for analysis of a high-resolution image using a plurality of prototypes having different importance levels for the high-resolution image.

[0002] Analysis of high-resolution images generated across various industrial sites in modern society has become essential for operational accuracy and efficiency. For example, in manufacturing, microscopic defects in production lines are detected; in the medical field, digital pathology images are used to diagnose cellular abnormalities; and in agriculture, drone-captured images are analyzed to assess crop growth. While these images contain vast amounts of information, their complexity requires expert analysis and massive computational resources, limiting their effectiveness. Diagnostic and analytical standards, which rely on expert knowledge, are subject to constant change as new technologies emerge and requirements evolve.

[0003] With recent advances in digitalization, various computational methodologies have been introduced to automate image analysis. Deep learning, in particular, has proven effective in understanding complex and irregular images. However, existing high-resolution image analysis methods follow a pipeline that divides the image into numerous small fragments (patches) and then processes them. While this "separate-then-integrate" strategy has shown some success, as image resolution increases, the number of patches increases exponentially, leading to serious inefficiency.

[0004] As image resolution increases, the number of patches increases, and many unnecessary or redundant patches can be generated for analysis. These unnecessary patches can contribute to noise in the analysis process, degrading the performance of high-resolution image analysis models. Furthermore, existing methods that require processing all patches consume excessive computational resources, and especially when utilizing deep learning models, they exhibit significant limitations in terms of training and inference times. These issues are a major factor in reducing the efficiency and accuracy of high-resolution image analysis.

[0005] Random sampling, introduced to address these issues, improves computational efficiency by selecting only a subset of patches. However, this method carries the risk of accidentally missing patches containing important information. In particular, when critical information, such as defects or disease signs, is contained in only a small portion of the entire image, random sampling can have devastating consequences. Therefore, there is a pressing need for a new learning method for high-resolution image analysis models based on a novel patch selection method that maintains computational efficiency while not missing crucial information.

[0006] According to various embodiments of the present disclosure, a method for learning a high-resolution image analysis model based on patch selection by prototypes, which can simultaneously improve the accuracy of a response and computational efficiency by defining a plurality of prototypes corresponding to various features of an entire high-resolution image and selecting important patches necessary for analysis of a high-resolution image from among a plurality of patches of the high-resolution image using a plurality of prototypes having different importance levels for the high-resolution image, and a method and system for providing a diagnostic response based on high-resolution image analysis are provided.

[0007] According to various embodiments of the present disclosure, there is provided a high-resolution image analysis model that effectively selects patch embeddings suitable for analysis of various characteristics of high-resolution images by determining the importance of high-resolution images of multiple prototypes through self-attention operations, and a method and system for providing a high-resolution image analysis-based diagnostic response utilizing the same.

[0008] According to various embodiments of the present disclosure, a method for learning a high-resolution image analysis model based on patch selection by prototypes is provided, in which auxiliary prediction values ​​are calculated based on a weighted sum calculated through a gated attention operation on a plurality of prototypes and selected patches, and parameters of a high-resolution image analysis model are updated so that a loss based on the auxiliary prediction values ​​and correct answer data is minimized, thereby dynamically determining each prototype in the learning process and optimizing it for high-resolution image analysis, and a method and system for providing a diagnostic response based on high-resolution image analysis are provided.

[0009] However, the technical problems that the various embodiments of the present disclosure seek to achieve are not limited to the technical problems described above, and other technical problems may exist.

[0010] One example is,

[0011] A method for learning an image analysis model performed by a computing device is provided, comprising the steps of: generating a plurality of patch embeddings from an image; calculating a similarity between a plurality of prototype embeddings and the plurality of patch embeddings; selecting some of the plurality of patch embeddings based on the similarity for each prototype embedding; calculating a prediction value based on the selected plurality of patch embeddings; and updating parameters of the image analysis model so that a loss based on the prediction value and correct answer data is minimized.

[0012] In another aspect, the step of selecting some of the plurality of patch embeddings may include the step of calculating a plurality of sample coefficients representing the relative importance of the plurality of prototype embeddings with respect to the image, the step of calculating the number of patch embeddings to be selected for each prototype embedding based on the total number of selections for the plurality of patch embeddings and the plurality of sample coefficients, and the step of selecting, from the plurality of patch embeddings, a number of patch embeddings equal to the number of selections for each prototype embedding in order of high similarity to the corresponding prototype embedding.

[0013] In another aspect, the plurality of prototype embeddings may have different values ​​to correspond to different features of the image.

[0014] In another aspect, the step of calculating the plurality of sample coefficients may include the step of calculating an attention matrix by performing a self-attention operation on the plurality of prototype embeddings, and the step of calculating the plurality of sample coefficients by applying a softmax function to the plurality of diagonal elements of the attention matrix.

[0015] In another aspect, the step of calculating the number of selections of patch embeddings for each prototype embedding may include the step of setting the total number of selections, and the step of calculating the number of selections of patch embeddings for each prototype embedding by multiplying the total number of selections by each of the plurality of sample coefficients.

[0016] In another aspect, the step of calculating the predicted value may include the step of calculating a weighted sum of the selected plurality of patch embeddings through a gated attention operation, and the step of calculating the predicted value based on a predetermined operation result for the weighted sum of the selected plurality of patch embeddings.

[0017] In another aspect, the step of updating the parameters of the image analysis model may include the step of calculating an additional weighted sum by adding the weighted sums of the selected plurality of patch embeddings and the weighted sums based on the plurality of sample coefficients of the plurality of prototype embeddings, the step of calculating an auxiliary prediction value based on a predetermined operation result for the additional weighted sum, and the step of updating the parameters of the image analysis model such that the sum of a first loss based on the prediction value and the correct answer data and a second loss based on the auxiliary prediction value and the correct answer data is minimized.

[0018] In another aspect, the step of updating the parameters of the image analysis model may include the step of updating the parameters of the image analysis model such that the sum of the first loss, the second loss, and the third loss defined by the KL divergence based on the predicted value and the auxiliary predicted value is minimized.

[0019] In another aspect, the image may include a tissue pathology image.

[0020] In another aspect, the number of prototype embeddings and the total number of selections for the plurality of patch embeddings can be fixed values.

[0021] One example is,

[0022] A method for providing a diagnosis response based on image analysis performed by a computing device is provided, comprising the steps of receiving a high-resolution image to be analyzed according to a user input, inputting the high-resolution image to be analyzed into the pre-learned image analysis model, and providing a response output for the high-resolution image to be analyzed output from the image analysis model to the user.

[0023] In another aspect, the high-resolution image to be analyzed includes a tissue pathology image, and the response output may include at least one of a prediction result regarding whether the high-resolution image to be analyzed includes a specific disease, a prediction result regarding the grade or progression stage of the disease, and a prediction result regarding the location of a lesion for the disease.

[0024] One example is,

[0025] The present invention provides an image analysis-based diagnostic response providing system, comprising at least one memory, and at least one processor for executing an image analysis-based response providing method by reading out at least one command stored in the memory, wherein the at least one processor generates a plurality of patch embeddings from an image, calculates a similarity between a plurality of prototype embeddings and the plurality of patch embeddings, selects some of the plurality of patch embeddings based on the similarity for each prototype embedding, calculates a prediction value based on the selected plurality of patch embeddings, updates parameters of an image analysis model so that a loss based on the prediction value and correct answer data is minimized, receives a high-resolution image to be analyzed according to a user input, inputs the high-resolution image to be analyzed into the image analysis model, and provides a response output for the high-resolution image to be analyzed output from the image analysis model to the user.

[0026] In another aspect, the image analysis-based diagnostic response providing system may further include a field programmable gate array (FPGA) implementation for a predetermined artificial neural network, including a plurality of neurons configured as an array including at least one register, at least one programmable logic, and at least one input interface, a plurality of synaptic circuits storing synaptic weights that adjust connection strengths between the plurality of neurons, and at least one routing network that controls data flow between the plurality of neurons, wherein each of the plurality of neurons is connected to at least one other neuron through the routing network to set a transmission path of the weights.

[0027] In another aspect, the image analysis-based diagnostic response providing system may include a plurality of neurons organized in an array including at least one register, at least one microprocessor, and at least one input, and a plurality of synaptic circuits storing synaptic weights that adjust connection strengths between the plurality of neurons, and each of the plurality of neurons may further include an application specific integrated circuit (ASIC) for a predetermined artificial neural network connected to at least one other neuron through one of the plurality of synaptic circuits.

[0028] According to various embodiments of the present disclosure, the inefficiency of high-resolution image analysis can be fundamentally resolved, significantly improving computational efficiency. Unlike existing methods that require enormous computational resources and long inference times to process a vast number of patches, the present disclosure utilizes a prototype that determines the importance of high-resolution images based on multi-head self-attention computation, thereby selectively processing only a small number of critical patches essential for image analysis. This minimizes unnecessary computation, effectively conserves memory and computing resources, and dramatically improves image analysis speed.

[0029] According to various embodiments of the present disclosure, the importance of each prototype is determined through a self-attention operation between prototypes, and based on this, patches that best reflect various characteristics of a high-resolution image are precisely selected, and then prediction values ​​that serve as the basis of a diagnostic response are generated using the selected patches, thereby improving the accuracy and reliability of a diagnostic response according to high-resolution image analysis.

[0030] This patch selection process eliminates the risk of random sampling missing important information and removes noise from redundant or unnecessary patches, allowing high-resolution image analysis models to focus solely on essential information. Consequently, it achieves state-of-the-art accuracy and Weighted-F1 scores compared to existing models.

[0031] According to various embodiments of the present disclosure, multiple prototypes having different importance levels are defined to correspond to various features of high-resolution images, and these prototypes are dynamically determined and optimized through a learning process, thereby enabling a high-resolution image analysis model to adapt to various patterns in the image and provide various diagnostic responses, such as not only cancer diagnosis but also disease grade, disease progression stage, and lesion location. Accordingly, the flexibility and versatility of the learned high-resolution image analysis model can be secured.

[0032] According to various embodiments of the present disclosure, a prediction value based on a patch selected according to a prototype ( ) as well as auxiliary predictions based on both the prototype and the selected patches ( ) can be effectively optimized for the patch selection process by training the model using a loss function that reflects the model parameters. This leads to stable parameter updates and ultimately contributes to building a model optimized for high-resolution image analysis.

[0033] However, the effects that can be obtained through various embodiments of the present disclosure are not limited to the effects mentioned above, and other effects that are not mentioned can be clearly understood from the description below.

[0034] FIG. 1 illustrates an example block diagram of a computing system that implements a high-resolution image analysis-based diagnostic response provision service according to one embodiment.

[0035] FIG. 2 is a schematic diagram illustrating the structure of a neuromorphic circuit that a processor may include according to one embodiment.

[0036] FIG. 3 is a block diagram of a computing device that implements a high-resolution image analysis-based diagnostic response provision service according to one embodiment.

[0037] FIG. 4 is a block diagram of a computing device that implements a high-resolution image analysis-based diagnostic response provision service according to another embodiment.

[0038] FIG. 5 is a block diagram of a computing device that implements a high-resolution image analysis-based diagnostic response provision service according to another embodiment.

[0039] FIG. 6 illustrates an exemplary configuration of a high-resolution image analysis model architecture based on patch selection by a prototype according to one embodiment.

[0040] FIG. 7 illustrates an example of a plurality of patches selected from a plurality of patches of a high-resolution image according to a plurality of prototypes according to one embodiment.

[0041] FIG. 8 is a table illustrating improved performance of a high-resolution image analysis model based on patch selection by a prototype according to one embodiment compared to existing models.

[0042] FIG. 9 is a diagram for explaining the optimized values ​​of the number of prototypes of a high-resolution image analysis model and the total number of patches to be selected according to one embodiment.

[0043] Fig. 10 is a flowchart of a high-resolution image analysis model learning method according to one embodiment.

[0044] FIG. 11 is a flowchart illustrating steps that may be included in the step of selecting some of the multiple patch embeddings of FIG. 10.

[0045] FIG. 12 is a flowchart illustrating steps that may be included in the step of calculating multiple sample coefficients of FIG. 11.

[0046] Fig. 13 is a flowchart of a method for providing a diagnostic response based on high-resolution image analysis according to one embodiment.

[0047] Various modifications can be made to the various embodiments of the present disclosure, and thus specific embodiments are illustrated in the drawings and described in detail in the detailed description. The effects and features of the various embodiments of the present disclosure, as well as the methods for achieving them, will become clear with reference to the embodiments described in detail below together with the drawings. However, the various embodiments of the present disclosure are not limited to the embodiments disclosed below and may be implemented in various forms. In the following embodiments, terms such as "first" and "second" are not used in a limiting sense but are used for the purpose of distinguishing one component from another. Furthermore, the singular expression includes the plural expression unless the context clearly indicates otherwise. Furthermore, terms such as "include" and "have" indicate the presence of a feature or component described in the specification, and do not preemptively exclude the possibility that one or more other features or components may be added. Furthermore, the sizes of components in the drawings may be exaggerated or reduced for convenience of explanation. For example, the size and thickness of each component shown in the drawings are arbitrarily shown for convenience of explanation, and thus the various embodiments of the present disclosure are not necessarily limited to those illustrated.

[0048] Hereinafter, various embodiments of the present disclosure will be described in detail with reference to the attached drawings. When describing with reference to the drawings, identical or corresponding components are given the same drawing reference numerals, and redundant descriptions thereof will be omitted.

[0049]

[0050] - A system (1000) that provides a diagnostic response service based on high-resolution image analysis.

[0051] A system (1000) according to one embodiment defines a plurality of prototypes corresponding to various features of an entire high-resolution image, and uses a method of selecting important patches necessary for analysis of a high-resolution image among a plurality of patches of a high-resolution image by using a plurality of prototypes with different importance for the high-resolution image, thereby providing a diagnostic response utilizing a high-resolution image analysis model based on patch selection by a prototype with improved response accuracy and computational efficiency.

[0052] Based on the analysis results of the selected patches by the high-resolution image analysis model, the system (1000) can output a final diagnostic response for the high-resolution image. This response can include predictions for not only the presence or absence of a disease but also the grade or stage of the disease and the location of the lesion. These diverse diagnostic outputs can be implemented because the model can effectively identify and analyze key morphological patterns within the high-resolution image.

[0053] FIG. 1 illustrates an example of a block diagram of a computing system (1000) that implements a high-resolution image analysis-based diagnostic response provision service according to one embodiment.

[0054] Referring to FIG. 1, a computing system (1000) implementing a high-resolution image analysis-based diagnostic response provision service according to one embodiment includes a user computing device (110), a server computing system (130), and a training computing system (150), and the devices are capable of communicating via a network (170).

[0055] A method for providing a high-resolution image analysis-based diagnostic response according to one embodiment may be implemented and provided locally by a user computing device (110), implemented and provided in the form of a web service by a server computing system (130) communicating with the user computing device (110), or implemented and provided by a user computing device (110) and a server computing system (130) in connection with each other.

[0056] At this time, in the embodiment, the user computing device (110) and / or the server computing system (130) can train the machine learning model (120 and / or 140) through interaction with a training computing system (150) communicatively connected via a network (170). The training computing system (150) may be separate from the server computing system (130) or may be part of the server computing system (130).

[0057] And at this time, the artificial intelligence model can be 1) trained directly locally by the user computing device (110), 2) trained by the server computing system (130) and the user computing device (110) interacting with each other through a network (170), and 3) trained by a separate training computing system (150) using various training techniques and learning techniques. And the artificial intelligence model trained by the training computing system (150) can be implemented in a manner of being provided / updated by transmitting the model to the user computing device (110) and / or the server computing system (130) through a network (170).

[0058] In some embodiments, the training computing system (150) may be part of a server computing system (130) or part of a user computing device (110).

[0059]

[0060] -User Computing Device (110)

[0061] The user computing device (110) may include any type of computing device, such as a smart phone, a mobile phone, a digital broadcasting device, a personal digital assistant (PDA), a portable multimedia player (PMP), a desktop, a wearable device, an embedded computing device, a tablet PC, an augmented reality (VR) device, and / or a virtual reality (AR) device.

[0062] Such a user computing device (110) may include at least one processor (111) and memory (112). Here, the processor (111) may be composed of at least one or a plurality of electrically connected processors among a central processing unit (CPU), a graphics processing unit (GPU), application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors, and / or other electrical units for performing functions.

[0063] In particular, depending on the embodiment, such a processor (111) may be configured based on a field programmable gate array (FPGA) implementation, which is a hardware technology for implementing a predetermined digital circuit, and / or an application specific integrated circuit (ASIC).

[0064] Here, a field programmable gate array (FPGA) can mean a flexible digital circuit that can be programmed according to user needs.

[0065] In an embodiment, the field programmable gate array implementation may include a register that temporarily stores data and controls the flow and timing of signals to maintain intermediate results of operations or state information to support synchronized operation of the FPGA; programmable logic that programs operations within the FPGA to perform specific functions or operations as logic circuits that can be configured according to user needs; and an input interface that receives signals from an external device or sensor as a path for receiving data from outside the FPGA and transmits them to the internal circuit.

[0066] By combining the above components, field programmable gate array implementations can provide flexible and diverse forms of digital circuits.

[0067] Meanwhile, an application-specific integrated circuit (ASIC) can refer to a custom integrated circuit that is specifically designed to perform a specific purpose or function.

[0068] As an example, the application-specific integrated circuit may include a register, which is a small memory device that temporarily stores and manages data and stores intermediate results of calculations or status information to support rapid processing of the operation of the ASIC; a microprocessor, which is a central processing unit that performs control and operations within the ASIC and, if necessary, performs various operations or generates control signals to coordinate the operation of the entire system; and an input block, which is an interface that receives data from the outside and receives data to be processed by the ASIC and transmits it to the inside, and receives various input data through a connection with a sensor or an external device.

[0069] By combining the above components, application-specific integrated circuits can perform specific tasks in an optimized manner.

[0070] For example, ASICs may have a neuromorphic circuit structure in the form of an array containing multiple neuron circuits.

[0071] FIG. 2 is a schematic diagram illustrating the structure of a neuromorphic circuit (300) that may be included in a processor (111, 131, 151) according to one embodiment.

[0072] Referring to FIG. 2, for example, a neuromorphic circuit (300) may include a plurality of presynaptic neuron circuits (310), a plurality of presynaptic lines (311) extending laterally from the plurality of presynaptic neuron circuits (310), a plurality of post-synaptic neuron circuits (320), a plurality of post-synaptic lines (321) extending longitudinally from the plurality of post-synaptic neuron circuits (320), and a plurality of synaptic circuits (330) provided at intersections of the plurality of presynaptic lines (311) and the plurality of post-synaptic lines (321).

[0073] A plurality of presynaptic neuron circuits (310) can transmit signals input from the outside in the form of electrical signals to a plurality of synaptic circuits (330) through a plurality of presynaptic lines (311).

[0074] Additionally, a plurality of post-synaptic neuron circuits (320) can receive electrical signals from a plurality of synaptic circuits (330) through a plurality of post-synaptic lines (321).

[0075] Furthermore, a plurality of post-synaptic neuron circuits (320) may transmit electrical signals to a plurality of synaptic circuits (330) through a plurality of post-synaptic lines (321).

[0076] A plurality of synaptic circuits (330) can store weights included in layers constituting a neural network system implemented by a neuromorphic circuit (300) and perform a predetermined operation based on the weights and input data.

[0077] For example, each of the plurality of synaptic circuits (330) may include a resistive memory cell having a variable resistance. In this case, the plurality of synaptic circuits (330) may have a resistance value that changes according to a voltage applied through the plurality of presynaptic neuron circuits (310) or the plurality of postsynaptic neuron circuits (320), and may store weight data according to this resistance change.

[0078] The neuromorphic circuit (300) is formed by simulating the neuron and synapse structures, which are essential elements of the human brain. When a deep neural network (DNN) is implemented using the neuromorphic circuit (300), data processing speed can be improved and power consumption can be reduced compared to when the existing von Neumann structure is utilized.

[0079] The memory (112) may include one or more non-transitory / transitory computer-readable storage media such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, and combinations thereof, and may include web storage of a server that performs a memory storage function on the Internet. The memory (112) may store data (113) and commands (114) necessary for the at least one processor (111) to perform functional operations such as training an artificial intelligence model or performing high-resolution image analysis using an artificial intelligence model.

[0080] In one embodiment, the user computing device (110) may store at least one machine learning model (120).

[0081] For example, the machine learning model (120) may be various machine learning models such as multiple neural networks (e.g., deep neural networks) for performing a high-resolution image analysis-based diagnostic response providing method, or other types of machine learning models including nonlinear models and / or linear models, and may be composed of a combination thereof.

[0082] For example, the machine learning model may store a linear regression, decision tree, random forest, gradient boosting, pre-trained language model, or / and a deep learning model. The neural network may include at least one of feed-forward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks, or / and other types of neural networks.

[0083] Additionally, according to various embodiments, the user computing device (110) may store a model to be used in each process and a prompt template that serves as the basis for input to the model in order to perform at least part of the process performed for the method of providing a high-resolution image analysis-based diagnostic response through a large language model (LLM).

[0084] In one embodiment, a user computing device (110) may receive at least one machine learning model (120) from a server computing system (130) via a network (170), store the model in a memory (112), and then execute the stored machine learning model (120) via a processor (111) to perform operations for providing a high-resolution image analysis-based diagnostic response.

[0085] In another embodiment, the server computing system (130) may include at least one machine learning model (140) to perform operations through the machine learning model (140), and may communicate data related thereto with the user computing device (110) to provide a high-resolution image analysis-based diagnostic response provision service to the user.

[0086] For example, a user computing device (110) can perform a high-resolution image analysis-based diagnostic response provision service by having a server computing system (130) provide output for a user's input using a machine learning model (140) via the web.

[0087] Additionally, the artificial intelligence model may be implemented in such a way that at least some of the machine learning models (120 and / or 140) are executed on a user computing device (110) and the rest are executed on a server computing system (130).

[0088] In addition, the user computing device (110) may include at least one input component (121) that detects a user's input. For example, the user input component (121) may include a touch sensor (e.g., a touch screen and / or a touch pad, etc.) that detects a touch of a user's input medium (e.g., a finger or a stylus), an image sensor that detects a user's motion input, a microphone, a button, a mouse, and / or a keyboard that detects a user's voice input, etc. In addition, the user input component (121) may include an interface and an external controller when receiving an input to an external controller (e.g., a mouse and / or a keyboard, etc.) through an interface.

[0089]

[0090] -Server Computing System (130)

[0091] The server computing system (130) can perform a series of processes to provide a high-resolution image analysis-based diagnostic response provision service.

[0092] In detail, in an embodiment, the server computing system (130) can provide a high-resolution image analysis-based diagnostic response provision service by exchanging data necessary to enable a high-resolution image analysis-based diagnostic response provision service process to be driven by an external device, such as a user computing device (110), with the external device.

[0093] In more detail, in an embodiment, a server computing system (130) may provide an environment in which an application for providing a high-resolution image analysis-based diagnostic response provision service may operate on a user computing device (110).

[0094] To this end, the server computing system (130) may include application programs, data and / or commands for the application to operate, and may transmit and receive various data based thereon with the external device.

[0095] The server computing system (130) may include at least one processor (131) and memory (132). Here, the processor (131) may be composed of at least one or a plurality of electrically connected processors among a central processing unit (CPU), a graphics processing unit (GPU), application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors, and / or other electrical units for performing functions.

[0096] For example, ASICs may have a neuromorphic circuit structure in the form of an array containing multiple neuron circuits (see Figure 2).

[0097] And the memory (132) may include one or more non-transitory / transitory computer-readable storage media such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. This memory (132) may store data (133) and commands (134) necessary for the processor (131) to perform functional operations such as training an artificial intelligence model or executing a method for providing a high-resolution image analysis-based diagnostic response through an artificial intelligence model.

[0098] In one embodiment, the server computing system (130) may be implemented to include at least one computing device. For example, the server computing system (130) may be implemented to operate multiple computing devices according to a sequential computing architecture, a parallel computing architecture, or a combination thereof. Additionally, the server computing system (130) may include multiple computing devices connected via a network (170).

[0099] Additionally, the server computing system (130) may store at least one machine learning model (140). For example, the server computing system (130) may include a neural network and / or other multi-layer nonlinear model as the machine learning model (140). Exemplary neural networks may include feedforward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks.

[0100] In an embodiment, the server computing system (130) may further include a data store computing system (hereinafter, “data store”), which is a storage for continuously storing and managing raw data that is the basis of a high-resolution image analysis-based diagnostic response provision service.

[0101] These data stores can include various forms of data storage, ranging from file systems to cloud storage. For example, a data store can include at least one of the following: a relational database that uses a structured query language (SQL) to define and manipulate data; a NoSQL database designed for flexibility and scalability to handle unstructured and semi-structured data; a data warehouse, which is a system used for reporting and data analysis, centralizes large amounts of data from multiple sources and is optimized for querying and analysis; a data warehouse that stores large amounts of raw data in its native formats of structured, semi-structured, and unstructured data; and a database on a local storage device or Network Attached Storage (NAS) that stores data in files in a format typically accessible by a computer operating system.

[0102]

[0103] -Training Computing System (150)

[0104] The training computing system (150) may include at least one processor (151) and memory (152). Here, the processor (151) may be composed of at least one or a plurality of electrically connected processors among a central processing unit (CPU), a graphics processing unit (GPU), application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors, and / or other electrical units for performing functions.

[0105] For example, ASICs may have a neuromorphic circuit structure in the form of an array containing multiple neuron circuits (see Figure 2).

[0106] And the memory (152) may include one or more non-transitory / transitory computer-readable storage media such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. This memory (152) may store data (153) and instructions (154) necessary for the processor (151) to perform learning of an artificial intelligence model, etc.

[0107] For example, the training computing system (150) may include a model trainer (160) that trains a machine learning model (120 and / or 140) stored in a user computing device (110) and / or a server computing system (130) using various training or learning techniques, such as backpropagation of errors.

[0108] For example, such a model trainer (160) may perform updates to one or more parameters of a machine learning model (120 and / or 140) for a diagnostic response provision service in a backpropagation manner based on a defined loss function.

[0109] In some implementations, performing backward propagation of errors may include performing truncated backpropagation through time. The model trainer (160) may perform a number of generalization techniques (e.g., weight reduction, dropout, and / or knowledge distillation) to improve the generalization ability of the trained machine learning model (120 and / or 140).

[0110] For example, a model trainer (160) may train a machine learning model (120 and / or 140) based on a series of training data (161). Here, the training data (161) may include data in different formats, such as, for example, images, audio samples, and / or text.

[0111] Examples of image types that may be used may include video frames, LiDAR point clouds, X-ray images, computed tomography scans, hyperspectral images, and / or various other forms of images.

[0112] Such training data (161) may be provided by a user computing device (110) and / or a server computing system (130). When the training computing device trains a machine learning model (120 and / or 140) on specific data of the user computing device (110), the machine learning model (120 and / or 140) may be characterized as a personalized model.

[0113] And the model trainer (160) includes computer logic utilized to provide the desired function.

[0114] Additionally, the model trainer (160) may be implemented as hardware, firmware, and / or software that controls a general-purpose processor. In one implementation, the model trainer (160) includes a program file stored in a storage device, which may be loaded into a memory (152) and executed by one or more processors (151). In another implementation, the model trainer (160) includes one or more sets of computer-executable data (153) and instructions (154) stored in a tangible computer-readable storage medium, such as a RAM hard disk or an optical or magnetic medium.

[0115] The network (170) includes, but is not limited to, a 3rd Generation Partnership Project (3GPP) network, a Long Term Evolution (LTE) network, a World Interoperability for Microwave Access (WIMAX) network, the Internet, a Local Area Network (LAN), a Wireless Local Area Network (Wireless LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), a Bluetooth network, a satellite broadcasting network, an analog broadcasting network, and / or a Digital Multimedia Broadcasting (DMB) network.

[0116] In general, communication over a network (170) may be performed using any type of wired and / or wireless connection, using various communication protocols (e.g., TCP / IP, HTTP, SMTP, and / or FTP), encodings or formats (e.g., HTML and / or XML), and / or protection schemes (e.g., VPN, Secure HTTP, and / or SSL).

[0117] FIG. 3 is a block diagram of a computing device (100) that implements a high-resolution image analysis-based diagnostic response provision service according to one embodiment.

[0118] Referring to FIG. 3, the computing device (100) included in the user computing device (110), the server computing system (130), and the training computing system (150) includes a plurality of applications (e.g., Application 1 to Application N). Each application may include a machine learning library and one or more machine learning models. For example, the applications may include an image processing (e.g., Detection, Classification, and / or Segmentation, etc.) application, a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, and / or a chat-bot application.

[0119] In an embodiment, the computing device (100) may include a model trainer (160) for training an artificial intelligence model, and may store and operate the trained artificial intelligence model to provide output data according to predetermined input data.

[0120] Each application of the computing device (100) may communicate with a number of other components of the computing device (100), such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In one embodiment, each application may communicate with each device component using an API (e.g., a public API). In one embodiment, the API used by each application may be specific to that application.

[0121] FIG. 4 is a block diagram of a computing device (200) that implements a high-resolution image analysis-based diagnostic response provision service according to another embodiment.

[0122] Referring to FIG. 4, a computing device (200) includes a plurality of applications (e.g., Application 1 to Application N). Each application may communicate with a central intelligence layer. For example, the applications may include an image processing application, a text messaging application, an email application, a dictation application, a virtual keyboard application, and / or a browser application. In one embodiment, each application may communicate with the central intelligence layer (and models stored therein) using an API (e.g., a common API across all applications).

[0123] The central intelligence layer may include multiple machine learning models. For example, as illustrated in FIG. 4, at least some of the machine learning models may be provided to each application and managed by the central intelligence layer. In other implementations, two or more applications may share a single machine learning model. For example, in some implementations, the central intelligence layer may provide a single model to all applications. In some implementations, the central intelligence layer may be included within the operating system of the computing device (200) or implemented differently.

[0124] The central intelligence layer may communicate with a central device data layer. The central device data layer may be a centralized data repository for the computing device (200). As illustrated in FIG. 4, the central device data layer may communicate with a number of other components of the computing device (200), such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, the central device data layer may communicate with each device component using an API (e.g., a private API).

[0125] The techniques described herein may refer to servers, databases, software applications, and other computer-based systems, as well as actions taken and information transmitted to or from such systems. It will be appreciated that the inherent flexibility of computer-based systems allows for a wide range of possible configurations, combinations, and division of labor and functionality between and among components. For example, the processes described herein may be implemented using a single device or component, or multiple devices or components operating in combination. Databases and applications may be implemented on a single system or in a distributed system across multiple systems. Distributed components may operate sequentially or in parallel.

[0126] FIG. 5 is a block diagram of a computing device (400) implementing a high-resolution image analysis-based diagnostic response providing service according to another embodiment. FIG. 6 illustrates an exemplary configuration of a high-resolution image analysis model architecture based on patch selection by a prototype according to an embodiment. FIG. 7 illustrates a plurality of patches selected from a plurality of patches of a high-resolution image according to a plurality of prototypes according to an embodiment. FIG. 8 is a table for explaining improved performance of a high-resolution image analysis model based on patch selection by a prototype according to an embodiment compared to a conventional model. FIG. 9 is for explaining optimized values ​​of the number of prototypes and the total number of selected patches of a high-resolution image analysis model according to an embodiment.

[0127] Referring to FIG. 5, a computing device (400) included in a user computing device (110), a server computing system (130), and a training computing system (150) may include a patching module (10), a prototype initialization module (20), a sample coefficient calculation module (30), a total selection count setting module (40), a similarity calculation module (50), a patch embedding selection module (60), a GA operation module (70), an auxiliary GA operation module (71), a prediction module (80), and a model learning module (90).

[0128] The patching module (10) can divide a high-resolution image (HRI) input from the outside into multiple patches and generate multiple patch embeddings corresponding to the multiple patches.

[0129] For example, referring to FIG. 6, the patching module (10) divides a high-resolution image (HRI) into N patches and generates N patch embeddings (w1, w2, …, w) corresponding to the N patches. N ) can be created.

[0130] In this case, for example, the patching module (10) can divide the high-resolution image (HRI) into multiple patches using a method such as a sliding window. However, this is not limited to this, and various methods other than the sliding window can be adopted for dividing the high-resolution image (HRI) into multiple patches.

[0131] The patching module (10) uses a pre-learned feature extraction model for multiple patches to generate multiple patch embeddings (w1, w2, …, w), which are high-dimensional vectors representing the visual features of each of the multiple patches. N) can be generated. In this case, the feature extraction model may include the structure of a convolutional neural network such as ResNet50, which is pre-trained with ImageNet. However, it is not limited thereto, and the feature extraction model may include various models other than ResNet50.

[0132] The prototype initialization module (20) can generate and initialize multiple prototype embeddings that serve to select patches associated with various features of a high-resolution image (HRI). The multiple prototype embeddings have a trainable vector form that corresponds to various features of the high-resolution image (HRI). Before learning begins, the prototype initialization module (20) can assign an initial value, such as a random value, to each prototype embedding.

[0133] Here, various features of the high resolution image (HRI) may include, for example, normal epithelial tissue, tumor cell clusters, necrotic tissue, inflammatory infiltrate, adipose tissue, fibrotic stroma, vascular structures, lymphatic vessels, glandular structures, and stain artifacts / background when the high resolution image (HRI) is a tissue pathology image.

[0134] The prototype embedding can then be provided to a similarity calculation module (50) and a sample coefficient calculation module (30). The prototype initialization module (20) generates a prototype embedding having the same dimension as the patch embedding, which enables similarity calculation between the prototype embedding and the patch embedding. The initialized prototype embedding can be updated during the model training process and gradually optimized to capture key morphological patterns of high-resolution images (HRI).

[0135] Referring to FIG. 6, the prototype initialization module (20) can generate and initialize M (positive integer) prototype embeddings. In this case, multiple prototype embeddings can be initialized and updated to have different values ​​to correspond to various features of a high-resolution image (HRI).

[0136] The number of these multiple prototype embeddings (M) is a hyperparameter that is either directly specified by the user or preset according to the model settings to balance the performance and computational efficiency of the high-resolution image analysis model, and may be a value that is not updated during the learning process.

[0137] For example, referring to Fig. 9, the model evaluation indices Weighted-F1 and ACC are calculated based on the combination of the number of various prototype embeddings (vertical axis) and the total number of selections for patch embeddings (horizontal axis). Considering the balance between model performance and computational efficiency, the optimal number of multiple prototype embeddings (M) may be 15.

[0138] The sample coefficient calculation module (30) can calculate the sample coefficients of each of the plurality of prototype embeddings initialized by the prototype initialization module (20).

[0139] Here, the sample coefficient is a value that quantifies the relative importance that the prototype embedding has for the high-resolution image (HRI), and is a value that can be used to represent multiple patch embeddings (w1, w2, …, wN ) can be used to determine the number of patch embeddings to be selected for each prototype during the selection process.

[0140] The sample coefficient calculation module (30) can calculate the sample coefficient for each prototype embedding by using the self-attention operation results for multiple prototype embeddings.

[0141] For example, the sample coefficient calculation module (30) can perform a multi-head self-attention operation on multiple prototype embeddings. This can be done by calculating multiple prototype embeddings (p1, p2, …, p M ) is a process of evaluating the importance of each prototype embedding in the entire set of prototype embeddings using the query, key, and value derived from the prototype embedding. Through this, the complex relationships between multiple prototype embeddings can be identified and the relative importance of each prototype embedding can be calculated for the analysis of various features of high-resolution images (HRI).

[0142] As a result of a multi-head self-attention operation on multiple prototype embeddings, an attention matrix of size M×M can be produced whose elements are multiple attention scores indicating the relevance of multiple prototype embeddings to each other.

[0143] The sample coefficient calculation module (30) can calculate sample coefficients for each prototype embedding by applying a softmax function to multiple diagonal elements of an attention matrix for multiple prototype embeddings according to the following equation (1). Accordingly, the sum of the multiple sample coefficients for multiple prototype embeddings becomes 1.

[0144]

[0145] Equation (1):

[0146] (step, : sample coefficient, (jth element of the diagonal elements of the attention matrix)

[0147]

[0148] Referring to FIG. 6, the sample coefficient calculation module (30) uses the multi-head self-attention operation results for the multiple prototype embeddings as above to generate multiple sample coefficients (c1, c2, …, c) indicating the importance of the high-resolution images (HRI) of the multiple prototype embeddings. M ) can be produced.

[0149] The total selection count setting module (40) embeds multiple patches of high resolution images (HRI) (w1, w2, …, w N ) can be set to the total number of patch embeddings to be finally selected (k). This total number of selections (k) is a hyperparameter that is directly specified by the user or preset according to the model settings to balance the performance and computational efficiency of the high-resolution image analysis model, and may be a value that is not updated during the learning process.

[0150] For example, referring to Fig. 9, the model evaluation indices Weighted-F1 and ACC are calculated based on the combination of the number of various prototype embeddings (vertical axis) and the total number of selections for patch embeddings (horizontal axis). Considering the balance between model performance and computational efficiency, the total number of selections (k) for multiple patch embeddings may be optimal at 100.

[0151] The similarity calculation module (50) generates multiple patch embeddings (w1, w2, …, w) generated by the patching module (10). N ) and the similarity between multiple prototype embeddings initialized in the prototype initialization module (20) can be calculated.

[0152] The similarity calculation module (50) can calculate the similarity between patch embeddings and prototype embeddings through a predetermined operation to determine the relationship between the two embeddings. The result of this operation includes the similarity score that each prototype embedding has with respect to all patch embeddings, and this score can be used by the patch embedding selection module (60) to determine which patches to select.

[0153] For example, the similarity calculation module (50) generates a plurality of patch embeddings (w1, w2, …, w) for each of the plurality of prototype embeddings. N ) by calculating the cosine similarity for multiple prototype embeddings and multiple patch embeddings (w1, w2, …, w N ) can be used to calculate the similarity between them.

[0154] The patch embedding selection module (60) can select a small number of core patch embeddings essential for high-resolution image analysis. The patch embedding selection module (60) can perform patch selection based on the similarity between the prototype embedding and the patch embedding, the sample coefficient (cj) generated by the sample coefficient calculation module (30), and the total number of selections (k) by the total number of selections setting module (40).

[0155] For example, referring to FIG. 6, the patch embedding selection module (60) can dynamically select multiple patch embeddings through multi-vector retrieval using multiple prototype embeddings.

[0156] The patch embedding selection module (60) can determine the selection count (kj) for each prototype embedding based on the sample coefficient (cj) and the total selection count (k). In this case, the patch embedding selection module (60) can calculate the selection count (kj) for each prototype embedding by multiplying the total selection count (k) by the sample of each prototype.

[0157] Thereafter, the patch embedding selection module (60) can select the top kj patches with the highest similarity to each prototype embedding, thereby finally selecting a total of k patch embeddings.

[0158] For example, referring to FIG. 7, the patch embedding selection module (60) can select 20 patch embeddings in order of high similarity for the first and second prototype embeddings having the largest sample coefficients from a plurality of patch embeddings of a tissue pathology image labeled as Normal, and can select 2 patch embeddings in order of high similarity for the 12th to 15th prototype embeddings having the lower ranked sample coefficients.

[0159] Similarly, the patch embedding selection module (60) can select 20 patch embeddings in order of high similarity for the first and second prototype embeddings having the largest sample coefficients from a plurality of patch embeddings of a tissue pathology image labeled as a tumor, and can select 2 patch embeddings in order of high similarity for the 12th to 15th prototype embeddings having the lowest sample coefficients.

[0160] The GA operation module (70) can integrate multiple patch embeddings selected by the patch embedding selection module (60) to produce an overall image-level representation for a high-resolution image (HRI).

[0161] The GA operation module (70) performs a gated attention operation based on the selected patch embeddings, thereby calculating a weight (ai) indicating how important each patch embedding is to the final prediction. Then, by multiplying each patch embedding by this weight (ai) and adding them all together, a weighted sum can be generated to produce a full image-level representation of a high-resolution image (HRI).

[0162] For example, the GA operation module (70) selects a plurality of patch embeddings selected by the patch embedding selection module (60). , … ), a learnable weight matrix (e.g., V, U) can be used to derive the weights (ai).

[0163] First, the GA operation module (70) embeds each patch according to the following equations (2) and (3). ) and the first weight matrix V are applied with a nonlinear function (e.g. tanh(·)) to obtain a new embedding h. i and generate this new embedding h i And the inner product of the second weight matrix U is taken to obtain a given score (score i ) can be produced.

[0164]

[0165] Equation (2):

[0166] Equation (3):

[0167]

[0168] Afterwards, the GA operation module (70) calculates all the scores (score1, …, score) according to the following equation (4). k ) can be applied to the softmax function to obtain the final attention weight (ai) between 0 and 1.

[0169]

[0170] Equation (4):

[0171]

[0172] Furthermore, the GA operation module (70) embeddings each patch ( ) by the corresponding weights (ai) and add them all up, resulting in a selected patch embedding (which is a full image-level representation for high-resolution images (HRI). i ) for the weighted sum ( ) can be produced.

[0173] In this way, the GA computation module (70) can perform computations on only a select few core patch embeddings, addressing the issues inherent in random patch sampling. This significantly improves computational efficiency compared to conventional methods that require processing all patches. The resulting weighted sum, i.e., the overall image-level representation, can then be passed to the prediction module (80) to generate the final diagnostic response.

[0174] The auxiliary GA operation module (71) learns multiple prototype embeddings (p1, p2, …, p) during the learning process of the high-resolution image analysis model. M ) plays a role in helping to effectively optimize the selection of patch embeddings. The auxiliary GA operation module (71) is activated only during the training stage and may not be used during inference.

[0175] The auxiliary GA operation module (71) selects multiple patch embeddings produced by the GA operation module (70). , … ) weighted sum( ) and multiple corresponding prototype embeddings (p1, p2, …, p M ) and multiple sample coefficients (c1, c2, …, c M ) is the weighted sum calculated by multiplying ) can be added together to produce an additional weighted sum.

[0176] The auxiliary GA computation module (71) can incorporate prototype embedding elements into a predetermined weighted sum that forms the basis of the final diagnostic response, thereby ensuring that the rate of change in the prototype embedding is transmitted during the learning process via backpropagation, thereby enabling the prototype embedding to be updated more efficiently during the patch selection process. The additional weighted sum thus produced, i.e., the additional representation at the full image level, can then be passed on to the prediction module (80) and utilized to generate the final diagnostic response.

[0177] The prediction module (80) can produce the final prediction value and auxiliary prediction value of the high-resolution image analysis model. The prediction module (80) receives different types of weighted sums from the GA operation module (70) and the auxiliary GA operation module (71).

[0178] First, the prediction module (80) calculates the weighted sum ( ) based on the model's prediction value ( ) can be produced.

[0179] Secondly, the prediction module (80) adds the additional weighted sum ( + ) based on the auxiliary prediction value ( ) can be produced.

[0180] In this case, the prediction module (80) can perform a predetermined operation on the input weighted sum to produce a predicted value or an auxiliary predicted value. For example, the predetermined operation may include an operation utilizing a linear classifier or an activation function, such as a sigmoid function or a softmax function.

[0181] The model learning module (90) outputs the predicted value ( ) and auxiliary predictors ( ) can be input and learning can be performed by updating the parameters of the high-resolution image analysis model based on this.

[0182] The model learning module (90) may be included in the computing device (400) as described above, but is not limited thereto, and may be dependent on the model trainer (160) of the training computing system (150).

[0183] For example, the model learning module (90) predicts the values ​​( ) and the first loss based on the correct answer data ( ) and auxiliary predictors ( ) and the second loss based on the correct answer data ( ) can be updated to minimize the parameters of the high-resolution image analysis model.

[0184] Here, the first loss ( ) is the predicted value ( ) is a cross entropy loss that calculates the difference between the correct data and the second loss ( ) is the auxiliary predictor ( ) is a cross-entropy loss that calculates the difference between the predicted data and the correct data.

[0185] Here, the correct answer data refers to the final correct answer for the entire high-resolution image (HRI). For example, if the high-resolution image (HRI) is a tissue pathology image, the correct answer data may include final diagnosis results such as "Normal" or "Cancer." If the high-resolution image (HRI) is a production line image, the correct answer data may include final inspection results such as "Good" or "Defective."

[0186] Additionally, for example, the model learning module (90) may be configured to generate a first loss ( ) and the second loss ( ) in the predicted value( ) and auxiliary predictors ( ) defined by the KL divergence based on the third loss ( ) can be updated to minimize the sum of the parameters of the high-resolution image analysis model.

[0187] The high-resolution image analysis model (XFS) learned in this way can simultaneously achieve accurate high-resolution image analysis performance and excellent computational efficiency.

[0188] For example, referring to FIG. 8, the high-resolution image analysis model (XFS) according to the embodiment of the present disclosure outperforms all other existing methods by recording 93.98 in Weighted-F1 score and accuracy using only 2.9% of the entire patches.

[0189]

[0190] - High-resolution image analysis model learning method (S100)

[0191] A method (S100) for training a high-resolution image analysis model can provide a learning process for training an artificial intelligence model that efficiently and accurately analyzes high-resolution images (HRI). The method (S100) can train the model by defining multiple prototype embeddings and utilizing them to select important patches within high-resolution images (HRI).

[0192] Method (S100) may include two main steps that embody the model's core learning mechanism. First, step (S105) of calculating sample coefficients for each prototype embedding establishes the model's patch selection criteria. Second, step (S113) of updating the parameters of the high-resolution image analysis model optimizes model performance by adjusting the parameters based on the model's prediction results.

[0193] These two key steps are closely linked. The patch selection criteria determined through sample coefficient calculation influence the model's predictions, and the model parameters are updated based on these predictions. These updates, in turn, influence the prototype and sample coefficient calculations, forming a feedback loop that continuously improves model performance.

[0194] FIG. 10 is a flowchart of a method (S100) for learning a high-resolution image analysis model according to one embodiment. FIG. 11 is a flowchart illustrating steps that may be included in the step (S105) of selecting some of the multiple patch embeddings of FIG. 10. FIG. 12 is a flowchart illustrating steps that may be included in the step (S1051) of calculating multiple sample coefficients of FIG. 11.

[0195] Referring to FIG. 10, a method for learning a high-resolution image analysis model (S100) according to one embodiment may include a step of generating a plurality of patch embeddings from an image (S101), a step of calculating a similarity between a plurality of prototype embeddings and the plurality of patch embeddings (S103), a step of selecting some of the plurality of patch embeddings based on the similarity for each prototype embedding (S105), a step of calculating a prediction value based on the selected plurality of patch embeddings (S107), and a step of updating parameters of an image analysis model so that a loss based on the prediction value and correct answer data is minimized (S109).

[0196] In one embodiment, the method (S100) may be performed by a processor (131) included in a server computing system (130). However, the present invention is not limited thereto, and at least a part of the method (S100) may be performed by a processor (111) of a user computing device (110) or a processor (151) of a training computing system (150), and another part may be performed by a processor (131) included in the server computing system (130).

[0197] For convenience of explanation, the method (S100) is described below as being performed by a processor (131) included in a server computing system (130).

[0198] In step (S101), the processor (131) divides the high-resolution image (HRI) into multiple patches, and generates multiple patch embeddings (w1, w2, …, w) corresponding to the multiple patches. N ) can be generated. Step (S101) is a preprocessing step for high-resolution image analysis, and the processor (131) can divide the input high-resolution image (HRI) into N small patches using the patching module (10).

[0199] Next, the processor can generate patch embeddings, which are high-dimensional vectors representing the visual features of each patch, using a pre-trained feature extraction model (e.g., ResNet50 pre-trained on ImageNet). These multiple patch embeddings (w1, w2, …, w N ) can be used to calculate the similarity with the prototype embedding in a later step and ultimately used as input for model prediction.

[0200] In step (S103), the processor (131) learns a plurality of prototype embeddings (p1, p2, …, p M ) and multiple patch embeddings (w1, w2, …, w N ) can be used to calculate the similarity between the prototypes. This similarity calculation serves as the basis for selecting patch embeddings and is a process for evaluating which patches each prototype should focus on.

[0201] In step (S105), the processor (131) can select some of the plurality of patch embeddings based on the similarity for each prototype embedding.

[0202] Here, the processor (131) can calculate the number of prototype embeddings to be selected, which serves as a criterion for selecting patch embeddings, based on the importance of high-resolution images (HRI) of multiple prototype embeddings.

[0203] In detail, referring to FIG. 11, step (S105) may include a step (S1051) of calculating a plurality of sample coefficients indicating the relative importance of the plurality of prototype embeddings for the image, a step (S1053) of calculating a number of patch embeddings to be selected for each prototype embedding based on the total number of selections for the plurality of patch embeddings and the plurality of sample coefficients, and a step (S1055) of selecting, from the plurality of patch embeddings, a number of patch embeddings equal to the number of selections for each prototype embedding in order of high similarity to the corresponding prototype embedding.

[0204] First, the processor (131) embeds multiple prototypes (p1, p2, …, p M ) can be used to calculate sample coefficients for each prototype embedding (S1051).

[0205] For example, referring to FIG. 12, the processor (131) may embed multiple prototypes (p1, p2, …, p M ) to evaluate the relationship between prototypes, and extract the diagonal elements of the attention matrix produced as a result of this attention operation (S511). The processor (131) can produce a sample coefficient (cj) indicating the relative importance of each prototype in the entire patch selection process by applying a softmax function to this diagonal element (S513).

[0206] Additionally, the processor (131) embeds multiple patches (w1, w2, …, w N ), the number of patches selected for each prototype embedding (kj) can be calculated (S1053).

[0207] In this case, the total number of selections (k) may be a fixed value determined in advance before learning to balance model performance and computational efficiency. The processor (131) may multiply this fixed total number of selections (k) by a sample coefficient (cj) to calculate the number of individual patches (kj) to be selected for each prototype embedding.

[0208] Furthermore, the processor (131) embeds multiple patches (w1, w2, …, w N ), multiple prototype embeddings (p1, p2, …, p M ) for each, some patch embeddings having a high similarity to the corresponding prototype embedding can be selected based on the sample coefficients. In this case, the processor (131) can perform patch selection (retrieval) using the patch embedding selection module (60).

[0209] In detail, the processor (131) embeds multiple patches (w1, w2, …, w N ), multiple prototype embeddings (p1, p2, …, p M ) For each, a number of patch embeddings (kj) can be selected for each prototype embedding in order of high similarity to the corresponding prototype embedding.

[0210] The processor (131) can select the top kj patch embeddings with the highest similarity to each prototype, based on the number of selections (kj) for each prototype determined in step (S1053). Through this process, a small number of patch embeddings containing only core information important for diagnosis are ultimately left.

[0211] In step (S107), the processor (131) selects a plurality of patch embeddings ( , … ) based on the predicted value ( ) can be produced.

[0212] Additionally, in step (S107), the processor (131) selects a plurality of patch embeddings ( , … ) based on the auxiliary prediction value ( ) can be additionally produced.

[0213] The processor (131) selects multiple patch embeddings through gated attention operations. , … ) weighted sum( ) based on the predicted value ( ) can be produced.

[0214] Additionally, the processor (131) selects a plurality of patch embeddings ( , … ) weighted sum( ) and multiple corresponding prototype embeddings (p1, p2, …, p M ) and multiple sample coefficients (c1, c2, …, c M ) is the weighted sum calculated by multiplying ) based on the additional weighted sum of the auxiliary prediction values ​​( ) can be produced.

[0215] In step (S109), the processor (131) predicts a value ( ) and the first loss based on the correct answer data ( ) can be updated to minimize the parameters of the high-resolution image analysis model.

[0216] Additionally, the processor (131) predicts the value ( ) and auxiliary predictors ( ) are used to calculate the loss with respect to the correct data. This loss is composed of three terms in total (main prediction loss ( ), auxiliary prediction loss ( ), KL Balsan Los ( )) is composed of the sum of the loss, and the parameters of the model including the prototype embedding are updated through backpropagation based on this loss, so that the high-resolution image analysis model can be optimized.

[0217] Fig. 13 is a flowchart of a method (S200) for providing a diagnostic response based on high-resolution image analysis according to one embodiment.

[0218] Referring to FIG. 13, a method for providing a diagnostic response based on high-resolution image analysis according to one embodiment (S200) may include a step of receiving a high-resolution image to be analyzed according to a user input (S201), a step of inputting the high-resolution image to be analyzed into an image analysis model pre-trained by a high-resolution image analysis model learning method based on hierarchical patch selection of the present disclosure (S203), and a step of providing a response output for the high-resolution image output from the image analysis model to the user (S205).

[0219] In one embodiment, the method (S200) may be performed by a processor (131) included in a server computing system (130). However, the present invention is not limited thereto, and at least a part of the method (S200) may be performed by a processor (111) of a user computing device (110) or a processor (151) of a training computing system (150), and another part may be performed by a processor (131) included in the server computing system (130).

[0220] For convenience of explanation, the method (S200) is described below as being performed by a processor (131) included in a server computing system (130).

[0221] In step (S201), the processor (131) can receive a high-resolution image (HRI) to be analyzed according to user input.

[0222] This step corresponds to the process of inputting a high-resolution image (HRI) provided by a user into the system (1000) to perform a diagnostic prediction.

[0223] Here, high-resolution imagery (HRI) can include various types of high-resolution images such as tissue pathology images, satellite images, aerial images, quality inspection images, surveillance camera footage, drone images, and celestial images in the form of whole slide images (WSI) related to various industries such as the medical industry, geographic information system (GIS), manufacturing, security, agriculture, and astronomy.

[0224] In step (S203), the processor (131) can input the received high-resolution image into a high-resolution image analysis model pre-learned according to the learning method (S100) of the present disclosure.

[0225] This model performs in-depth analysis of images, introduces prototype embeddings to select important patches from multiple patches of high-resolution images (HRI), and generates prediction values ​​using the selected patches.

[0226] In step (S205), the processor (131) can provide a response output for a high-resolution image output from a high-resolution image analysis model to the user.

[0227] Beyond a simple classification label, this response output can include at least one of a prediction result for whether the high-resolution image contains a particular disease, a prediction result for the grade or stage of the disease, and a prediction result for the location of a lesion for the disease.

[0228] The various embodiments of the present disclosure described above may be implemented in the form of program instructions that can be executed by various computer components and recorded on a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc., either singly or in combination. The program instructions recorded on the computer-readable recording medium may be specially designed and configured for the various embodiments of the present disclosure or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specifically configured to store and execute program instructions, such as ROMs, RAMs, and flash memories. Examples of program instructions include not only machine language codes generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc. Hardware devices may be modified into one or more software modules to perform processing according to the various embodiments of the present disclosure, and vice versa.

[0229] The specific implementations described in this disclosure are illustrative and do not limit the scope of various embodiments of the present disclosure in any way. For the sake of brevity of the specification, descriptions of conventional electronic components, control systems, software, and other functional aspects of the systems may be omitted. In addition, the lines connecting or connecting members between components depicted in the drawings are merely illustrative of functional connections and / or physical or circuit connections, and may be replaced or represented as various additional functional connections, physical connections, or circuit connections in an actual device. In addition, unless specifically mentioned as “essential,” “important,” etc., a component may not be absolutely necessary for the application of various embodiments of the present disclosure.

[0230] Although the detailed description of the present disclosure has been described with reference to preferred embodiments of the present disclosure, it will be understood by those skilled in the art or having ordinary knowledge in the art that various modifications and changes can be made to the various embodiments of the present disclosure without departing from the spirit and technical scope of the various embodiments of the present disclosure as set forth in the claims below. Accordingly, the technical scope of the various embodiments of the present disclosure should not be limited to the contents described in the detailed description of the specification, but should be defined by the claims.

[0231] Various embodiments according to the present disclosure have industrial applicability in that they can minimize unnecessary operations, effectively save memory and computing resources, and dramatically improve analysis speed by selecting only a small number of important patches essential for analysis from among multiple patches of an input image using prototype embedding.

Claims

1. A method for learning an image analysis model performed by a computing device, A step of generating multiple patch embeddings from an image; A step of calculating the similarity between a plurality of prototype embeddings and the plurality of patch embeddings; A step of selecting some of the plurality of patch embeddings based on the similarity for each of the above prototype embeddings; A step of calculating a prediction value based on the above-mentioned selected plurality of patch embeddings; and A method for learning an image analysis model, comprising: a step of updating parameters of an image analysis model so as to minimize loss based on the above predicted value and correct answer data; 2. In paragraph 1, The step of selecting some of the above multiple patch embeddings is: A step of calculating a plurality of sample coefficients representing the relative importance of the plurality of prototype embeddings for the image; A step of calculating the number of selections for patch embeddings for each prototype embedding based on the total number of selections for the plurality of patch embeddings and the plurality of sample coefficients; and A method for learning an image analysis model, comprising: a step of selecting, from the plurality of patch embeddings, a number of patch embeddings equal to the number of patch embeddings selected for each prototype embedding in order of high similarity to the prototype embedding; 3. In paragraph 1, A method for learning an image analysis model, wherein the above plurality of prototype embeddings have different values ​​corresponding to various features of the image.

4. In paragraph 2, The step of calculating the above multiple sample coefficients is: A step of performing a self-attention operation on the above plurality of prototype embeddings to produce an attention matrix; and A method for learning an image analysis model, comprising: a step of applying a softmax function to a plurality of diagonal elements of the attention matrix to produce a plurality of sample coefficients; 5. In paragraph 2, The step of calculating the number of selected patch embeddings for each prototype embedding above is: A step of setting the total number of selections; and A method for learning an image analysis model, comprising: a step of calculating the number of selections of patch embeddings for each prototype embedding by multiplying the total number of selections for each of the plurality of sample coefficients; 6. In paragraph 1, The step of calculating the above predicted value is: A step of calculating a weighted sum of the selected plurality of patch embeddings through a gated attention operation; and A method for learning an image analysis model, comprising: a step of calculating the predicted value based on a predetermined calculation result for a weighted sum of the selected plurality of patch embeddings; 7. In paragraph 6, The step of updating the parameters of the above image analysis model is: A step of calculating an additional weighted sum by adding the weighted sum of the selected plurality of patch embeddings and the weighted sum based on the plurality of sample coefficients of the plurality of prototype embeddings; A step of calculating an auxiliary prediction value based on a predetermined operation result for the above additional weighted sum; and A method for learning an image analysis model, comprising: a step of updating parameters of the image analysis model so that the sum of a first loss based on the predicted value and correct data and a second loss based on the auxiliary predicted value and correct data is minimized; 8. In paragraph 7, The step of updating the parameters of the above image analysis model is: A method for learning an image analysis model, comprising: a step of updating parameters of the image analysis model so that the sum of the first loss, the second loss, and the third loss defined by KL divergence based on the predicted value and the auxiliary predicted value is minimized; 9. In paragraph 1, A method for learning an image analysis model, wherein the image above includes a tissue pathology image.

10. In paragraph 1, A method for learning an image analysis model, wherein the number of the above prototype embeddings and the total number of selections for the above multiple patch embeddings are fixed values.

11. A method for providing a diagnostic response based on image analysis performed by a computing device, A step of receiving a high-resolution image to be analyzed according to user input; A step of inputting the high-resolution image to be analyzed into an image analysis model learned by the method of the first clause; and A method for providing an image analysis-based diagnostic response, comprising: providing a user with a response output for the high-resolution image to be analyzed output from the image analysis model; 12. In paragraph 11, The high-resolution images to be analyzed above include tissue pathology images, A method for providing an image analysis-based diagnostic response, wherein the response output includes at least one of a prediction result regarding whether the high-resolution image to be analyzed includes a specific disease, a prediction result regarding the grade or progression stage of the disease, and a prediction result regarding the location of a lesion for the disease.

13. At least one memory; and At least one processor for executing an image analysis-based response providing method by reading at least one command stored in the memory; At least one processor, Generate multiple patch embeddings from an image, Compute the similarity between multiple prototype embeddings and the multiple patch embeddings, Selecting some of the plurality of patch embeddings based on the similarity for each of the above prototype embeddings, A prediction value is calculated based on the above-selected multiple patch embeddings, Update the parameters of the image analysis model so that the loss based on the above predicted values ​​and correct data is minimized, Receive high-resolution images to be analyzed based on user input, Input the high-resolution image to be analyzed into the image analysis model, An image analysis-based diagnostic response provision system that provides a user with a response output for the high-resolution image to be analyzed output from the image analysis model.

14. In paragraph 13, A plurality of neurons comprising an array including at least one register, at least one programmable logic, and at least one input interface; a plurality of synaptic circuits storing synaptic weights that adjust connection strengths between the plurality of neurons; and at least one routing network that controls data flow between the plurality of neurons; An image analysis-based diagnostic response providing system, wherein each of the plurality of neurons further includes a field programmable gate array (FPGA) implementation for a predetermined artificial neural network that is connected to at least one other neuron through the routing network to set a transmission path of the weight.

15. In paragraph 13, A plurality of neurons organized in an array including at least one register, at least one microprocessor, and at least one input; and a plurality of synaptic circuits storing synaptic weights that adjust connection strengths between the plurality of neurons; An image analysis-based diagnostic response providing system, wherein each of the plurality of neurons further includes an application specific integrated circuit (ASIC) for a predetermined artificial neural network that is connected to at least one other neuron through one of the plurality of synaptic circuits.

Images