Method and apparatus for generating medical image analysis model

Abstract

According to some embodiments of the present disclosure, a medical image analysis method performed by a computing device is disclosed.SOLUTION: Acquiring a first data set including a medical image obtained by photographing a prosthetic implant inserted into a body by a first photographing device and a second data set including a medical image obtained by photographing the prosthetic implant inserted into the body by a second photographing device, the first data set including a medical image having a higher resolution than the second data set; The method may further include training an analysis model for determining the shell type information of the enclosure, and validating the analysis model using the first data set and the second data set.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a medical image analysis method, and more particularly to a method and apparatus for generating an analysis model for analyzing medical images of an implant inserted into the body. [Background technology]

[0002] Implants are devices inserted into the body for medical or cosmetic purposes. There are various types of implants, including breast implants, joint replacement implants, and dental implants. These implants can be inserted to assist a patient's physical functions or improve their appearance. However, implants inserted into the body can cause various side effects over time. Common side effects reported include implant rupture, deformation, inflammatory reactions within the tissues, and capsular contracture. If not detected early, these side effects can have serious consequences on the patient's health.

[0003] Regular checkups are essential after implant placement. Ultrasound is one of the diagnostic methods commonly used for this purpose. Ultrasound has the advantage of not exposing the patient to radiation and providing images in real time. Therefore, ultrasound is widely used as a non-invasive method to check the condition of the implant. However, the process of analyzing ultrasound images is quite subjective, making it difficult to accurately determine the condition of the implant and whether or not there are any side effects. In particular, when deformation or rupture of the implant is subtle, checking it with the naked eye requires a high level of skill.

[0004] Therefore, there is a need for a method and device that can more precisely monitor the state of an implant inserted into the body and detect any side effects that may occur at an early stage. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Korean Patent Registration No. 10-2535865 Summary of the Invention [Problem to be solved by the invention]

[0006] The present disclosure has been made in response to the above-mentioned background art, and aims to provide a method and apparatus for generating an analytical model for analyzing medical images of an implant inserted into the body. [Means for solving the problem]

[0007] To achieve the above object, a method for generating an analytical model for determining shell type information of an implant inserted into a body, executed by a computing device, is disclosed. The method may include acquiring a first dataset including medical images of the implant inserted into a body taken by a first imaging device and a second dataset including medical images of the implant inserted into a body taken by a second imaging device, the first dataset including medical images having a higher resolution than the second dataset; training an analytical model for determining shell type information of the implant using the first dataset; and validating the analytical model using the first dataset and the second dataset.

[0008] Alternatively, the first imaging device may be an imaging device manufactured by a different manufacturer than the second imaging device.

[0009] Alternatively, the method may further include a step of performing pre-processing to cut off side regions of the medical images included in the first data set or the second data set at a predetermined rate.

[0010] Alternatively, training the analytical model for determining shell type information of the prosthesis using the first dataset may include training the analytical model using a loss function that imposes a higher penalty on misclassification of a textured type compared to a smooth type among the shell type information of the prosthesis.

[0011] Alternatively, validating the analytical model using the first dataset and the second dataset may include performing cross validation on the analytical model using the first dataset; and performing external validation on the analytical model using the second dataset.

[0012] Alternatively, validating the analytical model using the first dataset and the second dataset may further include performing external validation of the analytical model using an additional fifth dataset including a publicly available image.

[0013] Alternatively, the method may further include performing quantitative validation of the analytical model using a medical image in which pixels with low classification contributions are masked based on a predetermined ratio, so as to determine shell type information of the prosthesis based on pixels corresponding to a layer of the prosthesis.

[0014] Alternatively, the method may further include acquiring a third dataset including medical images of a state in which a prosthesis inserted into the body has ruptured, and a fourth dataset including medical images of a state in which no prosthesis is inserted into the body; and performing uncertainty estimation on the analytical model using at least a portion of the first dataset, the second dataset, the third dataset, or the fourth dataset.

[0015] Alternatively, performing an uncertainty estimation for the analytical model using at least a portion of the first data set, the second data set, the third data set, or the fourth data set may include performing an uncertainty estimation for the analytical model based on a hypothesis that the third data set has higher prediction uncertainty than the first data set or the second data set, and that the fourth data set has higher prediction uncertainty than the third data set.

[0016] Alternatively, the method may further include performing a post-hoc explainable interpretation on the analytical model to generate region information representing pixels associated with the shell type information of the prosthesis by using classification contributions of pixels to the shell type information of the prosthesis.

[0017] Alternatively, the region information may be displayed in a heat map format.

[0018] To solve the above-mentioned problems, a computer program stored on a computer-readable storage medium is disclosed. The computer program includes instructions for causing one or more processors to execute a method, the method may include acquiring a first dataset including medical images of an implant inserted into a body captured by a first imaging device and a second dataset including medical images of the implant inserted into a body captured by a second imaging device, the first dataset including medical images having a higher resolution than the second dataset; training an analytical model using the first dataset to determine shell type information of the implant; and validating the analytical model using the first dataset and the second dataset.

[0019] A computing device for executing a method for generating an analytical model for solving the above-mentioned problems, the computing device including a memory and a processor, wherein the processor acquires a first dataset including medical images of an implant inserted into a body taken by a first imaging device and a second dataset including medical images of an implant inserted into a body taken by a second imaging device, the first dataset including medical images having a higher resolution than the second dataset, uses the first dataset to train an analytical model for determining shell type information of the implant, and validates the analytical model using the first dataset and the second dataset. [Effects of the Invention]

[0020] The present disclosure can provide a method and apparatus for generating an analytical model for analyzing medical images of an implant inserted into a body. [Brief explanation of the drawings]

[0021] [Figure 1]FIG. 1 is a block diagram of a computing device for performing a method for analyzing medical images and a method for generating an analytical model for analyzing medical images according to some embodiments of the present disclosure. [Figure 2] FIG. 2 is a schematic diagram illustrating network functions according to some embodiments of the present disclosure. [Figure 3] 1 is a schematic diagram of a medical image analysis system according to some embodiments of the present disclosure. [Figure 4] FIG. 1 illustrates a block diagram of an exemplary side effect information generator according to some embodiments of the present disclosure. [Figure 5] FIG. 2 illustrates a block diagram of an exemplary implant information generator according to some embodiments of the present disclosure. [Figure 6] FIG. 1 is a schematic diagram of an analytical model generation system according to some embodiments of the present disclosure. [Figure 7] 10A-10C are diagrams illustrating results regarding quantitative validation of analytical models according to some embodiments of the present disclosure. [Figure 8] FIG. 10 is another diagram illustrating results regarding quantitative validation of analytical models according to some embodiments of the present disclosure. [Figure 9] FIG. 10 is another diagram illustrating results regarding quantitative validation of analytical models according to some embodiments of the present disclosure. [Figure 10] FIG. 1 illustrates results regarding unpredictability estimation of analytical models according to some embodiments of the present disclosure. [Figure 11] FIG. 1 illustrates results regarding post-hoc explainable analysis of analytical models according to some embodiments of the present disclosure. [Figure 12] 1 is a flowchart of a method for analyzing medical images according to some embodiments of the present disclosure. [Figure 13] 1 is a flowchart of a method for generating an analytical model according to some embodiments of the present disclosure. [Figure 14] FIG. 1 illustrates a simplified general overview of an exemplary computing environment in which embodiments of the present disclosure may be implemented. DETAILED DESCRIPTION OF THE INVENTION

[0022] Various embodiments and / or aspects are disclosed with reference to the drawings. In the following description, for purposes of explanation, numerous specific details are disclosed to facilitate a thorough understanding of one or more aspects. However, it will be understood by those of ordinary skill in the art that these aspects may be practiced without such specific details. The following description and the accompanying drawings set forth certain exemplary aspects of one or more aspects in detail. However, these aspects are illustrative, and some of the various methods of the principles of the various aspects may be utilized, and the written description is intended to include all such aspects and their equivalents. In particular, the terms "embodiment," "example," "aspect," "exemplary," and the like, as used herein, may not be construed as implying that any described aspect or design is superior or advantageous over other aspects or designs.

[0023] Hereinafter, the same or similar components will be denoted by the same reference numerals regardless of the drawing numbers, and redundant descriptions thereof will be omitted. Furthermore, in describing the embodiments disclosed herein, if a detailed description of related known technology is deemed to detract from the gist of the embodiments disclosed herein, the detailed description will be omitted. Furthermore, the accompanying drawings are intended to facilitate understanding of the embodiments disclosed herein, and the technical ideas disclosed herein are not limited by the accompanying drawings.

[0024] The terms used herein are for the purpose of describing embodiments and are not intended to limit the present disclosure. In this specification, the singular form includes the plural form unless otherwise specified in the text. As used herein, "comprises" and / or "comprising" do not exclude the presence or addition of one or more other elements other than the elements mentioned.

[0025] Although terms such as "first" and "second" are used to describe various elements or components, it is understood that these elements or components are not limited by such terms. These terms are merely used to distinguish one element or component from another. Therefore, it is understood that a first element or component referred to below can also be a second element or component within the technical concept of the present disclosure.

[0026] Unless otherwise defined, all terms (including technical and scientific terms) used herein may be used in the sense commonly understood by a person of ordinary skill in the art to which this disclosure pertains. Furthermore, terms defined in commonly used dictionaries may not be interpreted ideally or excessively unless specifically defined otherwise.

[0027] Furthermore, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or." That is, unless otherwise specified or clear from the context, "X utilizes A or B" is intended to mean either of the natural implied permutations. That is, if X utilizes A; if X utilizes B; or if X utilizes both A and B, then "X utilizes A or B" can apply in either case. Additionally, as used herein, the term "and / or" should be understood to refer to and include all possible combinations of one or more of the associated listed items.

[0028] The term "at least one of A or B" should be interpreted as meaning "when containing only A," "when containing only B," or "when combined in the configuration of A and B."

[0029] Those skilled in the art should further appreciate that the various illustrative logical blocks, configurations, modules, circuits, means, logic, and algorithm steps described in connection with the embodiments disclosed herein can be embodied in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the various illustrative components, blocks, configurations, means, logic, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is embodied as hardware or software depends on the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in various ways for each particular application. However, such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

[0030] The description of the presented embodiments is provided to enable one of ordinary skill in the art to make or practice the invention. Various modifications to these embodiments will be apparent to those of ordinary skill in the art. The generic principles defined herein may be applied to other embodiments without departing from the scope of the present disclosure. Therefore, the present invention is not limited to the embodiments shown herein. The present invention is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0031] Furthermore, the term "etc.", such as "A, B, etc.", should be interpreted to mean "when including only A," "when including only B," or "when combined in the configuration of A and B."

[0032] The suffixes "module" and "section" used in the following description for components are given or mixed for the sole purpose of ease of writing the specification, and do not have any meanings or roles that are distinct from each other in themselves.

[0033] The objectives and effects of the present disclosure, as well as technical configurations for achieving them, will become apparent from the following detailed description of the embodiments together with the accompanying drawings. When describing the present disclosure, if it is determined that a detailed description of a known function or configuration may unnecessarily obscure the gist of the present disclosure, the detailed description will be omitted. Furthermore, the terms used below are defined in consideration of the functions in the present disclosure, and may vary depending on the intentions or practices of users or operators.

[0034] However, the present disclosure is not limited to the embodiments disclosed below and may be implemented in various other forms. However, the present embodiments are provided to complete the disclosure and to fully inform those skilled in the art of the disclosure of the categories of the disclosure, and the present disclosure is defined only by the categories of claims. Therefore, the definition should be based on the content of the entire specification.

[0035] FIG. 1 is a block diagram of a computing device for performing a method for analyzing medical images and a method for generating an analytical model for analyzing medical images according to some embodiments of the present disclosure.

[0036] 1, computing device 100 may include processor 110, memory 130, and network unit 150. The configuration of computing device 100 shown in FIG. 1 is merely a simplified example. In some embodiments of the present disclosure, computing device 100 may include other components for executing the computing environment of computing device 100, and only some of the disclosed components may constitute computing device 100.

[0037] The processor 110 may be configured with one or more cores and may include a processor for data analysis and processing, deep learning, such as a central processing unit (CPU) of a computing device, a general-purpose graphics processing unit (GPGPU), or a tensor processing unit (TPU). The processor 110 may read a computer program stored in the memory 130 and perform data conversion, calculation, generation, etc. to execute a medical image analysis method according to some embodiments of the present disclosure. For example, the processor 110 may execute steps for executing a medical image analysis method via a medical image analysis system as described with reference to FIGS. 3 to 5. To this end, the processor 110 may embody a medical image analysis system 1000 and its components. The processor 110 may also read a computer program stored in the memory 130 and perform data conversion, calculation, generation, etc. to execute a method for generating an analytical model according to some embodiments of the present disclosure. For example, the processor 110 may execute steps for executing a method for generating an analytical model via an analytical model generation system as described with reference to FIGS. 6 to 11. To this end, the processor 110 may embody an analytical model generation system and its components. According to some embodiments of the present disclosure, the processor 110 may perform computations for training a neural network using training data to execute a method for generating an analytical model. The processor 110 may perform computations for training a neural network in deep learning (DL), such as processing input data for training, extracting features from the input data, calculating errors, and updating weights of the neural network using backpropagation. At least one of the CPU, GPGPU, and TPU of the processor 110 may process computations for executing a medical image analysis method or a method for generating an analytical model.For example, a CPU and a GPGPU may both process computations to perform a medical image analysis method or a method for generating an analytical model. Also, in some embodiments of the present disclosure, processors of multiple computing devices may be used together to process data transformation, computation, generation, network function training, and data classification using the network function to perform a medical image analysis method or a method for generating an analytical model. Also, a computer program executed on a computing device according to some embodiments of the present disclosure may be a CPU, GPGPU, or TPU executable program.

[0038] According to some embodiments of the present disclosure, memory 130 may store any type of information generated or determined by processor 110 and any type of information received by network unit 150. For example, memory 130 may store data generated in the process of executing a medical image analysis method or a method for generating an analytical model by processor 110. Memory 130 may also store data received from an external source in the process of executing a medical image analysis method or a method for generating an analytical model by processor 110. However, without being limited thereto, memory 130 may store various information for executing a medical image analysis method or a method for generating an analytical model according to some embodiments of the present disclosure.

[0039] According to some embodiments of the present disclosure, memory 130 may include at least one of the following types of storage media: flash memory, hard disk, multimedia card micro, card-type memory (e.g., SD or XD memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, and optical disk. Computing device 100 may also operate in conjunction with web storage that performs storage functions of memory 130 over the internet. The foregoing memory descriptions are exemplary, and the present disclosure is not limited thereto.

[0040] The network unit 150 according to some embodiments of the present disclosure may use any form of known wired or wireless communication system.

[0041] The network unit 150 may transmit and receive information, a user interface, etc., processed by the processor 110 through communication with other terminals. For example, the network unit 150 may provide a user interface generated by the processor 110 to a client (e.g., a user terminal). The network unit 150 may also receive an external input from a user authorized as a client and transmit it to the processor 110. In this case, the processor 110 may process operations such as output, correction, modification, and addition of information provided through the user interface based on the external user input transmitted from the network unit 150.

[0042] Specifically, for example, the network unit 150 can transmit and receive various information for executing a medical image analysis method or a method for generating an analytical model according to some embodiments of the present disclosure. For example, the network unit 150 can receive one or more medical images stored in a database or a dataset including medical images. The network unit 150 can also transmit some data generated in the process of executing a medical image analysis method or a method for generating an analytical model described below to an external device. For example, the network unit 150 can transmit analysis results obtained by processing a medical image through an analytical model to an external device.

[0043] Meanwhile, a computing device 100 according to some embodiments of the present disclosure may include a server as a computing system that transmits and receives information through communication with a client. In this case, the client may be any type of terminal that can access the server. For example, the server computing device 100 may receive a query from a user terminal and generate a single information processing result corresponding to the query. In this case, the server computing device 100 may provide a user interface including the processing result to the user terminal. In this case, the user terminal may output the user interface received from the server computing device 100 and input or process information through user interaction.

[0044] In additional embodiments, computing device 100 may include any form of terminal to which data resources generated from any server may be delivered and which performs additional information processing.

[0045] FIG. 2 illustrates an example structure of an artificial intelligence-based model according to some embodiments of the present disclosure.

[0046] As used herein, artificial intelligence model, artificial intelligence-based model, computational model, neural network, network function, and neural network may be used interchangeably.

[0047] A neural network can be composed of a collection of interconnected computational units, generally called nodes. Such nodes are sometimes called neurons. A neural network is composed of at least one or more nodes. The nodes (or neurons) that make up a neural network can be interconnected by one or more links.

[0048] Within a neural network, one or more nodes connected via links can form a relative relationship of input node and output node. The concepts of input node and output node are relative, and any node that has an output node relationship with one node can also have an input node relationship with another node, and vice versa. As mentioned above, the relationship of output node to input node can be generated around links. One or more output nodes can be connected to one input node via links, and vice versa.

[0049] In a relationship between an input node and an output node connected via a link, the value of the data of the output node can be determined based on the data input to the input node. Here, the link interconnecting the input node and the output node may have a weight. The weight is variable and can be changed by a user or an algorithm so that the neural network performs a desired function. For example, when one or more input nodes are interconnected to one output node by respective links, the output node can determine the output node value based on the value input to the input node connected to the output node and the weight assigned to the link corresponding to each input node.

[0050] As described above, a neural network is a network in which one or more nodes are interconnected via one or more links, forming a relationship between input nodes and output nodes within the neural network. The characteristics of a neural network can be determined by the number of nodes and links within the neural network, the relationships between the nodes and links, and the weighting values ​​assigned to each link. For example, if two neural networks have the same number of nodes and links but different link weights, the two neural networks can be recognized as different from each other.

[0051] A neural network may be composed of a set of one or more nodes. A subset of the nodes constituting a neural network may constitute a layer. Some of the nodes constituting a neural network may constitute a layer based on their distance from the initial input node. For example, a set of nodes whose distance from the initial input node is n may constitute n layers. The distance from the initial input node may be defined by the minimum number of links that must be traversed to reach the node from the initial input node. However, this definition of a layer is arbitrary for illustrative purposes, and the order of layers in a neural network may be defined differently. For example, a node's layer may be defined by its distance from the final output node.

[0052] In one embodiment of the present disclosure, a collection of neurons or nodes may be defined in terms of a layer.

[0053] An initial input node may refer to one or more nodes in a neural network to which data is directly input without a link in relation to other nodes. Alternatively, it may refer to a node in a neural network that does not have other input nodes connected by links in relation to nodes based on links. Similarly, a final output node may refer to one or more nodes in a neural network that do not have output nodes in relation to other nodes. Furthermore, a hidden node may refer to a node that constitutes a neural network but is not an initial input node or a final output node.

[0054] In one embodiment of the present disclosure, the number of nodes in the input layer may be the same as the number of nodes in the output layer, or the number of nodes may decrease and then increase again as the layer progresses from the input layer to the hidden layer. In another embodiment of the present disclosure, the number of nodes in the input layer may be fewer than the number of nodes in the output layer, or the number of nodes may increase as the layer progresses from the input layer to the hidden layer. In another embodiment of the present disclosure, the number of nodes in the input layer may be greater than the number of nodes in the output layer, or the number of nodes may decrease as the layer progresses from the input layer to the hidden layer. In another embodiment of the present disclosure, the number of nodes in the input layer may be a combination of the above-described neural networks.

[0055] An artificial intelligence-based model according to an embodiment of the present disclosure may include a deep neural network (DNN). A deep neural network may refer to a neural network that includes multiple hidden layers in addition to an input layer and an output layer. A deep neural network can be used to understand the latent structures of data, such as photos, text, video, audio, protein sequence structure, gene sequence structure, peptide sequence structure, and music (e.g., what objects are in the photo, what is the content and emotion of the text, what is the content and emotion of the audio, etc.), and / or the binding affinity between peptides and MHCs. The deep neural network may include a convolutional neural network (CNN), a recurrent neural network (RNN), an autoencoder, a restricted Boltzmann machine (RBM), a deep belief network (DBN), a Q-network, a U-network, a Siamese network, a generative adversarial network (GAN), a transformer, etc. The foregoing descriptions of deep neural networks are for illustrative purposes only, and the present disclosure is not limited thereto.

[0056] The artificial intelligence-based model of the present disclosure can be represented by a network structure of any of the aforementioned structures, including an input layer, a hidden layer, and an output layer.

[0057] Neural networks that can be used in the artificial intelligence-based models of the present disclosure may be trained using at least one of supervised learning, unsupervised learning, semi-supervised learning, transfer learning, active learning, or reinforcement learning. Training a neural network may be a process in which the neural network applies knowledge to the neural network to perform a particular operation.

[0058] Neural networks can be trained to minimize output errors. Training a neural network involves repeatedly inputting training data into the neural network, calculating the error between the neural network's output and the target for the training data, and backpropagating the neural network's error from the output layer to the input layer of the neural network in a direction to reduce the error, thereby updating the weights of each node of the neural network. In supervised learning, training data in which a correct answer is labeled is used for each training data (i.e., labeled training data). In unsupervised learning, the correct answer may not be labeled for each training data. For example, in supervised learning for data classification, the training data may be data in which a category is labeled for each training data. Labeled training data is input into the neural network, and the error can be calculated by comparing the neural network's output (category) with the label of the training data. As another example, in unsupervised learning for data classification, the error can be calculated by comparing the input training data with the neural network's output. The calculated error is backpropagated in the reverse direction of the neural network (i.e., from the output layer to the input layer), and the connection weights of each node in each layer of the neural network can be updated through backpropagation. The amount of change in the connection weights of each updated node can be determined according to the learning rate. The neural network calculation for input data and backpropagation of the error can constitute a learning cycle (epoch). The learning rate can be applied differently depending on the number of iterations of the neural network learning cycle. For example, a high learning rate can be used in the early stages of neural network learning to quickly ensure a certain level of performance and improve efficiency, and a low learning rate can be used in the later stages of learning to improve accuracy.

[0059] In neural network training, training data is generally a subset of actual data (i.e., data to be processed using the trained neural network). Therefore, there can be a learning cycle in which errors decrease for training data but increase for actual data. Overfitting is a phenomenon in which excessive learning on training data increases errors for actual data. For example, a neural network that has learned cats by showing yellow cats may be unable to recognize cats that are not yellow as cats. Overfitting can increase errors in machine learning algorithms. Various optimization methods can be used to prevent overfitting. Methods that can be used include increasing the training data, regularization, dropout (which deactivates some network nodes during the training process), and the use of batch normalization layers.

[0060] According to one embodiment of the present disclosure, a computer-readable medium having stored thereon a data structure is disclosed, which can be stored in a memory unit, executed by a processor, and transmitted and received by a communication unit in the present disclosure.

[0061] A data structure can refer to the organization, management, and storage of data that allows efficient access and modification. A data structure can refer to the organization of data to solve a specific problem (e.g., data analysis, data retrieval, data storage, data modification). A data structure can also be defined as a physical or logical relationship between data elements designed to support a specific data processing function. Logical relationships between data elements may include user-defined interlinking relationships between data elements. Physical relationships between data elements may include actual relationships between data elements physically stored in a computer-readable storage medium (e.g., persistent storage). Specifically, a data structure may include a collection of data, relationships between data, and functions or instructions that can be applied to the data. An effectively designed data structure enables a computing device to perform operations with minimal use of computing device resources. Specifically, an effectively designed data structure enables a computing device to increase the efficiency of operations, reading, inserting, deleting, comparing, exchanging, and searching.

[0062] Data structures can be divided into linear and non-linear data structures depending on their type. A linear data structure is a structure in which only one piece of data is linked to another. Linear data structures may include lists, stacks, queues, and deques. A list may refer to a collection of data that has an internal order. Lists may also include linked lists. A linked list may be a data structure in which data is linked in a single line using a pointer. In a linked list, the pointer may contain information about the connection to the next or previous piece of data. Linked lists may be expressed as singly linked lists, doubly linked lists, and circularly linked lists depending on their type. A stack is a data structure in which data can be accessed in a limited way. A stack may be a linear data structure in which data can be manipulated (e.g., inserted or deleted) at only one end of the data structure. Data stored in a stack may be a LIFO (Last in First Out) data structure. A queue is a data structure that allows limited access to data, and unlike a stack, it may be a data structure that stores data later and retrieves it later (FIFO - First in First out). A deck is a data structure that allows data to be processed from both ends of the data structure.

[0063] A non-linear data structure is a structure in which multiple pieces of data are concatenated after one piece of data. A non-linear data structure may include a graph data structure. A graph data structure can be defined by vertices and edges, and the edges may include lines connecting two different vertices. A graph data structure may include a tree data structure. A tree data structure may have a single data path connecting two different vertices among the multiple vertices included in the tree. In other words, a graph data structure may be a data structure that does not form a loop.

[0064] As used herein, the terms artificial intelligence-based model, computational model, neural network, network function, and neural network may be used interchangeably. Hereinafter, they will be referred to as a neural network. A data structure may include a neural network. A data structure including a neural network may be stored on a computer-readable medium. A data structure including a neural network may also include data preprocessed for processing by the neural network, data input to the neural network, neural network weights, neural network hyperparameters, data obtained from the neural network, activation functions associated with each node or layer of the neural network, and loss functions for neural network training. A data structure including a neural network may include any of the components disclosed above. That is, a data structure including a neural network may include all or any combination of data preprocessed for processing by the neural network, data input to the neural network, neural network weights, neural network hyperparameters, data obtained from the neural network, activation functions associated with each node or layer of the neural network, and loss functions for neural network training. In addition to the above configurations, the data structure including the neural network may include any other information that determines the characteristics of the neural network. Furthermore, the data structure may include any form of data used or generated in the process of computing the neural network, and is not limited to the above. The computer-readable medium may include a computer-readable recording medium and / or a computer-readable transmission medium. A neural network may generally be composed of a collection of interconnected computational units called nodes. Such nodes are sometimes called neurons. A neural network is composed of at least one or more nodes.

[0065] The data structure may include data to be input to a neural network. The data structure including the data to be input to a neural network may be stored on a computer-readable medium. The data to be input to a neural network may include training data input during the training process of the neural network and / or input data to a neural network after training has been completed. The data to be input to a neural network may include data that has undergone pre-processing and / or data that is the target of pre-processing. Pre-processing may include a data processing process for inputting data to a neural network. Therefore, the data structure may include data that is the target of pre-processing and data generated by pre-processing. The above-described data structures are exemplary, and the present disclosure is not limited thereto.

[0066] The data structure may include neural network weights. (In this specification, the terms "weights" and "parameters" are used interchangeably.) The data structure including the neural network weights can be stored on a computer-readable medium. The neural network may include multiple weights. The weights are variable and can be adjusted by a user or an algorithm to enable the neural network to perform a desired function. For example, if one or more input nodes are interconnected to an output node by respective links, the output node can determine a data value to be output from the output node based on the values ​​input to the input nodes connected to the output node and the weights assigned to the links corresponding to each input node. The above data structure is illustrative, and the present disclosure is not limited thereto.

[0067] As non-limiting examples, the weights may include weights that change during neural network training and / or weights at which neural network training has been completed. The weights that change during neural network training may include weights at the start of a training cycle and / or weights that change during a training cycle. The weights at which neural network training has been completed may include weights at which a training cycle has been completed. Thus, a data structure including neural network weights may include a data structure including weights that change during neural network training and / or weights at which neural network training has been completed. Thus, the weights and / or combinations of weights described above are included in the data structure including neural network weights. The aforementioned data structures are exemplary, and the present disclosure is not limited thereto.

[0068] The data structure containing neural network weights can be stored in a computer-readable storage medium (e.g., memory, hard disk) after undergoing a serialization process. Serialization is a process of converting a data structure into a format that can be stored in the same or a different computing device and later reconstructed for use. A computing device can serialize the data structure to transmit and receive data over a network. The serialized data structure containing neural network weights can be reconstructed on the same or another computing device through deserialization. The data structure containing neural network weights is not limited to serialization. Furthermore, the data structure containing neural network weights may include a data structure (e.g., a non-linear data structure such as a B-tree, R-tree, trie, m-way search tree, AVL tree, or Red-Black tree) that increases computational efficiency while minimizing the use of computing device resources. The foregoing is merely exemplary, and the present disclosure is not limited thereto.

[0069] The data structure may include hyper-parameters of the neural network. The data structure including the hyper-parameters of the neural network can be stored in a computer-readable medium. The hyper-parameters may be user-variable variables. The hyper-parameters may include, for example, a learning rate, a cost function, the number of iterations of a learning cycle, weight initialization (e.g., setting a range of weights to be subjected to weight initialization), and the number of hidden units (e.g., the number of hidden layers, the number of nodes in the hidden layer). The above-described data structure is merely an example, and the present disclosure is not limited thereto.

[0070] An artificial intelligence-based model according to an embodiment of the present disclosure may include a large language model (LLM). In the present disclosure, a large language model may refer to an artificial intelligence-based model trained using a huge amount of training data to perform natural language processing. The large language model may include a transformer, a model of an encoder sequence of the transformer, and / or a model of a decoder sequence of the transformer. The model of the encoder sequence of the transformer may correspond to an artificial intelligence model using a transformer encoder structure. The model of the decoder sequence of the transformer may correspond to an artificial intelligence model using a transformer decoder structure.

[0071] In one embodiment, the Transformer may be configured with an encoder that encodes input data and a decoder that decodes the encoded data. The Transformer may have a structure that receives a series of input data, undergoes encoding and decoding steps, and outputs a series of output data. In one embodiment, the series of input data may be processed into a form that can be operated by the Transformer. The process of processing the series of input data into a form that can be operated by the Transformer may include a tokenizing process and an embedding process. The tokenizing process may refer to a process of dividing the series of input data into tokens of a certain unit. For example, the certain unit may include a word unit. The embedding process may refer to a process of converting at least one token tokenized from the series of input data into an embedded vector.

[0072] In one embodiment, the Transformer may obtain an embedded vector to be input to the encoder by combining a token-embedded vector in which at least one token corresponding to a series of input data is embedded, a segment-embedded vector that divides a sentence containing each token, and a position-embedded vector that reflects the position of the token. The Transformer's encoder-series model and decoder-series model may also obtain embedded vectors in the same manner.

[0073] In one embodiment, in order for the Transformer to encode and decode a series of input data, the encoder and decoder in the Transformer may utilize an attention algorithm. The attention algorithm may refer to an algorithm that, for a given query, calculates a similarity by applying a softmax function to an attention score obtained by multiplying the query by a key, and then calculates an attention value for the query by multiplying the calculated similarity by a value.

[0074] In one embodiment, the self-attention algorithm may refer to an attention algorithm that uses a query, a key, and a value generated by multiplying the same embedded vector by a query weight, a key weight, and a value weight, respectively. The cross-attention algorithm may refer to an attention algorithm that uses a query generated by multiplying a first embedded vector by a query weight and a key and a value generated by multiplying a second embedded vector by a key weight and a value weight, respectively. The query weight, the key weight, and the value weight may be trainable parameters that are updated through a training process of a large-scale language model.

[0075] In one embodiment, the encoder of the Transformer may include an embedding layer, a self-attention layer that applies a self-attention algorithm to the embedding vectors, a normalization layer, and a feed-forward neural network (FFN). The encoder may also have a configuration in which N unit structures, each including a self-attention layer, a normalization layer, and a feed-forward neural network, are connected. The decoder of the Transformer may include an embedding layer, a masked self-attention layer, a normalization layer, a cross-attention layer that applies a cross-attention algorithm, and a feed-forward neural network. The decoder may also have a configuration in which N unit structures, each including a masked self-attention layer, a normalization layer, a cross-attention layer, and a feed-forward neural network, are connected. The masked self-attention layer may correspond to a layer that calculates an attention value for each sequence of words included in a series of input data.

[0076] The Transformer may include not only an encoder and a decoder, but also additional components such as a linear layer, a softmax layer, etc. The Transformer encoder-series model and the Transformer decoder-series model may also include not only an encoder and a decoder, respectively, but also the aforementioned additional components. Methods for constructing a Transformer using an attention algorithm may include the methods disclosed in Vaswani et al., Attention Is All You Need, 2017 NIPS, which is incorporated herein by reference.

[0077] In one embodiment, an attention layer, such as a self-attention layer, a masked self-attention layer, or a cross-attention layer, can correspond to a multi-head attention layer including multiple parallel attention layers. The multi-head attention layer can concatenate attention values ​​output from the multiple attention layers and output an output attention value by multiplying the concatenated matrix by an output weight. The output attention value output from the multi-head attention layer may have the same size as the attention value output from a single attention layer.

[0078] In one embodiment, the Transformer may be trained through a Masked Language Model (MLM) process, a Next Sentence Prediction (NSP) process, etc. The MLM process may refer to a training process of predicting masked words through a series of training data in which some words are masked. The NSP process may refer to a training process of determining whether two sentences are concatenated from a series of training data containing any two sentences.

[0079] In one embodiment, the large-scale language model can process various data formats, such as not only natural language text but also image data, audio data, and video data. The large-scale language model can embed data to convert data having various data formats into a sequence of operable data. The large-scale language model can process additional data representing relative positions or phase relationships between a sequence of input data. Alternatively, a vector representing the relative positions or phase relationships between the input data may be added to the sequence of input data and embedded into the sequence of input data. In one example, the relative positions between the sequence of input data may include, but are not limited to, the order of words in a natural language sentence, the relative positions of each segmented image, the time order of segmented audio waveforms, etc. The process of adding information representing the relative positions or phase relationships between the sequence of input data may be referred to as positional encoding.

[0080] An example of a large-scale language model for processing image data (Vision Transformer, ViT) is disclosed in Dosovitskiy, et al., AN IMAGE IS WORTH 16x16 WORDS: TRANSFORMERS FOR IMAGE RECOGNITION AT SCALE, which is incorporated herein by reference.

[0081] An artificial intelligence model according to an embodiment of the present disclosure may include a multi-modal large-scale language model. The multi-modal large-scale language model may refer to a large-scale language model that can understand and process relationships between different data types, such as natural language text data, image data, audio data, and video data. The multi-modal language model may include multiple encoders that encode input data corresponding to each data type. The multi-modal language model may be trained to calculate similarities between embedded vectors encoded by encoders of each data type through training data including data of different data types, and to calculate higher similarities between same pairs and lower similarities between different pairs.

[0082] An example of a multimodal large-scale language model that understands and processes relationships between image data and natural language text data (Contrastive Language-Image Pre-training, CLIP) is disclosed in Alec Radford, et al., LEARNING TRANSFERABLE VISUAL MODELS FROM NATURAL LANGUAGE SUPERVISION, which is incorporated herein by reference.

[0083] Figure 3 is a schematic diagram of a medical image analysis system 1000 according to some embodiments of the present disclosure. Figure 4 is a block diagram of an exemplary side effect information generator 410 according to some embodiments of the present disclosure. Figure 5 is a block diagram of an exemplary prosthesis information generator 420 according to some embodiments of the present disclosure.

[0084] An exemplary embodiment of a medical image analysis system 1000 and its components for performing medical image analysis methods according to some embodiments of the present disclosure will now be described with reference to FIGS.

[0085] A medical image analysis system 1000 according to some embodiments of the present disclosure may provide analysis results for medical images of an implant inserted into the body. Specifically, the medical image analysis system 1000 according to some embodiments of the present disclosure may generate side effect information and implant information by processing medical images of an implant inserted into the body. For example, the medical image analysis system 1000 according to some embodiments of the present disclosure may generate binary information representing positive or negative information about the implant inserted into the body and information about side effects caused by the implant inserted into the body. In addition, the medical image analysis system 1000 according to some embodiments of the present disclosure may provide a user with area information displayed in a visualized format, such as a heat map, allowing the user to easily identify areas where side effects such as rupture have occurred. However, without being limited thereto, the medical image analysis system 1000 may provide analysis results for medical images of an implant inserted into the body in various ways.

[0086] 3, the medical image analysis system 1000 may include a medical image acquisition unit 200, an analysis model control unit 300, and a medical image analysis unit 400. However, without being limited thereto, the medical image analysis system 1000 may further include other components for providing analysis results for medical images, and the medical image analysis system 1000 may include only some of the disclosed components.

[0087] According to some embodiments of the present disclosure, the medical image acquisition unit 200 can acquire medical images of an implant inserted into the body.

[0088] In some examples, the medical image acquisition unit 200 can acquire medical images 10 to be analyzed by the analytical model. For example, the medical image acquisition unit 200 can receive registered medical images via a user interface. In some examples, the medical images may include various types of images used in medical diagnosis, treatment, and research. For example, the medical images may include radiography (X-ray) images, computed tomography (CT) images, magnetic resonance imaging (MRI) images, or ultrasound images. However, without being limited thereto, the medical images may include various types of images capable of capturing images of implants inserted into the body. Furthermore, although the following describes an embodiment relating to medical images capturing images of implants inserted into the breast, the present disclosure is not limited thereto.

[0089] In some examples, the analytical model control unit 300 can control the medical image analysis unit 400, which processes medical images. For example, when a medical image is acquired by the medical image acquisition unit 200, the analytical model control unit 300 can control the medical image analysis unit 400 (or an analytical model included in the medical image analysis unit 400) to process the medical image and generate prosthesis information and side effect information as analysis results. In some examples, as described below, the analytical model control unit 300 can transmit binary information generation instructions 20 and region information generation control instructions 40 to the medical image analysis unit 400 to control the medical image analysis unit 400. For example, the analytical model control unit 300 can transmit a binary information generation instruction to the medical image analysis unit 400 that activates a binary information generation operation of the analytical model included in the medical image analysis unit 400. The analytical model control unit 300 can also transmit a region information generation control instruction to the image analysis unit 400 that activates or deactivates a region information generation operation of the analytical model included in the medical image analysis unit 400.

[0090] In some examples, the side effect information may include information about side effects caused by the implant inserted into the body. For example, the side effect information may include capsule thickening side effect information, folding side effect information, seroma side effect information, upside-down rotation side effect information, capsule calcification side effect information, capsule nodule side effect information, etc. However, the side effect information may include various information without being limited thereto.

[0091] In some examples, the prosthesis information may include information about the prosthesis inserted into the body. For example, the prosthesis information may include information about the type of prosthesis, the shape of the prosthesis, information about the manufacturer of the prosthesis, information about the ingredients of the prosthesis, information about the location of the prosthesis, etc. However, the prosthesis information may include various information without being limited thereto.

[0092] According to some embodiments of the present disclosure, the prosthesis information and the side effect information may include binary information and region information corresponding to the binary information. In some examples, the binary information may represent information about the prosthesis inserted into the body and information about side effects caused by the prosthesis inserted into the body as positive or negative. For example, binary information related to rupture side effect information may indicate whether rupture of the prosthesis has occurred on the analyzed medical image as positive or negative. As another example, binary information related to prosthesis manufacturer information may indicate whether the prosthesis manufacturer information can be identified on the medical image as positive or negative.

[0093] In some examples, the region information may include information indicating regions related to the prosthesis information and the side effect information. For example, the region information related to the rupture side effect information may display information indicating the region where the rupture occurred. In some examples, the region information may be displayed in a heat map format. For example, the region information related to the rupture side effect information may display the location where the rupture of the prosthesis occurred in the medical image in a heat map format.

[0094] In some examples, the medical image analysis unit 400 may generate side effect information and prosthesis information by processing medical images. For example, as shown in FIG. 3, the medical image analysis unit 400 may include a side effect information generation unit 410 and a prosthesis information generation unit 420. In some examples, the side effect information generation unit 410 may include multiple analysis models for generating various side effect information. For example, as shown in FIG. 4, the side effect information generation unit 410 may include a rupture analysis model 411, a thickened capsule analysis model 412, folding analysis information 413, a seroma analysis model 414, a reverse rotation analysis model 415, a capsule calcification analysis model 416, and a capsule nodule analysis model 417. However, without being limited thereto, the side effect information generation unit 410 may include various analysis models.

[0095] In some examples, as described below, the side effect information generator 410 may include a first side effect analysis group 410a configured with an analysis model in which a region information generation operation is activated or deactivated depending on burst side effect binary information. Also, the side effect information generator 410 may include a second side effect analysis group 410b configured with an analysis model in which a region information generation operation is always activated regardless of burst side effect binary information.

[0096] In some examples, the implant information generator 420 may include a plurality of analysis models for generating various types of implant information. For example, as shown in Fig. 5, the implant information generator 420 may include an implant type analysis model 421, an implant shape analysis model 422, an implant manufacturer analysis model 423, an implant component analysis model 424, and an implant position analysis model 425. However, without being limited thereto, the implant information generator 420 may include various analysis models.

[0097] In some examples, as described below, the prosthesis information generator 420 may include a first prosthesis analysis group 420a configured with an analysis model in which a region information generation operation is activated or deactivated depending on the rupture side effect binary information. Also, as described below, the prosthesis information generator 420 may include a second prosthesis analysis group 420b configured with an analysis model in which a region information generation operation is always activated regardless of the rupture side effect binary information.

[0098] Hereinafter, an exemplary embodiment will be described in which the analysis model control unit 300 controls the region information generation operations of the first side effect analysis group 410a and the first prosthesis analysis group 420a.

[0099] According to some embodiments of the present disclosure, the medical image analysis system 1000 can generate binary information by activating the binary information generation operation of the first side effect analysis model group 410a and the first prosthetic object analysis model group 420a using the analysis model control unit 300.

[0100] Specifically, when the medical image acquisition unit 200 acquires a medical image 10, the analysis model control unit 300 may receive an analysis command for the acquired medical image 10 from the user interface. In response to the analysis command, the analysis model control unit 300 may transmit a binary information generation command to the medical image analysis unit 400 to activate binary information generation operations of the first side effect analysis model group 410a and the first prosthesis analysis model group 420a. In this case, the medical image analysis unit 400 may generate binary information by activating the binary information generation operations of the first side effect analysis model group 410a and the first prosthesis analysis model group 420a. For example, the medical image analysis unit 400 may generate rupture side effect binary information indicating whether or not a prosthesis has ruptured on the medical image 10 as positive or negative by activating the binary information generation operation of the rupture analysis model 411 in the first side effect analysis model group 410a. As another example, the medical image analysis unit 400 may generate prosthetic shape binary information indicating whether information about the shape of the prosthetic object has been identified in the medical image as positive or negative by activating the binary information generation operation of the prosthetic object shape analysis model 422 in the first side effect analysis model group 420a. In other words, the medical image analysis unit 400 may generate multiple pieces of binary information by activating the binary information generation operation of all analysis models belonging to the first side effect analysis model group 410a and the first prosthetic object analysis model group 420a.

[0101] According to some embodiments of the present disclosure, the medical image analysis system 1000 can activate or deactivate the region information generation operations of the first side effect analysis model group 410a and the first prosthesis analysis model group 420a, which generate region information corresponding to the binary information based on the rupture side effect binary information among the binary information.

[0102] The medical image analysis system 1000 can efficiently analyze medical images by activating or deactivating the region information generation operation of the analysis model using binary information for clinically significant side effects, such as a rupture side effect, among various side effects that can be identified by analyzing medical images. Specifically, the medical image analysis system 1000 can determine whether to activate the region information generation operation of the first side effect analysis model group 410a and the first prosthesis analysis model group 420a using specific binary information. Compared to binary information, region information may require calculation for each pixel included in a medical image. Therefore, the medical image analysis system 1000 can efficiently use resources by activating the region information generation operation in a limited manner. For example, when analyzing medical images of an implant inserted into the body, such as a breast, the rupture side effect may be more clinically significant than other side effects. In other words, for a medical image in which a rupture side effect is detected, region information related to other side effects may be less utilized. Therefore, even if binary information regarding other side effects indicates positive for a medical image in which a rupture side effect is found, the medical image analysis system 1000 can save resources by deactivating the region information generation operation regarding other side effects other than rupture.

[0103] In some examples, the medical image analysis system 1000 may activate region information generation operations of the first side effect analysis model group 410a and the first prosthesis analysis model group 420a in response to the binary information of the rupture side effect indicating negative, thereby generating region information corresponding to the binary information through the region information generation operations of the first side effect analysis model group 410a and the first prosthesis analysis model group 420a. Specifically, the analysis model control unit 300 may receive the rupture side effect binary information from the medical image analysis unit 400. If the rupture side effect binary information indicates negative, the analysis model control unit 300 may transmit a region information generation control command 40 to the medical image analysis unit 400 to activate the region information generation operations of the first side effect analysis model group 410a and the first prosthesis analysis model group 420a to generate region information of other side effects on the medical image. In this case, the first side effect analysis model group 410a and the first prosthesis analysis model group 420a may each generate region information corresponding to the binary information. In other words, when the rupture side effect binary information indicates negative, the analysis model control unit 300 can control the first side effect analysis model group 410a and the first prosthesis analysis model group 420a to generate both the binary information and the area information corresponding to the binary information, respectively.

[0104] In some examples, the medical image analysis system 1000 may deactivate region information generation operations of the first side effect analysis model group and the first prosthesis analysis model group in response to positive rupture side effect binary information. Specifically, the analysis model control unit 300 may receive rupture side effect binary information from the medical image analysis unit 400. When the rupture side effect binary information indicates positive, the analysis model control unit 300 may transmit a region information generation control command 40 to the medical image analysis unit 400 to deactivate region information generation operations of the first side effect analysis model group 410a and the first prosthesis analysis model group 420a so as not to generate region information for side effects other than rupture on the medical image. In this case, the first side effect analysis model group 410a and the first prosthesis analysis model group 420a may each generate only binary information and not region information. In other words, when the burst side effect binary information indicates a positive result, the analytical model control unit 300 can control the first side effect analytical model group 410a and the first prosthesis analytical model group 420a to generate only binary information.

[0105] In some examples, the analytical models belonging to the first side effect analysis model group 410a and the first prosthesis analysis model group 420a may perform binary information generation operations and region information generation operations in various manners. For example, the analytical model may be composed of a binary information generation sub-model that performs a binary information generation operation and a region information generation sub-model that performs a region information generation operation. In this case, the analytical model control unit 300 may deactivate the region information generation sub-models of the first side effect analysis model group 410a and the first prosthesis analysis model group 420a in response to deactivating the region information generation operations of the first side effect analysis model group 410a and the first prosthesis analysis model group 420a. In other words, if the analytical model is composed of two or more sub-models that individually generate binary information and region information, the analytical model control unit 300 may deactivate the region information generation sub-models in response to deactivating the region information generation operation.

[0106] In some examples, the analytical model may be configured as a single model that performs both the binary information generation operation and the region information generation operation. For example, the analytical model may perform a region information generation operation that calculates the classification contribution of pixels included in a medical image to binary information. In this case, the analytical model control unit 300 may deactivate the operation of calculating the classification contribution of pixels included in a medical image to binary information of the first side effect analysis model group 410a and the first prosthesis analysis model group 420a in response to deactivating the region information generation operation of the first side effect analysis model group 410a and the first prosthesis analysis model group 420a. In other words, if the analytical model is a single model that performs both the binary information generation operation and the region information generation operation, the analytical model control unit 300 may perform only the binary information generation operation in response to deactivating the region information generation operation.

[0107] According to some embodiments of the present disclosure, the medical image analysis system 1000 can generate binary information and area information corresponding to the binary information by processing the acquired medical image 10 using the second side effect analysis model group 410b and the second prosthesis analysis model group 420b.

[0108] The medical image analysis system 1000 can efficiently analyze medical images by using two types of analysis models classified according to whether a region information generation operation is controlled using binary information for a specific side effect, such as a rupture side effect. Specifically, the medical image analyzer 400 may include a first side effect analysis model group 410a and a second prosthesis analysis model group 420a as analysis models controlled to restrict the execution of a region information generation operation according to binary information for a specific side effect, such as a rupture side effect. In some examples, as described above, the medical image analysis system 1000 can control the region information generation operation of the first side effect analysis model group 410a and the first prosthesis analysis model group 420a by activating or deactivating the region information generation operation in response to the binary information for the rupture side effect indicating positive or negative. Meanwhile, regardless of whether the rupture side effect binary information indicates positive or negative, the medical image analysis system 1000 can generate binary information and area information by activating both the binary information generation operation and the area information generation operation of the second side effect analysis model group 410b and the second prosthesis analysis model group 420b.

[0109] In some examples, the second side effect analysis model group 410b may include at least one of a silicone membrane infiltration analysis model 418 or a silicone lymph node infiltration analysis model 419. Furthermore, the second prosthesis analysis model group 420b may include at least one of an implant component analysis model 424 or an implant location analysis model 425. However, without being limited thereto, the second side effect analysis model group 410b and the second prosthesis analysis model group 420b may include various analysis models.

[0110] FIG. 6 illustrates an analytical model generation system 2000 according to some embodiments of the present disclosure. FIG. 7 illustrates a diagram for explaining results related to quantitative validation of an analytical model according to some embodiments of the present disclosure. FIG. 8 illustrates another diagram for explaining results related to quantitative validation of an analytical model according to some embodiments of the present disclosure. FIG. 9 illustrates another diagram for explaining results related to quantitative validation of an analytical model according to some embodiments of the present disclosure. FIG. 10 illustrates a diagram for explaining results related to unpredictability estimation of an analytical model according to some embodiments of the present disclosure. FIG. 11 illustrates a diagram for explaining results related to post-hoc explainable analysis of an analytical model according to some embodiments of the present disclosure.

[0111] In some examples, the analytical model generation system 2000 can generate an analytical model that provides an analysis result regarding medical images of an implant inserted into the body. For example, the analytical model generation system 2000 can generate analytical models included in the side effect information generation unit 410 and the implant information generation unit 420. Hereinafter, with reference to FIGS. 6 to 11 , exemplary embodiments of the analytical model generation system 2000 and its components that execute methods for generating analytical models according to some embodiments of the present disclosure will be described.

[0112] 6 , the analytical model generation system 2000 may include a dataset acquisition unit 2100, a model training unit 2200, and a model evaluation unit 2300. However, without being limited thereto, the analytical model generation system 2000 may further include other components for generating an analytical model, and only some of the disclosed components may constitute the analytical model generation system 2000.

[0113] Hereinafter, an embodiment will be described in which the analytical model system 2000 generates an analytical model that generates shell type information, such as the object type analytical model 421. However, the analytical model system 2000 is not limited to this, and can generate various analytical models.

[0114] According to some embodiments of the present disclosure, the analytical model generation system 2000 can acquire, via the dataset acquisition unit 2100, a first dataset including medical images of an implant inserted into the body taken by a first imaging device, and a second dataset including medical images of an implant inserted into the body taken by a second imaging device.

[0115] Specifically, the dataset acquisition unit 2100 may acquire a dataset used to generate an analytical model in various ways. For example, the dataset acquisition unit 2100 may acquire a dataset including medical images registered through a user interface. As another example, the dataset acquisition unit 2100 may acquire a dataset including medical images stored in an external database. However, without being limited thereto, the dataset acquisition unit 2100 may acquire a dataset in various ways.

[0116] In some examples, the dataset acquisition unit 210 can acquire various types of datasets. For example, the dataset acquisition unit 210 can acquire medical images captured using different imaging devices. For example, the dataset acquisition unit 210 can acquire a first dataset including medical images of an implant inserted into the body captured by a first imaging device, and a second dataset including medical images of an implant inserted into the body captured by a second imaging device. Here, the first imaging device may be an imaging device manufactured by a different manufacturer from the second imaging device. For example, the first imaging device may be an imaging device manufactured by Canon, and the second imaging device may be an imaging device manufactured by General Electric. Here, the manufacturers of the imaging devices are described by way of example, and the present disclosure is not limited thereto.

[0117] In some examples, the first imaging device may be an imaging device that acquires medical images at a higher resolution than the second imaging device. In this case, the first dataset may include medical images with a higher resolution than the second dataset. As described below, the analytical model generation system 2000 may use the first dataset to train, validate, and test an analytical model. The second dataset may also be used to externally validate the analytical model. In other words, the analytical model generation system 2000 may train an analytical model using only the first dataset including high-resolution medical images among two or more datasets including medical images with different resolutions. The inventors have found that an analytical model trained using only a dataset including high-resolution medical images exhibits higher classification accuracy than an analytical model trained using a combination of multiple datasets including medical images with various resolutions. Therefore, the analytical model generation system 2000 according to some embodiments of the present disclosure may efficiently generate an analytical model using a relatively small amount of training data by training an analytical model using the first dataset and then validating the trained analytical model using the first dataset and the second dataset. Furthermore, the analytical model generation system 2000 can generate analytical models that provide high classification accuracy for medical images captured by various types of imaging equipment.

[0118] In some examples, the medical images may be ultrasound images stored in PACS rendered JPEG format. A 128-bit MD5 hashing algorithm may be used to eliminate data leakage between the training and test datasets. Additionally, the medical images included in the dataset may have shell type information designated by a breast surgeon with extensive experience in ultrasound examination of breast implants.

[0119] Furthermore, the dataset acquisition unit 210 can acquire a third dataset including medical images of a ruptured implant inserted into the body and a fourth dataset including medical images of a state in which no implant is inserted into the body. The third dataset and the fourth dataset may be used as out-of-distribution (OOD) datasets for analyzing the analytical model. For medical images included in the third dataset, it may be difficult to identify the shell type of the implant due to a damaged shell. For medical images included in the fourth dataset, the absence of the implant in the image makes it impossible to identify the shell type of the implant. Therefore, as described below, the third dataset and the fourth dataset can be used to understand the analytical model's ability to estimate the model's uncertainty regarding shell type.

[0120] Additionally, the dataset acquisition unit 210 may include a fifth dataset containing shared images. For example, the fifth dataset may include medical images searched using keywords such as "breast implant ultrasound" and "breast implant ultrasonography." In some examples, the fifth dataset may be used for external validation of the analytical model. In that the fifth dataset includes medical images searched without any restrictions on imaging equipment, the fifth dataset can be used to verify the analytical capabilities of the analytical model for medical images captured with various imaging equipment.

[0121] Table 1 discloses an exemplary dataset used to generate the analytical model.

[0122] [Table 1]

[0123] According to some embodiments of the present disclosure, the analytical model generation system 2000 may use the model training unit 2200 to train an analytical model that determines shell type information of a prosthesis using the first dataset. In some examples, the analytical model may use a convolutional neural network architecture with parameter size adjustment to achieve high performance. The analytical model may use a lightweight model instead of a model that requires expensive computational costs along with large amounts of data due to inductive bias. For example, the analytical model may use a ResNet-50 consisting of 50 layers as a backbone. As another example, the analytical model may use a model architecture with many parameters, such as a Vision Transformer or a Swin Transformer. In some examples, the model training unit 2200 may perform transition training on a pre-trained ResNet-50 that has trained ImageNet classification. The pre-trained ResNet-50 may include a binary classification layer that replaces a classification layer with a 1000-dimensional vector for multiple classification to generate binary information. A binary classification layer can return a 2D vector for shell surface topologies such as smooth and texture types.

[0124] In some examples, the model training unit 2200 may train the analysis model using a loss function that penalizes misclassification of textured types among the shell type information of the prosthesis compared to smooth types. For example, the model training unit 2200 may use weighted binary cross-entropy as an objective function for parameter optimization. The weighted binary cross-entropy penalizes misprediction of minor classes due to class imbalance among shell type information (texture types being minor classes), thereby enabling efficient training of the analysis model. In the weighted binary cross-entropy, the weight for the minor class may be calculated as the inverse ratio of the minor class (texture type) from the training dataset (first dataset).

[0125] In some examples, the model training unit 2200 may perform preprocessing on the medical images included in the datasets. For example, the model training unit 2200 may perform preprocessing by cutting off the side regions of the medical images included in the first dataset or the second dataset by a predetermined percentage (e.g., 10%, 20%, 30%, etc.). Specifically, the medical images may further include additional information, such as text information and metadata, along with the image. For example, a PACS-rendered image may further include information about patient information, examination information, image metadata, annotations or overlays, related documents, etc., along with the image. Because additional information is generally located on the side of the medical image, the model training unit 2200 may improve the learning efficiency of the analysis model by cutting off the side regions of the medical images by a predetermined percentage.

[0126] The model training unit 2200 may also resize the medical images to fit the input data of the analytical model. For example, the model training unit 2200 may perform preprocessing to resize the medical images included in the first dataset to 224 x 224, the size of the input data of the analytical model, using binary interpolation. However, the preprocessing is not limited to this, and the model training unit 2200 may perform various preprocessing operations.

[0127] In some examples, the model training unit 2200 can use an Adam optimizer to train an analytical model with a learning rate of 0.001 and a batch size of 32. The total training epochs were 20, but the actual training epochs were fewer than 20 due to early stopping by monitoring validation loss with a patience of 7.

[0128] In some examples, the model evaluator 2300 may include various components for evaluating the analytical model trained by the model trainer 2200. As shown in Figure 6, the model evaluator 2300 may include a cross- and external validation unit 2310, a quantitative validation unit 2320, an unpredictability estimation unit 2330, and a post-hoc explainable analysis unit 2340. The model evaluator 2300 may further include other components for evaluating the analytical model, and only some of the disclosed components may constitute the model evaluator 2300.

[0129] According to some embodiments of the present disclosure, the analytical model generation system 2000 may validate the analytical model using the first and second datasets via the cross and external validation unit 2310.

[0130] As described above, the analytical model generation system 2000 can train an analytical model using a first dataset including relatively high-resolution medical images, and then validate the analytical model using the first dataset and a second dataset. Specifically, the cross-validation and external validation unit 2310 can perform cross-validation on the analytical model using the first dataset. Then, the cross-validation and external validation unit 2310 can perform external validation on the analytical model using the second dataset.

[0131] In some examples, the dataset acquisition unit 2100 can divide the first dataset into training data (60%), validation data (20%), and test data (20%). The model training unit 2200 can then use the training data, which accounts for 60% of the first dataset, to train the analytical model. The cross-validation and external validation unit 2310 can cross-validate the analytical model using the validation data, which accounts for 20% of the first dataset. Specifically, for example, the cross-validation and external validation unit 2310 can perform stratified 5-fold cross-validation to identify the generalized performance of the analytical model under class imbalance between smooth and text types. For cross-validation, the cross-validation and external validation unit 2310 can identify the area under the curve (AUC) using receiver operating characteristic (ROC) curves and precision-recall (PR) curves with different cutoffs. The crossover and external validation unit 2310 may externally validate the analytical model using a second dataset to evaluate the analytical model using medical images (online acquired ultrasonography) instead of the training data. The crossover and external validation unit 2310 may check the AUC of both the ROC and PR using such data. The crossover and external validation unit 2310 may also externally validate the analytical model using a fifth dataset including shared images. However, the crossover and external validation unit 2310 may validate the analytical model in various ways, without being limited thereto.

[0132] According to some embodiments of the present disclosure, the analytical model generation system 2000 may perform quantitative validation of the analytical model by the quantitative validation unit 2320 using a medical image in which pixels with low classification contributions are masked according to a predetermined ratio to determine shell type information of the prosthesis based on pixels corresponding to the layer of the prosthesis.

[0133] In some examples, the quantitative verification unit 2320 may use an explainable AI approach (XAI) to verify whether the analysis model can accurately determine the shell type information of the prosthesis using layer and echogenicity features on a medical image, such as an ultrasound image. Specifically, the quantitative verification unit 2320 may perform quantitative verification to evaluate classification performance according to masked portions of the medical image. The quantitative verification unit 2320 may perform quantitative verification under the assumption that important pixels for determining the shell type of the prosthesis are located in the layer of the prosthesis. The quantitative verification unit 2320 may also perform quantitative verification under the assumption that performance will not deteriorate even if some pixels that are not in the layer are removed.

[0134] In some examples, the quantitative verification unit 2320 may calculate the importance of pixels using an algorithm that quantifies classification contribution. For example, the quantitative verification unit 2320 may calculate the importance of pixels using "Grad-CAM." The quantitative verification unit 2320 may remove 90% of pixels with low classification contribution from all pixels in the medical image and replace these values ​​with 0 before calculating the AUROC and PRAUC. In this way, the quantitative verification unit 2320 can evaluate whether shell type information has been determined using pixels in the prosthesis layer. However, the quantitative verification unit 2320 may perform quantitative verification in various ways, without being limited thereto.

[0135] According to some embodiments of the present disclosure, the analytical model generation system 2000 can perform uncertainty estimation for the analytical model using at least a portion of the first data set, the second data set, the third data set, or the fourth data set via the uncertainty estimation unit 2330.

[0136] Specifically, the unpredictability estimator 2330 may calculate entropy to estimate the prediction uncertainty of the medical image. In some examples, the entropy may include Shannon entropy. The entropy may have a value between 0 and 1. The entropy may have a higher value when the prediction uncertainty is higher.

[0137] In some examples, the unpredictability estimation unit 2330 may estimate the unpredictability of the analytical model based on a hypothesis that the third dataset has higher prediction uncertainty than the first dataset or the second dataset, and that the fourth dataset has higher prediction uncertainty than the third dataset. Specifically, because the analytical model was trained using the first dataset including medical images with the shell integrity intact, the third dataset including medical images of a ruptured implant inserted into the body may have higher entropy than the other datasets. Furthermore, the fourth dataset including medical images of a state in which no implant is inserted into the body may have higher entropy than the third dataset. Therefore, the unpredictability estimation unit 2330 may cause the analytical model to calculate a model reliability value for the unpredictability estimation of the analytical model based on the hypothesis that the third dataset has higher prediction uncertainty than the first dataset or the second dataset, and that the fourth dataset has higher prediction uncertainty than the third dataset. Under such assumptions, the analytical model generator 2000 can provide an analytical model that can provide a model confidence level that helps a clinician make a decision that reflects diagnostic uncertainty. However, without being limited thereto, the unpredictability estimator 2330 can perform unpredictability estimation in various ways.

[0138] According to some embodiments of the present disclosure, the analytical model generation system 2000 may perform a post-hoc explainable interpretation on the analytical model by the post-hoc explainable analysis unit 2340, such that the post-hoc explainable analysis unit 2340 uses the classification contribution of pixels to the shell type information of the prosthesis to generate region information displaying pixels related to the shell type information of the prosthesis.

[0139] Specifically, the post-hoc explainable analysis unit 2340 can perform post-hoc explainable analysis on the analytical model to gain insight into the decision-making processes of the analytical model. In some examples, the post-hoc explainable analysis unit 2340 can perform post-hoc explainable analysis using the Gradient-weighted Class Activation Mapping (Grad-CAM) technique. The Grad-CAM technique can be used to visually estimate whether there is a match between established medical expertise on shell type information and the analytical model's prediction pattern. The post-hoc explainable analysis unit 2340 can cause the analytical model to generate region information displaying pixels related to the prosthesis's shell type information using the classification contribution of pixels to the prosthesis's shell type information. In some examples, the region information can be displayed in a heat map format. In this case, the region information can be used to confirm whether the analytical model has generated a prediction pattern using pixels displaying clinically important regions by highlighting pixels important for determining shell type information on the medical image.

[0140] Although the analysis model has been described as a model that determines shell type information, the post-explainable analysis unit 2340 may also be used to generate an analysis model that generates side effect information such as rupture. When generating the rupture analysis model 411, the post-explainable analysis unit 2340 may cause the rupture analysis model 411 to generate region information that displays a region where a rupture has occurred on a medical image. However, the present invention is not limited to this, and the post-explainable analysis unit 2340 may perform post-explainable analysis on various analysis models.

[0141] Below, evaluation results of an exemplary analytical model according to some embodiments of the present disclosure are described.

[0142] [Table 2]

[0143] Referring to Table 2, the exemplary analytical model performed with an AUROC of 0.998 and a PRAUC of 0.994 for the test dataset included in the first dataset. In a hierarchical five-fold cross-validation, the analytical model demonstrated an average AUROC of 0.98 and a PRAUC of 0.88 for the first dataset, which included medical images taken with a first imaging device. For the second dataset, which included images taken with a second imaging device, the analytical model also demonstrated an AUROC of 0.985 and a PRAUC of 0.748. For the fifth dataset, which included public images, the analytical model demonstrated an AUROC of 0.909 and a PRAUC of 0.958. This suggests that the analytical model produces accurate analytical results for various imaging devices, not just the imaging device that captured the medical images included in the training data. Referring to FIG. 7, the results of quantitative validation to confirm whether the analytical model classifies medical images based on medical knowledge showed that the analytical model demonstrated an AUROC of 0.999 when masking 90% of the pixels with low classification contribution. The analytical model also showed an AUROC of 0.997 when 100% of the pixels were masked. However, the PRAUC remained at 0.999 when 90% of the pixels were masked, but dropped to 0.493 when 100% of the pixels were masked. Referring to Figure 8, for each individual case, the confidence for the texture shell type remained at 0.993 when 80% or less of the pixels contributing to the prediction were masked. When 90% of the pixels were masked, the model confidence dropped to 0.968, and when all pixels were masked, it reached 0.497. Similarly, for the other cases related to the texture shell type, the model confidence remained at 0.994 until 80% of the pixels were masked. Then, when 90% of the pixels were masked, the model confidence dropped to 0.960. Finally, when 100% of the pixels were masked, the model confidence dropped to 0.947. In other words, the analytical model maintained high accuracy even when pixels with low classification contribution were removed, suggesting that the analytical model relies on core information to make predictions.

[0144] Referring to Figure 10, the results for uncertainty estimation show that the analytical model did not exhibit significantly lower entropy for the first dataset compared to the second dataset, which was externally validated (mean [SD], 0.072 [0.201] vs. 0.066 [0.21]; p = 0.350). This indicates that the analytical model exhibits similar predictive stability for medical images taken with the first imaging device and medical images taken with the second imaging device. The mean entropy for the third dataset was significantly higher than that for the first dataset (mean [SD], 0.371 [0.318] vs. 0.072 [0.201]; p < 0.001). This indicates that the analytical model has greater predictive uncertainty when analyzing images of ruptured implants. Additionally, the analytical model showed statistically significantly higher entropy for the fourth dataset, which included images without an implant, compared to the third dataset (mean [SD], 0.777 [0.199] vs. 0.371 [0.318]; p<0.001). This suggests that the analytical model has greater predictive uncertainty when analyzing images without an implant compared to images with a ruptured implant. This suggests that the analytical model's estimate of unpredictability can be robustly quantified. Therefore, the analytical model generator 2000 according to some embodiments of the present disclosure can provide an analytical model that can provide a model confidence measure that helps clinicians make decisions that reflect uncertainty.

[0145] Referring to Figure 11, for a qualitative case study, one medical image was sampled from the first dataset and another from the third dataset. Both medical images were generated using the same imaging equipment. The analytical model showed a model confidence of 0.998 for the texture type of the medical image. The Grad-CAM score for the texture type showed a high value in the heat map (white horizontal line in Figure 11A). Furthermore, for the ruptured texture type implant image, the analytical model showed a model confidence of 0.664 for the texture type. This score was higher than the classification threshold (0.5), but lower than that of the intact texture type implant. However, despite the rupture, the Grad-CAM score was higher in the intact layer adjacent to the area where the ruptured shell was located. The arrow in the left image of Figure 11B indicates the location of the rupture in the implant shell. Referring now to the right image of Figure 11B, a heat map is displayed along the layer adjacent to the location of the rupture in the implant shell. This suggests that the analytical model responds reliably to intact layers, regardless of whether they are ruptured or not.

[0146] FIG. 12 is a flowchart of a method for analyzing medical images according to some embodiments of the present disclosure.

[0147] According to some embodiments of the present disclosure, the medical image analysis method may include a step (S100) of acquiring a medical image of an implant inserted into a body.

[0148] According to some embodiments of the present disclosure, the medical image analysis method may include generating binary information by activating a binary information generation operation of a first side effect analysis model group and a first prosthesis analysis model group (S200). Here, the binary information may represent information about the prosthesis inserted into the body and information about side effects caused by the prosthesis inserted into the body as positive or negative.

[0149] According to some embodiments of the present disclosure, the medical image analysis method may include activating or deactivating region information generation operations of the first side effect analysis model group and the first prosthetic object analysis model group, which generate region information corresponding to the binary information based on rupture side effect binary information among the binary information (S300).

[0150] Alternatively, the step of activating or deactivating region information generating operations of the first side effect analysis model group and the first prosthesis analysis model group, which generate region information corresponding to the binary information based on the rupture side effect binary information among the binary information (S300), may include: activating region information generating operations of the first side effect analysis model group and the first prosthesis analysis model group in response to the rupture side effect binary information indicating negative, thereby generating region information corresponding to the binary information through the region information generating operations of the first side effect analysis model group and the first prosthesis analysis model group; and deactivating region information generating operations of the first side effect analysis model group and the first prosthesis analysis model group in response to the rupture side effect binary information indicating positive.

[0151] Alternatively, the step of deactivating region information generation operations of the first side effect analysis model group and the first prosthetic object analysis model group in response to the rupture side effect binary information indicating positive may include the step of deactivating region information generation sub-models of the first side effect analysis model group and the first prosthetic object analysis model group in response to deactivating region information generation operations of the first side effect analysis model group and the first prosthetic object analysis model group.

[0152] Alternatively, the step of deactivating an operation of generating region information of the first side effect analysis model group and the first prosthetic object analysis model group in response to the binary information of the rupture side effect indicating positive may include the step of deactivating an operation of calculating a classification contribution of a pixel included in the medical image of the first side effect analysis model group and the first prosthetic object analysis model group to the binary information in response to deactivating the operation of generating region information of the first side effect analysis model group and the first prosthetic object analysis model group.

[0153] Alternatively, the first side effect analysis model group may include at least one of a thickened capsule analysis model, a folding analysis model, a fluid collection analysis model, an upside-down rotation analysis model, a capsular calcification analysis model, or a capsular nodule analysis model.

[0154] Alternatively, the first prosthesis analysis model group may include at least one of a prosthesis type analysis model, a prosthesis shape analysis model, or a prosthesis manufacturer analysis model.

[0155] Alternatively, the medical image analysis method may further include generating the binary information and region information corresponding to the binary information by processing the acquired medical image using a second side effect analysis model group and a second prosthetic object analysis model group.

[0156] Alternatively, the second side effect analysis model group may include at least one of a silicone capsule infiltration analysis model or a silicone lymph node infiltration analysis model.

[0157] Alternatively, the second implant analysis model group may include at least one of an implant component analysis model or an implant position analysis model. Alternatively, the region information may be displayed in a heat map format.

[0158] The steps of the medical image analysis method described above are presented for illustrative purposes only, and some steps may be omitted, other steps may be added, and the steps described above may be performed in any order.

[0159] FIG. 13 is a flowchart of a method for generating an analytical model according to some embodiments of the present disclosure.

[0160] According to some embodiments of the present disclosure, a method for generating an analytical model for determining shell type information of an implant inserted into a body may include acquiring (S1100) a first dataset including medical images of the implant inserted into the body taken by a first imaging device and a second dataset including medical images of the implant inserted into the body taken by a second imaging device, wherein the first dataset may include medical images having a higher resolution than the second dataset.

[0161] According to some embodiments of the present disclosure, the method for generating the analytical model may include training (S1200) an analytical model that determines shell type information of the prosthesis using the first data set.

[0162] According to some embodiments of the present disclosure, the method for generating the analytical model may include a step (S1300) of validating the analytical model using the first dataset and the second dataset.

[0163] Alternatively, the first imaging device may be an imaging device manufactured by a different manufacturer than the second imaging device.

[0164] Alternatively, the method for generating the analytical model may further include a step of performing pre-processing to cut out side regions of the medical images included in the first data set or the second data set at a predetermined percentage.

[0165] Alternatively, the step of training an analysis model for determining shell type information of the prosthesis using the first dataset (S1200) may include training the analysis model using a loss function that imposes a higher penalty on misclassification of a textured type compared to a smooth type among the shell type information of the prosthesis.

[0166] Alternatively, the step of validating the analytical model using the first dataset and the second dataset (S1300) may include a step of performing cross validation on the analytical model using the first dataset, and a step of performing external validation on the analytical model using the second dataset.

[0167] Alternatively, validating the analytical model using the first dataset and the second dataset (S1300) may further include performing external validation of the analytical model using a fifth dataset including a publicly available image.

[0168] Alternatively, the method for generating the analytical model may further include performing quantitative validation of the analytical model using a medical image in which pixels with low classification contributions are masked based on a predetermined ratio to determine shell type information of the prosthesis based on pixels corresponding to a layer of the prosthesis.

[0169] Alternatively, the method for generating the analytical model may further include acquiring a third dataset including medical images of a state in which a prosthesis inserted into a body has ruptured, and a fourth dataset including medical images of a state in which no prosthesis is inserted into a body, and performing uncertainty estimation on the analytical model using at least a portion of the first dataset, the second dataset, the third dataset, or the fourth dataset.

[0170] Alternatively, performing an uncertainty estimation for the analytical model using at least a portion of the first data set, the second data set, the third data set, or the fourth data set may include performing an uncertainty estimation for the analytical model based on a hypothesis that the third data set has higher prediction uncertainty than the first data set or the second data set, and that the fourth data set has higher prediction uncertainty than the third data set.

[0171] Alternatively, the method for generating the analytical model may further include performing a post-hoc explainable interpretation on the analytical model to generate region information representing pixels associated with the shell type information of the prosthesis by using classification contributions of pixels to the shell type information of the prosthesis.

[0172] Alternatively, the region information may be displayed in a heat map format.

[0173] The steps of the method for generating an analytical model described above are presented for illustrative purposes, and some steps may be omitted, other steps may be added, and the steps may be performed in any order. FIG. 14 is a simplified general schematic diagram of an exemplary computing environment in which embodiments of the present disclosure can be implemented.

[0174] While the present disclosure has been described above as generally capable of being embodied in computing devices, those skilled in the art will appreciate that the present disclosure can also be embodied in combination with computer-executable instructions and / or other program modules that can be executed on one or more computers and / or as a combination of hardware and software. Generally, program modules include routines, programs, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Those skilled in the art will also appreciate that the methods of the present disclosure can be practiced with other computer system configurations, including single-processor or multi-processor computer systems, minicomputers, mainframe computers, as well as personal computers, handheld computing devices, microprocessor-based or programmable consumer electronics, and the like, each of which can operate in conjunction with one or more associated devices.

[0175] Moreover, the embodiments described in this disclosure may be practiced in distributed computing environments where certain tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices. A computer typically includes a variety of computer-readable media. Any medium accessible by a computer can be computer-readable, including volatile and non-volatile media, transitory and non-transitory media, and portable and non-portable media. By way of example and not limitation, computer-readable media can include computer-readable storage media and computer-readable transmission media. Computer-readable storage media include volatile and non-volatile media, transitory and non-transitory media, portable and non-portable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer-readable storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital video disk (DVD) or other optical disk storage devices, magnetic cassettes, magnetic tape, magnetic disk storage devices or other magnetic storage devices, or any other medium that can be accessed by a computer and used to store information.

[0176] Computer-readable transmission media typically embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave or other transport mechanism, and includes all information delivery media. The term modulated data signal means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, computer-readable transmission media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared, and other wireless media. Combinations of any of the above media are also intended to be included within the scope of computer-readable transmission media. An exemplary environment 1100 for implementing various aspects of the present disclosure is shown, including a computer 1102, which includes a processing unit 1104, a system memory 1106, and a system bus 1108. The system bus 1108 couples system components, including, but not limited to, the system memory 1106, to the processing unit 1104. The processing unit 1104 can be any of a variety of commercially available processors. Dual processors and other multi-processor architectures can also be utilized as the processing unit 1104.

[0177] The system bus (1108) can be any of several types of bus structures that can be further interconnected to a memory bus, a peripheral bus, and a local bus using any of a variety of commercially available bus architectures. The system memory (1106) includes read-only memory (ROM) (1110) and random access memory (RAM) (1112). The basic input / output system (BIOS) is stored in non-volatile memory (1110), such as ROM, EPROM, or EEPROM. The BIOS contains the basic routines that support the exchange of information between the various components within the computer (1102), such as during start-up. The RAM (1112) can also include high-speed RAM, such as static RAM, for caching data. The computer 1102 also includes an internal hard disk drive (HDD) 1114 (e.g., EIDE, SATA)—the internal hard disk drive 1114 can be used externally in a suitable chassis (not shown)—a magnetic floppy disk drive (FDD) 1116 (e.g., for reading from and writing to a removable diskette 1118), and an optical disk drive 1120 (e.g., for reading from a CD-ROM disk 1122 or for reading from and writing to other high-capacity optical media such as DVDs). The hard disk drive 1114, magnetic disk drive 1116, and optical disk drive 1120 can be connected to the system bus 1108 by a hard disk drive interface 1124, a magnetic disk drive interface 1126, and an optical drive interface 1128, respectively. The interface (1124) for implementing an external drive includes at least one or both of USB (Universal Serial Bus) and IEEE 1394 interface technologies.

[0178] These drives and their associated computer-readable media provide non-volatile storage for data, data structures, computer-executable instructions, etc. In the case of computer 1102, the drives and media accommodate storing any data in a suitable digital format. While the foregoing description of computer-readable storage media refers to hard disk drives, removable magnetic disks, and removable optical media such as CDs or DVDs, those skilled in the art will recognize that other types of computer-readable storage media, such as zip drives, magnetic cassettes, flash memory cards, cartridges, etc., can also be used in the exemplary operating environment, and that any such media can contain computer-executable instructions for performing the methods of the present disclosure. A number of program modules may be stored on the drives and RAM 1112, including an operating system 1130, one or more application programs 1132, other program modules 1134, and program data 1136. All or portions of the operating system, applications, modules, and / or data may also be cached in RAM 1112. It will be appreciated that the present disclosure may be implemented with various commercially available operating systems or combinations of operating systems.

[0179] A user can enter commands and information into the computer 1102 through one or more wired or wireless input devices, such as a keyboard 1138 and a pointing device such as a mouse 1140. Other input devices (not shown) may include a microphone, an IR remote control, a joystick, a game pad, a stylus pen, a touch screen, and the like. These and other input devices are often connected to the processing unit 1104 through an input device interface 1142 connected to the system bus 1108, but may also be connected through other interfaces such as a parallel port, an IEEE 1394 serial port, a game port, a USB port, an IR interface, and the like. A monitor 1144 or other type of display device is also connected to the system bus 1108 through an interface, such as a video adapter 1146. In addition to the monitor 1144, computers typically include other peripheral output devices (not shown), such as speakers, printers, etc.

[0180] The computer 1102 can operate in a networked environment using logical connections, via wired and / or wireless communications, to one or more remote computers, such as remote computer(s) 1148. The remote computer(s) 1148 can be a workstation, computing device computer, router, personal computer, handheld computer, microprocessor-based entertainment device, peer device, or other conventional network node, and can include many or all of the components typically described associated with the computer 1102, although for simplicity, only a memory storage device 1150 is shown. The logical connections shown include wired or wireless connections to a local area network (LAN) 1152 and / or larger networks, e.g., a long-range network (WAN) 1154. Such LAN and WAN networking environments are commonplace in offices and companies and facilitate enterprise-wide computer networks, such as intranets, all of which can connect to worldwide computer networks, e.g., the Internet. When used in a LAN networking environment, the computer 1102 connects to the local network 1152 through a wired and / or wireless communication network interface or adapter 1156. The adapter 1156 can facilitate wired or wireless communication to the LAN 1152, which may also include a wireless access point attached thereto for communicating with the wireless adapter 1156. When used in a WAN networking environment, the computer 1102 may include a modem 1158 or other means for establishing communications over the WAN 1154, such as connecting to a communications computing device in the WAN 1154 or through the Internet. The modem 1158, which may be internal or external and may be a wired or wireless device, connects to the system bus 1108 through a serial port interface 1142. In a networked environment, program modules depicted for computer 1102, or portions thereof, may be stored in remote memory / storage device 1150. It will be appreciated that the network connections shown are exemplary and other means of establishing a communications link between computers may be used.

[0181] The computer 1102 is operable to communicate with any wireless device or unit configured and operating in wireless communication, such as printers, scanners, desktop and / or handheld computers, portable data assistants (PDAs), communication satellites, any equipment or location associated with a radio-detectable tag, and telephones. This includes at least Wi-Fi and Bluetooth® wireless technologies. Thus, communication can be in a predefined structure, such as a traditional network, or in the simple case, ad hoc communication between at least two devices. Wi-Fi (Wireless Fidelity) enables connectivity to the Internet and other networks without wires. Wi-Fi is a wireless technology similar to cell phones, allowing devices such as computers to send and receive data indoors or outdoors, anywhere within the coverage area of ​​a base station. Wi-Fi networks use IEEE 802.11 (a, b, g, etc.) radio technology to provide secure, reliable, and high-speed wireless connections. Wi-Fi can be used to connect computers to each other, the Internet, and wired networks (using IEEE 802.3 or Ethernet). Wi-Fi networks can operate in the unlicensed 2.4 or 5 GHz radio bands, at data rates of, for example, 11 Mbps (802.11a) or 54 Mbps (802.11b), or in products that include both bands (dual bands).

[0182] Those skilled in the art will appreciate that information and signals may be represented using any of a variety of different technologies and techniques. For example, the data, instructions, commands, information, signals, bits, symbols, and chips referred to in the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof. Those skilled in the art will appreciate that the various illustrative logic blocks, modules, processors, means, circuits, and algorithm steps described in the description of the embodiments disclosed herein can be implemented as electronic hardware, various forms of program or design code (referred to herein as "software" for ease of description), or a combination of all of these. To clearly illustrate this interoperability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been generally described above by focusing on their functionality. Whether such functionality is implemented as hardware or software is determined by the particular application and design constraints imposed on the overall system. Those skilled in the art will appreciate that the described functionality can be implemented in various ways for each particular application, and such implementation decisions should not be interpreted as a departure from the scope of the present disclosure.

[0183] The various embodiments described herein may be implemented as a method, apparatus, or article of manufacture using standard programming and / or engineering techniques. The term "article of manufacture" includes a computer program, carrier, or media accessible by any computer-readable device. For example, computer-readable storage media include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, magnetic strips, etc.), optical disks (e.g., CDs, DVDs, etc.), smart cards, and flash memory devices (e.g., EEPROMs, cards, sticks, key drives, etc.). Additionally, the various storage media described herein include one or more devices and / or other machine-readable media for storing information. It should be understood that the specific order or hierarchy of process steps presented herein is an example of an exemplary approach. Based on design priorities, it should be understood that the specific order or hierarchy of process steps can be rearranged within the scope of this disclosure. The accompanying method claims present elements of the various steps in a sample order, but are not meant to be limited to the specific order or hierarchy presented.

[0184] The description of the embodiments set forth herein is provided to enable any person skilled in the art to use or practice the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the scope of the present disclosure. Thus, the present disclosure is not intended to be limited by the embodiments set forth herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. 1. A method for generating an analytical model for determining shell type information of an implant inserted into a body, executed by a computing device, comprising: acquiring a first dataset including medical images of the implant inserted into the body taken by a first imaging device, and a second dataset including medical images of the implant inserted into the body taken by a second imaging device; the first data set includes medical images having a higher resolution than the second data set; training an analytical model for determining shell type information of the prosthesis using the first data set; and validating the analytical model using the first data set and the second data set. method.

2. The first photographic device is manufactured by a different manufacturer from the second photographic device. The method of claim 1.

3. performing pre-processing to cut off a side region of the medical image included in the first data set or the second data set at a predetermined percentage; The method of claim 1.

4. The step of training an analytical model for determining shell type information of the prosthesis using the first dataset includes: training the analysis model using a loss function that imposes a higher penalty on misclassification of a texture type than a smooth type among the shell type information of the prosthesis; The method of claim 1.

5. Validating the analytical model using the first data set and the second data set includes: performing cross validation on the analytical model using the first dataset; performing external validation of the analytical model using the second dataset; The method of claim 1.

6. Validating the analytical model using the first data set and the second data set includes: performing external validation of the analytical model using an additional fifth dataset including publicly available images; The method of claim 4.

7. The method further includes performing quantitative validation of the analytical model using a medical image in which pixels with low classification contributions are masked based on a predetermined ratio so as to determine shell type information of the prosthesis based on pixels corresponding to a layer of the prosthesis. The method of claim 1.

8. acquiring a third dataset including medical images of a state in which the implant inserted into the body has ruptured, and a fourth dataset including medical images of a state in which the implant is not inserted into the body; performing an uncertainty estimation for the analytical model using at least a portion of the first data set, the second data set, the third data set, or the fourth data set. The method of claim 1.

9. performing uncertainty estimation for the analytical model using at least a portion of the first data set, the second data set, the third data set, or the fourth data set, comprising: performing an unpredictability estimation for the analytical model based on a hypothesis that the third dataset has higher prediction uncertainty than the first dataset or the second dataset, and that the fourth dataset has higher prediction uncertainty than the third dataset; The method of claim 8.

10. The method further includes performing a post-hoc explainable interpretation on the analysis model to generate region information representing pixels associated with the shell type information of the prosthesis by using a classification contribution of the pixel to the shell type information of the prosthesis. The method of claim 1.

11. The region information is displayed in a heat map format. The method of claim 10.

12. A computer program stored on a computer-readable storage medium, comprising: The computer program includes instructions to cause one or more processors to perform a method, The method comprises: acquiring a first dataset including medical images of the implant inserted into the body taken by a first imaging device, and a second dataset including medical images of the implant inserted into the body taken by a second imaging device; the first data set includes medical images having a higher resolution than the second data set; training an analytical model for determining shell type information of the prosthesis using the first data set; and validating the analytical model using the first data set and the second data set. A computer program stored on a computer-readable storage medium.

13. A computing device for performing a method, comprising: the computing device includes a memory and a processor; The processor: Obtain a first dataset including medical images of the implant inserted into the body taken by a first imaging device, and a second dataset including medical images of the implant inserted into the body taken by a second imaging device; the first data set includes medical images having a higher resolution than the second data set; training an analytical model for determining shell type information of the prosthesis using the first data set; Validating the analytical model using the first data set and the second data set. Computing equipment.

Images