Anatomical positioning frame

Abstract

A frame for anatomical positioning. According to one aspect, input text is mapped to normalized coordinates using an artificial neural network. And determining and presenting a position corresponding to the normalized coordinate in the target image. According to another aspect, a user selection of a point of interest in a medical image is received. And determining a contextual point set closest to the point of interest. A hint containing the point of interest and the contextual point set is constructed. The large language model may then generate text data associated with the point of interest in response to the cue.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of U.S. Provisional Application No. 63 / 585,743, filed on September 27, 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0002] The present disclosure relates generally to data processing and, more particularly, to a framework for anatomical localization. Background Art

[0003] Written radiology reports are the primary work product of diagnostic radiologists and serve as the primary method of communicating the results of a radiology study or procedure, including findings in medical images. Radiology reports are reusable after the image reading is complete. When a report is reused, clinicians often have to manually find the corresponding items in the image. These items can include findings at the organ and subtissue level and are often described in terms of their location relative to some known anatomical landmarks.

[0004] This process can be automated by automatically mapping textual entries in radiology reports to anatomical landmarks in images. Navigating to specific anatomical landmarks is not a trivial problem because (1) anatomical landmarks may appear in different forms and orders; and (2) storing information in unstructured text limits the ability to process data at scale. For example, identifying patients with liver disease may require taking all types of mentions (e.g., "liver", "hepatic") into account.

[0005] Previous methods seek to identify the longitudinal extent of an image given Digital Imaging and Communications in Medicine (DICOM) text input. However, there are several limitations to these methods. The extent is estimated only one-dimensionally and has limited applicability since only header data is used. It may not work well with natural language sentences present in radiology reports.

[0006] Parsing methods can be considered as a baseline for associating textual input with anatomical coordinates, since image identifiers (e.g., image and sequence numbers) mentioned in the text follow a well-structured pattern. However, each keyword must be manually designed and extracted, limiting its application. Another approach is to use the locations of anatomical landmarks in the image identified by segmentation or landmark marking methods. However, the scope is limited to landmarks and organs with well-defined structures. Summary of the invention

[0007] A framework for anatomical localization is described herein. According to one aspect, input text is mapped to normalized coordinates using an artificial neural network. Locations in a target image corresponding to the normalized coordinates are determined and presented. According to another aspect, a user selection of a point of interest in a medical image is received. A set of context points closest to the point of interest is determined. A prompt is constructed that includes the point of interest and the set of context points. A large language model can then generate text data associated with the point of interest in response to the prompt. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] A more complete understanding of the present disclosure and its many attendant aspects will be readily obtained by reference to the following detailed description when considered in conjunction with the accompanying drawings as the same becomes better understood.

[0009] Figure 1 A block diagram illustrating an exemplary system is shown;

[0010] Figure 2 is a flow chart of an exemplary method for estimating image position based on text input;

[0011] Figure 3 is a schematic diagram of an exemplary method for estimating image position based on text input;

[0012] Figure 4 is a flow chart of an exemplary method of determining text data associated with a point of interest in a medical image; and

[0013] Figure 5 is a schematic diagram of an exemplary method for determining text data associated with a point of interest in a medical image. DETAILED DESCRIPTION

[0014] In the following description, many specific details are set forth, such as examples of specific components, devices, methods, etc., in order to provide a thorough understanding of the implementation of the present framework. However, it is apparent to those skilled in the art that these specific details do not need to be used to practice the implementation of the present framework. In other cases, in order to avoid unnecessarily obscuring the implementation of the present framework, well-known materials or methods are not described in detail. Although the present framework is susceptible to various modifications and alternative forms, its specific embodiments are shown by way of example in the accompanying drawings and will be described in detail herein. However, it should be understood that it is not intended to limit the present invention to the specific form disclosed, but rather, it is intended to cover all modifications, equivalents, and substitutes that fall within the spirit and scope of the present invention. In addition, for ease of understanding, certain method steps are described as separate steps; however, these separately described steps should not be interpreted as necessarily relying on order in their execution. Independent of the use of grammatical terms, individuals with male or female gender identities are included in the term.

[0015] Unless otherwise indicated, it will be apparent from the following discussion that it should be understood that terms such as "segmentation," "generation," "registration," "determination," "alignment," "positioning," "processing," "computation," "selection," "estimation," "detection," "tracking," and the like may refer to the actions and processes of a computer system or similar electronic computing device that manipulates and transforms data represented as physical (e.g., electronic) quantities within registers and memories of a computer system into other data similarly represented as physical quantities within the memory or registers of the computer system or other such information storage, transmission, or display device. The embodiments of the methods described herein may be implemented using computer software. If written in a programming language that conforms to a recognized standard, the instruction sequences designed to implement the methods may be compiled for execution on a variety of hardware platforms and interfaced with a variety of operating systems. In addition, the implementation of the present framework is not described with reference to any particular programming language. It should be understood that a variety of programming languages ​​may be used.

[0016] For the sake of brevity, an image corresponding to an object (e.g., a tissue, an organ, a tumor, etc. of a subject (e.g., a patient, etc.) or a portion thereof (e.g., a region of interest (ROI) in an image) may be referred to as an image of the object or the object itself or a portion thereof (e.g., ROI) or an image including the object or the object itself or a portion thereof (e.g., ROI). For example, an ROI of an image corresponding to the lungs or the heart may be described as the ROI including the lungs or the heart. As another example, a chest or an image including the chest may be referred to as a chest image, or simply as the chest. For the sake of brevity, a portion of an image corresponding to an object being processed (e.g., extracted, segmented) may be described as the object being processed. For example, extracting the portion of the image corresponding to the lungs from the rest of the image may be described as the lungs being extracted.

[0017] There is a growing interest in establishing correspondence between radiology reports and corresponding images. Typically, regions of interest are found via attention maps or neural network activations corresponding to textual information. The present framework uses semantic coordinates of anatomical relations instead of using attention maps and network activations directly. The present framework maps information between anatomical and textual domains. Anatomical information may include the locations of anatomical landmarks in a normalized coordinate system. Textual information may include textual entries in the radiology report corresponding to anatomical landmarks. The present framework can be used in either or both forward and reverse directions.

[0018] In the forward use of the method, the framework can estimate the location of specific anatomical landmarks in terms of normalized coordinates using textual input entries. A continuous normalized coordinate system mapped to a standard human body is used instead of specific organ labels in the output. This mapping can describe anatomical structures at various granularities. In the reverse use of the method, a large language model (LLM) with human anatomical knowledge can be used to identify the semantic meaning of the presented normalized coordinates. Anatomical landmarks can be identified using only position information from the image data.

[0019] Keyword-level detectors of anatomical structures may only work to a certain extent. On the other hand, machine learning models can provide better generalization capabilities. The present framework leverages existing landmarks and LLMs to generate more accurate estimates. These and other exemplary advantages and features are described in more detail in the following description.

[0020] Figure 1 1 is a block diagram illustrating an exemplary system 100. System 100 includes a computer system 101 for implementing the framework described herein. In some implementations, computer system 101 operates as a standalone device. In other implementations, computer system 101 can be connected (e.g., using a network) to other machines, such as medical imaging device 102 and workstation 103. In a networked deployment, computer system 101 can operate in the capacity of a server (e.g., as a client user machine in a server-client user network environment, or as a peer machine in a peer-to-peer (or distributed) network environment).

[0021] In one implementation, the computer system 101 includes a processor device or central processing unit (CPU) 104 coupled to one or more non-transitory computer-readable media 105 (e.g., computer storage or memory devices), a display device 108 (e.g., a monitor), and various input devices 110 (e.g., a mouse, touchpad, or keyboard) via an input-output interface 121. The computer system 101 may also include support circuits such as caches, power supplies, clock circuits, and communication buses. Various other peripheral devices, such as additional data storage devices and printing devices, may also be connected to the computer system 101.

[0022] The present technology can be implemented in various forms of hardware, software, firmware, dedicated processors, or a combination thereof, or as part of microinstruction code, or as part of an application or software product, or a combination thereof, which is executed via an operating system. In some implementations, the technology described herein is implemented as computer-readable program code tangibly embodied in one or more non-transitory computer-readable media 105. Specifically, the present technology can be implemented by processing module 106 and database 107.

[0023] The non-temporary computer readable medium 105 may include random access memory (RAM), read-only memory (ROM), magnetic floppy disk, flash memory and other types of memory, or a combination thereof. The computer readable program code is executed by the processor device 104 to process the data provided by, for example, the medical imaging device 102. In this way, the computer system 101 is a general-purpose computer system, which becomes a special-purpose computer system when executing the computer readable program code. The computer readable program code is not intended to be limited to any specific programming language and its implementation. It should be understood that various programming languages ​​and their encodings can be used to implement the teachings of the disclosure contained herein. The same or different computer readable media 105 can be used to store databases, including but not limited to image data sets, knowledge bases, individual subject data, electronic medical records (EMR), subject's diagnostic reports (or documents) or combinations thereof.

[0024] The medical imaging device 102 acquires medical image data 132. Such medical image data 132 may be processed by the processing module 106. The medical imaging device 102 may be a radiology scanner and / or appropriate peripherals (e.g., a keyboard and a display device) for acquiring, collecting, and / or storing such medical image data 132. The medical imaging device 102 may acquire medical image data 132 from a subject or patient using techniques such as high-resolution computed tomography (HRCT), magnetic resonance (MR) imaging, computed tomography (CT), spiral CT, X-ray, angiography, positron emission tomography (PET), fluoroscopy, ultrasound, single photon emission computed tomography (SPECT), or a combination thereof. Other types of imaging modalities are also useful. Medical imaging software may be used to control the medical imaging device 102.

[0025] The medical image data 132 represents a medical image (or in some examples, more than one medical image). For example, the medical image data 132 may include an array or list of pixel or voxel values. When processed by suitable image viewing software, the medical image data 132 results in a presentation of the medical image (or multiple medical images) it represents. An image file containing the medical image data 132 may also include one or more attributes and attribute values. A specific example of an image file is a Digital Imaging and Communications in Medicine (DICOM) file. The one or more attribute values ​​are separate and distinct from the medical image data 132, but include a text string indicating the content of the medical image data 132. In some examples, such attribute values ​​may be referred to as metadata for the image file. In some examples, the portion of the image file that stores attributes and attribute values ​​may be referred to as a header of the image file, and the attributes and attribute values ​​may be referred to as header data of the image file.

[0026] The workstation 103 may include a computer and appropriate peripherals, such as a keyboard and a display device, and may operate in conjunction with the entire system 100. For example, the workstation 103 may communicate with the medical imaging device 102 so that the medical image data 132 may be presented or displayed at the workstation 103. The workstation 103 may communicate directly with the computer system 101 to display processed data and / or output results 144. The workstation 103 may include a graphical user interface to receive user input via an input device (e.g., a keyboard, a mouse, a touch screen, a voice or video recognition interface, etc.) to manipulate the visualization and / or processing of data.

[0027] It should also be understood that because some of the constituent system components and method steps described in the accompanying drawings can be implemented in software, the actual connections between the system components (or process steps) can vary depending on how the present framework is programmed. Given the teachings provided herein, one of ordinary skill in the relevant art will be able to envision these and similar implementations or configurations of the present framework.

[0028] Figure 2 2 is a flow chart of an exemplary method 200 for estimating image position based on text input. It should be understood that the steps of method 200 may be performed in the order shown or in a different order. Additional, different or fewer steps may also be provided. In addition, method 200 may be used Figure 1 The system 100 may be implemented by a different system or a combination thereof.

[0029] At 204, the processing module 106 receives input text and a target image of the patient. The input text may be extracted from a radiology report associated with the target image. The input may also be provided by a user through a user interface. The target image is a medical image of the current patient acquired by, for example, a medical imaging device 102. The target image may be, for example, a PET, MR, SPECT, X-ray, or other type of medical image.

[0030] In some implementations, the processing module 106 divides the radiology report into different parts, each of which corresponds to a specific location in the target image. The report can be preprocessed first, for example, to remove stop words (e.g., "a", "and", "to") and punctuation that add noise to the data. Then, a tokenization technique (e.g., tokenization) can be applied to segment the radiology report into meaningful elements, such as sentences or tokens. The tokenization technique can include, for example, a rule-based method or an artificial neural network-based method. Then, each token can be processed as input text by a trained artificial neural network. In some implementations, the tokens are optionally filtered to provide only valid tokens as input text to the trained artificial neural network. The tokens can be filtered by using a trained classifier to determine whether each token describes an anatomical feature. Such a classifier is trained separately from the artificial neural network. The user interface can provide the user with an option to select a filter for the input text.

[0031] In some implementations, the input text is selected by the user via a user interface (e.g., a graphical user interface) displayed on, for example, the workstation 103. The user can select, for example, one or more words (e.g., phrases, sentences) in a radiology report as the input text. A free text box can also be provided to allow the user to enter the input text. The user can create a hyperlink at the selected input text. The hyperlink can also be automatically inserted into the input text by the processing module 106.

[0032] At 206, the processing module 106 maps the input text to target normalized coordinates using an artificial neural network. As previously described, the input text may be extracted from a radiology report manually or automatically by the processing module 106. In addition, the target normalized coordinates may be defined with reference to a selected anatomical atlas of a standard body or a portion thereof.

[0033] An artificial neural network can be trained with a training set of keywords and corresponding normalized coordinates. An artificial neural network can be trained to map any input text to normalized coordinates. An artificial neural network can include, for example, a deep convolutional neural network, a recurrent neural network, a recurrent neural network, a multilayer perceptron, a long short-term memory, a shallow neural network, or a sequence-to-sequence model. Other types of artificial neural networks are also useful.

[0034] Keywords in the training set are one or more words that describe anatomical features (e.g., landmarks or organs) in an anatomical atlas (e.g., a human body atlas). Examples of such keywords include, but are not limited to, upper lobe, pulmonary artery, pulmonary vein, middle lobe, aorta, aortic arch, left ventricle, right atrium, renal center, etc. Keywords can describe corresponding anatomical features at different levels of granularity (e.g., body, chest, heart, tricuspid valve). Each keyword is mapped to the corresponding normalized coordinates of a landmark or bounding box of an organ in the anatomical atlas.

[0035] An anatomical atlas is a reference image representing a standard body (e.g., a human body) or a portion thereof. The anatomical atlas may be stored in database 107 and accessed by processing module 106. For example, database 107 may include one or more anatomical atlases representing various types of people with corresponding characteristics (e.g., body shape, height, weight, gender, age, race). Processing module 106 may select an anatomical atlas that most closely corresponds to a particular set of characteristics of the patient currently being imaged.

[0036] A normalized coordinate system is a continuous coordinate system that is scaled to a predefined bounding box (e.g., a scale of 0 to 1 in each dimension) or an atlas patient image frame with a predefined origin. Such normalization can often advantageously improve neural network training. The normalized coordinates can be two-dimensional or three-dimensional. In some implementations, each anatomical position coordinate is scaled between 0 and 1 by dividing the voxel position by the volume size (i.e., width, height, breadth) of the anatomical atlas. Thus, position (0, 0, 0) can be the lower right front corner of a bounding box predefined for the normalized coordinate system, while position (1, 1, 1) can be the upper left corner of the bounding box.

[0037] In other implementations, normalization is performed by subtracting predefined landmark coordinates. In this normalization method, a known landmark is predefined as the origin of the coordinate system. This predefined landmark becomes a protuberance with coordinates (0, 0, 0). Any other coordinates in the initial frame of the atlas (or target image) can be calculated by subtracting the offset of the predefined landmark. In this case, the distance is specified in standard measurement units (e.g., millimeters), which may be easier to use in the real world.

[0038] At 208, the processing module 106 determines a position in the target image that corresponds to the target normalized coordinates estimated by the trained artificial neural network. In some implementations, the corresponding position is determined by performing a search to find a position in the target image that has normalized coordinates that match the target normalized coordinates. To perform the search, the framework can start with a first image position in the target image, normalize the coordinates of the first image position, and estimate where to find the target normalized coordinates. A trained recurrent neural network can be used to normalize the coordinates. The recurrent neural network can include, for example, an 8-layer residual network. Other types of artificial neural networks are also useful. The input to the neural network can be the first image position and associated image intensity data, and the output of the neural network is the normalized coordinates of the first image position.

[0039] In other implementations, a landmark detector is used to determine a location in the target image that corresponds to the normalized coordinates of the target. The landmarks can be assigned to the normalized coordinate space, and the corresponding location can be calculated from nearby landmarks using, for example, a linear interpolation method.

[0040] At 210, the processing module 106 presents the location in the target image. The determined location can be highlighted in the target image and displayed, for example, at the workstation 103. In some implementations, when the user selects a hyperlink at the input text, the user is directed to the corresponding location in the target image. The user can use, for example, a mouse, a cursor, or other user interface components to select the hyperlink. Whenever a hyperlink at the input text is selected, the corresponding location in the target image can be highlighted and displayed in a window, for example, next to the radiology report. Other ways of presenting the location are also possible.

[0041] Figure 3 is a schematic diagram of an exemplary method 200 for estimating image locations based on text input. A patient's radiology report 302 is processed by the processing module 106 to extract the input text. The input text is fed to a trained artificial neural network 304, which maps the input text to normalized coordinates. A corresponding location 306 in a target image of the patient is determined based on the normalized coordinates. For example, when a user selects a hyperlink inserted at the input text on the radiology report 302, the corresponding location 306 can be highlighted on the target image.

[0042] Figure 4 4 is a flow chart of an exemplary method 400 for determining text data associated with a point of interest in a medical image. It should be understood that the steps of method 400 may be performed in the order shown or in a different order. Additional, different, or fewer steps may also be provided. In addition, method 400 may be used Figure 1 The system 100 may be implemented by a different system or a combination thereof.

[0043] At 404, the processing module 106 receives a user selection of a point of interest in the medical image and determines normalized coordinates of the point of interest. The medical image may be acquired from a current patient by, for example, the medical imaging device 102. The medical image may be a CT, MR, PET, SPECT, or any other type of image. The user may select the point of interest via, for example, a user interface (e.g., a graphical user interface) displayed at the workstation 103.

[0044] In some implementations, the processing module 106 determines the normalized coordinates of the point of interest. The normalized coordinates can be determined using the medical image pixel (or voxel) values ​​associated with the point of interest. In some implementations, a trained recurrent neural network can be used to estimate the normalized coordinates of the point of interest. The recurrent neural network can include, for example, an 8-layer residual network. Other types of artificial neural networks are also useful. The input of the trained recurrent neural network can be the voxel (or pixel) intensity value associated with the point of interest, and the output of the neural network is the corresponding normalized coordinates.

[0045] At 406, the processing module 106 generates a set of context points closest to the point of interest. The set of context points can be identified from, for example, a database 107 of landmarks. The landmarks in the database 107 can be identified as possible points of interest in an anatomical atlas and defined in a normalized coordinate system. As previously described, the normalized coordinate system is a continuous coordinate system scaled to a predefined bounding box. The set of context points may include multiple landmarks and their corresponding normalized coordinates of the closest point of interest. These closest landmarks can be determined by searching a database of landmarks using the normalized coordinates of the point of interest and calculating the distance between the landmarks in the database 107 and the point of interest to find the closest landmark with the shortest distance (e.g., less than a predetermined threshold distance).

[0046] At 408, the processing module 106 constructs a prompt containing a point of interest and a set of context points. The prompt may include a set of context points and normalized coordinates and names of the point of interest. The prompt may then be sent to a trained large language model (LLM) to generate text data associated with the point of interest. An LLM is an artificial intelligence model that can be used to understand and generate human-like responses after training with a large amount of data. An LLM may include one or more artificial neural networks (e.g., transformers) trained using self-supervised and / or semi-supervised learning. The LLM uses deep learning techniques and encodes anatomical knowledge to ensure a better understanding of the relationship and relative position between anatomical features. The LLM may be trained with medical health record data in which anatomical information often appears. Such medical health record data may be obtained from an electronic health record dataset of a large number of patients (e.g., more than one million) and include real medical images and related radiology reports.

[0047] In some implementations, the prompt requests the LLM to generate text data associated with the anatomical feature at the point of interest using the context point set. For example, the prompt may request a description (eg, name, label) of the anatomical feature at the point of interest. An exemplary prompt is as follows: "What anatomical landmark / organ might be at position [10.30, -83.40, 9.179259] if the spatial position of the suprasternal notch is at [7.6, -71.22, 49.87], the spatial position of the origin of the common carotid artery (r) is at [-1.51, -43.21, 36.80], the spatial position of the aorta is at [-5.77, -43.17, -17.51], the spatial position of the origin of the common carotid artery (l) is at [15.25, -41.54, 38.27], the spatial position of the pulmonary trunk is at [23.26, -22.09, -12.0], and the spatial position of the pulmonary artery is at [28.3936, -16.13, -19.0]?" An exemplary response is as follows: "At or near the sternum."

[0048] At 410, the trained LLM is used to generate text data associated with the point of interest in response to the prompt. The processing module 106 can receive the generated text data from the trained LLM and display the text data, for example, at the workstation 103. In response to the prompt, the generated text data can describe or provide a semantic meaning of the point of interest. The knowledge encoded in the LLM model is richer than the limited landmarks present in the database of landmarks 107. This information is useful for both report generation and information retrieval of the discovery of any selected point of interest. For example, the generated text data can be compared with the patient's existing medical history to find mentions at corresponding locations.

[0049] Figure 5 is a schematic diagram of an exemplary method 400 for determining text data associated with a point of interest in a medical image. The method employs an LLM 502 with knowledge of human anatomy to identify anatomical landmarks using only position information from a medical image I. Normalization of the coordinates of a point of interest 504 creates a relationship between the current medical image and an atlas image with an associated landmark database. Thus, nearby landmarks can be directly identified using distances in the normalized coordinate space. Given the locations of these landmarks in the normalized coordinate system, the LLM 502 can identify any randomly selected point 504 in the medical image I.

[0050] The positions (p) of n landmarks l in image I are known in a set s, where s = {l i :p i} i＝1，...，n , and i is an index, and n is a positive integer. The user can submit an indication of the point of interest r on image I j(504) selects a request 506. Given a point of interest r j (504), the m closest points in the set s are selected as context c: c={l i :p i} i＝1,...,m --------(1) where i is an index and m is a positive integer.

[0051] The processing module 106 constructs a set containing context c and interest points r i The hint is sent to LLM 502. LLM 502 identifies the most relevant landmarks It corresponds to the interest point r of a given set c i (504): LLM 502 then generates a description of the point of interest r i In this case, text data "lower edge of the liver" is generated.

[0052] The following is a list of non-limiting illustrative embodiments disclosed herein:.

[0053] Illustrative embodiments 1. A system comprising: a non-transitory memory device for storing a computer-readable program code; and a processor device in communication with the non-transitory memory device, the processor device operating together with the computer-readable program code to perform the following steps, including: receiving input text and a target image of a patient, mapping the input text to target normalized coordinates using an artificial neural network, determining a location in the target image corresponding to the target normalized coordinates, and presenting the location in the target image.

[0054] Illustrative embodiment 2. The system of illustrative embodiment 1, wherein the step further comprises determining the input text by segmenting the radiology report into different parts using a tokenization technique to generate one or more valid tokens.

[0055] Illustrative embodiment 3. The system of any of illustrative embodiments 1-2, wherein the steps further include providing a user interface that enables a user to select input text from a radiology report.

[0056] Illustrative embodiment 4. The system of any of illustrative embodiments 1-3, wherein the artificial neural network is trained using a training set of keywords with corresponding normalized coordinates, wherein the keywords describe anatomical features in an anatomical atlas.

[0057] Illustrative embodiment 5. The system of illustrative embodiment 4, wherein the corresponding normalized coordinates are scaled to a predefined bounding box.

[0058] Illustrative embodiment 6. The system of illustrative embodiment 4, wherein subtracting predefined landmark coordinates is used to determine corresponding normalized coordinates.

[0059] Illustrative embodiment 7. The system of any of illustrative embodiments 1-6, wherein determining a location in the target image corresponding to the target normalized coordinates comprises performing a search to find a location in the target image having normalized coordinates that match the target normalized coordinates.

[0060] Illustrative embodiment 8. A method for determining text data associated with a point of interest, comprising: receiving a user selection of a point of interest in a medical image; generating a set of context points closest to the point of interest; constructing a prompt including the point of interest and the set of context points; and generating text data associated with the point of interest by a large language model in response to the prompt.

[0061] Illustrative embodiment 9. The method of illustrative embodiment 8, further comprising determining normalized coordinates of the point of interest.

[0062] Illustrative embodiment 10. The method of illustrative embodiment 9, wherein determining normalized coordinates of the point of interest includes using a trained recurrent neural network.

[0063] Illustrative embodiment 11. The method of any of illustrative embodiments 8-10, wherein generating the context point set comprises identifying the context point set from a database of landmarks.

[0064] Illustrative embodiment 12. The method of illustrative embodiment 11, wherein the database of landmarks is defined in a normalized coordinate system.

[0065] Illustrative embodiment 13. A method according to illustrative embodiment 12, wherein identifying a context point set includes searching a database of landmarks using the normalized coordinates of the point of interest, and calculating the distance between one or more landmarks in the database of landmarks and the point of interest to find the context point set closest to the point of interest.

[0066] Illustrative Embodiment 14. The method of any of Illustrative Embodiments 8-13, wherein constructing the prompt comprises constructing a prompt requesting a description of an anatomical feature at a point of interest.

[0067] Illustrative embodiment 15. The method of illustrative embodiment 14, wherein the prompt requests a description of the anatomical features at the point of interest given the normalized coordinates of the point of interest and a set of context points.

[0068] Illustrative embodiment 16. The method of any of illustrative embodiments 8-15, wherein medical health record data is used to train a large language model.

[0069] Illustrative embodiment 17. One or more non-transitory computer-readable media comprising computer-readable instructions, which, when executed by a processor device, cause the processor device to perform the following steps, including: receiving a user selection of a point of interest in a medical image; determining normalized coordinates of the point of interest; generating a set of context points closest to the point of interest, wherein the set of context points is identified from a database of landmarks defined in a normalized coordinate system; constructing a prompt comprising the point of interest and the set of context points; and generating text data associated with the point of interest by a large language model in response to the prompt.

[0070] Illustrative embodiment 18. The one or more non-transitory computer-readable media of illustrative embodiment 17, wherein constructing the prompt comprises constructing a prompt requesting a description of an anatomical feature at a point of interest.

[0071] Illustrative Embodiment 19. The one or more non-transitory computer-readable media of any of Illustrative Embodiments 17-18, wherein the hint comprises normalized coordinates of a point of interest and a set of context points.

[0072] Illustrative embodiment 20. The one or more non-transitory computer-readable media of any of illustrative embodiments 17-19, wherein determining normalized coordinates of the point of interest comprises using a trained recurrent neural network.

[0073] Although the present framework has been described in detail with reference to exemplary embodiments, it will be appreciated by those skilled in the art that various modifications and substitutions may be made thereto without departing from the spirit and scope of the present invention as set forth in the appended claims. For example, within the scope of the present disclosure and the appended claims, elements and / or features of different exemplary embodiments may be combined with each other and / or substituted with each other.

Claims

1. A system comprising: a non-transitory memory device for storing computer readable program code; as well as a processor device in communication with the non-transitory memory device, the processor device operating with the computer readable program code to perform the following steps, comprising: receiving input text and a target image of a patient, Use an artificial neural network to map input text to target normalized coordinates, determining the location in the target image corresponding to the target normalized coordinates, and The location is rendered in a target image.

2. The system of claim 1, wherein the step further comprises determining the input text by segmenting the radiology report into different parts using a tokenization technique to generate one or more valid tokens.

3. The system of claim 1, wherein the step further comprises providing a user interface that enables a user to select input text from a radiology report.

4. The system of claim 1, wherein the artificial neural network is trained using a training set of keywords with corresponding normalized coordinates, wherein the keywords describe anatomical features in an anatomical atlas. The system of claim 4 , wherein the corresponding normalized coordinates are scaled to a predefined bounding box. The system of claim 4 , wherein subtracting predefined landmark coordinates is used to determine corresponding normalized coordinates.

7. The system of claim 1, wherein determining a location in the target image corresponding to the target normalized coordinates comprises performing a search to find a location in the target image having normalized coordinates that match the target normalized coordinates.

8. A method for determining text data associated with a point of interest, comprising: receiving a user selection of a point of interest in a medical image; Generate a set of context points closest to the point of interest; Constructing a prompt containing a set of points of interest and context points; as well as Text data associated with the point of interest is generated by a large language model in response to the prompt.

9. The method of claim 8, further comprising determining normalized coordinates of the point of interest.

10. The method of claim 9, wherein determining normalized coordinates of the points of interest comprises using a trained recurrent neural network. The method of claim 8 , wherein generating the context point set comprises identifying the context point set from a database of landmarks.

12. The method according to claim 11, wherein: A database of landmarks defined in a normalized coordinate system.

13. The method of claim 12, wherein identifying the context point set comprises searching a database of landmarks using the normalized coordinates of the point of interest, and calculating a distance between one or more landmarks in the database of landmarks and the point of interest to find the context point set closest to the point of interest.

14. The method of claim 8, wherein constructing the prompt comprises constructing a prompt requesting a description of an anatomical feature at a point of interest.

15. The method according to claim 14, wherein: The hint requests a description of the anatomical features at the point of interest given the normalized coordinates of the point of interest and a set of context points.

16. The method of claim 8, wherein medical health record data is used to train a large language model.

17. One or more non-transitory computer-readable media comprising computer-readable instructions that, when executed by a processor device, cause the processor device to perform the following steps, comprising: receiving a user selection of a point of interest in a medical image; Determine the normalized coordinates of the point of interest; generating a set of context points closest to the point of interest, wherein the set of context points is identified from a database of landmarks defined in a normalized coordinate system; Constructing a prompt containing a set of points of interest and context points; as well as Text data associated with the point of interest is generated by a large language model in response to the prompt.

18. The one or more non-transitory computer-readable media of claim 17, wherein constructing the hint comprises constructing a hint requesting a description of an anatomical feature at a point of interest.

19. The one or more non-transitory computer-readable media of claim 17, wherein the hint comprises normalized coordinates of a point of interest and a set of context points.

20. The one or more non-transitory computer-readable media of claim 17, wherein determining normalized coordinates of the point of interest comprises using a trained recurrent neural network.

Images