Method and system for automatically tracking and reading medical image data

The method automates the selection and registration of medical images based on body regions and comparability, addressing the inefficiencies in manual comparison of large datasets, improving the efficiency of medical image evaluation.

JP2024528381A5Active Publication Date: 2024-11-20SIEMENS HEALTHINEERS AG
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
JP2023574229
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2021-08-05
Filing Date
2022-04-20
Publication Date
2024-11-20
Estimated Expiration
2042-04-20

AI Technical Summary

Technical Problem

Medical professionals face challenges in efficiently comparing medical images taken at different times due to the time-consuming and burdensome manual assessment of multiple large image datasets, which are often stored remotely, leading to network resource concentration issues.

Method used

A method and system for automatically selecting and registering reference medical images based on body regions and comparability, using machine learning and image registration techniques to generate display data for efficient comparison.

Benefits of technology

Facilitates efficient and flexible selection of appropriate previous medical images for comparison, reducing the need for manual analysis and network data transfer, thereby enhancing the efficiency of medical image evaluation.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader

Abstract

A method and system for generating display data for a medical image dataset is provided, the method and system being configured to identify, for a set of target medical images of a patient at a first time point, a set of reference medical images of the patient acquired at a second time point distinct from the first time point. In one example, the selection is based on a comparison of respective depicted body regions of the patient. The method and system are also configured to generate display data for displaying a rendering of the reference medical images on a display device based on a registration between the target medical images and the reference medical images.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical Field]

[0001] This application relates to medical image processing, such as image processing of X-ray images or magnetic resonance images or computed tomography images. In particular, this application relates to processing separate medical image datasets of a patient to automatically identify image datasets taken at different time points for later comparison by a physician. [Background technology]

[0002] Medical imaging, such as MRI (Magnetic Resonance Imaging) and CT (Computed Tomography), is an important tool in medical diagnosis. In determining clinical decisions, the progression of a patient's disease over time can be as important, if not more important, than the current state of that disease. To aid in assessing the progression of a patient's disease, medical professionals often desire to compare a patient's subject medical image with one or more appropriate reference medical images of the patient. Specifically, for example, the subject medical image may be an image from the patient's current examination, while the reference medical image may be an image from a previous examination of the patient.

[0003] There are often many possible reference medical images for a patient, which may be designated as candidate medical images. For example, a patient may have multiple examinations performed at multiple previous time points (prior examinations). Furthermore, each examination may have multiple sets of medical images, each with different characteristics. Manually evaluating the validity of previous medical images for comparison with or relevance to a current medical image is time-consuming and burdensome for medical professionals. Furthermore, medical images are often stored on storage devices remote from the medical professional's device, and collecting all of a patient's previous medical images for evaluation by the medical professional is network-resource-intensive due to the large size of the images. Summary of the Invention

[0004] Accordingly, it is an object of the aspects described herein to provide a system and method for automatically selecting a medical image (e.g., a previous or, collectively, a reference medical image) that is relevant for comparison with or related to a given medical image (e.g., a current or, collectively, a target medical image).

[0005] This and other objects are achieved by the display data of the independent claim. of This is achieved by a generating method, a system corresponding to the method, a computer program (or a product storing the program) corresponding to the method, and a computer-readable storage medium. Alternative aspects are the subject of the dependent claims and are aspects described herein.

[0006] In the following, the technical solution according to the invention will be described in relation to the claimed apparatus as well as the claimed method. Features, advantages or alternative aspects described therein also apply to the other subject matter of the claims, and vice versa. In other words, a claim directed to the inventive method can be improved by features described or claimed in relation to the apparatus. In such a case, for example, functional features of the method are implemented by suitable units or elements of the apparatus.

[0007] According to one aspect, a method for generating display data for a medical image dataset is provided. The method includes a plurality of steps. A first step is directed to receiving a set of target medical images of a patient at a first time point. A second step is directed to obtaining a region of interest in the set of target medical images. A second step is directed to determining a target body region represented by the set of target medical images. A second step is directed to selecting a reference medical imaging study from a plurality of candidate medical imaging studies based on a comparison of the plurality of candidate body regions with the plurality of target body regions. Each of the plurality of candidate body regions corresponds to one of the plurality of candidate medical imaging studies, and each candidate medical imaging study includes a plurality of candidate medical images of the patient at a second time point. A second step is directed to selecting a reference medical imaging study from the plurality of candidate medical images of the reference medical imaging study that is related to the set of target medical images. Degree of comparabilityThe object of the present invention is to select a set of reference medical images based on the region of interest. The next step is to perform registration of the set of target medical images with the set of reference medical images. The next step is to generate display data and display on a display device a rendering of the set of reference medical images based on the region of interest and the registration.

[0008] In one example, the medical image collection (subject medical image collection, candidate medical image collection, reference medical image collection) relates to one or more two-dimensional images. In one example, the medical image collection is a three-dimensional image. In one example, the medical image collection is a four-dimensional image, with three spatial dimensions and one temporal dimension.

[0009] As an example, the type of medical image collection relates to the type of medical imaging device used to acquire the medical image collection. For example, a first set of X-ray images and a second set of CT images are comparable even if they were recorded by different medical imaging devices. As an example, two medical image collections are comparable if they correspond to the same anatomical region of the human body (body region). For example, a first X-ray image of a human lung and a second X-ray image of a human knee are not comparable even if they relate to the same patient and were acquired with the same medical imaging modality.

[0010] As an example, a type of medical image collection can be characterized by the modality used to generate the medical images and by the body region that the medical images cover. Alternatively, a type of medical image can be characterized by the parameters (of the imaging modality) used to generate the medical images (e.g., one could distinguish between "low-dose images" and "high-dose images").

[0011] Each medical image set comprises image data, for example in the form of an array of pixels or voxels, which represent intensity, absorption or other parameters as a function of three-dimensional position and which may be obtained, for example, by suitable processing of measurement signals obtained by a medical imaging modality.

[0012] A medical image collection may be the same as or contained in one or more image data files, in particular DICOM files, where reference to DICOM is understood to refer to the "Digital Imaging and Communications in Medicine" (DICOM) standard, e.g., according to the DICOM PS3.1 2020c standard (or earlier or later versions of that standard).

[0013] "Received" may mean, for example, that the subject medical images are obtained from a medical imaging modality. "Received" may also mean, for example, that the subject medical images are obtained from a suitable memory, such as a Picture Archiving and Communication System (PACS) or other suitable medical image storage facility.

[0014] The target medical image group may relate to a patient examination at a first time point (first implementation time), while the candidate medical image group of the candidate examination may relate to a patient examination at a second time point (second implementation time) different from the first time point. The second time point may be a few hours or hours before, a few days or weeks before, a few months before, a few months after, or a few years after the first time point. There may also be an intervening scan or procedure between the first and second time points.

[0015] For example, one or more medical image sets may be associated with a corresponding medical imaging study. In other words, a medical imaging study may include multiple medical image sets. The medical image sets included in a medical imaging study may be of different types. For example, the medical image sets of a medical imaging study may be acquired with different modalities and / or may be acquired using different parameters when generating the medical image sets. For example, a medical imaging study may include various MRI medical image sets acquired with different MRI imaging parameters. For example, one medical image set may be acquired with a T2FLAIR setting, while another medical image set is a T1 set.

[0016] The medical images contained in a medical imaging study represent regions of a patient's body, and therefore it is possible to assign at least one body region to the medical imaging study. By way of example, the medical images contained in a medical imaging study may (although not necessarily) represent (at least approximately) the same region of a patient's body.

[0017] Although the medical images included in a medical imaging study will necessarily be acquired at slightly different times, they can be considered to relate to the same point in time with respect to the follow-up review workflow described herein, i.e., a difference of a few minutes (or even a few hours) between the acquisition times of the images of the study still pertains to the same state of the patient in a clinical sense.

[0018] In one example, the target medical imaging study and the candidate medical imaging studies relate to a single patient. In some examples, the candidate medical imaging studies may relate to multiple patients other than the patient of the target medical imaging study. In the latter case, similar cases can be compared.

[0019] A region of interest represents a region within the target medical images, which is of particular interest to a user analyzing the target medical images. The region of interest typically relates to the portion of the target medical images that the user is currently reviewing. This region relates, for example, to an image volume or an image plane. For example, the region of interest may be an image slice of the target medical images, also called a "target slice." The region of interest may also be, for example, a portion of an image slice of the target medical images. Thus, the region of interest may have any shape; preferably, the region of interest is circular or quadratic in form. In either case, the region of interest may be understood as a group of picture elements, such as pixels or voxels, within the target medical images.

[0020] The region of interest may be defined by a user, or may be defined semi-automatically or automatically by a computer-implemented method.

[0021] Thus, obtaining the region of interest may be based on, for example, processing one or more user inputs specifying the region of interest in the target medical images, for example, by scrolling to a target slice and / or defining the region of interest in the target slice.

[0022] Obtaining the region of interest may also include automatically identifying anatomical features in the target medical images, the anatomical features representing a pathological condition of the patient, for example, by applying a detection function to the target medical images configured to identify anatomical features in the medical image data.

[0023] A body region, for example, relates to a patient's body part depicted in an image set. Thus, a body region of a medical imaging study represents a patient's body part depicted in image data associated with the medical imaging study. In this case, the body region is not a rigid descriptor of the patient's anatomical structure clipped by the image data, for example in the form of an anatomical coordinate system, but rather a high-level or coarse-grained descriptor of the image data content of the study. For example, the body regions of the individual medical images of the study are aggregated to derive a body region for the medical imaging study. According to some examples, the body region is a range of a normalized or generalized coordinate system relative to the patient's anatomical structure. In one example, these normalized or generalized coordinate systems are defined along the craniocaudal or longitudinal axis of the human / patient.

[0024] The target and candidate body regions may be determined based on, for example, image data and non-image data associated with each of the medical imaging studies. As an example, metadata of the medical imaging studies that describes the contents of the medical imaging studies may be evaluated to determine the target and candidate body regions.

[0025] The degree of comparability indicates, for example, how well a user can compare two separate medical image data sets. As an example, the degree of comparability of two medical image data sets is based on the respective types of the two medical image data sets. Accordingly, the type of a medical image set may be characterized, as described above, by the modality used to generate the medical image set, by one or more image parameters used to generate the medical image set, and / or by the body region that the medical image set covers.

[0026] Thus, selecting a set of reference medical images from a set of candidate medical images for a reference medical imaging study may, for example, include determining, for each set of candidate medical images, a degree of comparability with a set of target medical images, where determining the degree of comparability is, for example, based on comparing the type of the set of target medical images with each type of the set of candidate medical images.

[0027] Performing registration between a group of target medical images and a group of reference medical images includes, as an example, roughly aligning a target image (e.g., a region of interest or a target slice) of the group of target medical images with a reference image (e.g., in the form of a reference slice) of the group of reference medical images.

[0028] According to one example, this step is based on evaluating the similarity of the image data and / or detecting locations of interest as described elsewhere herein. According to another example, this step includes obtaining a deformation field between the target medical images and the reference medical images, which determines the relationship between the coordinate systems of the target medical images and the reference medical images, such that each anatomical location in the target medical images is mapped to the same anatomical location in the reference medical images (or vice versa). That is, the deformation field includes, for example, a plurality of individual displacement vectors associated with pixels / voxels of the target and reference images, respectively.

[0029] According to one example, the registration comprises rigid registration, in which the coordinate systems of pixels / voxels in one image are subject to rotation and translation in order to align the image with another image. According to some examples, the registration comprises affine registration, in which the coordinate systems of data points in one image are subject to rotation, translation, scaling, and / or shear in order to align the image with another image. That is, rigid registration can be considered a specific type of affine registration. According to some examples, the registration comprises non-rigid registration, in which, for example, a separate displacement is provided for each pixel / voxel in the images to be registered and, for example, a non-linear transformation can be used, in which the coordinate systems of pixels / voxels in one image are subject to flexible deformation in order to align the image with another image. The nonlinear transformation is determined, according to one example, using a vector field, such as a warp field or other field or function, that defines an individual displacement for each pixel / voxel in the image. For details on image registration, US2011 / 0081066 and US2012 / 0235679 are incorporated herein by reference. Rigid image registration is very effective when anatomical changes or deformations are not expected. Compared to rigid image registration, nonrigid image registration has greater flexibility and can manage local distortions (e.g., anatomical structure changes) between two image sets, but is more complex to handle.

[0030] Providing rendering based on the registration and the reference medical image set includes, by way of example, identifying, based on the registration, image data portions of the target medical image set that correspond to the region of interest, where the rendering is based on the identified image data portions of the target medical image set.

[0031] According to some examples, rendering is performed such that the region of interest is displayed in conjunction with the identified image data portion of the target medical image set.

[0032] The method of this aspect allows for stepwise and directional selection and presentation of reference medical image data for a user, captured for easy comparison with a set of target medical images. This provides for efficient selection of previous medical images suitable for comparison with target medical images, compared to, for example, opening and evaluating all of a patient's previous medical images. Moreover, selection based on the body region of each imaging study allows for selection without the need to extract or analyze the medical imaging data itself, thus enabling, for example, efficient selection. Additionally, selection based on the body region and the degree of comparability allows, for example, flexible selection. For example, comparing body regions is more robust and / or flexible with respect to inexact matching between parameters, compared to, for example, attempting to directly match imaging coordinate systems.

[0033] According to one example, the target medical imaging study and each candidate medical imaging study each include one or more attributes with attribute values ​​including text strings indicative of the content of the medical imaging study. Thus, determining the target body region may include, for example, obtaining one or more of the text strings of the target medical imaging study, inputting the obtained one or more text strings into a trained machine learning model, and obtaining output from the trained machine learning model to determine the body region represented by the target medical imaging study. The machine learning model is trained to output a body region based on the input of the one or more text strings. Additionally, at least one of the candidate body regions may be determined by, for example, obtaining one or more of the text strings of the candidate medical imaging study, inputting the obtained one or more text strings into a trained machine learning model, and obtaining output from the trained machine learning model to determine the body region represented by the target medical imaging study. is determined byThe machine learning model is trained to output body regions based on the input of one or more such text strings.

[0034] Generally, a trained machine learning model mimics the cognitive functions humans use to interact with other human minds. By way of example, training based on training data enables a trained machine learning model to adapt to new situations and detect and infer patterns. Other terms for a trained machine learning model include, for example, a trained function, a trained mapping specification, a mapping specification with trained parameters, a function with trained parameters, an artificial intelligence-based algorithm, or a machine-learned algorithm.

[0035] In general, the parameters of a trained machine learning model can be adapted through training. By way of example, supervised training, semi-supervised training, unsupervised training, reinforcement learning, and / or active learning can be used. Furthermore, representation learning (also known as "feature representation learning") can be used. By way of example, the parameters of a trained machine learning model can be iteratively adapted through several steps of training.

[0036] By way of example, the trained machine learning model may consist of a neural network, a support vector machine, a decision tree, and / or a Bayesian network, and / or the trained function may be based on k-means clustering, Q-learning, a genetic algorithm, and / or association rules. By way of example, the neural network may be a deep neural network, a convolutional neural network, or a deep convolutional neural network. Moreover, the neural network may be an adversarial network, a deep adversarial network, and / or a generative adversarial network.

[0037] According to one example, the machine learning model is trained to output body regions in the form of extents in a normalized or generalized coordinate system relative to the patient's anatomy, where, by way of example, these normalized or generalized coordinate systems are defined along the craniocaudal or longitudinal axis of the human / patient.

[0038] For example, attributes and attribute values ​​typically relate to non-image data or metadata associated with each medical imaging study. Attribute values ​​may, for example, be stored in or separately from the image data file for each medical imaging study. As an example, attribute values ​​may also be stored in the patient's electronic medical record.

[0039] The attributes and attribute values ​​provide additional information about the image data, such as the body part examined, the imaging modality and protocol used, and / or any findings or annotations reported or made by the user, etc. Additionally, the non-image data includes information not directly related to the image data, such as patient personal information, such as sex, age, weight, insurance details, information about the treating physician, etc.

[0040] According to one example, each medical imaging study is associated with one or more DICOM image files, and one or more attributes having attribute values ​​containing text strings indicative of the contents of the medical imaging data are the DICOM attributes "Study Description," "Series Description," and "Series Description." covered The body part under investigation may include one or more of the following:

[0041] Determining body regions based on text strings extracted from medical imaging study attributes is less resource intensive, and therefore more efficient, than, for example, determining body regions by extracting and analyzing the medical imaging data itself (which is relatively large in terms of bits). If the candidate medical images are stored remotely over a network from a processing device configured to perform the methods described herein, determining body regions based on (relatively small) text strings has the advantage that the (relatively large) medical imaging data does not need to be transmitted over the network to determine the body regions represented thereby, resulting in efficient use of network resources.

[0042] Determining body regions by inputting text strings into a trained machine learning model (e.g., a trained neural network) can provide efficient, flexible, and / or robust body region determination, as compared to, for example, determining body regions by applying hard-coded rules to text strings. For example, hard-coded rules require the text string to be matched exactly to the rule to provide a body region (and thus are inflexible regarding the text strings for which a match can be determined, and / or are inefficient due to the comprehensive nature of the set of rules that must be coded for all possible text strings). On the other hand, a trained machine learning model (e.g., a trained neural network) is derived from a training dataset for training and therefore can relatively efficiently obtain and determine appropriate body regions even for different text strings in the training dataset, and is therefore relatively flexible / robust.

[0043] According to one example, the trained machine learning model is a trained character-based neural network, e.g., a character-based recurrent neural network configured to take as input individual characters of one or more retrieved text strings, whereby inputting the one or more retrieved text strings into the trained neural network includes inputting the individual characters of the one or more retrieved text strings into the trained character-based neural network.

[0044] The advantage of a character-based language model is its small vocabulary and flexibility in handling any text string and other document structure, which is beneficial in medical applications where attribute values ​​are often not standardized across multiple medical institutions.

[0045] According to one example, the trained neural network is configured to output body regions in the form of one or more numerical values ​​representing regions of the human body, the output portion of the trained neural network being calculated with respect to one or more input text strings.

[0046] Outputting the body region as one or more numerical values ​​allows, for example, an accurate and / or flexible determination of the body region represented by the medical imaging data of each medical imaging study. For example, the numerical values ​​are continuous and therefore allow for more accurate and more flexible definition of the body region compared to, for example, using a limited set of predefined classes. This, in turn, allows, for example, flexible and / or accurate comparison of body regions.

[0047] According to one example, the trained neural network is configured to output the body region in the form of a selection of a body region from a list of predefined body regions, which has the advantage, for example, of reducing the fluidity of the results and providing a clearer association when a medical imaging exam is mapped to a set of predefined body regions.

[0048] According to one example, the neural network is based on a training dataset including a plurality of training text strings. Each training text string is from attribute values ​​of a medical image data file and is indicative of the content of a medical imaging study associated with the medical image data file. Each of the plurality of training text strings is labeled with the training text string's corresponding body region. The labels are used as ground truth in training the neural network.

[0049] According to one example, the training data set is generated using a graphical user interface (GUI) configured to present to a user one or more training text strings and a representation of a human body divided into selectable body regions, receive user input selecting a body region from the representation for each of the one or more presented training text strings, and label the training text string with a label indicating the selected body region.

[0050] According to one example, each of the target and candidate image data sets is associated with one or more attributes, each having an attribute value indicative of the imaging parameters used to capture each of the medical imaging data sets, and the degree of comparability is based on determining a match of one or more attribute values ​​between the target medical image set and each of the plurality of candidate medical image data sets.

[0051] According to one example, each medical image set may include one or more DICOM image files and a corresponding medical imaging data set. Capture and one or more attributes having attribute values ​​containing text strings indicating the imaging parameters used to generate the image.

[0052] The above aspects enable, for example, the selection of one or more reference medical images captured using imaging parameters that are the same as or similar to those of the image data of the target medical image group. This provides efficient selection of previous medical images suitable for comparison with the target medical image data, compared to, for example, opening and evaluating all of the previous medical images related to the patient. Moreover, selection based on the attribute value AV enables selection without the need to extract or analyze the medical imaging data itself, thereby enabling efficient selection.

[0053] According to one example, the step of determining the degree of comparability includes obtaining a first feature vector for the group of target medical images, obtaining a second feature vector for each of the group of candidate medical images constructed in the reference medical examination, and determining a comparability metric indicating the degree of comparability between the first feature vector and each of the second feature vectors of the group of candidate medical images.

[0054] A feature vector is generally thought of as a descriptor that represents or characterizes, for example, an underlying collection of medical images. The feature vectors described herein are based, for example, on image data and non-image data associated with the collection of medical images. In one example, the feature vector is based on one or more attribute values.

[0055] In some examples, the feature vector is determined by encoding the image data and non-image data associated with each of the medical images using a specific encoding algorithm (also called a "vectorizer"). The features of the feature vector are, for example, visual features such as visual manifestations of medical abnormalities, patterns, anatomical structures, medical landmarks, etc., exhibited by each image of the medical images. The feature vector also comprises non-image features, for example, related to the imaging modality or imaging parameters used. The feature vector is calculated online, for example, during selection of a reference medical imaging exam. Alternatively, the feature vector is kept available as a (pre-)generated data item, for example, stored in a database in association with a candidate medical imaging exam.

[0056] According to one example, the degree of comparability can be calculated, for example, by applying a metric ("comparability metric") that represents how similar or comparable a group of candidate medical images is to a group of target medical images. Another term for a comparability metric is a similarity metric or a distance metric. In one example, the comparability metric can be configured to quantify the vector or feature space distance between a first feature vector and a second feature vector. According to one example, a predetermined mathematical function can be used to calculate the distance, such as cosine similarity or Euclidean distance. According to one example, the comparability metric comprises a trained metric derived by machine learning. According to one example, the extraction of feature vectors and / or the evaluation of the overall similarity metric are performed by one or more trained functions.

[0057] According to one example, the target medical image set and the reference medical image set are associated with one or more attributes, each having an attribute value indicating the imaging parameters used to capture the medical imaging data of the image files, wherein a first feature vector is obtained based on one or more of the attribute values ​​indicating the imaging parameters used to capture the target medical image set, and a second feature vector is obtained based on one or more of the attribute values ​​indicating the imaging parameters used to capture each of the candidate medical images.

[0058] According to one example, the step of determining the degree of comparability includes inputting the target medical image group and the reference medical image group into a trained function configured to determine the degree of comparability of two medical image data groups for follow-up reading.

[0059] In one example, the trained function may comprise a neural network, whereby a first group of neural network layers is applied to extract first and second feature vectors. These feature vectors are sent as inputs to a second group of network layers. The second group of network layers functions to determine a degree of comparability based on the extracted feature vectors. However, both of these neural network functions may equally be performed by separate, independent neural networks. That is, image analysis related to feature vector extraction may be performed by a first neural network, and classification according to comparability may be performed by a second neural network.

[0060] Selecting matching groups based on feature vectors allows for flexible selection, for example, comparing feature vectors in feature space may be more robust and / or flexible with respect to inexact matching between parameters compared to, for example, directly matching imaging parameters.

[0061] According to one example, the first and second feature vectors include indicators of the imaging modalities of the subject and candidate medical image data groups, and the comparability metric includes an imaging modality relevance score between the imaging modality of the medical image data group and the imaging modality of each of the candidate medical image data groups.

[0062] The imaging modality relevance score between two modalities is, for example, a value that represents the degree to which medical imaging data captured using one modality is related to (i.e., suitable or useful for comparison with) medical imaging data captured using another modality. In some examples, the imaging modality relevance score may be such that, if two candidate medical image sets both have the same body region as a target medical image set, but one set was captured using MR and the other set was captured using CT, the set captured using MR may be selected in preference.

[0063] Consideration of the modality used when capturing each of the sets of medical image data further improves the comparability of the selected sets of reference medical images.

[0064] According to one example, the imaging modality relevance score accounts for the probability that, given a first set of medical images associated with a particular first imaging modality, a user will select a second set of medical images with a particular second imaging modality for comparison with the first set of medical images, the probability being based on a statistical analysis of user selections recorded on previous occasions.

[0065] The imaging modality relevance score, based on statistical analysis of actual user interactions with the medical imaging data, helps to ensure, for example, that second medical imaging data of an appropriate modality is selected.

[0066] User interaction is also used, for example, for online (ad-hoc) automatic updating of the system's behavior.

[0067] According to one example, the set of reference medical images depicts a reference image volume (e.g., a body region represented by the set of reference medical images). The method further includes obtaining a plurality of candidate slices, each of which depicts a particular section of the reference image volume. Performing registration also includes identifying at least one reference slice from the plurality of candidate slices based on image similarity between image data included in the region of interest and each of the candidate slices. Generating display data also includes generating the display data to display a rendering of the reference slice on a display device.

[0068] In some examples, the plurality of slices represents a subset of the slices included in the set of reference medical images. According to one example, the plurality of slices relates to all of the slices included in the set of reference medical images.

[0069] Image similarity measures, for example, how similar the image data is between two medical images (in this case, slices from a region of interest and a set of reference medical images) in terms of, for example, contrast, tone, texture, density, distortion, singularities, patterns, landmarks, masks, and displayed anatomical structures.

[0070] Identifying a reference slice means, for example, evaluating an image similarity metric representing the image similarity between the image data of the region of interest and each of the slices in the reference medical image set for each of a plurality of candidate slices in the reference medical image set. Based on this evaluation, for example, one or more slices having a high image similarity to the image data of the region of interest are selected and identified. For example, all slices having an image similarity above a predetermined threshold may be selected. Alternatively, the slice with the highest image similarity may be selected. For example, the result of determining the similarity metric between two images is the image similarity between the images.

[0071] In other words, according to this example, one or more slices in a set of reference medical images are automatically identified as corresponding to the region of interest. This automatic search is based on quantifying the image similarity between the target image data (here, the region of interest) and the reference image data (here, each slice in the set of reference medical images). This provides a tool that can automatically search for slices in a set of reference medical images that are related to the region of interest in terms of the reference image data. Therefore, the physician does not need to manually scroll through the set of reference medical images to track down the related slices. This not only saves considerable time, but also makes the process less error-prone and improves reproducibility. Matching slices in multiple medical imaging studies is based on evaluating the similarity between individual images. Compared to other methods, this has the advantage of a comparatively fast processing time. Moreover, image similarity often produces reasonable results even in the case of other methods, such as methods based on conventional image registration, or when anatomical landmark detection is not applicable. This may be the case when registration is impossible due to the large variation between individual slices, or when landmarks cannot be detected. Thus, it is also possible to easily synchronize widely different image data acquired using, for example, different imaging protocols or imaging modalities.

[0072] According to one example, the target medical image data set includes a plurality of target slices that depict a target image volume (e.g., a body region represented by the target medical images), each of the target slices depicting a specific section of the target image volume. Thus, performing registration includes, for example, identifying one matching candidate slice for each of the plurality of target slices to determine slice matches between the target medical images set and the reference medical image data set, where a similarity measure is based on the similarity between each target slice and each candidate slice of the reference medical images. Then, determining at least one reference slice is based on the slice matches.

[0073] In other words, the method thus unambiguously matches slices in the target medical image set with matching slices in the reference medical image set based on image similarity considerations. Accordingly, the slices of the two different medical image sets are synchronized. Thus, when scrolling through the target medical image set, the user is automatically provided with matching slices from each of the reference medical image sets (and vice versa). That is, the user is provided with a position where they can directly compare the target medical image set and the reference medical image set side-by-side. According to some examples, the results of such a matching process are provided in the form of a data association that unambiguously assigns each of the slices of the first medical data set to a matching slice from the slices of the second medical image set (CMIS). Such an association may be referred to as a slice correspondence.

[0074] In cases where the identification of a matching slice is ambiguous, the method utilizes auxiliary conditions to pinpoint the matching slice. The auxiliary conditions may be, for example, the slice order in the target medical image group and / or the reference medical image group, and / or a measure of overall similarity. Thus, according to one example, the step of identifying a matching slice in the reference medical image group for each of a plurality of slices in the target medical image group is additionally performed based on the slice order in the target medical image group or the reference medical image group and / or additionally under a condition that maximizes the overall image similarity. In this regard, the overall image similarity may be, for example, the cumulative image similarity (the sum of the individual image similarities) between the matching slices in the target medical image group and the reference medical image group. The slice order may be considered as the natural sequence of slices as determined by the imaging process or simply by the patient's anatomical structure.

[0075] According to one example, performing registration includes determining an offset in slice numbering between the target medical images and the candidate medical images based on the slice matching, and determining at least one reference slice based on the offset.

[0076] Determining the offset can provide a simple method for aligning the target and reference medical images with each other, such that when scrolling through the target medical images, the user is automatically provided with matching slices of each of the reference medical images (and vice versa).

[0077] According to one example, performing registration includes extracting at least image descriptors from the target medical images and extracting matching image descriptors from each of a plurality of candidate slices, whereby the image similarity is based on a comparison of the extracted image descriptors of the target medical images with the image descriptors of the candidate slices.

[0078] In some examples, the image descriptors are vectors that represent or characterize image data underlying either the region of interest or a slice of a set of reference medical images. In some examples, the image descriptors are determined by encoding the region of interest or each slice of a set of reference medical images using a particular image descriptor encoding algorithm. For example, features of the region of interest or each slice, such as visual manifestations of medical abnormalities, patterns, anatomical structures, medical landmarks, etc., are encoded in the respective image descriptors.

[0079] According to one example, image similarity can be calculated by applying an image similarity metric that represents how similar individual image descriptors are. In one example, the image similarity metric is configured to quantify the distance in feature space between the image descriptor of the region of interest and each image descriptor of a slice of a set of reference medical images. According to one example, a predetermined mathematical function can be used to calculate the distance, such as cosine similarity or Euclidean distance. According to one example, the image similarity or distance metric comprises a trained metric derived by machine learning. According to one example, the extraction of image descriptors and / or the evaluation of the overall image similarity metric is performed by one or more trained functions.

[0080] Selection of matching image data based on image descriptors allows for flexible selection, for example, comparison of image descriptors in feature space is more robust and / or flexible with respect to inexact matches between parameters compared to, for example, attempts at rigid or non-rigid image registration.

[0081] According to one example, the method further includes extracting an image descriptor from each of the plurality of target slices, wherein the similarity measure is based on a comparison between the extracted image descriptors of the target and candidate slices, respectively.

[0082] According to one example, obtaining the plurality of candidate slices includes resampling the candidate medical dataset based on the subject medical image dataset to determine the plurality of candidate slices in the reference medical image dataset.

[0083] Resampling is equally applicable to medical image sets that already have defined slices and to medical image data sets that do not have defined slices and that more or less depict an isotropic image volume. According to some examples, resampling includes determining slices of the reference medical image set so that the slices have slice thicknesses that correspond to slice thicknesses of the target medical image set. Additionally or alternatively, for example, the stacking orientation of the slices is set or adapted to match the target medical image set. Resampling the reference medical image set can result in the reference medical image set being adapted to the target medical image set. This can, for example, improve the comparability of the two sets and provide overall better results for subsequent follow-up reading.

[0084] According to one example, the step of identifying at least one reference slice includes applying a trained function to the subject and reference medical image data sets. doIn this case, the trained function is configured to determine similarity between two-dimensional medical images. Optionally, the trained function further applies a learned metric to determine similarity between two-dimensional medical images, where the trained function preferably comprises a deep metric trained network.

[0085] According to some examples, the trained function comprises a distance metric network, and more specifically, a deep distance metric learning network.

[0086] Distance metric learning (or simply distance learning: metric learning) is a machine learning method that aims to automatically construct task-specific distance or image similarity metrics from supervised data. The learned distance metric can then be used to perform various tasks, such as identifying similar image data in this example. Compared to using a preset distance metric, the advantage of using a learned distance metric is, for example, that the learned metric is better adapted to the specific data and task of interest.

[0087] Deep distance metric learning networks, for example, further transform the data into a new, more discriminatory feature space before a metric (either trained or standard) is applied, such that semantically similar extracted image features are mapped to nearby locations while dissimilar image features are pushed apart using an appropriate distance metric.

[0088] According to some examples, the trained function comprises a Siamese network. According to one example, the trained function comprises a fully convolutional Siamese network.

[0089] A Siamese network is a type of neural network that learns to compare two inputs based on a distance or similarity metric, where an input that is closer in a certain semantic sense has a lower distance compared to two inputs that are further apart according to the same semantic sense. The semantic that needs to be captured is implicitly provided to the network during the training process. The semantic can be viewed as a vector space of image descriptors. Thus, the semantic determines which image descriptors are extracted. The semantic is extracted, for example, using sub-network branches with identical structure and parameters. The extracted image descriptors are considered, for example, as a representation of the learned semantic. As an example, a Siamese network includes a structure with two sub-network branches with identical structure and parameters.

[0090] Based on a fully convolutional Siamese network, for example, at least one convolutional process and at least one pooling process are performed on, for example, a first slice of a first image dataset, thereby obtaining image features (or image descriptors) for the first image slice. Also, for example, at least one convolutional process and at least one pooling process are performed on, for example, a second slice of a second image dataset, thereby obtaining image features (or image descriptors) for the second image slice. The output image features (or image descriptors) after the convolutional and pooling processes are, for example, one or more feature vectors or maps of the same size. Parameters in the convolutional or pooling processes, such as the size and number of convolutional kernels used for each convolutional layer or each pooling layer, are pre-configured, for example, through a training process of the fully convolutional Siamese network. For example, the fully convolutional Siamese network also has a structure with two branches of sub-networks with the same structure and parameters.

[0091] In some examples, the trained function consists of a triplet network.

[0092] A triplet network can be thought of as an extension of a Siamese network, since it constitutes, for example, three branches of the same feedforward network. Given three samples, the network outputs an intermediate value in the form of a pairwise distance. This allows one sample to be treated as a reference (or anchor) against which the other samples are compared. The intermediate value is then fed into a comparator, which determines the output. Rather than directly comparing data labels, triplet networks allow learning by comparing samples, which lowers training data requirements and allows for use as an unsupervised learning model.

[0093] According to some examples, the trained function is trained using a triplet loss function.

[0094] The triplet loss function is a loss function for machine learning algorithms in which a reference (anchor) input is compared against a positive (true) input and a negative (false) input. The distance from the baseline (anchor) input to the positive input is minimized, and the distance from the reference input to the negative input is maximized. In this example, for example, the positive input is a slice whose similarity to a predetermined reference slice is verified. The similarity can be verified, for example, by a user or due to a verified neighbor relationship of the slice to the reference slice of an imaging study. The latter has the advantage that training data is much easier to obtain. The negative input is, for example, a slice that is less similar to the reference slice than the positive input. As an example, the negative input is a slice that is not related to the reference slice, for example, a slice from another group or patient. By adhering to such a similarity ranking, the triplet loss model embeds (i.e., extracts image descriptors) in a manner in which pairs of similar slices have a smaller distance (or are more similar) than dissimilar slices. One advantage of this approach is that the triplet loss function is very flexible in terms of the training data required.

[0095] As another example, other loss functions can also be used, such as a contrastive loss function calculated by contrasting two or more similarities of slice pairs, or a categorical cross-entropy loss function.

[0096] According to one example, the trained function is configured to determine similarity between two-dimensional medical images by comparing a first candidate image to a target image and a second candidate image to the target image, and determining whether the first candidate image or the second candidate image is more similar to the target image, where the target image is extracted from a set of target medical images and the candidate image is extracted from a set of reference medical images.

[0097] According to one example, the step of performing registration includes determining a location in the set of reference medical images that matches the region of interest. For example, this step includes generating a first local image descriptor based on image data of the region of interest, generating a second local image descriptor for each of a plurality of candidate locations in the set of reference medical images (the second local image descriptor is generated based on image data of the reference medical images located relative to each candidate location), calculating a local image similarity metric for each of the plurality of candidate locations indicating a local image similarity between the first local image descriptor and the second local image descriptor for the candidate location, selecting a candidate location from the plurality of candidate locations based on the calculated local image similarity metric, and determining a location in the set of reference medical images that matches the region of interest based on the selected candidate location. Furthermore, in the step of generating display data, rendering is generated based on image data of the reference medical images for the set selected candidate location.

[0098] Local image descriptors are, for example, vectors that represent or characterize underlying image data and image data extracted from a reference medical image set with respect to candidate locations of regions of interest in the subject medical image set. In some examples, local image descriptors are determined by encoding matching images using a specific image descriptor encoding algorithm. Features of each region of interest or slice, such as visual manifestations of medical abnormalities, patterns, anatomical structures, medical landmarks, etc., are encoded in each local image descriptor, for example. According to some examples, local image descriptors consist of pattern descriptors that characterize one or more patterns in the underlying image data.

[0099] According to some examples, the local image similarity is calculated by applying a local image similarity metric that represents how similar individual local image descriptors are. In some examples, the local image similarity metric is configured to quantify the distance in feature space between the local image descriptor of the region of interest and the local image descriptor extracted for the candidate location in the set of reference medical images. According to some examples, a predetermined mathematical function can be used to calculate such distance, such as cosine similarity or Euclidean distance. According to one example, the local image similarity or distance metric comprises a learned metric derived by machine learning. According to some examples, the extraction of the local image descriptors and / or the evaluation of the overall local image similarity metric are performed by one or more trained functions. According to one example, the local image similarity metric is implemented as a pattern similarity metric, where the local image similarity is the pattern similarity between image patterns associated with each location.

[0100] As a result, a technique is provided for determining the location where a predetermined pattern or feature of a region of interest is represented in a set of reference medical images. As an example, known locations where a predetermined pattern or feature is represented in a set of target medical images (i.e., a region of interest) are used to determine the location where the predetermined pattern or feature is represented in a set of reference medical images. This reduces the user's burden in locating relevant locations in a set of reference medical images (RMIS), for example. Moreover, this determination based on determining local image similarity provides fast, efficient, and / or flexible feature location, especially in cases where traditional image registration is not feasible. Furthermore, determining local image similarity results in robust, consistent, and stable feature location that is less prone to human error.

[0101] According to one example, the candidate locations are locations distributed throughout the set of reference medical images in a predetermined sampling pattern. Extracting local image descriptors based on a predetermined distribution or sampling pattern allows for global sampling of the set of reference medical images.

[0102] According to one example, the step of performing registration further includes determining a plurality of candidate sub-resolution locations in the reference medical image set based on the selected candidate locations and obtaining a third local image descriptor for each of the plurality of candidate sub-resolution locations in the reference medical image set. Each third local image descriptor is generated based on image data of the reference medical image set located relative to each of the candidate sub-resolution locations. Then, for each of the plurality of candidate sub-resolution locations, a local image similarity metric is calculated indicating a local image similarity between the first local image descriptor and the third local image descriptor for the candidate sub-resolution location. Furthermore, a candidate sub-resolution location is selected from the plurality of candidate sub-resolution locations based on the calculated local image similarity metric indicating a local image similarity between each of the first local image descriptor and the third local image descriptor. Next, a location that matches the region of interest in the reference medical image set is selected based on the selected candidate sub-resolution location. The candidate sub-resolution locations are locations distributed throughout the reference medical image set of the sub-resolution sampling pattern in the region of the selected candidate location. In this case, the distance between the candidate sub-resolution locations in the predetermined pattern of sub-resolution locations is less than the distance between the candidate locations in the predetermined sampling pattern.

[0103] In this way, the method of layering two sampling patterns with different spatial resolutions allows for the accurate determination of the location of a region of interest in a fast and efficient manner, even when the location of any feature in a set of reference medical images is unknown.

[0104] According to one example, candidate locations are determined based on a learned policy network that executes a learned policy to iteratively determine a next candidate location based on at least one of the previous candidate locations to analyze the reference medical image set for locations that match regions of interest in the reference medical image set.

[0105] Furthermore, the learned policy network is trained, for example, to output a probability distribution p(a) corresponding to an action a for determining the next candidate position at each iteration, and the next candidate position is determined by sampling the probability distribution p(a).

[0106] As a result, the learned policy network is configured to define one or more of the candidate locations based on mapping the location of the region of interest to the reference medical image dataset.

[0107] A policy network can be considered a trained function that follows a specific learned policy for repeatedly executing multiple actions to accomplish a learned task. The policy of the learned policy network can be defined as the ability to select a next action for selecting a candidate location, for example, to repeatedly find a location that matches a region of interest so that long-term rewards are favored. In other words, the manner in which an individual action is selected starting from a given state of the trained function is called a "policy." The learned policy network thus includes a policy for actions on how to find a location in a set of reference medical images that matches a region of interest. As a result, the iterative repositioning process evolves a current estimate of the matching location according to the policy, ultimately identifying the matching location in the set of reference medical images. Implementing the learned policy network as a policy network can therefore be considered as one aspect of a reinforcement learning trained function.

[0108] One consequence of dividing the problem of finding a match location into multiple iterative steps is that traditional learning schemes, such as supervised learning, fall short. This is because typically only the final state label, i.e., the actual match location, exists, not the labels of the intermediate states along the way. Therefore, the learned policy network must learn to take actions in the environment to maximize some notion of future cumulative reward (which will emerge when the actual match location is reached). In this regard, reinforcement learning is a technique that facilitates learning as an exhaustive cognitive process for a trained function, instead of a predefined methodology. That is, the trained function gains the ability to develop strategies to deal with uncertain environments (here, medical image data), thereby increasing its flexibility.

[0109] By relying on reinforcement learning, it is possible to provide a trained function that can decompose a complex problem into multiple individual steps. At each step, the trained function evaluates its current state based on input data and decides on the next action to proceed to the next state. When applied to a landmark detection problem, for example, the current state is considered to be the current candidate location as the current estimate of the match location, and the state is sampled using the aforementioned local image similarity. An action, for example, involves moving the current candidate location in the reference medical images to proceed to a new (improved) candidate location. This process continues iteratively until, for example, the action falls outside the image space of the reference medical images (indicating that no valid landmarks are included in the image data) or until convergence is reached (i.e., no action remains to further improve the candidate location). Breaking down a problem into multiple subproblems in machine learning has the advantage of improved convergence behavior, meaning the trained function will perform faster and produce better results. Furthermore, the trained function becomes more flexible, for example, in the sense that it can be applied to a wide variety of medical image data. In this way, the use of reinforcement learning trained functions synergistically contributes to improved methods in terms of speed, accuracy, and flexibility.

[0110] According to one example, the learned policy network is trained as a Markov decision process with policy gradients.

[0111] Employing a Markov decision process effectively means that each state and action is considered statistically independent of previous states. This is the basis for the approach to iteratively determining the optimal policy during training. For each state, the action that maximizes the future (cumulative) reward (cumulative, since the individual rewards for each future action are summed until convergence is reached) is considered good. Using policy gradients is even more beneficial when the training data is complex and / or the inputs to the trained function are undefined.

[0112] According to one example, the learned policy networks are each considered to be trained to maximize the cumulative future reward value for a sequence of actions that identify a match location, as described above, preferably by evaluating the gradient of each cumulative future reward with respect to one or more parameters of each trained function.

[0113] The behavior of the learned policy network can be viewed as, for example, intelligent selection of the next action to advance a candidate location in a manner that maximizes cumulative future reward. That is, the learned policy network learns to determine the most favorable sequence of position changes required to accurately find a matching location. Thus, the state of the learned policy network is modeled, for example, as the local image similarity of the current candidate location. The subsequent behavior of the candidate location responds to what is observed within the state.

[0114] Basing the next action on a probability distribution (rather than calculating a specific reward value for each action) has the advantage that the method is more easily applicable to continuous, finely discretized actions and generally large action spaces.

[0115] According to one example, the probability distribution reflects a learned policy of a learned policy network, which policy is learned by evaluating the gradient of each accumulated future reward with respect to one or more parameters of each learned policy network.

[0116] According to one example, the learned policy network is configured to refine the candidate locations until a convergence point of possible matching locations is detected or a trajectory of refined predicted candidate locations reaches a convergence point from the image space of the reference medical images.

[0117] According to one example, the step of selecting at least one of the candidate medical imaging studies as the reference medical imaging study includes identifying a plurality of related medical imaging studies from the candidate medical imaging studies based on a comparison of the body regions; Interface providing a user with multiple related medical imaging examination indicators via Interface receiving a user selection designating at least one of the associated medical imaging studies via the reference medical imaging study selection; and selecting the at least one designated associated medical imaging study as a reference medical imaging study.

[0118] The plurality of related medical imaging studies may, for example, be considered an automatically provided pre-selection from which the user may select. An indication of the plurality of related medical imaging studies may be provided to the user in the form of a list of studies from which the user may select. According to one example, the list may include further details associated with each related medical imaging study, such as the time the study was acquired, the modality and other imaging parameters, clinical parameters of the patient or study, one or more medical records or excerpts thereof, etc. According to one example, the user may Interface Providing an indication of a plurality of related medical imaging studies to a user via the medical imaging system may include, for example, providing highlighting of a plurality of related medical imaging studies in a set of available candidate medical imaging studies.

[0119] By providing the user with potentially related studies and selecting one or more reference studies based on the user's selection of one or more potentially related studies, continuous human-machine interaction is achieved for the directed selection of imaging studies for follow-up reading. According to one example, the user selection is used / recorded as the user's previous action, and based thereon, the next study selection process for the user can be optimized.

[0120] The step of identifying the plurality of related medical imaging examinations also includes determining, for each of the related medical imaging examinations, a degree of match with the target medical imaging examination based on a comparison of the respective body regions, and the step of providing an indication of the plurality of related medical imaging examinations includes providing the degree of match to a user. Interface whereby the goodness of fit is preferably based on anatomical overlap between each medical imaging examination based on a comparison of the subject and candidate body regions.

[0121] The degree of match is, for example, an objective measure of how well two imaging studies match each other. For example, the degree of match is calculated based on the body regions of the target medical imaging study and each candidate medical imaging study. As an example, the degree of match comprises an overlap of body regions determined based on the body regions of the target medical imaging study and each candidate medical imaging study. Alternatively or in addition, the degree of match may take into account, for example, the above-mentioned modality relevance score or the compatibility of imaging parameters of the medical images composed of the target medical imaging study and each candidate medical imaging study.

[0122] By providing a goodness of fit, the user is provided with an objective measure of how well two medical imaging studies match, thus providing the user with further assistance in selecting an appropriate medical imaging study.

[0123] According to one example, the step of acquiring a region of interest in the target medical image group includes: Interface The method includes receiving user input specifying a region of interest via the input.

[0124] According to one example, the user input includes at least one of: a) selection of a (target) slice from multiple slices included in the target medical image set (the target medical image set depicts an image volume, and multiple slices each depict a specific section of the image volume); b) manual boundary drawing of an image region within the target medical image set; and c) designation of a point of interest within the target medical image set (around which a region of interest can later be established).

[0125] By selecting a region of interest, the user can, in other words, establish an object on which the reference medical image set will be selected, and registration will be performed. Thus, the appropriate reference medical image set is specifically selected depending on the portion of the reference medical imaging study that the user is currently focused on. For example, if the region of interest shows a lung lesion, the search for the study and set will be directed to obtain previous images of the patient's lungs, particularly of the same portion of the lungs, acquired with comparable imaging parameters.

[0126] According to one example, in the process of generating display data and displaying a rendering of the reference medical image group on a display device, user input, and in particular manual boundary drawing, is preferably applied to the reference medical image group based on the registration to select a matching image slice (reference slice) from the reference medical image group or to crop the reference medical image group in accordance with the manual boundary drawing.

[0127] This allows the display data to be adapted to the region of interest, thereby providing the user with a visualization adapted to the region of interest, for example improving comparative reading for the user.

[0128] According to one example, the step of selecting at least one of the candidate images as the reference medical image group includes identifying, from the candidate medical image group, a plurality of related medical image groups that are related to the target medical image group based on a degree of comparability; Interface providing a user with an index of a plurality of related medical image groups via the index (which index may comprise a degree of comparability); Interface receiving a user selection via the reference medical image set selection interface (the user selection being directed to a set of related medical images of interest); and selecting the set of related medical images of interest as the set of reference medical images.

[0129] The plurality of related medical image groups may, for example, be considered as an automatically provided pre-selection from which the user may select. An indication of the plurality of related medical image groups may be provided to the user in the form of a list of groups from which the user may select.

[0130] This achieves continuous human-machine interaction regarding the directed selection of image groups for follow-up reading. According to one example, user selections are used / recorded as the user's prior actions as a basis for optimizing the next group selection process for the user.

[0131] According to one example, each of the subject and reference medical images is associated with at least one attribute having an attribute value indicative of the imaging parameters used to capture each of the medical images. Interface Providing an indication of the plurality of related medical images to the user via the medical image processing method includes providing an indication of each attribute value.

[0132] According to one example, the reference medical imaging study includes one or more annotations that match the reference images. The method further includes, for example, obtaining at least a reference annotation from the one or more annotations that matches the target medical images and / or the region of interest, and providing the reference annotation. The reference annotation preferably includes one or more first words.

[0133] Preferably, providing the reference annotations comprises annotating the target medical images and / or the region of interest with at least one reference annotation.

[0134] Obtaining at least one reference annotation includes, for example, determining, for each annotation, a position in the reference medical image portion to which each of the annotations is associated, determining, for each position, preferably based on registration, whether the position has a matching position in the target medical image group or region of interest, and selecting the annotation as the reference annotation if a matching position can be determined for the annotation position.

[0135] Thus, for example, at least one reference annotation specifically relates to the reference slice.

[0136] Annotating the target medical images and / or regions of interest may include, for example, annotating the target medical images and / or regions of interest with at least one reference annotation of the determined match locations.

[0137] According to the above example, for example, previous annotations are automatically provided to facilitate the follow-up image review process by the user. By automatically annotating the target medical images and / or regions of interest, the user can directly compare the previous annotations with the image features of the target medical images.

[0138] According to one example, obtaining at least one reference annotation includes obtaining at least one location in the set of reference medical images for each annotation, and determining, for each location, a matching (or target) location and / or region of interest in the set of target medical images. The matching (or target) location matches the at least one location. Thus, annotating includes, for example, annotating the set of target medical images and / or region of interest with the reference annotation of the at least one target location.

[0139] Determining the matching position is based on, for example, the registration described above.

[0140] Furthermore, determining the matching (or target) location, or determining for each location whether the location has a matching location in the target medical images or region of interest, is based on, for example, an image similarity assessment. According to one example, these steps include obtaining at least one reference image patch in the reference medical images for each annotation, the reference image patch being positioned according to the location of the annotation. These steps also include, for example, determining, for each of a plurality of target image patches in the target medical images and / or region of interest, a local image similarity metric indicating the local image similarity between the reference image patch and each of the target image patches. These steps also include selecting a matching image patch from the plurality of target image patches based on the similarity metric (if possible), and determining the matching (or target) location based on the matching image patch.

[0141] The local image similarity can be calculated with a local image similarity metric essentially as described in connection with the registration process.

[0142] For example, the target patch having the highest local image similarity to the reference patch is selected as the matching image patch. Also, a matching image patch is identified, for example, if the local image similarity exceeds a predetermined threshold. Similarity If no target image patches are identified for which σ exceeds a predetermined threshold, the associated annotations will not match in the target medical image set and therefore will not be applied to the target medical image set, for example.

[0143] According to one example, the method further includes obtaining a medical report associated with the reference medical imaging study, obtaining one or more sections of text from the medical report, each section including one or more second words for each of the one or more sections, comparing the one or more second words with one or more first words of the reference annotations to identify matches for each of the reference annotations, and associating at least one of the reference annotations with at least one of the first sections based on the identified matches. Optionally, the associated first sections may be displayed to a user. Interface may be provided to the user via

[0144] For example, based on the automatic annotation of the target medical images, matching sections of previous medical reports are identified and are suitable for reading the target medical images. In this way, the user is freed from manually searching for the previous reports and relevant passages therein, and can immediately determine which findings reported in the previous medical reports have been changed.

[0145] The comparison of the one or more words is performed, for example, by applying a machine-learned classifier to the first and second words. The machine-learned classifier is trained to map similar words to similar regions in a feature space. For example, the words are encoded into vector representations using word embeddings. For example, word embeddings map words to a vector space, where a vector for each word can be generated by applying a word embedding model to a corpus of training text. An example of a known word embedding model is "Word2vec," which uses a neural network to learn word embeddings from a corpus of training text.

[0146] According to an example, the method further includes identifying a user who provides the display data and obtaining one or more previous actions of the user. The previous (prior) action refers to at least one of an examination selection action for selecting a reference medical imaging examination from a plurality of candidate medical imaging examinations and a group selection action for selecting a group of reference medical images from a group of a plurality of candidate medical images of the medical imaging examination. Thus, the step of selecting the reference medical imaging examination is based, for example, on the at least one examination selection action. Also, the step of selecting the group of reference medical images is based, for example, on the at least one group selection action.

[0147] By taking into account the user's prior actions, the (semi-)automatic selection process described herein can be adapted to the user's preferences, allowing for the selection of relevant groups and tests, and allowing the user to choose whether to manually explore the available tests / groups. Furthermore, if the test and / or group selection is performed by a machine-learned function, the prior actions can be used, for example, to further train the underlying machine-learned function.

[0148] Basing the selection process on at least one prior action may include, for example, evaluating (e.g., statistically) one or more prior actions to determine a probability that a user will select a particular candidate medical imaging study or group. Thus, the probability provides an additional basis, for example, in selecting a reference medical imaging study or a group of reference medical images.

[0149] According to one example, the method further includes quantifying changes between image data of the region of interest in the target medical image group and matching image data of the reference medical image group, generating change display data to visualize and display the changes on a display device.

[0150] In some examples, the changes are determined by transforming the target medical image group into the coordinate system of the reference medical image group based on registration, or vice versa, to generate a transformed medical image group, and determining the changes based on this transformed medical image group (for example, by comparing or subtracting the target medical image group or the reference medical image group from the transformed medical image group).

[0151] The change may relate, for example, to a patient's pathological condition. The change may relate, for example, to tissue changes between the subject medical image set and the reference medical image set. The change may relate, for example, to the growth, shrinkage, appearance, or disappearance of an abnormality from the subject medical image set to the reference medical image set (i.e., from a first time point to a second time point). Examples include the growth or shrinkage of nodules, the appearance of new nodules and / or lesions, etc.

[0152] Visualizing changes can include, for example, displaying renderings of the target medical images (regions of interest) and / or reference medical images with changes highlighted. In this example, "highlighting" can mean, for example, visually increasing brightness, color, and / or intensity to indicate the changes. Additionally or alternatively, symbols can be used to highlight changes. Also, for example, labels comprising semantic representations can be used to highlight changes. Highlighting can further mean, for example, using a heat map where the amount of change is color-coded. For example, shrinking nodules can be colored differently from growing nodules.

[0153] As an example, generating the change display data comprises at least one of: including visualization of the changes in a rendering of the reference medical image group, preferably overlaid on a rendering of the reference medical image group; including visualization of the changes in a rendering of the region of interest, preferably overlaid on a rendering of the region of interest.

[0154] By providing the user with a rendering that highlights the changes, the user can immediately infer what changes have occurred and where they have occurred, which helps guide image interpretation, thereby increasing the usefulness of the method and providing the user with better assistance in guiding medical diagnoses.

[0155] According to one example, the target medical images are acquired using a first medical imaging modality and the reference medical images are acquired using a second medical imaging modality, where the first medical imaging modality is based on an imaging technique different from that on which the second modality is based, such that, for example, the first modality is based on an X-ray imaging technique (such as CT or X-ray projection) and the second medical images (CMIS) are preferably based on a magnetic resonance imaging technique.

[0156] In other words, the method allows for the synchronization of medical image datasets acquired with functionally different imaging modalities.

[0157] According to one example, the candidate medical imaging studies and / or groups are stored in a remote storage device, and the method includes retrieving from the remote storage device medical image data of the candidate medical imaging studies and / or groups selected as reference medical imaging studies and / or groups, without retrieving medical image data of the candidate medical imaging studies and / or groups not selected as reference medical imaging studies and / or groups.

[0158] In other words, this means that, for example, no transfer of image data occurs until a candidate medical imaging study or group is selected as a reference medical imaging study or group.

[0159] According to one example, selecting a reference medical imaging study from a plurality of candidate medical imaging studies is based on non-image data associated with the candidate medical imaging studies. For example, determining the body region of the candidate medical imaging study is based on, for example, the non-image data associated with the candidate medical imaging study. As a result, selecting the reference medical imaging study includes retrieving the non-image data of the candidate medical imaging study from a remote storage device without retrieving the medical image data of the candidate medical imaging study.

[0160] Similarly, according to one example, selecting a reference group of medical images from a plurality of candidate groups of medical images is based on non-image data associated with the candidate groups of medical images. For example, determining the degree of comparability of the candidate groups of medical images is based on the non-image data associated with the candidate groups of medical images. As a result, selecting a reference group of medical images includes, for example, retrieving the non-image data of the candidate groups of medical images from a remote storage device without retrieving the medical image data of the candidate groups of medical images.

[0161] Determining the body region / degree of comparability based on (computationally relatively small) non-image data eliminates the need to transmit (computationally relatively large) medical image data over the network to determine the body region / degree of comparability represented by that data, thus enabling efficient use of network resources.

[0162] According to one example, the method further includes providing an anomaly detection function configured to detect one or more anatomical anomalies in the medical image data, detecting the anatomical anomalies in the set of target medical images by applying the anomaly detection function to the set of target medical images, and obtaining an anomaly location of the anatomical anomaly, wherein obtaining the region of interest is based on the anomaly location.

[0163] As one example, an abnormality (or alternatively, an "abnormal structure") in a patient is an anatomical structure that distinguishes that patient from other patients. As one example, an abnormality can be associated with a particular medical condition of the patient. An abnormality can be located within various organs of the patient (e.g., within the patient's lungs or within the patient's liver), or an abnormality can be located between organs of the patient. As one example, an abnormality can be a foreign body.

[0164] By way of example, the abnormality is a neoplasm (also called a "tumor"), particularly a benign neoplasm, a non-invasive neoplasm, a malignant neoplasm, and / or an indeterminate / unidentified neoplasm. By way of example, the abnormality is a nodule, particularly a pulmonary nodule. By way of example, the abnormality is a lesion, particularly a pulmonary lesion.

[0165] The anomaly detection function may be implemented, for example, as a computer-aided detection (CADe) and computer-aided diagnosis (CADx) system. Such systems are technologies that help radiologists interpret medical images. A common application of CAD systems is to automatically identify suspicious regions in medical images. The suspicious regions may contain image patterns that indicate abnormalities, such as cancerous growths, masses, abscesses, lacerations, calcifications, lesions, and / or other irregularities in biological tissue, which, if left undetected, may cause serious medical problems. In principle, a plethora of functions and methods are known for computer-aided detection of anomalies, all of which may be implemented in the anomaly detection function. For example, US2009 / 0092300A1, US2009 / 0067693A1, and US2016 / 0321427A1 are cited. For example, the anomaly detection function may be based on one of a support vector machine, a naive Bayesian algorithm, a neural network, a decision tree, and / or a ResNet.

[0166] By obtaining a region of interest via an automated anomaly detection workflow, a user's attention can be automatically focused on the most relevant portions of, for example, a set of target medical images, and based on this, appropriate reference medical images can be specifically retrieved, which further assists the user in, for example, providing a medical diagnosis.

[0167] According to one example, a system for assisting in evaluation of a set of target medical images of a patient acquired at a first time point is provided. The system includes an interface unit and a computing unit. The interface unit is configured to provide a rendering of the target medical images to a user and receive user input from the user to specify a region of interest in the target medical images based on the rendering of the target medical images. The computing unit is configured to obtain the region of interest based on the user input, determine a target body region represented by the target image data set, select a reference medical imaging study from a plurality of candidate medical imaging studies based on a comparison of the plurality of target body regions to a plurality of candidate body regions, where each of the plurality of candidate body regions corresponds to one of a plurality of candidate medical imaging studies, each candidate medical imaging study including a plurality of candidate medical images of the patient at a second time point, select a reference medical image from the plurality of candidate medical images of the reference medical imaging study based on a degree of comparability with the target medical image set, perform registration between the target medical images and the reference medical images, and generate display data for displaying the rendering of the reference medical images on the interface unit based on the registration.

[0168] The computing unit may include, for example, an exam selection module or unit configured to select a reference medical imaging exam as described herein. The computing unit may include, for example, a group selection module or unit configured to select a group of reference medical images as described herein. The computing unit may include, for example, an image registration unit configured to generate at least one image registration as described herein. The computing unit may include, for example, a visualization module or unit configured to generate display data based on at least the group of reference medical images as described herein.

[0169] Optionally, the computing unit includes, for example, an annotation module or unit configured to annotate the subject medical images based on annotations included in the reference medical image set RMIS (if any) and / or to match annotations associated with the subject medical images to existing sections of the patient's medical report as described herein. Optionally, the computing unit also includes, for example, a comparison unit configured to compare the medical image data and determine variations therefrom as described herein.

[0170] The computing unit may be implemented, for example, as a data processing system or as part of a data processing system. Such data processing systems include, for example, cloud computing systems, computer networks, computers, tablet computers, smartphones, and / or the like. The computing unit may, for example, consist of hardware and / or software. The hardware may, for example, include one or more processors, one or more memories, or a combination thereof. The one or more memories may, for example, store instructions for executing the steps of the method according to the present invention. The hardware may be configurable and / or operable by software. Generally, all units, subunits, or modules are in data communication with each other, at least temporarily, for example, via a network connection or respective interfaces. That is, the individual units may be located remotely from each other.

[0171] The interface unit includes an interface for data exchange with a local server or a central web server via an internet connection, for example to receive reference image data or tracking image data. The interface unit may further provide a user interface, for example by displaying results of processing by the computing unit to a user (e.g., a graphical user interface). Interfaceat), or configured to collaborate with one or more users of the system by allowing the users to adjust parameters for image processing or visualization and / or for medical imaging exams and groups.

[0172] According to one example, a system is configured to perform the method of the present invention in its various aspects to determine abnormal changes in an anatomical region of a patient, and the advantages described in connection with the method aspects are also achieved by correspondingly configured system components.

[0173] According to one example, the interface unit is further configured to visualize an assessment tool, where the assessment tool is configured to be placed in a rendering of the target medical images, and to receive input from a user regarding the placement of the assessment tool in the rendering of the target medical images to define a region of interest, and the computing unit is further configured to, for example, place the assessment tool relative to the region of interest in the rendering of the target medical images.

[0174] According to one example, the interface unit is further configured to display the assessment tool as a circular or rectangular assessment tool of user configurable size placed as an overlay on a rendering of the target medical images.

[0175] According to one example, the interface unit is further configured to provide an evaluation function configured to quantify changes between the image data of the region of interest and corresponding image data of the set of reference medical images and to receive user input from a user related to a selection of the evaluation function, and the computing unit is configured to execute the evaluation function based on the user input, generate change representation data, and cause the interface to display a visualization of the changes.

[0176] That is, a user-friendly image evaluation / analysis tool is provided that can be arbitrarily positioned within, on, or overlaid on a displayed image data set. In this manner, the evaluation / analysis tool allows the user to easily select a location or region of interest within the displayed image volume by directly moving the tool to that location. When positioned at a region of interest, the evaluation tool is configured to provide a number of different evaluation / analysis functions for the user to select, with each evaluation function configured to provide an evaluation / analysis result for the image data within the selected location. Furthermore, the evaluation tool can be configured to provide a palette of medical image data, a palette of evaluation functions, and an evaluation result. In parallel A graphical user interface configured to visualize the data is provided.

[0177] According to one example, there is provided an image analysis system comprising the above system and a medical information system configured to acquire, store and / or transfer at least subject and reference medical imaging studies and / or groups, whereby the interface unit is configured to receive the subject and reference medical imaging studies and / or groups from the medical information system.

[0178] According to some examples, the medical information system includes one or more archiving stations that store subject, candidate, and reference medical imaging studies and / or groups. The archiving stations may be implemented as remote storage, e.g., in the form of cloud storage, or as local or distributed storage, e.g., as a Picture Archiving and Communication System (PACS). The medical information system also includes one or more medical imaging modalities, such as a computed tomography system, a magnetic resonance system, an angiography (or C-arm X-ray) system, a positron emission tomography system, a mammography system, etc.

[0179] According to one aspect, the invention also relates to a computer program product (having recorded thereon) including program elements which, when loaded into the memory of a computing unit of a system, cause said computing unit to carry out the steps according to one or more aspects of the method described above.

[0180] According to one aspect, the invention also relates to a computer-readable medium having recorded thereon program elements that can be read and executed by a computing unit of a system, the computing unit executing the program elements performing the steps according to one or more aspects of the method described above.

[0181] The implementation of the present invention by a computer program product and / or computer readable medium has the advantage that existing delivery systems can be easily adapted with software updates to operate as proposed by the present invention.

[0182] A computer program product is a computer program and may, for example, include other elements linked to such a program. Such other elements may be hardware, for example a memory device having the computer program stored thereon, a hardware key for using the computer program, etc., and / or software, for example documentation or a software key for using the computer program. The computer program product may further include, for example, development materials, a runtime system and / or databases or libraries. The computer program product may, for example, be distributed among several computer instances.

[0183] The features, characteristics, and advantages of the invention described above, as well as the operations for achieving them, will become clearer and more deeply understood from the following detailed description of the embodiments, which is given with reference to the drawings. The following description does not limit the invention to the embodiments contained therein. The same reference numerals are used to designate the same components, parts, or steps in the multiple drawings, as far as possible. Generally, the drawings are not drawn to scale. [Brief explanation of the drawings]

[0184] [Figure 1] 1 is a diagram illustrating an embodiment of a system for assisting in the evaluation of a set of medical images of a patient. [Figure 2] FIG. 2 is a diagram schematically showing an image file according to the embodiment. [Figure 3] FIG. 2 is a diagram schematically showing an image file according to the embodiment. [Figure 4] FIG. 2 is a diagram illustrating a method for generating display data for a group of medical images according to an embodiment. [Figure 5] 1 is a diagram illustrating a method for determining a body region of a medical imaging examination according to an embodiment; [Figure 6] FIG. 2 is a diagram illustrating a data flow between components according to an embodiment. [Figure 7] FIG. 2 is a schematic diagram illustrating a graphical user interface (GUI) according to an embodiment. [Figure 8] 1A and 1B are diagrams illustrating a method for selecting a reference medical imaging exam according to an embodiment; [Figure 9] FIG. 2 is a diagram illustrating a schematic data flow between components in a method for selecting a reference medical imaging study according to an embodiment. [Figure 10] FIG. 2 is a diagram illustrating a method for selecting a group of reference medical images according to an embodiment. [Figure 11] FIG. 2 is a diagram illustrating a data flow between components in a method for selecting a group of reference medical images according to an embodiment. [Figure 12] 1 is a flowchart illustrating a method for registering two medical image sets by identifying matching slices according to an embodiment. [Figure 13]1A and 1B are diagrams illustrating a data stream associated with a method for registering two medical image sets by identifying matching slices according to an embodiment; [Figure 14] 10 is a flowchart further illustrating steps of a method based on identifying matching slices of a collection of medical images according to an embodiment. [Figure 15] 10 is a flowchart further illustrating steps of a method based on identifying matching slices of a collection of medical images according to an embodiment. [Figure 16] 1 is a flowchart illustrating a method for selecting one or more sets of medical images according to an embodiment. [Figure 17] 1 is a flowchart illustrating a method based on identifying matching slices in a collection of medical images according to an embodiment. [Figure 18] 1 is a flowchart illustrating a method for training a trained function to identify matching slices in a collection of medical images according to an embodiment. [Figure 19] 1 is a flowchart illustrating a method for registering two medical image sets by identifying matching image patterns according to an embodiment. [Figure 20] FIG. 2 is a diagram schematically showing a group of target medical images according to the embodiment. [Figure 21] FIG. 2 is a diagram schematically showing a group of reference medical images according to the embodiment. [Figure 22] FIG. 2 is a diagram schematically showing a group of reference medical images according to the embodiment. [Figure 23] FIG. 2 is a diagram schematically showing a group of reference medical images according to the embodiment. [Figure 24] FIG. 2 is a diagram schematically showing a group of reference medical images according to the embodiment. [Figure 25] FIG. 4 is a diagram schematically showing output data according to the embodiment. [Figure 26] FIG. 4 is a diagram schematically showing output data according to the embodiment. [Figure 27] 1 is a flowchart illustrating a method for annotating a collection of target medical images according to an embodiment. [Figure 28] 10 is a flowchart illustrating a method for determining annotations for a set of target medical images based on a set of reference medical images according to an embodiment. [Figure 29]2A and 2B are schematic diagrams illustrating image components in the form of image slices of a target medical image set according to an embodiment; [Figure 30] 1 is a schematic diagram illustrating image components in the form of image slices of a collection of reference medical images and associated annotations according to an embodiment; [Figure 31] 1 is a diagram illustrating a schematic diagram of a data stream associated with a method for annotating a collection of target medical images according to an embodiment; [Figure 32] 5A and 5B are schematic diagrams illustrating rendering of display data according to an embodiment. [Figure 33] 5A and 5B are schematic diagrams illustrating rendering of display data according to an embodiment. [Figure 34] FIG. 2 is a schematic diagram illustrating elements of a graphical user interface (GUI) according to an embodiment. [Figure 35] FIG. 1 is a diagram showing a system for supporting evaluation of a group of target medical images according to an embodiment. [Figure 36] FIG. 2 is a diagram illustrating an evaluation tool according to an embodiment in a first operating state. [Figure 37] FIG. 10 is a diagram illustrating an evaluation tool according to an embodiment in a second operating state. [Figure 38] FIG. 10 is a diagram illustrating an evaluation tool according to an embodiment in a third operating state. [Figure 39] 1 is a flowchart showing a method for supporting evaluation of a group of target medical images according to an embodiment. [Figure 40] FIG. 10 is a diagram showing an evaluation tool for determining and visualizing reference medical image data according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0185] Figure 1 shows the target medical image set TMIS. Reference medical imaging examination RS Identification In order to do so, the target medical image group TMIS Reference medical images RMIS Identification to annotate the target medical image set TMIS, and / or 、 The base as To generate display data 、 System 1 is shown. In this example System 1 is shown in Figure 2- Fig. 34 See As will be described later , in one or more embodiments Related The users of the system 1 may generally relate to medical professionals such as physicians, clinicians, technicians, radiologists, pathologists, etc., according to some examples. do .

[0186] System 1 is User Interface 10 (as part of the interface unit) and a processing system 20 (as part of the computing unit 30). Furthermore, the system 1 comprises: for example, Medical Information System 40 Contains or connected to The medical information system 40 is For example, roughly , a reference medical imaging study RS, a target medical imaging study TS, a reference medical image set RMIS, and / or a target medical image set TMIS. Configured to retrieve and / or store and / or transmit . in particular , the medical information system 40 is For example, reference medical imaging studies R.S., Target medical imaging examination Configured to handle image data file IF in which TS, reference medical image set RMIS, and / or target medical image set TMIS are stored will be .for example, Medical Information System 40 is Matches One or more archive / review stations (not shown) may be included for storing image data files IF for medical imaging studies and groups. The archive / review stations may be stored in one or more databases. implement It is possible. More specifically: ,The archiving / review station can be realized in the form of one or more cloud storage modules. or ,archive / review stations, e.g. PACS (Picture Archiving and Communication System: medical image management system) As, Local or distributed storage Furthermore, the archive / review station may be implemented as an image data file. IF Additional information related to Clinical information You can save The clinical information For example, relevant medical findings, patient case Prepared through preliminary research Key image, one or more previous (structured) Medical Report,Personal information relating to the patient under consideration,Patient records (Medical Record) etc. include According to some embodiments, Medical Information The system 40 may be a computed tomography system, a magnetic resonance system, an angiography (or C-arm X-ray) system, positron emission tomography systems, mammography systems, systems for acquiring digital pathology images, etc. One or more medical imaging Modality (not shown) include .

[0187] User Interface 10 is Display unit Enter 11 unit 12 and have . User Interface 10 is Mobile devices such as smartphones and tablet computers By May be implemented Furthermore, users Interface 10 as a workstation in the form of a desktop PC or laptop It can also be implemented The input unit 12 is connected to a display unit 11, for example in the form of a touch screen. Even if incorporated Alternatively or additionally, the input unit 12 may include: keyboard ,mouse 、 or a digital pen, and combinations of these may include: Display unit 11 is the target medical image group TMIS and / or reference medical image group RMIS expression can be configured to display More specifically , Display unit 11 can be configured to display individual slices of the target medical image set TMIS and / or the reference medical image set RMIS.

[0188] User Interface 10 further states, Display unit 11 and input unit 12 To handle At least one software component an interface computing unit 13 configured to execute A set of target medical images TMIS from the target medical imaging study TS so that the user can select the target medical imaging study TS to review. and / or to receive a user selection of ,In the target medical image group TMIS Region of interest (ROI) Definition In order to、 Graphical User Interface (For example, Figure 36- Fig. 38 ) to provide Also, the interface The computing unit 13 For example, communicating with the medical information system 40 or the processing system 20 to receive one or more reference medical image sets RMIS to compare with the subject medical image set TMIS, or display data associated with the reference medical image sets RMIS. The user is configured as Interface through 10 Software Components You can also launch e.g. Internet Application Store By downloading from Software Components According to one example, the software component may be a client-server computer program in the form of a web application that runs in a web browser. Interface The computing unit 13 For example, a general-purpose processor , central processing unit, control processor, graphics processing unit, digital signal Processor, 3D rendering Processor , Image Processor ,Application Specific Integrated Circuits, Field Programmable Gate Arrays, Digital Circuits, Analog Circuits, A combination of these or process image data Other known devices .

[0189] The processing system 20 includes: for example, Processor is .this Processor teeth, For example, a general-purpose processor , central processing unit , control processor, graphics processing unit ,digital signal Processor, 3D rendering Processor , Image Processor , application specific integrated circuits, field programmable gate arrays, digital circuits, analog circuits, combination or process image data Other known devices The processor is a single device may be in series, parallel, or separately Operation Multiple deviceThe processor may be the main processor of a computer, such as a laptop or desktop computer, or it may be a processor for handling several tasks in a larger system, such as a medical information system or server. The steps described herein may be performed by instruction, design, hardware, and / or software. of To run Alternatively, the processing system 20 may be configured to: for example, So-called "clusters" or "clouds" Consists of a group of real or virtual computers . Such a server system teeth, for example, Central server , For example, a cloud server , or For example, a local server at a hospital or radiology site. Furthermore, the processing system 20 For example, medical images RAM and other memory for temporarily loading the data sets MIDS-1 and MIDS-2 include .or, The memory is the user Interface It may be included in 10.

[0190] ( for example Forms of target medical image group TMIS, reference medical image group RMIS, and candidate medical image group CMIS of )Here explain A medical image set is, for example, a three-dimensional image data set acquired using a computed tomography system or a magnetic resonance imaging system or other system. be Image information is for example, A three-dimensional array of m×n×p voxels is encoded to . Medical image group RMIS, TMIS, CMIS medical images teeth, For example, the image slices S-1, S-2 (also referred to simply as "slices") may be stacked in a stacking direction across the image volumes covered by the respective medical image sets RMIS, TMIS, and CMIS. .

[0191] Furthermore, the medical image groups RMIS, TMIS, and CMIS are: for example, m×n pixel Array of is encoded in Image information accompanied by One or more individual 2D medical images include In some instances, these two-dimensional medical images are extracted from a set of three-dimensional medical images. It was Another example is two-dimensional medical imaging, which is similar to X-rays. Imaging Modality etc. appropriate Imaging Using modalities Separately acquisition It was .

[0192] A set of voxels or pixels is For example, the following data is represented as image data for each data set: In general, ultrasound, X-ray, angiography, Fluoroscopy , positron emission tomography, single-photon emission computed tomography etc., images To get the data Various imaging Any modality and scanner can be used. Generally speaking The medical image groups RMIS, TMIS, and CMIS provide images of various anatomical structures and organs. Consists of Illustrates patient body parts. If we consider the chest area as a body part, The medical image groups RMIS, TMIS, and CMIS are used to visualize, for example, lung lobes, thorax, heart, and lymph nodes. Draw For example, a suitable image Viewer Once processed by the software, Imaging The data ID is one or more medical images that the data represents Rendering of or Generate an expression .

[0193] The medical image groups RMIS, TMIS, and CMIS are: For example, one or more Save in image file IF format will be An example of an image file interface is shown in Figure 2. The image file interface is for example, Actual medical use Imaging Data ID While preserving, medical use Imaging Data ID content of point out one or more attributes A each having an attribute value AV containing a text string; Save .

[0194] One or more attribute values ​​AV are Imaging Data ID It is separate and distinct from ,Instead, medical Imaging Data ID content of point out Such attribute values ​​AV may, in some instances, be used as metadata for image files IF. be called In some examples, the part of the image file IF that stores the attribute A and the attribute value AV is the header of the image file IF. called, attribute A and attribute value AV are the header data of the image file IF. be called .

[0195] A specific example of an image data file interface is a DICOM (Digital Imaging and Communications in Medicine) file. Example of a DICOM file is shown in Figure 3 and is described in detail below. . In summary , the DICOM file contains the specified data element IF-312 pixel data For medical use Imaging Save the data ID and, As one or more other data elements IF-310, one or more attributes A each having an attribute value AV containing a text string indicating the contents of the medical imaging data ID. Save. Like this DICOM attribute IF-310 An example is the "Study Description" Attribute value IF-314 is a text string describing the study of which the medical imaging data is a part (e.g., "NERUO^HEAD" where the medical imaging data is the patient's head region), thereby medical use Imaging Data 316 Point out content . Like this Other examples of DICOM attributes However, "Series description" and "Body Part Examined" are also included. is.

[0196] In one example , For a given patient at a given time , the radiologist For example, medical use Imaging Conduct an inspection do . Other tests may be performed at other times. Certain tests are performed on specific body parts of the patient. It is the opposite of specific Imaging Use modality (e.g., MR) do In some cases, multiple tests may be performed on a patient. Separate On body parts carried out and / or Different imaging Modality Use (i.e., by multiple devices Capture will be) Sometimes The test results were: for example, Saved in one or more medical image data file interfaces will be . predetermined Inspection CS, RS, TS are For example, one or more medical images The group includes TMIS, CMIS, RMIS. nothing .for example, In the prescribed groups TMIS, CMIS, and RMIS of the tests TS, CS, and RS , medical imaging teeth , Same imaging parameters (e.g., patient direction , Echo Time and other MR Unique imagingparameter) Being captured Inspection TS, CS, RS in TMIS, CMIS, RMIS for each group are Imaging If the parameters are different too In some instances, the image file IF may be a medical image representing an image within a particular group and within a particular study. Imaging Save the data ID. One Medical images that are part of the examination TS, CS, RS Imaging The image file IF that stores the data ID is, for example, The image file interface that stores medical imaging data IDs representing medical images that have the same unique study identifier and are part of one group, TMIS, CMIS, or RMIS, is ,for example, have the same unique group identifier . In either case , predetermined The test is for example, At least one such image file Has at least one group .

[0197] Here explain The method and system review Regarding the medical imaging examination TS for patients who are Medical Information System 40 In The objective of this study is to automatically select an appropriate reference medical imaging study RS from multiple available candidate medical imaging studies CS of a patient. The methods and systems described herein also include: The purpose is to automatically select an appropriate reference medical image set RMIS from multiple candidate medical image sets CMIS included in a selected reference medical image examination RS. Let's say Furthermore, here explain The method and system aim to automatically register a set of subject medical images TMIS to a set of reference medical images RMIS and generate display data thereon. To this end, the processing system 20 can include a number of sub-units 21-27 configured to process the involved studies TS, CS, TS and sets TMIS, CMIS, RMIS accordingly.

[0198] Subunit 21 is the data search In modules or units can be, medical imaging inspection TS, CS, RS and / or Match Medical image groups TMIS, CMIS, RMIS and / or MatchImage file IF Regarding medical information System 40 of Access and / or search For example, subunit 21 is currently review Regarding the target medical image group TMIS of the target medical image examination TS death , Prior testing of potentially involved patients (i.e., candidate medical imaging examination CS) Medical Information The subunit 21 may be configured to query the system 40. Specifically, the subunit 21 may Formulating a search query death, these to the medical information system 40. In particular, the subunit 21 may be configured to extract data identifiers from the available information and use these to query the medical information system 40.

[0199] Subunit 22 serves as a test selection module or unit. It can be captured, Multiple candidate Medical Imaging CS from , Reference Medical Imaging RS and Especially for regions of interest (ROIs) prominent The subunit 22 is adapted to select at least one relevant target medical imaging study TS. For this purpose, the subunit 22 comprises: To determine the body regions to be compared for each medical imaging study TS, CS, each of which represents a medical imaging study TS, CS, It can be configured. Also, Subunit 22 is It is thus derived The body regions may be configured to be compared to one another to determine a reference medical imaging study RS. By doing so , Candidate Medical Imaging Examination CS Each of and the target medical imaging examination TS Goodness of fit To decide may include Specifically, subunit 22 is Explained process It can be configured to execute S30 and S40. Also , subunit 22 is For example, as explained in conjunction with Figs. 5-9 method Process and / or 、 Explained in this context do Image processing functions Carry out like 、 composition will be .

[0200] Subunit 23 serves as a group selection module or unit. It can be captured, The subunit 23 is configured to select at least one reference medical image group RMIS that can be easily compared with the target medical image group TMIS from a plurality of candidate medical image groups CMIS included in the reference medical imaging study RS. Specifically, the subunit 23 can be configured to determine a degree of comparability between each candidate medical image group CMIS and the target medical image group TMIS. The candidate medical image group CMIS having the highest degree of comparability can then be selected as the reference medical image group RMIS. Specifically, the subunit 23 for example, Here Explained process Configure S50 to run will be . Also , subunit 23 is for example, Figure 10 and Explained together with Figure 11 Method process and / or 、 Explained in this context do Image processing functions Carry out like 、 composition will be .

[0201] Subunit 24 is a registration module or unit. can be captured The subunit 24 is configured to perform registration of the target medical image set TMIS with the reference medical image set RMIS. So ,Here Explaining Process S It is configured to run 60. specifically , subunit 24 is fundamentally One set of images image The data is stored in the coordinate system of the other image group. fart The calculation results provided by the subunit 24 can be for example, a 2D or 3D transformation matrix or Deformation field Form is Subunit 24 performs rigid image registration, affine image registration, non-rigid image registration, and Any combination of theseTo improve the registration results, subunit 24 may be configured to apply one or more image registration techniques, including: Selectively, Soft tissue deformity Regarding One or more exercise models The calculation results Constructed to fit mathematically You can also . Alternatively Alternatively, in addition to the above registration techniques, the subunit 24 may be configured to correlate or synchronize slices of the target medical image set TMIS and the reference medical image set RMIS with each other based on image similarity between the individual slices, thereby providing registration between the reference medical image set RMIS and the target medical image set TMIS. In this regard, the subunit 24 may be configured to: for example, Figure 12- This will be explained together with Figure 17. method Process and / or 、 Explained in this context do Image processing functions Carry out like 、 composition will be Furthermore, the subunit 24 directly detects similar patterns in the reference medical image set RMIS and the target medical image set TMIS. Identification death, The positions of these patterns in the reference medical image set RMIS and the target medical image set TMIS are correlated. In this regard, subunit 24 may be configured to provide registration by: for example, Figure 19- Explained together with Figure 26 method Process and / or 、 Explained in this context do Image processing functions Carry out like 、 composition will be .

[0202] The subunit 25 is an annotation module or unit as It can be captured, Reference Medical Image Set RMIS Notes contained in (if any) Based on the target medical image group TMIS To annotate and / or , to match annotations associated with the target medical image set TMIS with existing sections F1-F3 of the patient's medical report RM-504; The subunit 25 can be configured as follows: for example, Figure 27- Explained together with Figure 34 method Process and / or、 Explained in this context do Image processing functions Carry out Configured as follows will be .

[0203] Subunit 26 is for example, Configured as a comparison module or unit will be The subunit 26 may, for example, store a reference medical image set RMIS. and Target medical image group TMIS Images included in The subunit 26 may be configured to compare the data IDs with each other to determine changes between the reference medical image set RMIS and the target medical image set TMIS. in The subunit 26 may be configured to correlate different representations of the anomaly A. the For this purpose, the subunit 26 may be configured to quantify the change in the abnormality based on the correlation. leading up to Abnormal Size Change and / or Volume change and / or Strength change and / or Texture Change and / or other parameter changes.

[0204] The subunit 27 includes a visualization module configured to generate display data based on at least a reference medical image set RMIS. can be captured Specifically, the subunit 27 calculates the region of interest (ROI) based on the registration. Match Reference medical image collection RMIS Render a representation of It can be configured as follows. Also , the subunit 27 performs rendering of the region of interest ROI and / or the reference medical image set RMIS, Emphasize the change and add a note 、 Such a representation may be configured to provide: Changes and / or annotations are visually encoded Auxiliary image format It can be said that .this is, For example, change and / or annotations are expanded in their representation , which means Specifically, subunit 27 is The changes and / or annotations from the matching rendered region of interest (ROI) and / or the reference medical image set (RMIS) to be overlaid on the matching region of interest (ROI) Semi-transparent overlay image Render Algorithm , run or execute It can be configured as follows.

[0205] individual The designation of subunits 21-27 should be taken as illustrative and not limiting. That is, , the sub-units 21-27 may be integrated to form one single unit (e.g., in the form of a "computing unit 30"), or 、 The processor of the processing system 20 Activated correspondence method Process by a computer code segment configured to execute implementation The same thing can be done. but , and Interface Computing Unit 13 Applies Each of the subunits 21-27 and the interface computing unit 13 is Process other subunits of System 1 with which data exchange is required to perform and / or other component For example, subunits 21 and subunit 27 is To search for medical IF, Connected to medical information system 40 via interface 29 can be Similarly, interface 29 is For example, to transmit the results of a calculation to the user and collect user input, Interfaces subunits 21-27 Computing Connected to Unit 13 do .

[0206] The processing system 20 and the interface computing unit 13 together may comprise a computing unit 30 . Naturally The layout of the computing unit 30, i.e. the physical distribution of the interface computing unit 13 and the subunits 21-27, is in principle arbitrary. For example, the subunit 27 (or its individual elements or specific algorithm sequences) may also be configured by the user. InterfaceIt can be localized in 10. The same is true for the other subunits 21-26. Furthermore , the processing system 20 Interface It may be integrated into 10. As mentioned above , the processing system 20 may alternatively be, for example, a cloud server, or, for example, a hospital or Radiology site to placed Server systems, such as local servers as To be carried out Such example According to the user Interface 10 can be designated as the user-facing "front end" or "client," while 、 The processing system 20 acts as a "backend" or server. can be captured User Interface The communication between 10 and the processing system 20 is performed using, for example, https protocol Using to be carried out The system's computing power is server and Client (i.e., User Interface 10) Between Distributed by It can be done. Thin Client " In the system, calculation ability Most of it resides on the server. "Thick Client" The system requires more computing power, and As much as possible The data , to the client exist.

[0207] System 1 Individual component may be at least temporarily connected to one another for data transfer and / or exchange. User Interface 10, for example, patient data TPD, data descriptors, or to exchange the results of calculations with the processing system 20 via the interface 29. For example, the processing system 20 may Start with , this The request is Interface 10. Furthermore, the processing system 20 Target patients medical information system to search for cases 40 can communicate with. Alternatively Or in addition, the user Interface 10 can communicate directly with a medical information system 40 . Medical Information System 40 is similar. for example, Request-based Start with , this The request may be sent to the processing system 20 and / or the user Interface By 10 sent The interface 29 for data exchange may be a hardware or software interface, for example a PCI bus, USB, or Firewire. implementation Data transfer can be performed by for example, Using a network connection To be carried out The network is Local Area Network (LAN) , e.g., an intranet 、 or , Wide Area Network (WAN) as implementation The network connection is preferably , for example, wireless methods such as wireless LAN (WLAN or Wi-Fi) In addition, the network may be a collection of multiple network instances. combination The users 11 and 12 may include Interact for component data exchange with conduct The interface 29 can be considered to constitute an interface unit of the system 1 .

[0208] Figure 4 shows the target medical image set TMIS. against a method for obtaining a reference medical image set RMIS; and / or , based on that A method for generating display data is presented. The method includes several Process (step) It consists of: process The order of process Numbering of It's not that I'm responding , the present invention Various embodiments Between Things can change Furthermore, individual process or a series of The process is as follows: Repeat too can.

[0209] In the first step S10 , a target medical image set TMIS is received. The target medical image set TMIS is Consider it as the target image data that will be the basis for the interpretation analysis that the user is about to perform. It is possible. This process is stored in, for example, a medical information system 40, MatchesSelecting a target medical image group TMIS from multiple groups in the target medical image examination TS may include This choice is for example, User Interface in 10 Operation do In the graphical user interface By selecting the appropriate target medical image set TMIS, Can be run manually by the user Alternatively, the target medical image set TMIS may be uploaded to the computing unit 30. twist A set of target medical images TMIS can be provided to the computing unit 30 by a user. process The S10 is Either the user interface 10 or the processing system 20 It can be implemented at least in part. The exchange of relevant data is included in this process. .

[0210] process S20 In , the region of interest (ROI) for the target medical image set TMIS is to be acquired Region of interest (ROI) Acquisition of teeth, for example, User Interface 10. Manually execute the area manually by the user Definition User input containing Including obtaining Alternatively, select the region of interest (ROI) acquisition teeth, for example, Automatically or semi-automatically Can be advanced According to some embodiments, FIG. As explained in conjunction with Figure 39 , namely, the evaluation tool ML-40 Using Region of interest (ROI) can be obtained manually and semi-automatically. Definition of teeth, For example, a graphical user interface (preferably user interface 10) The representation of the target medical image set TMIS is display To do include The user can, for example, Expression image As an overlay on RI 、 User whereas Visualized Region of Interest Tool ML-40 arrangement It is possible. selectively , The tool The size and / or shape of Further adjustments can then follow . Alternatively Or in addition, the region of interest (ROI) Definition teeth, This includes freehand selection of the region of interest ROI, for example by tracing the outline of the region of interest ROI using any suitable input unit 12.. Also , Region of Interest (ROI) Definition teeth, for example, Based on identifying features in the target medical image set TMIS Made .this is, for example, Anatomical features that indicate the patient's condition is . In addition In other words, the position, size, and shape of the region of interest (ROI) are Preferably, Abnormal or atypical features present in the target medical image set TMIS Based on The unusual features are: For example, localized tissue density increases or decreases, cysts, calcifications, etc., corresponding to deviations of anatomical structures, organs, or tissues, such as the lungs, heart, blood vessels, or brain. . Atypical anatomical features are therefore Patient pathological conditions Represents. Furthermore, ,Its features are: Characteristics of pathological images or related to pathological features obtain . this Features include: for example, Visually inspected or identified by a radiologist / pathologist or is the result of a feature extraction and / or target detection step optionally included in step S20. . process The S20 is Either the user interface 10 or the processing system 20 At least partially in execution If necessary, The exchange of relevant data is included in this process. .

[0211] process S30 In ,Target medical image group TMIS is expressed by The body region 408 is determined. decision teeth, for example, Based on the target medical image group TMIS Zukuka , or , the target medical image group TMIS matches Target medical images inspection TS Based on To determine the body region 408, the target medical image set TMIS and / or the target medical imaging study TS are image For example, in medical imaging data, anatomical structure An image recognition algorithm configured to recognize the medical image data ID of the target medical imaging study TS may then be applied to determine one or more anatomical structures represented by the target medical imaging study TS. the Based on the anatomy, a body region 408 can be determined. Alternatively In addition, medical imaging tests included in the target medical imaging test TS Imaging Data ID content The body region 408 can be derived by evaluating the attribute value AV indicative of the body region of the medical imaging study and / or medical image collection. decision Further details regarding are provided in connection with FIGS. process S30 is mainly a processing system 20 in If necessary, The exchange of relevant data is included in this process. .

[0212] process S40 In At least one reference medical imaging study RS is selected from the plurality of candidate medical imaging studies CS. As mentioned above, the reference study RS is used for comparison with the target medical imaging study TS currently opened by the user. Reasonable It is possible It is something . Specifically, this The choice is For example, the process Based on the body region 408 determined in S30 Made In this regard, process S40 determines candidate body regions 408 for each of the candidate medical imaging studies CS. Selective sub-processes aimed at S41 may be included. This , of candidate body regions 408 decision teeth , basically a process In relation to the S30 can be performed as described . Selective sub-processes S42 In ,Target medical image examination TS and With each of the candidate medical imaging examination CS but Is it comparable? In order to determine whether Regarding The determined body region 408 is compared individually with each of the candidate body regions 408. Specifically, For example, the displayed anatomical structure Based on , Each of the candidate medical imaging examination CSs How well does the TS match the target medical imaging test? As a measure For each candidate medical imaging test CS, calculate the fitness do One or more reference medical imaging studies RS are selected from the candidate imaging studies CS based on a comparison of the body regions 408 (goodness of fit). will be 408 body regions Further details regarding the determination, subsequent comparison of body regions, and the identification of the reference test RS based on this are provided in Figure 8 and Figure 9 Provided in connection with.

[0213] moreover, process The S40 is Selective sub-processes In S43, from multiple medical imaging examination candidates, 、 Based on a comparison of 408 body regions, Multiple potentially related Imaging tests Identification This may include Possibly relevant Imaging tests are for example, Other candidate imaging testsCS More comparable fit than Candidate imaging test showing CS is Next Selective sub-processes In S44, Multiple potentially related Imaging test indicators: for example, User Interface Provided to users via 10 do . In response , user selection but , For example, selective sub-processes In S45, the user Interface Received via 10 And , This user selection is , at least the associated image data files Targeting one specific Next, process In S46, the user selects Specify with was A particular one is selected as the reference imaging study RS, for example. .

[0214] As mentioned above, Either selected In most cases, the reference medical imaging study RS contains multiple medical images. but The candidate medical image set CMIS is used to automatically provide the user with at least one related medical image set of the selected reference medical imaging study RS. process S50 selects candidate medical images from the CMIS for follow-up reading by the user. Purposefully At least one reference medical image set RMIS that provides a good basis Identification do The purpose is to For this purpose, In optional substep S51, for example, a degree of comparability is determined for each of the candidate medical image sets CMIS, the degree of comparability being a measure of how well each of the candidate medical image sets CMIS compares with the target medical image set TMIS. . This , Medical image groups TMIS and CMIS Included in image Data ID characteristics of , The relevant Further metadata from TMIS and CMIS, e.g., each medical image group Capture Used to was Image parameters , along withIt can be evaluated. specifically , This process teeth, For example, medical image groups TMIS and CMIS are linked to each other. Evaluating the attribute value AV include . process Further details regarding the group matching process of S50 are shown in Figures 10 and 11. Figure 11 Provided in connection with.

[0215] moreover, process The choice of S50 is for example, A user interaction for selecting at least one reference medical image group RMIS. include .in particular, process The S50 is Selective sub-processes In S52, for example, Candidate medical image set from CMIS Multiple potentially related Image group Identification To do include . This process is , For example, the process S51 Based on the determined degree of comparability . Possibly relevant The image group is for example, Other candidate medical image groups: CMIS A higher degree of comparability than Show Candidate medical image group CMIS Next Selective sub-processes In S53, Multiple potentially related Image group index but , for example, User Interface Provided to users via 10 will be . Following this, User Selection but , For example, selective sub-processes In S54, the user Interface Received via 10 And , This user selection is , at least one of the related images A specific one of Target Next, Process In S55, the user selects A specific thing Reference medical image group selected as RMIS will be .

[0216] Process The S60 During the registration process Registration between the reference medical image set RMIS and the target medical image set TMIS is provided. will beThere are various ways to achieve such registration. The options available in this regard include: Rigid body Registration, affine registration, Non-rigid Registration, non-affine (non-affine) Registration and all of the combination Specifically, the coordinate systems of the target medical image set TMIS and the reference medical image set RMIS are Link Registration can be obtained. In other words, On the other hand Medical image groups TMIS and RMIS each The image data is The other set of images A transformation can be calculated that can be used to convert the RMIS and TIMS into coordinate systems. For this purpose, at least a portion of the target medical image set TMIS is but , at least a part of the reference medical image set RMIS and It is registered. Basically, this process is Two image groups and identifying matching data points in Such Match By identifying the data points, The local offset between these matched points can be calculated, providing a measure of the local shift in the coordinate systems between the two images. This is the basis It has become Multiple well-distributed images Matching Data Points to Follow What you do is: each A good indicator of displacement and deformation between images early provide It will be Apart from this formal registration using coordinate transformation, medical image groups TMIS and RMIS Matches in Slice or image pattern Identification Registration can be performed based on this. in this case , slices in the target medical image set TMIS, especially slices in the region of interest ROI, can be transformed to slices in the reference medical image set RMIS without the need to calculate a complete coordinate transformation. Match Image slice (also designated as reference slice REF-SLC, described later) obtain ) Assign to It is possible. Process S60 Further options that can be implemented are presented in connection with Figures 12 to 26. .

[0217] ProcessS70 generates display data, User Interface 10. Rendering of the reference medical image group RMIS Intended for display on a display device . specifically , rendering is For example, based on the registration determined in step S60, meaning that the image data ID extracted from the reference medical image set RMIS matches the image data ID of the target medical image set TMIS currently being reviewed by the user. . especially , the rendering is , see for example Image slice based on REF-SLC Made .

[0218] Referring to Figure 5, Process S30 and Process At S41 be executed , medical image stored in the first image file Imaging By data expression A computer that determines the body area to be affected execution The method is illustrated.

[0219] In brief ,The method in Fig. 5 is Next step include: In step BR-1, for example, Match By evaluating the image file IF, Imaging examination TS, CS each Obtaining one or more of the text strings AF,IF-314; Process In BR-2, Acquired inputting one or more text strings AF,IF-314 into a trained machine learning model (e.g., see trained neural network 406 in FIG. 6); Determining the body region 408 represented by the medical imaging data ID by obtaining output from a trained machine learning model 406, where the trained machine learning model 406 has been trained to output a body region based on an input of one or more such text strings.

[0220] Such a text string may be used to determine body regions based on the input of the text string. Trained A machine learning model 406 (e.g., a neural network) is applied to the target medical image examination TS. Connected Enter one or more text strings AV,IF-314 to Imaging By Data ID expression Determining the body region 408 to be treated may include: subject and / or medical imaging data included in candidate imaging studies TS, CS is expressed by Efficient and / or flexible determination of body region 408 may be provided.

[0221] For example, the text string in the file IF (These are relatively small in terms of bits) Determining a body region based on Imaging data the same (bit In terms of Determine body region by extracting and analyzing (relatively large) Compared to , less resource intensive and therefore more efficient be . Image files are stored remotely over a network from the processing device In this case, determining the body region based on the (relatively small) text strings can be used to identify the (relatively large) medical Imaging The data, Represented by the data Network to determine body region Through send This will prevent the , therefore, Enables efficient use of network resources.

[0222] Another example is a trained machine learning model (e.g., trained Neural Networks ) to determine the body region, e.g., by entering a text string Hard-coded rules The efficiency of determining the body area is improved compared to determining the body area by applying ,flexible, and / or provide robust decisions. For example, hard-coded rules can: body area To provide a rule, it requires exact matching of text strings (hence lacking flexibility in terms of the text strings that can be matched) and / or is difficult to use. May be available A set of rules that must be coded for every text string Comprehensive nature On the other hand, pre-trained machine learning models (e.g., Neural Networks )teeth, Train it training data set from It is induction , and therefore the training data set of Even if the text string is different from the original text string, an appropriate body region can be obtained relatively efficiently. can be determined, and therefore relatively Flexible / Robust is.

[0223] From the above , medical Imaging By data expression Efficient and / or flexible Such an automated decision may be provided.

[0224] According to some examples, a trained machine learning model It can also complement (or be complemented by) rule-based systems, allowing induction of rule-based systems when they are insufficient. A rule-based system is for example, One or more choices for selecting the reference medical imaging study RS rule of Execute .choice rule For example, Regulatory requirements and / or derived from medical guidelines Jill .

[0225] In some examples, the imaging data ID of the target medical imaging study TS is used. be expressed Body Area 408 decision is then included in the target medical image examination TS image The candidate medical imaging study CS associated with the data ID (e.g., suitable for comparison) Reference Image Automatic data selection Can be promoted For example, the medical records of the current examination file Imaging By data be expressed Determining the body region 408 may involve determining the medical records of a previous examination of the same body region of the patient. Imaging This can be used to select data, as will be described in more detail below with reference to Figures 6 and 7.

[0226] As mentioned above, in some examples, the medical image data file IF is a DICOM file IF (i.e., for example DICOM standard “NEMA PS3 / ISO 12052, Digital Imaging and Communications in Medicine (DICOM) Standard, National Electrical Manufacturers Association, Rosslyn, VA, USA) follow DICOM files Format image filesIF). Referring again to FIG. 3, the DICOM file IF includes, in more detail, a header IF-302 and a dataset IF-308. The header IF-302 includes a 128-byte preamble IF-304 (if unused, all bytes are set to zero) and a 4-byte prefix 306 containing the string "DICM". The dataset IF-308 includes data elements IF-310 and IF-312. Each data element IF-310 and IF-312 includes: Contains tags , tag Each tag is in the format (xxx,xxx), where each "x" is a hexadecimal digit. The DICOM file 300 stores a unique identifier for the study that the file is part of in the "Study ID" data element (not shown). can be saved in (i.e., identified by the DICOM tag (0020,0010)) 、 And the file of the group that is part of it The unique identifier is stored in the "Group ID" data element (not shown). can be saved to (i.e., identified by the DICOM tag (0020,000E)) 。

[0227] The DICOM file IF stores the medical image data ID (in this case, pixel data) in the "Pixel Data" data element IF-312 of the DICOM file IF (i.e., identified by the DICOM tag (7FE0,0010)). Imaging Data ID content 3. Store one or more attributes IF- 310 (provided by one or more other data elements 310) each having an attribute value IF- 314 containing a text string indicating One such attribute is "Study Description" (i.e., identified by the DICOM Tag (0008,1030)), whose attribute value IF-314 is a text string that describes the study of which the medical imaging data is part (e.g., "NERUO^HEAD" where the medical imaging data is the patient's head region or "PELVIS^PROSTATE" where the medical imaging data is the patient's pelvic region), and thus indicates the contents of the Medical Imaging Data ID. In some examples, the attribute IF-310 and the attribute value IF-314 are Like that It is not included in the header IF-302, but rather in the dataset IF-308. Get , attribute IF-310 and attribute value IF-314 (medical Imaging (not including the data element IF-312 that stores the data ID itself) ImagingBecause it is not the data ID itself but data related to the medical image data ID, it is sometimes called the header data of the DICOM file interface. obtain .

[0228] medical use Imaging Data content Text indicating string Other examples with attribute values ​​containing of DICOM attributes can also be used For example, such Other examples DICOM attributes are used for medical Imaging the group that the data is part of (e.g., "ax t1 whole pelvis" indicates that the medical imaging data is of the whole pelvis, captured in an axial orientation using a T1 MRI), " Series Description " (i.e., identified by the DICOM Tag (0008,103E)). For example, other such For example DICOM attributes are "Body Part Examined" (i.e., DICOM Tag (0018,0015)) ) and this is, investigation Part of the body that has been removed (e.g., "PELVIS" to indicate that the medical imaging data is of the pelvis) Other such appropriate It will be appreciated that DICOM attributes may exist and be used. In some examples, the DICOM attribute " Reason for the Requested Procedure " (i.e., identified by DICOM Tag (0040,1002)) may also be used.

[0229] moreover, some In some examples, it will be appreciated that such image files other than DICOM files may be used. For example, in some examples, the image file may be Can be a Portable Network Graphics (PNG) file , this is, One of the chunks is medical Imaging Save Data and then in another one of those chunks, medical use Imaging Data content one or more attributes, each with an attribute value containing a text string indicating Included Metadata Save .

[0230] In some examples, one such text string may be used to train a machine learning model (e.g., a trained Neural NetworksHowever, in other examples, the medical image data files associated with the target medical imaging study and / or target medical image collection TMIS may be entered into Multiple Attributes of IF from multiple attribute values ​​of Multiple of the above The text string is 1st File medical Imaging By data expression To determine the body part to be imaged, a trained machine learning model (e.g., a trained Neural Networks ) together. For example, this allows for more accurate and / or Robust For example, a trained machine learning model (e.g., a trained Neural Networks ) determines the body region with a high degree of accuracy. The decision is based Input data ( No. 1 from the file The above As another example, if one of the text strings happens to be unreliable and / or missing (e.g., by a radiologist), As per regulations or precisely Because it is not recorded, For example, in the case of "investigated body part" ), inputting multiple text strings alleviates this, Nevertheless , To be determined Reliable in the body area decision This makes it possible.

[0231] FIG. 6 shows an example. Related 1 illustrates the flow between the components of the method described above with reference to FIGS. Figure 6 Referring to the above, Target medical use One or more of the text strings 404 corresponding to the imaging study TS are input into a trained machine learning model 406 , which outputs a body region 408 .

[0232] In some instances As shown, one or more text strings 404 may be extracted from the image file IF associated with the target medical imaging study TS. Specifically, the one or more text strings 404 may be extracted from the image file IF associated with the target medical imaging study TS. ,without extracting the medical imaging data.For example, in a DICOM file, the image file identifies attributes whose attribute values ​​are to be extracted. One More predefined tags (e.g., "Inspection" Description "attribute In the case of DICOM Tag (0008,1030), " Series Description " In the case of DICOM Tag (0008,103E) locate It is analyzed for obtain Once a tag is found, attribute values ​​(i.e., text strings 404) of one or more attributes identified by the one or more tags may be extracted from the image file IF associated with the target medical imaging study TS. The extracted text strings 404 may be, for example, Or medical image data thereof Identifier for to In some examples, images related to the target medical imaging study TS may be stored in association with the file IF text string 404 teeth , extracted from the image file IF related to the target medical imaging examination TS Are you , or Otherwise, this method is obtained before Tei , e.g., image files related to the target medical imaging study TS Identifier of IF or its medical image data ID Stored in relation to are In either case, one or more text strings 404 are taken and input into a trained machine learning model 406.

[0233] In some cases, only one text string can be used However, in some examples, multiple text strings 404 of an image file IF associated with a target medical imaging study TS or a candidate medical imaging study CS can be obtained and input to a trained machine learning model 406. For example, the attribute "Study" in a DICOM file IF can be Description "," Series Description " and " Body part to be investigated " and input the text strings of the attribute values ​​of all the attributes of ", and the trained machine learning model 406 can then generate a medical Imaging By data expression Body region For example, the text strings can be concatenated and input together into the trained machine learning model 406. In either case, an output 408 from the trained machine learning model 406 is obtained, which can be used to Imaging With data expression The body region 408 to be examined is determined.

[0234] As mentioned above, the machine learning model 406 One or more of the above The machine learning model is trained to output body regions based on input of a text string, i.e., the ... Imaging Data content medical imaging data of imaging studies TS, CS based on the input of one or more text strings indicating is expressed by It is trained to output body regions.

[0235] In some examples, the trained machine learning model 406 may be a trained neural network 406. Indeed, in the examples described below, the trained neural network 406 Mention about However, other examples may output body regions based on the input of one or more text strings. Trained to Other types of machine learning models may be used , said For example, in another example, the trained machine learning model may be Random Forest It can take the form of algorithms, etc. For example, the random forest algorithm: It consists of an ensemble of decision trees trained on training data containing training text strings labeled with relevant body regions (e.g., as determined by an expert). , i.e., to minimize the error in predicting body regions compared to the body regions labeled for the training text strings. Ensemble of decision trees However, As already mentioned, in the following 406 examples of trained neural networks Mention The use of trained neural networks 406 is described in more detail below. As stated , some Advantages do .

[0236] In some examples, the neural network 406 deep It may be a neural network (i.e., have one or more hidden layers). In some examples, the neural network may With teacher For example, the neural network 406 may be trained using a number of training text strings (in practice, Hundreds or thousands of training text strings , each training text string may be derived from an attribute value of an attribute of an image file. It is , medical data further stored in the image file Imaging Data content For example, training text strings may be generated from the attribute values ​​of appropriate attributes in DICOM image files (e.g., "Study Description", " series Description, Subject of investigation Text strings extracted from the is In some cases, Just as I did Each training text string is generated from multiple images of different appropriate attributes of the image file. Each of the above In any case, each training text string can represent a concatenation of the training text strings. Matching Labeled by body region obtain For example, in some instances, the training text strings are arise File medical Imaging By data expression The actual body region may be labeled with the actual body region to be used, as determined, for example, by an expert. In some instances (as described in more detail below with reference to FIG. 7), the actual body region may be labeled with the actual body region to be used, as determined, for example, by an expert. will be , or corresponds to the text string itself The text string itself By expression The body area determined to be in , training text strings teeth Labeling will be In either case, the body region labels for each text string are used to train the neural network. In the teacher signalIt can be used as.

[0237] In some embodiments, the trained neural network 406 may include: In the form of a body region classification, the classifier portion of the trained neural network 406 selects from among a plurality of body region classifications for one or more input text strings, The system may be configured to output a body region 408. For example, each classification may be a standardized word representing a body region, such as "ABDOMEN," "PELVIS," "CHEST," etc. In such an example, each training text string is labeled with the classification to which it belongs, This is , for training neural networks teacher signal Used as obtain For example, training Deep learning For example, training involves comparing the classification predicted by the classifier for each of the training text strings with the actual classification of each of the training text strings as defined by their respective labels. 、 The method may include updating the weights of the connections between layers of neurons in the neural network to minimize the error between

[0238] In some embodiments, the trained neural network 406 may generate one or more numerical values ​​representing regions of the human body. and configured to output the body region 408 in a manner Trained neural network 406 Regressors The portion is calculated for one or more input text strings 404. are For example, one or more numbers can be Body Ruler ", i.e., It can be a ruler or scale marking defined on the human body, for example: A value of 0.0 human toes A value of 1.0 represents the top of a person's head Show death , in this case, 0 to 1 Between The value is Toes and top of the head Between Areas of the human body In some instances, two such numbers can be used to represent a body region of the human body. For example, The relevant The two values ​​indicate where the body region is defined. Can be shown . for example , the bladder has a body ruler value of [0.5,0.55] is assignedIn such an example, each training text string is labeled with one (or more, e.g., two) numeric values ​​representing the body region to which it corresponds. And , which is used to train the neural network 406 teacher signal Used as obtain For example, training Deep learning For example, training involves the following steps: Regressors and the actual value of each of the training text strings defined by their respective labels. 、 The method can include updating the weights of the connections between layers of neurons in the neural network to minimize the error between

[0239] Outputting the body region as one or more numerical values ​​is a medical Imaging By data expression For example, the values ​​may be continuous and therefore may allow for accurate and / or flexible determination of the body area to be measured. For example, Compared to using a limited set of defined classes 、 Body regions can then be defined more precisely and flexibly, for example, as described in more detail below with reference to FIG. 1st medical use Imaging When selecting the Medical Imaging Study RS as the data related, Flexible and / or may allow for accurate comparison.

[0240] some example In this example, the trained neural network 406 analyzes the individual characters of one or more retrieved text strings 404. Take as input In these examples, inputting the one or more obtained text strings 404 into the trained neural network 406 may include inputting individual characters of the one or more obtained text strings into the trained neural network 406. can be seenFor example, the neural network 406 may convert each character in the input text string into Encode to a vector configured to Encoder For example, this could be a 1-hot character vocabulary that includes the alphabet, numbers 1 through 9, and special characters. encoding This is done using character embedding such as obtain These vectors can be used by the neural network 406 as a basis for determining body regions. can also be used .

[0241] For example, some example So, neural network 406 is It consists of a character-based recurrent neural network (RNN), for example, a long-short-term memory RNN such as a bidirectional long-short-term memory (LSTM) RNN. . A vector for each character in the text string but Continuous input to RNN And ,Then, the RNN maintains a certain internal state (e.g., the vector for the last character of the text string) of a vector representing the value of that neuron will be This internal state is then transferred to the neural network 406. Regressors Or it can be passed to a classifier, where the internal state can be mapped onto the body region 408.

[0242] As another example, the neural network 406 may Base Convolution Neural Networks (CNNs) Consists of In these examples, vectors of successive characters from a text string can be ordered and assembled to create a matrix. Convolution and Pooling process is applied to the matrix to represent the features present in the text string. Condensation Features Vector (condensed feature vector) The feature vector can then be calculated using the neural network 406. Regressors or passed to the classifier And , these are The feature vectors can then be mapped onto the body regions. of Accurate mapping obtain Features is itself, can be learned during training of the neural network 406.

[0243] In some instances, other neural networks, e.g., base Neural networks can be used. however , character-based neural networks Consists of The neural network 406 uses a part of the training data Missing abbreviations, misspellings, and other words Regarding Robust of the body area where decision For example, "ABDOMEN" can be abbreviated as "ABD," but because the first few letters are the same, a character-based neural network can find similar positions in vector space. We can generate vectors for these two words: (Thus, the appropriate body area is determined. can be ) On the other hand, the word-based neural network out of vocabulary This is because out-of-vocabulary words can reduce accuracy. Later, It may help in making accurate decisions.

[0244] As mentioned above, each training text string is Match In some examples, the training text strings may be labeled with the body region that corresponds to the training text string. In a decision by , in a text string Does it match? Alternatively, the body region may be labeled with a determined text string. Then ,In such a case, training dataset ( training text strings and their labels) Graphical User Interface The GUI 502 may be generated using training The GUI may be configured to present one or more of the text strings 508 and a representation of a human body 510 divided into user-selectable body regions 511. For each of the one or more presented training text strings 508, the GUI may: receiving user input to select a body region 511 from the representation 510;, may be configured to label the training text strings 508 with labels 509 indicating the selected body regions 511. In this manner, a training data set can be generated that includes training text strings labeled with body region labels.

[0245] For example, with reference to the specific example of Figure 7, GUI 502 includes a progress bar 504 that indicates to the user the particular pane of the GUI that is currently displayed. The title of the particular pane is also displayed 506. It is displayed as In this case, " Annotate Key Words " is displayed. are In this example, the training text strings 508 are Imaging Data keep These are keywords extracted from the attribute values ​​of the image file attributes. In the example shown, the keywords include "aaa", "ab", "abdpel", "abdroutine", "aquired", "aif", "angiogram", and "ascities". for , specific keywords is presented ( In this example, "aaa" is in bold ), and, This corresponds to human body Display of Expression 510 body The user is asked to select one of the areas 511. The user may be, for example, a medical professional. Imaging Knowing that "aaa" in the context stands for "abdominal aortic aneurysm" Crate , displayed human body Select the abdominal portion of the representation 511 and accordingly add the body region label "ABDOMEN" to the text string "aaa". Labeling Other suggested keywords but, "ab" - "ABDOMEN", "abdpel" - "ABDOMEN", "abdroutine" - "ABDOMEN", "aif" - "HEAD", "angiogram" - "CHEST", and " ascities " - "ABDOMEN" and, A similar selection can be made: the text string "acquired" is not associated with a specific body region, It is not labeled with a body region and therefore will not be part of the training dataset.Note that GUI 502 also includes selectable buttons 512, That is, ", "Uterus", "Prostate", "Unknown" and "None". Uterus "and" Prostate The "" buttons are for providing the body region labels "PROSTATE" and "UTERUS" respectively. The "Unknown" button allows the user to body I don't know the region labels, but for example, body The "None" button is for cases where the user knows there is no body region that can be assigned to a text string (e.g., the " acquired " as in the case of

[0246] The GUI 502 allows a simple and efficient way to generate a plurality of body region labels (e.g., hundreds )of training Contains a text string training The data set can be labeled as follows: human body representations (e.g., stylized figure ) and This expression is visual and easy for users (including medical professionals, not necessarily programming experts) to interact with. This simple and efficient interface between the user and the GUI Interactive Then the training dataset (i.e. the text string to be labeled) This allows for efficient generation of

[0247] In the above example, the neural network 406 is trained to output body regions 408 based on an input text string. example The neural network 406 may then be further trained to output laterality of the output body region 408 based on the input of one or more text strings. Which side (i.e., "left" or "right"), Imaging By data expression The body area to be examined is located say For example, the output laterality may be either "left" or "right" as appropriate. In these examples, the method of FIG. Process 104, based on the obtained output of the trained neural network 406, Imaging By data expression In one example, the method may further include determining the laterality of the body region being examined. This process is configured to output laterality based on one or more text string inputs. Trained to This may be done by providing a dedicated laterality portion of the neural network 406. For example, This process is a text string Matching Differences between the left and right body regions Label each As another example, the training dataset can be: This process returns the entered text string Neural Network 406 Mapping Body Region Classification Set Extending the " left " and "Right" versions by .for example, This process are the training text strings Matching Body area and Left-right difference The system may be trained using a training dataset containing training text strings each labeled with a classification. Determining the laterality of body regions allows for more precise body region determination, This then, as will be described in more detail below, Imaging This may allow for a more accurate selection of data.

[0248] As noted above, in some instances: subject By examination TS or target medical image set TMIS expression The determined body region 408 is subject A second medical image related to the study TS or target medical image set TMIS (e.g., for appropriate comparison) Imaging For example, the body region 408 may be used to select data. predetermined patient RegardingDetermined for the target medical image set TMIS based on the image file IF of the current examination And ,This is the current target medical image set TMIS, i.e. subject Suitable for comparison with test TS (e.g., same or similar body area) , given patient Regarding It may be used to select one or more image datasets (eg, stored in one or more image files IF) of one or more previous examinations (reference medical imaging examinations RS).

[0249] Two medical imaging tests TS, CS Degree of comparability or suitability may be calculated based on each body region. For this purpose, The overlap can be determined For example, if the body region is against Normalized or generalized coordinates Using the range of Between two body areas, if provided Overlap can be calculated based on the respective ranges. Degree of comparability or suitability can be determined by calculating the Dice score between the respective body regions to measure the similarity of the two output values.

[0250] Now referring to Figure 8, TMIS / Target inspection At least one reference as related to TS inspection A method for selecting RS is illustrated.

[0251] In step BR-3, the method comprises: subject The first medical image stored in the examination TS / target medical image group TMIS Imaging By data expression will be subject The body region 408 is then divided into a plurality of candidate medical imaging studies CS. Each of The method includes comparing the body region to each of a plurality of candidate body regions represented by Process At BR-4, selecting one or more of the candidate medical imaging studies CS as associated with the target medical image set TMIS based on the comparison of the body regions 408.

[0252] each Candidate medical imaging tests CS includes multiple (candidate) medical image sets CMIS of the same type as the target medical image set TMIS. can be seen Similarly, each candidate medical imaging study CS and its associated image set CMIS are stored in one or more image files IF. will be That is, each candidate medical image set CMIS and, therewith, each candidate medical imaging study CS is further stored in a corresponding image file IF. Sometimes , Candidate Medical Imaging Examination CS content has an attribute value AV containing a text string indicating One It is associated with the above attribute A. Target body Region 408 may be configured to add one or more text strings of the target medical imaging study TS to the text strings in the manner described above with reference to FIG. Process By applying BR-1 and BR-2 、 decided are Alternatively or additionally, at least one (and in some cases all) of the candidate body regions 408 may be At least one (and possibly all) Candidate Medical Imaging CS Each of One or more text strings may be coded in accordance with the method described above with reference to FIG. Process By applying BR-1 and BR-2 、 can be determined.

[0253] Figure 9 shows An example of 9 illustrates the flow between the components of the method described above with reference to Figure 8. Referring to Figure 9, in this example, similar to that described above with reference to Figure 6, one or more text strings 404 from the image files IF of the target medical image set TMIS or the target medical imaging study TS are input to a trained neural network 406, which Target body Region 408 is output. Target body Area 408 contains one or more image files, or separate files. To, identifier In association with It may be stored.

[0254] Furthermore, in this example, medical care From Information Systems 40 search Candidate for Medical Imaging CS Multiple Image data file interface Each of from, One or more The text string 716 is multiple For example, multiple candidates are extracted. inspection Each of the Previous separate tests Related to obtain One or more text strings 716 Multiple Each one 、 input to a trained neural network 720, next, Multiple candidate body regions 720 Each of Each of the candidate body regions 720 is stored as one or more image files IF or separate files To, identifier Save as Then, Target body The region 408 is divided into a plurality of candidates by a comparator 710. body Compared to each of the regions 720, subject and candidates body For each pair of regions Goodness of fit can be determined, this Suggestions based on comparison inspection In particular, one or more of the following may be selected 712: Reference Test RS is a candidate inspection From CS In this way choice can be .

[0255] In some cases, the selection is based solely on comparison. Made Other examples are described in more detail below. explanation As such, selection is based on further criteria. Made In some instances, the selection is based on the body region Saved in association with The selected identifier may be: Make an inquiry to Medical Information System 40 Medical Information System 40 (e.g., online or near-line DICOM archive devices) Related candidate medical image data files from Search (e.g., prefetch) It can be used for

[0256] In some instances, Medical image data or One or more selected images set Image files for those studies (e.g., only those studies) that contain one of the medical care From Information Systems 40 search These files can be, for example, Imaging The data may be determined by checking the "examination identifier" attribute of the file containing the data. In some examples, as described below, Medical image data or Second medical use Imaging Image files of those groups (e.g., only those groups) that contain one or more of the selected sets of data; medical care From Information Systems 40 search These files can be used to store, for example, selected medical Imaging This can be determined by matching the "group identifier" attribute of the file containing the data.

[0257] In any case, the second medical image data Searched Set One or more renderings of Display unit 11 It can be displayed.

[0258] In some examples, the target medical image set TMIS may include a current medical image or a series of medical images of a patient. Express The candidate medical image set CMIS is a set of previous medical images or a series of medical images of a patient. can be expressed For example, the target medical image set TMIS corresponds to the patient's current examination. death , each of the candidate medical images CMIS is a patient's Previously separate Supports inspection obtain . medical care Experts, for example, inspection To assess the progression of the disease during inspection One or more images of TS from the patient's previous inspection CS Our Related 1 inspection RS One or more images The key criteria that allow for a valid comparison are: Current Inspection TS and previous inspection CS (i.e., Current and previous status of these tests medical images) are of the same or similar body parts (e.g., head, abdomen, legs) Or That is the thing. subject Medical image data and candidate medical use Imaging Based on a comparison of the determined body area to the data reference medical use Imaging By automatically selecting data, the method allows medical professionals to: subject Medical images of the examination TS are Previously medical images (e.g., of previous examinations of the same or similar body regions) medical imaging ) only and allows for comparison with This teeth, The user Need to compare with all previous medical images There is It is more efficient compared to

[0259] moreover, some In this example, the candidate medical imaging CS Multiple sets of teeth, A medical device that can be accessed from a user's terminal 10 As part of Information System 40 be Remote Storage Device to keep will be In these cases, the reference medical imaging study RS One or more selected sets of (or individual image data ID of the reference study RS) is the candidate medical image data Multiple sets of Other things search from a remote storage device without search (for example, medical care Part of Information System 40 is DICOM Archive device Prefetch from ) The body area is determined based on the text string in the image file attributes. Because it will be , those file candidates medical use Imaging data teeth , When selected as relevant remote Storage from Just search , as related or not selected to be searched candidate medical use Imaging data teeth , Search at all There is no need. Like this , Network Resources efficiently Can be utilized .

[0260] In some instances, the comparison of body regions is ImagingThe body region classification for the data (e.g., "ABDOMEN," "CHEST") is classification In some examples, the method may include determining whether the match (e.g., both have the same body area classification "ABDOMEN"), Imaging The data is for the target medical Imaging It may be selected as relevant to the data.

[0261] some example So, the comparison of body area 408,720 is: Numerical values ​​defining the body regions for the subject and candidate medical imaging studies TS, CS For example, this can include comparing Targeted medical imaging The values ​​of the body regions for the data are Candidate Medical Imaging The numerical value of the data is the same as or similar to the numerical value of the data, or Overlap For example, the method may include determining whether the target medical image is inspection If the value of TS is 0.5, the candidate medical image has a value of 0.45. inspection CS Set , similar (e.g., The difference is less than the predefined amount ) As another example, subject If the numerical value of the medical imaging examination TS is [0.5,0.55], Number has [0.5,0.55] candidate Medical Imaging CS Assuming that the set of choice And , Number has [0.4,0.55] candidate medical use Imaging data Assuming that the sets of overlap choice can be .

[0262] As mentioned above, the target medical image group TMIS to , especially in the region of interest (ROI) Associated with one or more reference medical imaging studies RS (e.g., predetermined the patient's previous examinations) The choice is Based on comparing the body regions can be achieved However, in some instances, the selection may be different, as described in more detail below. explanation The threshold may be based on further factors or criteria, as described above.

[0263] Showing the same body part Current Inspection TS and Previous inspection Besides RS, another important criterion that allows for an effective comparison of those images is TMIS and CMIS of the reference medical imaging study RS , the same, similar or comparable modality (i.e., Capture Used to Ta medical use Imaging The mode or type of imaging (e.g., CT, MRI, X-ray) is required.

[0264] therefore , some example In This method For each of a plurality of groups of candidate medical imaging data (i.e., a plurality of candidate medical image groups CMIS), a first Imaging The second set of candidate medical images of the selected medical imaging examination RS is the CMIS. Imaging Between modalities Imaging Modality Relevance This can include determining a score. Imaging The modality relevance score is Is it included in the similarity? , or , which can be considered as a factor of similarity. .

[0265] In these examples, the target medical Imaging As related to data Candidate medical imaging data It was decided to select one or more groups of CMIS Imaging It may also be based on modality relevance scores. For example, Imaging One of a group of reference medical imaging studies RS with modality relevance scores CMIS but , lower Imaging Group of reference medical imaging studies RS with modality relevance scores CMIS Other things Priority is given to the target medical Imaging A data ROI, TMIS, may be selected for comparison. In some instances, the ROI may include the same body region as the target medical image set TMIS. At the same timeThe candidate medical image group CMIS having the highest imaging modality relevance score can be selected as the reference medical image group RMIS. In some examples, candidate medical image groups CMIS having the same or similar body regions as the target medical image group TMIS may be pre-selected, for example, according to the method described above with reference to Figure 8, and then one or more candidate medical image groups CMIS from the pre-selected set may be selected as the reference medical image group RMIS based on the imaging modality relevance score.

[0266] In some instances, the file Imaging Data Capture used to Imaging Modality is medical Imaging Image file IF where data is saved Imaging The image modality can be determined from the attribute value AV of the modality attribute A. For example, the image modality can be obtained directly from the attribute value of the "Modality" attribute (e.g., identified by DICOM tag 0008,0060) in the DICOM file IF. will be The DICOM "modality" attribute value is Imaging Identifies the type of device that acquired the data. The DICOM standard allows for multiple Imaging Values ​​that can express modality But in advance For example, "CT" stands for "Computed Tomography" and "MR" stands for "Magnetic Resonance." expressing Furthermore, in some examples, Imaging The modality attribute value is automatically set to the appropriate value by the software used with the device to generate the image file. obtain .therefore, Imaging The modality can be reliably obtained directly from the file's "modality" attribute.

[0267] The imaging modality relevance score between two modalities was calculated using one modality. Capture Medical imaging data but , using a different modality Capture Medical Imaging Data and associations do (i.e., For comparison appropriate or In some examples, the value may represent the degree to which the Imaging The modality relevance score is Imaging Modality transition It can be determined using a matrix. For example, each element of the matrix can be a specific Imaging Modality and another specific Imaging Modality and One between Imaging It may correspond to the modality relevance score, i.e., the element s of the matrix ij teeth, Imaging A subject having modality i (e.g., current ) medical imaging data; and Imaging between candidate (e.g., prior) medical imaging data with modality j Imaging It can be a modality relevance score, e.g., i=MR and j=MR (i.e., s MRMR ) and Between Imaging The modality relevance score was 0.6. can be , on the other hand, i = MR and j = CT (i.e., s MRCT ) and Between Imaging The modality relevance score was 0.22. be In this case, for example, subject When a medical image group TMIS is captured using MR, Two candidates Medical image group CMIS but both candidate Medical image set with the same body region as CMIS There are but, One set is captured using MR , The other set was captured using CT. In this case, the set captured using MR can be selected preferentially.

[0268] In some instances, Imaging Modality Relevance score (i.e., transition Each element of the matrix s ij ) is a specific first Imaging Modality i Related medical imaging data RegardingA user (e.g., a medical professional) may request a specific second Imaging has modality j reference medical use Imaging data Target It can represent the probability of selecting a medical image data for comparison. Imaging with data record User interactions ( Preemptive Action ) can be determined based on a statistical analysis of the data log. predetermined session In the specified by medical professionals predetermined About the patient search Medical Imaging Statistical processing may be applied to these logs to determine the probability. For example, in current medical applications where the modality is MR, Imaging When reviewing a file, If it is determined that medical professionals are likely to review a patient's previous medical imaging files with an MR imaging modality 60% of the time, but are likely to review a patient's previous medical imaging files with a CT imaging modality 22% of the time, , s MRMR is determined as 0.6, and s MRCT can be determined as 0.22. This , transition Matrix Add In order to Imaging All combinations of modalities are performed obtain . Imaging Modality Relevance score (i.e. transition Each element of the matrix s ij ) is a medical Imaging data against It is based on statistical analysis of actual user interactions and is used for second medical use in appropriate modalities. Imaging Ensures data is selected useful for .

[0269] In the above example, for a given patient Regarding Candidate medical image group CMIS medical imaging data teeth , for example Body area and / or Imaging For a given patient, based on modality Regarding Target medical image group TMIS medical Imaging relevant to the data (e.g., suitable for comparison) Something As mentioned above, predeterminedWithin an examination, there may be multiple sets of medical images. For example: predetermined Inspection (especially predetermined Modality ,for example MR , within ) in The plurality of groups may be different Image parameters (e.g., MR: Echo time, flip angle, echo train length, patient orientation, etc. Regarding ) obtained using Imaging May contain data from medical professionals the current and another important criterion that allows for effective comparison of previous medical images. the current and previous medical images Capture Used when Ta medical use Imaging The parameters are the same mosquito or similar That is what it means. . therefore In some cases, a second medical Imaging The selection of data is Alternatively or additionally, This can be done based on a comparison of image parameters, for example: predetermined patient Regarding Previous inspection Candidate group CMIS within RS candidate medical use Imaging The data includes images from the previous CMIS and the current TMIS. Capture Used to Ta medical use Imaging Based on the comparison of parameters, the current group of TMIS target physicians Imaging Data-related (e.g., suitable for comparison) It can be selected as.

[0270] See Figure 10 Then , Imaging Related medical data based on parameters Imaging A method for selecting data is illustrated.

[0271] In these examples, each of the image files IF associated with the CMIS of the reference medical imaging study RS contains the medical imaging data of the image file. Capture Used to Imaging Attribute value AV indicating the parameter Each oneSave one or more attributes A that have

[0272] For example, Similarly , each image file IF, DICOM file IF Its medical Imaging Data Capture Used to Imaging Parameters Show attribute value Having AV One or more first attributes A , Contains one or more of the following DICOM attributes: "Image Orientation Patient" (identified by DICOM Tag (0020,0037) The value is the medical information related to the patient. Imaging Data No. 1 line and the first column Azimuth cosine Specify death , For example, the value is [1,0,0, 0, 1,0] ) ; series Description" (DICOM tag (0020,0037) Identified by 、 Its value contains a description of the group, for example The value is "axt1 whole pelvis" which indicates the axial direction. ) ; Echo Time " (DICOM tag (0018,0081) Identified by 、 Its value is Excitation of echoes produced in MR imaging Pulse center and peak and The time in milliseconds between Identify , for example The value is "4.2" ) ; 「 Repeat Time 」( Identified by DICOM tag (0018,0080) 、 its value teeth , M.R. Imaging Initiation of pulse sequence at and initiation of subsequent pulse sequence and The time in milliseconds between identification death, For example, the value is "8"); "Flip Angle" DICOM Tag(0018,1IF-314 ) Identified by 、 its value teeth , M.R. Imaging Primary in place is flipped from the magnetic vector of for the magnetic vector The steady state angle is Identify , For example, the value is "90" ); "Eco Train Chief"(DICOM attribute Tag(0018,0091) Identified by , the value is , excitation per image Every Acquired k-space (an array of numbers representing spatial frequencies in an MR image) in Specify the number of lines , For example, the value is "1"); " scanning Sequence" (identified by DICOM Tag (0018,0020) and its value capture The type of MR data generated indicates , For example, the value is "SE" indicates spin echo type MR ); "Sequence Name" (DICOM Identified by Tag (0018,0024), its value teeth , M.R. Imaging in scanning User-defined names for sequence and sequence variant combinations Identify , for example The value is "spcir_242" ); "Protocol Name" (DICOM tag Identified by (0018,1030) and its value teeth, Name the CT protocol Identify , for example The value is "T2W_TSE SENSE"). some In this example, it will be appreciated that other such attributes and indeed other types of image files may be used.

[0273] Process To S50 Related For selecting reference medical image groups RMIS An example of the method is shown in Figure 10. The corresponding data stream is shown in Figure 11.

[0274] In the embodiment shown in FIG. This method teeth, Process In BR-5, the target medical image group TMIS About Obtaining the first vector FV Includes , The first vector FV is Target medical images inspection The target medical image group TMIS of TS Capture Used to Imaging Generated based on one or more attribute values ​​AV indicating parameters It has been . This method teeth, Process Included in the Reference Medical Imaging Examination RS in BR-6 Multiple For each of the set of candidate medical images CMIS, a plurality of second of obtaining a vector SV, where the second of medical use Imaging For each set of data, we define a set of candidate medical images CMIS for the reference medical imaging study RS. Capture Used to Imaging The second parameter is based on one or more of the attribute values ​​AV. of A vector SV is created.

[0275] This method teeth, Process In BR-7, for each of the plurality of second vectors SV, a target medical image group TMIS and a candidate medical image group CMIS are Each of Determine a similarity metric indicating the similarity between the first vector FV and the second vector SV to determine the degree of comparability between to do, process In BR-8, Based on the determined degree of comparability, Target medical use Imaging As related to data (and, accordingly, as Reference Medical Image Set RMIS) ,select one or more of the candidate medical image sets CMIS. thing, For example, in some cases, the highest CMIS score among multiple candidate medical image sets is similarity A candidate medical image group CMIS having a metric / degree of comparability can be selected as a reference medical image group RMIS. In some examples, the two or more highest similarity among the plurality of candidate medical image group CMISs can be selected as a reference medical image group RMIS. metric / Comparability degree Two or more candidate medical image sets CMIS having set can be selected as the reference medical image set RMIS.

[0276] In some instances, the selection of reference medical image data is 、 Determined Similarity Sex Metrics / Degree of comparability Based further on , i.e., the above body regions and / or Imaging Comparison of modality relevance scores In additionFor example, in some instances, Previous studies (e.g., including multiple candidate medical image sets CMIS) medical use Imaging The data may be collected from the aforementioned body regions and / or Imaging Based on a comparison of modalities, current medical Imaging As related to the data, patients Previous testing for can be preselected from 。 And the preselected inspection From among the multiple groups in the RMIS, the reference medical image group is selected based on the determined similarity. metric / Based on the degree of comparability, Imaging data, which may be selected as relevant to the data, and in this case be shown to the user as display data. reference Image data can be used for comparison of body regions, Imaging Modality relevance scores and determined similarities metric may be selected as relevant to the region of interest based on

[0277] The method described with reference to FIG. 10 involves obtaining image data or regions of interest that are the same as or similar to the image data or regions of interest of the target medical image set TMIS. Imaging Using parameters Capture It is possible to automatically select one or more reference medical image sets RMIS that have been to This is because, for example, patients Regarding Compared to opening and evaluating all previous medical images to ,Provides efficient selection of suitable previous medical images for comparison with the target medical image. do Furthermore, by selecting based on the attribute value AV of the image file IF (which is relatively small in size), Imaging Selection can be made efficiently without extracting or analyzing the data itself (which is relatively large in size). becomes .moreover, Generated based on attribute value AV Vector FV ,SV Allows flexibility in selection based on become For example, comparing vectors FV and SV in feature space is, for example, Imaging Compared to attempts to match parameters directly, there is a greater likelihood of inexact matching between parameters. Robust and / or may be flexible.

[0278] Figure 11 shows An example of 11 illustrates the flow between the components of the method described with reference to FIG. 10. The image file IF of the target medical image set TMIS is Imaging Data, such as the region of interest (ROI) and the target medical image set (TMIS) Capture Used to Imaging Save the attribute A with the attribute value AV indicating the parameter. of Extract the attribute value AV, Vectorizer Provided to 906, Vectorizer 906 is the first of The vector FV is output. The set of candidate medical images CMIS is medical care may be part of the information system 40 Storage In some examples, each of the plurality of candidate medical image sets CMIS is stored in a different set within the reference medical imaging study RS. Each of It may be one or more Second Multiple attribute values ​​AV set (The candidate medical image group CMIS stored in each medical image data file IF that contains them is Capture Used to Imaging parameters Each shows ) are extracted. The set of these attribute values ​​AV is next , vectorizers 906, which each provide a plurality of A second vector SV (one for each of the input set of one or more attribute values ​​AV) Output The first vector FV and / or the second vector SV can be stored in association with the file of the target medical image set TMIS or candidate medical image set CMIS (or its identifier) ​​from which it was generated. Each of the first vector FV and the plurality of second vectors SV can be stored in association with the file of the target medical image set TMIS or candidate medical image set CMIS (or its identifier). The signal is input to a comparator 910, which then outputs Determine a similarity metric between each of the first vectors FV and the second vectors SV. death, Target medical image group TMIS and candidate medical imaging group CMIS and The comparator 910 determines the degree of comparability between the degree of, one or more selections 912 of the target medical image set TMIS or candidate medical image set CMIS (or identifiers thereof) can be output. set of, Storage can be obtained from

[0279] some In the example, as already mentioned, multiple set Candidate medical image group CMIS Remote Storage These can be saved. example wherein the method comprises: Selecting one or more sets of candidate medical images CMIS (or a set of medical imaging data including the selected set of second medical imaging data) from the remote storage; Multiple set Candidate medical image group CMIS other set This provides efficient use of network resources. do .

[0280] some example In This method generates the display data and Display Device (e.g., 11 of (see 、 Region of Interest (ROI) Rendering and , one or more selected candidate medical image groups of CMIS image Based on data Rendering based on the This can include: medical care Experts, patients Regarding Region of interest (ROI) (e.g. Target or current ) and reference (e.g., previous) medical Imaging Provides a visual assessment of the differences between data do The candidate medical image group CMIS is the target medical image Imaging Data / regions of interest (ROIs) are appropriate for comparison do Since it is selected, the user can select the modality and Imaging Parameters Not related to diseases such as Not a difference due to factors, Image group Differences due to disease progression represented by More like You can focus.

[0281] In some instances, This method is the first vector FV and / or one or more (e.g., all) It may include generating a second vector SV.

[0282] In an example where one or more of the first attribute values ​​AV or the second attribute values ​​AV include a text string, generating the first vector FV or the second vector SV, respectively, may include converting the text string into a vector representation. Encoding For example, the text string may include: Word embeddings Encode it into a vector representation using will be For example, word embeddings can map words from a dictionary onto a vector space Map to , where the words in the dictionary and the vectors for each word can be generated by applying a word embedding model to a corpus of training text. Examples of known word embedding models use neural networks to extract word In some cases, for example, large corpora of general-purpose text are used. in Pre-trained is , advance Pre-trained word embeddings may be used. In some examples, training text may be text strings of attribute values ​​from radiology reports, medical literature, and / or training image files. etc. It can contain medical texts. Medical field Words and abbreviations specific to Semantic Meaning (within the context of the training text) Learning Make it possible do If the text string consists of multiple words, each word Word embeddings for combine vectors from, for example, each word in a text string Word embeddings for The vector of join or by taking the average 1st Vector (or 1st You can generate a vector (part of a vector). Other methods may be used.

[0283] In examples where one or more of the first attribute values ​​AV or the second attribute values ​​AV include numeric values, generating the first vector FV or the second vector SV, respectively, may include formatting the numeric values ​​into a vector representation. For example, one or more of the attribute values ​​AV may be numeric. For example, as described above, An example DICOM attribute “ Echo Time "of Example The value is "4.2". In this example, the attribute value is Vector It may be used as an element of FV or a second vector SV. In some cases, attribute values ​​may be optionally 、 1st or 2nd of Normalize before including it as an element of the vector You may For example, Echo time The attribute values ​​may be divided by 10,000 before being included as elements of the vectors FV and SV. As another example, one or more of the attribute values ​​may include multiple numeric values, such as a sequence of values. In such an example, the sequence of values ​​may have one numeric value per element. column vector For example, as mentioned above, An example An example of the attribute value for the DICOM attribute "Image Orientation Patient" is "[1,0,0,0,1,0]". column vector formatted to the first Vector It may be used as FV or as the second vector SV (or part thereof).

[0284] In some examples, generating the first vector FV or the second vector SV includes generating the first or second attribute values ​​AV For each, respectively , a third vector based on the first or second attribute value AV each Generate To do , the first vector FV or the second vector SV combined with the third vector each For example, multiple attribute values ​​can be generated using Series Description ","echo time " and "image direction patient." In this case, the third vector can be For each of these three attribute values, a SD ,v ET,v IOP Next, these three Third Concatenate each of the vectors and add 1st Vector FV or Second Vector SV can be generated 、 For example, [v SD ,v ET ,v IOP ].

[0285] some In the example, similarity Determining the metric / degree of comparability is crucial for each second of For the vector SV, the first of Vector FV and second of The method may include determining the cosine similarity between the vectors SV. As is known, the cosine similarity is the dot product between two vectors, and represents the similarity between the two vectors. For example, a cosine similarity of 1 indicates that the vectors are identical or Very Similar vectors (thus identical or Very corresponds to similar imaging parameters), and a cosine similarity of 0 indicates that the vectors are orthogonal (hence, The exact opposite or Very different In some examples, the first of the target medical image set TMIS / region of interest ROI is shown. of Vector FV and high cosine similarity Offal No. 2 of have a vector SV candidate Medical image group CMIS but , the second with lower cosine similarity of have a vector SV candidate In some instances, other similarity measures, such as the Euclidean distance between a first vector FV and a second vector SV, may be used. metric In these examples, the comparator 910 measures the cosine similarity (or other similarity) between the first and second vectors. Similar Calculation to calculate the similarity measure unit (not shown) may be provided.

[0286] some example In the example, the comparator 910 determines the degree of comparability based on the input of the two vectors FV and SV. Style It is trained to output a value indicating the similarity between two vectors FV and SV. was In some examples, the neural network may include: Similarity Metrics Determining the first vector FV and the second vector SV is performed by inputting the first vector FV and the second vector SV to the trained neural network. To do , thereby obtaining an output from the trained neural network, similarity For example, the neural network may include one or more hidden layers of neurons between the initial layer and the final layer. deep It may also be a neural network. initial The layer can be configured to take as input a first vector FV and a fixed-size second vector SV. The neural network is configured to map the vector representation from the final layer of the neural network to a value (e.g., between 0 and 1) that indicates the similarity between the two input vectors. Regressors (not shown).

[0287] In some instances, the neural network is trained based on a training dataset. And , the training dataset is Multiple pairs of vectors like the above (For example, each vector in a pair has the same format as the first vector FV and the second vector SV, respectively.) Multiple pairs of such vectors ), each versus training similarity value( training degree of comparability) Leveling And, The training similarity values ​​are During neural network training teacher Provides signal do In practice, training involves such labeled training pairs. Hundreds or thousands For example, training can include Deep LearningFor example, training can consist of updating the weights of the connections between layers of neurons in the neural network to obtain Regressors predicted by similarity values ​​and the actual values ​​of each of the training pairs defined by their respective labels. similarity This may include minimizing the error between the values.

[0288] In some cases, training similarity The value labels are, for each training pair of vectors, represents the probability, and this probability is A first medical image having a particular first attribute value represented in one vector of training pairs. Imaging Data Assumed case to , user but , For comparison with the first medical imaging data, A second medical image having a particular second attribute value represented in the other vector of the pair. Imaging data would choose probability is For example, this probability is Imaging The data log may be determined based on a statistical analysis of recorded user interactions with the data. For example, the data log may include: predetermined During the session predetermined by medical professionals predetermined About the patient search Medical Imaging Statistical processing may be applied to these logs to determine the probability. For example, for a set of target medical images TMIS with imaging parameters X, , medical professionals, imaging The parameter It is Y Review previous reference medical image collection RMIS What is the percentage probability that you will try (e.g., 60%)? , and , medical professionals, imaging Parameter Z is Review previous reference medical image collection RMIS What is the percentage probability of trying (e.g., 20%)? In this case, two training pairs can be generated, one with vectors representing X and Y. At the same time 0.6 training similarity one with value labels and the other with vectors representing X and Z At the same time 0.2 training similarity It has value labels, which are used to generate a training dataset. Imaging Many of the parameters combination performed against obtain The training dataset is Imaging It is based on statistical analysis of actual user interactions with the data and is appropriate Imaging Using parameters Capture The second medical Imaging Ensures data is selected It helps to .

[0289] FIG. 12 shows the example Process As a method for performing registration according to S60, we have used the medical image sets TMIS and RMIS. Match Slice Identification The corresponding data stream is shown in Figure 13 。 This method has several Process There is. Process The order of Process Corresponding to the numbering of There is no need , may vary between different embodiments of the present invention. Process or a series of Process can be repeated.

[0290] The target medical image set TMIS and the reference medical image set RMIS relate to three-dimensional image data depicting a patient's body part. Specifically, the target medical image set TMIS includes: subject Image volume specific Each section The image slices S-1 (hereafter, "First of Slice S-1" ) The reference medical image set RMIS similarly includes the reference image volume specific Each section A plurality of image slices S-2 (hereinafter, "Second of Slice S-2" ) may include:

[0291] According to some examples, the target medical image set TMIS and the reference medical image set RMIS relate to the same anatomical structure of the same patient, but were acquired at different times. According to some examples, the target medical image set TMIS relates to a current imaging examination that the user is expected to analyze, and the reference medical image set RMIS relates to a previous examination showing the same body part, but at an earlier time point. By analyzing the target medical image set TMIS in conjunction with the reference medical image set RMIS, the user can therefore infer how a disease has progressed or whether a particular treatment has been successful. To assist the user in this regard, the method of FIG. 12 The subject medical image set TMIS and the reference medical image set RMIS are synchronized or registered in that matching slices CS in the subject medical image set TMIS and the reference medical image set RMIS are identified. .

[0292] Process SLS-10 in , and receives the target medical image set TMIS. Process The SLS-10 provides a user interface 10 Use by users for medical imaging examinations Manually selecting the TS, and / or 、 From Medical Information System 50 Target medical images TMIS search Including nothing .moreover, Process SLS-10 is subject The medical image group TMIS, Cases and tasks at hand To this end, a data identifier, such as a patient or study identifier, may be used to query the medical information system 40 for the appropriate group. Process SLS-10 can be implemented, at least in part, in either the user interface 10 or the processing system 20. Corresponding data exchange, if necessary. but this Included in the process .

[0293] Process SLS-20 in , a reference medical image group RMIS is obtained. This ,The process is shown in Fig. 8. From Figure 9 outlined in relation to Process to continuation, selecting a suitable reference medical imaging study RS from a plurality of candidate medical imaging studies CS, and selecting a reference medical image group RMIS from a plurality of candidate medical image groups CMIS associated with the reference medical imaging study RS.

[0294] Process In SLS-30, slices between the target medical image set TMIS and the reference medical image set RMIS are Match Establish CS and separate slices S-1 of the target medical image set TMIS and, if possible, slices S-2 of the reference medical image set RMIS. Link That is, Process SLS-30 calculates the slice S-1 of the target medical image set TMIS by dividing the slice S-1 of the reference medical image set RMIS. Match Slice CS Just one Therefore, the target medical image group TMIS of Reference medical image set RMIS for a given slice S-1 Match Finding the slice CS It may happen that the opposite is not possible. This is a medical image collection TMIS ,RMIS to Included Anatomical location of slices S-1 and S-2 But the other Medical image groups not covered by RMIS and TMIS It may be the case In this case, according to some examples, Match Slice Association Ha row I can't . Process SLS-30 (and all optional Process ) image processing is mainly performed by the processing system 20 in This can be done.

[0295] In the following, the optional Lower process SLS-31 from SLS-35, Process SLS-30 Match Identification of slice CS How is it implemented? some Illustrating Examples These provide some examples in this regard, should not be considered as limiting the scope of Process SLS-30 How can the similarity between image slices S-1 and S-2 be determined? Another Eliminate the possibility It is not something to doFor example, image descriptors D-1 and D-2 Clear extraction of and their subsequent comparison of Alternatively, a Fourier-based analysis scheme can be used execution It is possible. in this case , individual images or image regions will be transformed into Fourier space and similarity will be calculated by applying a mathematical convolution function.

[0296] Optional Process SLS-31 is The purpose is to resample the reference image set RMIS based on the target image set TMIS, in order to make it better suited for slice-by-slice registration of the next two image sets. This involves creating multiple slices S-2 with appropriate slice thickness and stacking direction in the reference medical image group RMIS. Definition This may include: This process Furthermore, The slice S-2 already existing in the reference medical image group RMIS is They are suitable slice thickness and direction so that Resampling Including nothing In particular, slice S-2 of the reference medical image set RMIS is different from slice S-1 of the target medical image set TMIS in terms of thickness, spacing, slice direction, etc. is comparable to The reference medical image set RMIS may be resampled as follows: Furthermore, , This process is ,The target medical image set TMIS is matched with the reference medical image set RMIS. to make , You can also do it the other way around. .moreover, Process SLS-31 is a system that performs other image processing to improve the comparability between the reference medical image set RMIS and the target medical image set TMIS. Process It is possible. This process is the target medical image set TMIS Image data contained in (For example, in the metadata file of the target medical image set TMIS Encoded ) Ta Image Processing Process The same image processing is performed on the reference medical image group RMIS. Process The method can include applying

[0297] Next Any Process SLS-32 provides a trained function TF that is specifically configured to determine and quantify image similarity between two-dimensional medical images. The purpose is toIn particular, the trained function is Subsequent processes SLS-33 to SLS-35 carry out However, Subsequent processes SLS-33 to SLS-35 teeth ,without using a trained function, i.e., e.g. image To select features from the data and determine similarity based on them, One The above deterministic rules execution has a hard-coded function to Image Analysis Functions By IAF 、 It should be understood that this can also be done.

[0298] Process In SLS-33, an image descriptor D-1 is extracted from each slice S-1 of the target medical image set TMIS. The image descriptor D-1 is in the form of an image feature vector and is a representative feature of the slice S-1. or characterization The target medical image set TMIS may include features. usually , Image descriptors consist of image data and are associated with non-image data. D-1 is similar to the image feature Signature and non-image features It can be based on The image feature signature is present in any image data included in the target medical image set TMIS. Includes identification, analysis and / or measurement of objects, local and / or global structure, and / or texture Generated image features may be generated by image analysis methods. Signature For example, the presence of landmarks 、 or organs the size of, or the structure, texture and / or Image features may include anatomical features and / or structures such as density, etc. Signature Similarly, the colors and / or gray Scale scheme or contrast characteristics or local gray scale gradient The image feature may include a parameter that characterizes the image. Signature is preferably Not just one feature, but many features that together characterize the analyzed image The target medical image group TMIS includes non- image data from The extracted non-image features include metadata associated with the image data. can be seen In addition, they are used in electronic health records, clinical laboratory data, etc. Features extracted from such as , providing further contextual information about the target patient image Related to data independent of data obtain .

[0299] Process In SLS-34, a matching image descriptor D-2 is extracted from the reference medical image set RMIS. The image descriptor D-2 of the reference medical image set RMIS is compared with the image descriptor D-1 of the target medical image set TMIS. In the same way Generate obtain . some According to the example, the image descriptor D-2 of the reference medical image group RMIS has already been generated and is stored together with the reference medical image group RMIS. medical care Stored in Information System 40 will be .

[0300] Process In SLS-35, the extracted image descriptors D-1 and D-2 are used to Match Slice CS Identification The comparison is shown in Figures 10 and 11. figure Introduced in relation to 11 will be Similarity or distance metric The comparison can be done again by applying Paired comparison That is, each slice S-1 of the target medical image set TMIS is compared with each slice S-2 of the reference medical image set RMIS. Alternatively, each slice may have two more similar slices retained for subsequent comparison. other Medical image datasets MIDS-1 and MIDS-2 Each of them Compare two slices, compare in triplets be able to For each slice S-1, S-2 in the medical image set, in order to provide an unambiguous relationship between the slices S-1, S-2 between the two medical image sets, other Medical image datasets MIDS-1 and MIDS-2 each of Match Slice CS Identify only one Or not at all Do not identifyIf the slices of one medical image set are outside the imaging area of ​​each other medical image set, or if the slice resolution of the medical image set is different from that of the other medical image set, e.g. Compared with either slice S-1 or S-2 of If it is too coarse to provide a match (If this is not corrected by optional process SLS-31) , Match Slice Can't find Reference Medical Image Collection (RMIS) and between slices S-1 and S-2 of the target medical image set TMIS The identified allocation of To ensure that each slice order is preserved, and Ambiguity Resolution In order to achieve this, additional constraints or auxiliary conditions that the final result must satisfy may be provided. Such constraints may be provided by the order and / or the overall similarity (which should be maximized) of the slices in the reference medical image set RMIS and the target medical image set TMIS.

[0301] The results of the SLS-30 step are compared with the reference medical image set RMIS. and the target medical image group TMIS An assignment that explicitly assigns slices S-1 and S-2 to each other can be considered This result is a "slice" Match " CS and can also be expressed as In this way, slice Match CS is a form of registration between the target medical image set TMIS and the reference medical image set RMIS. can be considered .

[0302] In step SLS-40, this result is sent to the user or to further processing. Process Step SLS-40 can be performed, at least in part, by either the user interface 10 or the processing system 20. Corresponding data exchange, if necessary, can be performed. but this Process Included be captured .

[0303] FIG. 14 shows some examples. Related to Medical image collection Match Any method that can be implemented based on the method for identifying slices is shown. In particular, the option shown in FIG. of method The process , Process According to SLS-40 obtain . Process The order of Process The numbering of Without , which may vary between different embodiments of the present invention. Process The method introduced and described in connection with the embodiment of FIG. Process Corresponds to.

[0304] In particular, FIG. 14 shows the process SLS Slice determined by -30 If This example illustrates how a match CS can be used to synchronize the views of two medical image sets. Usage examples related to The user has opened the target medical image set TMIS, is looking at a specific slice S-1 or region of interest ROI, and the reference medical image set RMIS Comparison If it is taken as a reference , arise Here, Figures 12 and figure A similar slice search was introduced in relation to 13 to find the target medical image set TMIS. Either Slice S-1 and reference medical image group RMIS Match Slice S-2 and automatically Can be matched ,When ,the ,user ,selects ,one ,slice ,for ,viewing ,e.g. Each match The slice S-2 of the reference medical image set RMIS corresponding to the region of interest ROI is automatically provided as a "reference slice" REF-SLC. Also represented .

[0305] in particular, Process At SLS-50, input is received from a user indicating a selection of slices S-1, S-2 in either the target medical image set TMIS or the reference medical image set RMIS. The user input may be entered into the system 1, for example, via the user interface 10. The user input may include: subject Scroll through one of the medical image sets TMIS or reference medical image sets RMIS to reach the desired slices S-1 and S-2. Scrolling behavior It can be in the form For example, such scrolling may be achieved by means of a scroll wheel or jog dial or wheel, an appropriate graphical user interface, Action button or slider , gestures, and voice commands Using etc. Furthermore, such user input can directly indicate the desired slice that the user wants to view, for example, by clicking on the z-axis bar indicating the z-position in each stack of slices S-1 and S-2, or by directly entering the slice number. In particular, slice S-1 of the target medical image set TMIS can be implicitly selected by defining a region of interest in the target medical image set TMIS. Process SLS-50 may be implemented, at least in part, in either the user interface 10 or the processing system 20. Corresponding data exchange, if necessary. but this Process Included be captured .

[0306] Process In SLS-60, the user selects the slices S-1 and S-2. Each match Slice CS Identification To this end, the method Process Slices determined by SLS-30 Match To CS return Next, slice Match Based on CS, System 1 can analyze other medical image groups. Each Which slice is the one currently selected (viewed) by the user? Match It is possible to calculate whether reference In the case of a medical image set RMIS, this involves identifying the reference slice REF-SLC corresponding to the region of interest ROI. Connect . Process SLS-60 may be implemented primarily on processing system 20.

[0307] Process Identified in SLS-70 Match The slice CS and / or the reference slice REF-SLC are provided. Identification was Match slice CS and / or reference slice REF-SLC ( Identification was Matchand displaying the slices to the user in an appropriate graphical user interface (by generating corresponding display data based on the slices). obtain According to some examples, the synchronized medical image set Identification was Match The slice CS / reference slice REF-SLC can be displayed along with the current selection / region of interest ROI. for example ,The two slices REF-SLC,S-1 can be displayed side-by-side in a suitable graphical user interface. Process The SLS-70 is a user interface 0 more Processing System 2 0's The corresponding data exchange can be implemented, at least in part, in either but this Process Included be captured .

[0308] Process SLS-80 is Process SLS-50 from SLS-70, i.e. receiving user input (SLS-50) , and Matching Slice CS Identification (SLS-60) , Match Slice CS Output (SLS-70) Repeat to indicate that can be repeated multiple times Process Therefore, Match The slice CS can be dynamically adapted to the slice S-1 currently selected by the user. User When a new slice is selected in a set of medical images, Match The slice CS is updated, allowing the user to scroll through two sets of images simultaneously. Process SLS-80 is, at least in part, User Interface 10 or processing system 20. Corresponding data exchange, if necessary. but this Process Included be captured .

[0309] FIG. 15 shows an example of the present invention. Related toMedical image collection Match Optional methods that can be implemented based on the method for identifying slice CS are shown. In particular, the optional method shown in FIG. Process teeth, Process SLS-40 continuation , Process Addition to determine reference medical image group RMIS in S50 scale It can be used as: Process The order of Process The numbering of Without The methodology shown in FIG. 12 may vary between different embodiments of the present invention. Process is the method introduced and described in connection with the embodiment of FIG. Process Furthermore, individual Process or Process The sequence can be repeated.

[0310] In particular, FIG. 15 is a diagram illustrating the configuration of the casing described in relation to FIG. similar Slice search allows comparison of two sets of medical images. There We will explain how slices can be additionally used to infer whether they are suitable for comparative reading by a user. Match Using CS, the imaging volumes of the target medical image group TMIS and the candidate medical image group CMIS are Overlap Determine this. Overlap The larger the value, the more accurate the two medical image sets. well comparison This means that it is possible .

[0311] therefore , Process SLS-90 is Process SLS-40 in Served slices Match Based on CS, Match Anatomical data of medical image groups TMIS and CMIS Overlap Determine The purpose is to . anatomical Overlap The two medical image collections, TMIS and RMIS, Leave each other Match do Each Number of slices Determined fromBy way of further example, anatomical Overlap The two medical image groups, TMIS and CMIS, Overlap Marking the ends of an area Match can be determined from image pairs. Overlap is based on two medical image data sets, TMIS and CMIS. Okeru Measurement can be made in any unit along the stacking direction of the slices S-1 and S-2. Process SLS-90 may be implemented primarily on processing system 20.

[0312] Process The SLS-100 uses two medical image groups: TMIS and The degree of comparability of CMIS was evaluated based on anatomical Overlap According to some examples, the degree of comparability is calculated based on, e.g. the anatomical Overlap By way of further example, additional information may be factored in. Such additional information may be, for example: Has been acquired Two medical image groups: TMIS and Between CMIS time span ( Low comparability Two image groups teeth (Sometimes longer than this), two medical image collections TMIS and Used to retrieve CMIS Ta Each imaging modality type (data acquired with similar modalities may be more Highly comparable TMIS, two medical image collections and CMIS Included in Metadata, etc. Process The SLS-100 may be implemented primarily on a processing system 20.

[0313] Process In SLS-110, the degree of comparability is provided. "Provided" means the determined degree of comparability. Get your users' attention means that obtain If two sets of images are difficult to compare, the user will receive a warning message. receive You can compare Image interpretation For another thingFurthermore, "provided" means that the degree of comparability between the two sets of medical images, TMIS and CMIS, is recorded, i.e., the data is stored for later use. medical care This means that the information is stored in the information system 40. obtain .moreover, "Provide" The degree of comparability is used to automatically select a candidate medical image group CMIS as a reference medical image group RMIS. Process Meant to be used in S50 obtain . Process The SLS-110 is a user interface 0 more Processing System 2 0's The corresponding data exchange can be implemented, at least in part, in either but this Process Included be captured .

[0314] FIG. 16 shows a reference medical image group RMIS and one candidate medical image group CMIS. About A method for determining the degree of comparability of sets of medical imaging data is presented. Process It consists of: Process The order of Process The numbering of Without , may vary between different embodiments of the present invention. figure of the method indicated by similar reference numbers such as 15 Process of the methods introduced and described in connection with each example. Process Furthermore, each Process or a series of Process can be repeated.

[0315] The embodiment shown in FIG. It is a problem From multiple candidate medical image sets CMIS of Comparative Reading About In order to provide a method by which a reference medical image group RMIS can be automatically evaluated and / or selected, the above-described embodiment It is based on .

[0316] First stepSLS-120 receives the target medical image set TMIS The purpose is to The target medical image group TMIS is Trying to read Considered as a current examination for patients can be Apart from that, Process SLS-120 Process Equivalent to SLS-10.

[0317] Process SLS-130 searches for multiple candidate medical image groups CMIS that are problematic for comparative interpretation along with the target medical image group TMIS. The purpose is to This includes: Process In relation to the S40 Explained Reference Medical Imaging RS Identification This includes obtain Alternatively, the candidate medical image set CMIS can be In another method, For example, candidate medical imaging CS Each test Related to do For suitable candidate medical image sets CMIS, medical care By directly querying the information system 40 、 For example, suitable case or patient identifiers, such as patient names or IDs, can be extracted from the target medical image collection TMIS and used as search terms. Process The SLS-130 is a user interface 、 or processing system 20 、 or Medical Information System 40 of The corresponding data exchange can be implemented, at least in part, in either but this Process Included be captured .

[0318] Next, for each of the retrieved candidate medical image sets CMIS, Process Repeat SLS-30, SLS-100, and SLS-110 Process SLS-140 Through Repeat. These Process may be executed primarily on the processing system 20. As previously mentioned, Process SLS-30 is the target medical image group TMIS andReference Medical Image Set RMIS each Slices between Match CS Identification do. This , Process SLS-30 will be repeated for each candidate medical image set CMIS in particular. Lower process Contains SLS-31, SLS-34 and / or SLS-35 can be seen . Process Not related to candidate medical image sets CMIS, such as SLS-32 and / or SLS-33 Other optional steps S130' Preferably, the candidate medical image set CMIS Every 16. Steps SLS-100 and SLS-110 are not repeated but are performed only once in the method of FIG. 16. Steps SLS-100 and SLS-110 are shown in conjunction with FIG. Process is identical to Produce a degree of comparability for each candidate medical image group CMIS, which indicates the comparability of each candidate medical image group CMIS to the target medical image group TMIS. .

[0319] process In the SLS-150, the degree of comparability may be provided to a user or provided for further processing. For example, the degree of comparability may be calculated based on the candidate medical image set CMIS. Each of , which can be displayed to the user along with candidate The user is asked to select which medical images in the CMIS are suitable for follow-up reading. Overview can be provided. therefore , the user must select the appropriate candidate A medical image set CMIS can be intentionally selected as a reference medical image set RMIS. be put in a position . process SLS-150 may be implemented, at least in part, in either the user interface 10 or the processing system 20. Corresponding data exchange, if necessary. but this process Included be captured .

[0320] Furthermore, the degree of comparability is process The SLS-160 may be used to automatically select or pre-select suitable candidate medical image sets CMIS as reference medical image sets RMIS. For example, the system 1 may select or pre-select suitable candidate medical image sets CMIS with a degree of comparability (anatomical or physical) greater than a predetermined threshold. Overlap )of candidate Medical image collection CMIS Everything You can select it. candidate The medical image set CMIS then process Available in S50 will be Therefore, " offer " means, Presenting them to the user via the user interface 10 for review and further selection and / or loading them into temporary memory of the system 1. , meaning obtain . process The SLS-160 has a user interface of 10 、 or processing system 20 、 or medical information systems 40 The corresponding data exchange can be implemented, at least in part, in either but this process Included be captured .

[0321] FIG. 17 is a diagram illustrating a method for manufacturing a cellular phone according to one or more of the above embodiments. similar Use slice search to identify the temporal location of one or more lesions. development Any method for determining process This method shows several process It consists of: process The order of process The numbering of Without may vary between different embodiments of the present invention. process of the method introduced and described in connection with each example. process Furthermore, each process or a series of process can be repeated.

[0322] process The SLS-170 is a system that extracts one or more relevant slices from the target medical image set TMIS. The purpose is to identify The relevant slices are taken from the patient's lungs. or one or more lesions in a patient's body part, such as lesions in liver tissue drawing Additionally or alternatively, connection A slice can be, for example, One or more Presence of lesions point out , user's One or more Previous annotations may include Characterized by points can be Relevant slices are identified by applying a suitable computer-aided detection algorithm adapted to detect lesions in the medical image data, for example. can be Additionally, the relevant slices are displayed according to the user's One or more Related to Region of Interest (ROI) obtain . process SLS-170 may be implemented primarily on processing system 20.

[0323] process The SLS-180 detects one or more lesions in one or more relevant slices, for example, by using the computer-aided detection algorithm. The purpose is to . First, by detecting the slice with the lesion at the location If the relevant slice is identified, process SLS-180 is process S510 Included in obtain. process SLS-180 may be implemented primarily on processing system 20.

[0324] process In the SLS-190, Match The slice is in the reference medical image set RMIS. Identification And, Match The slices correspond to the related slices. Match Slice Identification teeth, process Calculated in SLS-30, Fig. 12 process Slices provided in SLS-40 Match It is based on CS. process SLS-190 may be implemented primarily on processing system 20.

[0325] process In the SLS-200, one or more corresponding lesions process Identified in SLS-190 Match This is also detected in the slice. Match Slice imageThis can be done by applying computer-aided detection algorithms to the data. process SLS-200 may be implemented primarily on processing system 20.

[0326] process Lesions detected in the target medical image set TMIS using SLS-210 but , with the corresponding lesion To be matched In other words, the lesions in the target medical image set TMIS are unambiguously assigned to the corresponding lesions in the reference medical image set RMIS, so that Lesion and corresponding lesions The pair of Obtained Especially Lesions not assigned in this way are unpaired and, in some cases, designated new or lost lesions. Such new or lost lesions may be, for example, process Displayed on S70 image data In By outputting the appropriate markings, Optionally The user can be alerted to the lesions, their position in each slice, relative to each other or to other anatomical structures By assessing their relative position, their shape, their properties (degree of calcification, structure at the borders, etc.), size, optical properties, etc. matching It can be done. process Lesions in SLS-210 of matching Regarding , a specified computer-assisted lesion matching algorithm can be applied. process SLS-210 may be primarily executed on processing system 20.

[0327] process SLS-220 is process SLS-210 matching Based on the above, the temporal relationship between one or more lesions detected in the reference medical image group RMIS and the target medical image group TMIS is development Determine The purpose is to For this purpose, one or more measurements are extracted for each lesion, MatchingFor example, such measurements may relate to geometric values ​​such as lesion size, diameter, or volume. obtain . The temporal development is Showing how a particular lesion has progressed Get results Furthermore, the mean / temporal values ​​of the various lesion pairs were development may be calculated. process SLS-220 may be implemented primarily on processing system 20.

[0328] process In SLS-220, the time period determined in this way development Specifically, the time development can be displayed, for example, in the form of a trend graph or by the user over the lesion. When hovered (e.g., with the mouse cursor) Open to Pop-up windows Furthermore, the time-varying development teeth , them By relating to individual lesions can be provided Additionally or alternatively, temporal Development is medical Information Systems 40 to Archived in structured medical reports To be offered, i.e., to be put in It is possible. process The SLS-220 is a user interface 0 or more Processing System 2 0's The corresponding data exchange can be implemented, at least in part, in either but this process Included be captured .

[0329] If you add ,The above is not only applicable to two sets of medical images, One or more temporal developments from, Two or more time points It also shows the progression of individual lesions over time. Regarding temporal development The automatic extraction of correct It is compatible with the automatic selection of the reference medical image set RMIS. Specifically, first, the target medical image set TMIS and sufficient anatomical Overlap One or more reference medical image sets RMIS are determined, each having next , Target medical image group included in TMIS Temporal progression of the lesion Development and The reference medical image group RMIS thus identified is guess It can be thought that this will be the case.

[0330] As already described in detail in connection with other embodiments, in the medical image groups TMIS, CMIS, RMIS Match Processing to identify slices process may be implemented by a trained function TF configured to determine similarity between two-dimensional medical images.

[0331] Generally speaking ,Such a trained function TF is learned task According to image Suitable for classifying data intelligent agents or related to classifiers do They are a pair of two-dimensional medical images. image It can predict how similar the data are Some kind of relating to a method or apparatus obtain This definition includes, but is not limited to, support vector machines, decision trees, naive Bayes or Convolution ) Data mining tools and techniques such as neural networks. Specifically, Example According to the,trained function TF is, Convolution In one embodiment, the trained function TF arrangement is completely Convolution Neural Networks. Alternative Networks arrangement For example, 3D Very Deep Convolutional Network (3D-VGGNet) can be used, this VGGNet is Stacking many layer blocks, including narrow convolutional layers followed by max-pooling layers . Another embodiment According to the skip connection , or a shortcut that jumps over several layers, Use residual Neural Network (ResNet) can be used .

[0332] Convolution A neural network is defined in terms of a number of successive layers. "continuous" is used to show the general flow of how feature values ​​output from a layer are input to the next layer. the This is fed to the next layer, and so on, until the final output layer. It can be feedforward only, or along the direction, including some feedback to the previous layer. There are also layers Weighting Furthermore, each layer is generally weighted. too Many node Essentially, each node is a mathematical function that maps one or more input values ​​to an output value. Calculation The nodes in each layer are Just before and / or may connect to all or only a subset of the nodes in subsequent layers. The two nodes are If the input and / or output are connected, "connection" The input layer node The input value for is Each area of ​​interest The image element values, preferably pixel values, of the ROI or slices S-1 and S-2 are used. The last layer is the output layer, which outputs the similarity between the input image data. The output is a continuously changing value indicating the similarity. form According to another example, the output (similarity) can be in the form of a binary value indicating whether the two images are similar or not. Possible Between the input and output layers there are a number of hidden layers. The first group of neural network layers can be applied to extract features from images. In this case, medical images, That is, , the image individual Image element Every The grayscale and / or color values ​​serve as inputs to a neural network. 、 Contrast, gradient, Texture , density, etc. Such characteristics are , The second group of network layers (also known as a classifier) ​​as inputs, These network layers are: Further assigning objects and / or features to at least one of the extracted features present in the image. It functions like Convolution, pooling (e.g., max value Pooling or averaging value Pooling), upsampling, Deconvolution , Fully connectedVarious types of layers may be used, such as , or other types of layers. Convolution The layer takes the input as Convolution , and the result 、 In the next layer, we move the image filter kernel over the input. Transfer . Pooling Layers are the nodes that are connected to the output of a cluster in one layer and then to the output of a cluster in the next layer. One To the node combine This allows data size This reduces the σ, thereby making the underlying computation more efficient. and deconvolution Layers are abstractions transformation In terms of level, Convolution and Pooling Layer action Reverse the. Fully connected Layers connect every node in one layer to every node in another layer, essentially ensuring that every feature receives a "vote." One Example According to the,skip connections can be used, so that the layer, It is possible to output continuously to layers other than the next layer, One Residuals above We can introduce blocks or layers. Such a structure is also called ResNet. Using residual blocks, this 、 known from very deep neural networks Vanishing gradient Ability to train deeper networks as the problem is alleviated You can get .

[0333] According to some examples, the trained function TF extracts image descriptors from two-dimensional images. Tasks (also known as encoding) and compare the image descriptors of these images By (also called decoding) , assessing the similarity between two images (i.e., determining similarity); The method can be configured to perform the following. therefore , the trained function must be at least Encoder Branch and decoder branch Including can be seen .

[0334] Following some examples, the trained function can be implemented using multiple encoders. branch and decoder branch Each EncoderThe branch is capable of processing a two-dimensional image to extract image descriptors D-1, D-2 therefrom. Decoder Branch teeth, Encoder Branch The image descriptors D-1 and D-2 obtained from Merged Processing latent image descriptor data structures. Processing multiple 2D images. Encoder Branch are copies of each other sharing the same parameters. branch is the convolution layer In principle, it can be used with a CNN backbone such as ResNet or Custom Design The encoder can be used. branch The weights of the encoder branch This allows for efficient training and processing of each layer of the network. Encoder Branch can share the same parameters, i.e., all encoders branch Same for Weight This means that you can use Weight By sharing and modifying , during training Multiple Encoder Branch The concept of sharing parameters between Siam It's called a copy Sometimes Specifically, two Encoder Branch A network containing Siam It can be called a network, while three Encoder Branch A network containing Triplet It can be called a network. Encoder Branch is larger in the feature space of image descriptors than the image descriptors D-1 and D-2 of a similar image pair. Dissimilar Image descriptors D-1 and D-2 of an image pair is pushed further away It can be configured as follows.

[0335] In the decoding branch Processing the image descriptors D-1 and D-2 involves using learned or preset distance or similarity metric is applied to evaluate the similarity between two image descriptors D-1 and D-2. obtain . Similarity Metricscan be constructed such that the distance between two image descriptors D-1 and D-2 is appropriately quantified in the feature space of the image descriptors D-1 and D-2. metric can take the form of a mathematical function, for example, by outputting the cosine similarity or L1-norm for each image descriptor D-1, D-2. metric is one with properly adapted weights or parameters. The above By the network layer implementation It can be done.

[0336] Generally speaking ,The trained function TF in this embodiment is used to calculate the individual layers and node of Weight or The weighting parameters are By adapting Learning: Potential similarities in medical images sign Instead of pre-programming the TF, we define the architecture of a trained function TF, which learns these patterns at different levels of abstraction based on the input data. With teacher For learning follow It is preferable to train the algorithm using a backpropagation method, which is well established and is the basis of the present invention. Example During training, we apply the trained function TF to the training input values ​​and calculate the corresponding output values, whose target values ​​are known. Generate What is the difference between the generated output values ​​and the target output values ​​(e.g., in the form of the mean square error (MSE) of the difference between the generated and target values)? Good or bad The trained function TF is It works As a measure of the training performance, a cost or loss function can be introduced. The goal of training is to find a (locally) minimal loss function by iteratively adjusting the weights of the trained function TF, which ultimately enables the trained function to produce acceptable results over a (sufficiently) large training data cohort. This optimization problem can be performed using stochastic gradient descent or other approaches known in the art.

[0337] As a rule, one or more Encoder Branchand Decoder Branch The trained function TF, which includes encoding (i.e., extracting image descriptors) or Decoding (i.e., quantification of image similarity) or both. For example, one or more decoder Branches are Exceptionally The decoder can be adapted to extract meaningful image descriptors D-1 and D-2. branch is an appropriate analogy metric may be learned and / or trained to be applied.

[0338] According to some examples, Loss function is a triplet Loss function It can contain triplets loss is the baseline (anchor) input when the positive (similar) input and Negative ( dissimilar ) compared to the input loss The function minimizes the distance from the baseline (anchor) input to the positive input and maximizes the distance from the baseline input to the negative input.

[0339] According to an embodiment, a general training scheme may take the following form: First, a trained function TF is received. The trained function is a function that has already been trained. Dolphin , or not trained at all do .next, training You can provide a dataset, training dataset teeth, Multiple images and training Similarity between images in an image dataset Show Predetermined similarity and In particular, the images contained in the training image dataset are 1 person Extracted from the medical images of the above patients It can be something that Then, the trained function TF training Images from the image dataset versus To determine the similarity between The images in the training image dataset are input to the trained function TF.The similarity thus determined can be compared to a predetermined similarity. Finally, based on this comparison, the trained function TF is adjusted, and the adjusted trained function TF can be provided for further training or deployment.

[0340] Additionally or alternatively, FIG. 18 illustrates a specific method for training a trained function TF to determine similarity between two-dimensional medical images. process Shows. process The order of process The numbering of Without , may vary between different embodiments of the invention.

[0341] First step T10 is Pre-trained or untrained Receive the trained function TF The purpose is to: The trained function TF is received from memory (for example, in the form of a library of KI models). 、 In temporary memory keep It can be done.

[0342] After that process T20 aims to provide a training image dataset including at least three two-dimensional medical images, each representing an image volume of a patient's body part. drawing The medical images are extracted from one or more medical image groups RMIS, CMIS, TMIS. and Easily learned Trained Functions It is desirable that the slices S-1 and S-2 processed by the TF are of the same type. therefore , medical imaging teeth ,each Similarly, cross-sections of a patient's body parts can be shown, depicting multiple anatomical structures and organs; As mentioned above medical use Imaging acquired using one of the modalities are In some examples, the two-dimensional medical images are extracted from the same medical image set. No. 2 medical images , 1st For medical images Third Medical images have similaritythan expensive Characterized by similarities (So ​​selected) That is, the first of Medical images are anchors It is perceived as , 2nd of Medical images are positive (positive) Image is the third of Medical images are negative (negative) This is an image.

[0343] To be continued process In T30, the first, second, and third medical images are input to the trained function TF; process In the T40, The trained function TF is A first similarity between the first medical image and the second medical image and a second similarity between the first medical image and the third medical image are determined. Specifically, the trained function TF determines a first similarity between the first medical image and the second medical image and a second similarity between the first medical image and the third medical image. each , image descriptors D-1, D-2 can be extracted. In other words, the trained function TF can classify the first, second, and third medical images as Encode each Subsequently, the trained function can be applied to the extracted image descriptors each The first and second similarities are calculated by comparing can be guessed In this regard, the trained function TF is encoding space in Image Descriptors each It is possible to output the distance Similarity Metrics Apply obtain .

[0344] Next, the subsequent process In T50, the trained function TF is of Similarity is second of The similarity is adjusted to be larger than the encoding , i.e., the process of image descriptor extraction, and / or Decoding , i.e., appropriate similarity metric The process of applying Adapt However, according to some examples, similarity metric For example, Euclid Fixed as distance or cosine similarity Hold atIn that case, the trained function Conformity is either , One or more encoding branches Focus on It becomes something .

[0345] FIG. 19 shows the example process As a method for performing registration according to S60, a method for determining the location where a given feature is represented in medical image data is shown. An example medical use Imaging Data expression Figure 20 ~Figure Shown in Figure 23.

[0346] Figure 20 shows a representation of the target medical image set TMIS. ~Figure 23 show the representation of the reference medical image set RMIS. ~Figure As can be seen from 23, the target medical image set TMIS and the reference medical image set RMIS may differ. The group , There is a difference in the location of the area from which imaging data was captured for the patient...the area in question is offset to the right in the reference medical image set RMIS in Figures 21 to 23 compared to the target medical image set TMIS in Figure 20. In addition, the slices of the target medical image group TMIS and the reference medical image group RMIS match. Not yet , or slightly different body Area covered are In this sense, Figure 20 ~Figure 23 image planes Regarding , the target medical image set TMIS and the reference medical image set RMIS are vertically misaligned. There is a difference between the two in that Nevertheless, Features PM-226, PM-242 specific pattern but , which are expressed in both the target medical image set TMIS and the reference medical image set RMIS. are In this example, Imaging The predetermined feature PM-226 represented in the data is a lesion PM-226 in the patient's right lung. In one example, a predetermined The feature is medical Imaging data In be expressed Some kind of Features, for example, Imaging Target be Possible specific parts (e.g., including internal cavities) is a matter of course .

[0347] Usage example So (for the sake of explanation) ,The physician reviews the rendering of the target medical image set TMIS (as shown in FIG. 20). is doing Specifically, the physician reviews the region of interest (ROI). is doing Physicians compared the RMIS with the reference medical image group. predetermined Evaluate features, e.g., progression of lesion PM-226 trying to In the target medical image group TMIS predetermined Pattern PM-226 is represented It has been Location PM-224 is known However, predetermined Pattern PM-226 is represented in the reference medical image set RMIS. It has been The location is Not known , the doctor must check visually only teeth Difficulty or burden is The method shown in Figure 20 is predetermined Features PM-226 reference medical image group RMIS In expression It has been Determine the location.

[0348] Referring again to FIG. 20, the method includes: overview In this regard, including: process PM-10 In obtaining a first local image descriptor for a first position PM-224 in the target medical image set TMIS, the first position PM-224 being a position in the target medical image set TMIS at which a predetermined pattern or feature PM-226 is represented; can be , the first local image descriptor is the value of an element PM-222 (i.e., pixel or voxel) of the target medical image set TMIS located relative to a first position 224 according to a first predefined sampling pattern. Represents ; process In PM-20, multiple candidate second-order images in the reference medical image set RMIS were of Locations PM-334, PM-340, PM-440, and PM-448 About Obtain the second local image descriptor thing,Each second local image descriptor corresponds to a first predefined sampling pattern. Therefore Candidate No. 2 of Position PM-334, PM-340, PM-440, PM-448 each represents the values ​​of elements PM-332, PM-338 (i.e., pixels or voxels) of the reference medical image group RMIS located relative to PM-30 In , multiple candidates Second For each of positions PM-334, PM-340, PM-440, and PM-448, No. 1 Descriptors of and Candidate No. 2 Locations PM-334, PM-340, PM-440, PM-448 No. 2 Between the descriptors Local Image Similarity Show Local image Calculate the similarity metric thing ; process In PM-40, calculated local images Similarity Metrics Based on Multiple candidate second positions PM-334, PM-340, PM-440, PM-448 The second candidate position PM-334, PM-448 of Select thing ; process In PM-50, the predetermined feature PM-226 is expression The positions PM-334 and PM-446 are the second candidates selected. of Determined based on locations PM-334 and PM-448 thing .

[0349] Therefore, the predetermined pattern or feature PM-226 is Imaging Data TMIS ,At RMIS Techniques for determining expressed positions PM-334 and PM-446 but Specifically, predetermined The pattern or feature 226 is represented in the first set of medical images TMIS. are Using the known position, predetermined The pattern or feature is a second, e.g. Previous , In the medical image group, the patient's reference medical image group RMISThis determines the location to be represented, for example, in the reference medical image set RMIS. In the specified This reduces the burden on doctors in finding the location where feature PM-226 is expressed. image The data is process This can be the basis for generating display data in S70.

[0350] Furthermore, this decision is based on determining the similarity between local image descriptors for a known location where a pattern or feature PM-226 is represented in a reference medical image set RMIS and for each of a number of candidate locations in a target medical image set TMIS, which can provide fast, efficient, and / or flexible feature location.

[0351] The predetermined pattern or feature PM-226 can be obtained based on a region of interest ROI. specified The pattern or feature PM-226 is included in the region of interest (ROI). image In some instances, the data can be obtained. predetermined The pattern or feature PM-226 is placed in the region of interest (ROI). Matches .

[0352] For example, known No. 1 Location PM-224 and Candidate No. 2 Location PM-334 ,PM-340 to Related topics Determine the similarity between image descriptors That is By optimizing the cost function, hand This can be computationally significantly cheaper compared to image registration techniques where every pixel / voxel in one image is mapped to every pixel / voxel in the other image. Thus, for a given computational budget, The present disclosure The method is significantly faster than methods based on image registration. Ku This may then allow for real-time or near real-time interaction with the image data.

[0353] a predetermined distribution or sampling pattern in Elements located relative to a predetermined position PM-224 (e.g., Pixel or voxel )value Based on descriptors related to This allows us to understand the surrounding and spatial context of feature PM-226. local In image descriptor Encoding This allows predetermined Pattern or feature PM-226 The position expressed in the reference medical image set RMIS is , determined in a reliable, efficient and / or flexible manner. That ,provide.

[0354] For example, such a local image descriptor may be According to image registration techniques All pixels in an image of Rather than attempting to map to pixels in another image, the pattern or feature and the surroundings of the candidate location of interest are Encode . therefore , target medical image set TMIS and reference medical image set RMIS of The image Relatively different (e.g., the bodies they depict whole area In the case Even if there is , specified The location where the pattern or feature is represented in the reference medical image set RMIS can nevertheless be reliably determined (e.g., typically, to attempt to map every pixel between images, Relatively Limited to similar images, Conventional (compared to image registration techniques). Therefore, The present disclosure The technique provides accurate results for a wider range of images and can therefore be applied more flexibly.

[0355] As another example, at the first known location PM-224 Related topics Image descriptors and candidate Second position PM-334 ,PM-340 Between local Determine image similarity That is, Landmark detection-based methods Like , The classifier To detect in the image trainedThere is no need to rely on the presence of "landmarks" in the medical image sets TMIS and RMIS. therefore , The present disclosure The method is: method It may be more flexible with respect to the types of medical image sets to which it can be effectively applied. local By doing so based on the similarity between image descriptors, The present disclosure Techniques include, for example, landmark detection Base method Like classification vessel training It has been Not a landmark, Either It can be applied to certain features. Therefore , The present disclosure The technology provides accurate results for a wider range of features, therefore , can be applied more flexibly. By determining the similarity between descriptors, Imaging data In the specified The position where the feature is expressed teeth , landmark detection Base technology Like Trained Classifier vessel Decide without using To be able to , therefore, classification vessel The time and effort involved in preparing the training dataset for classification vessel The computational load of training savings It is possible. therefore , The present disclosure The method is medical Imaging data specified in Features of PM-226 can be efficiently determined where .

[0356] As mentioned above As mentioned above, the method process In PM-20, the target medical image group TMIS The first in Location PM-224 The first regarding This involves obtaining local image descriptors. 1st The position PM-224 is a position in the target medical image set TMIS where a predetermined feature PM-226 is represented. The first local image descriptor is calculated by calculating the value of the element PM-222 of the target medical image set TMIS located relative to the first position PM-224 according to a first predefined sampling pattern. represent.

[0357] In some instances, 1st The local image descriptors are 1st The descriptor model may be output from a descriptor model applied to the target medical image set TMIS for location PM-224. in advance According to the defined sampling pattern, specified At location PM-224 Regarding Based on the value of the element it is located at, specified The image processing unit 100 may be configured to calculate a local image descriptor for the position PM-224.

[0358] In some instances, 1st The local image descriptors may be obtained from a database (not shown). For example, 1st The local image descriptor for location PM-224 has already been calculated (e.g., by applying a descriptor model), e.g., 1st Stored in the database in association with location PM-224 are For example, the database may include a first local image descriptor determined by This became the basis for medical use Imaging a plurality of first local image descriptors each associated with a corresponding first location in the data; keep It is possible. therefore , some example In the method, the method may include selecting a first location PM-224 from the plurality of locations and extracting a first descriptor associated with the selected first location PM-224.

[0359] In either case, for a given position PM-224 local Image descriptors are made up of multiple entry may be a vector containing entry is one The above element set Value of represents , one The above element set teeth, According to a first predefined sampling pattern For the given position PM-224 Located For example, each entryis a plurality of predefined boxes PM-223 (i.e., rectangle Area) Within each of one or more The value of the element at the position Representable . medical use Imaging Data in three spatial dimensions Quit It will be appreciated that in some cases the term "box" as used herein may refer to a cubic area or volume.

[0360] In some instances, each Descriptor entry Predefined multiple boxes PM-223 Within each Set the value of the element at Representable .for example, Each local Image descriptor entry multiple predefined boxes Within each of the PM-223 It may be the average of the values ​​of the elements located at each entry is the sum of the values ​​of the elements located in a particular box PM-223 in , divided by the number of elements included in the box PM-223. For example, as illustrated in FIG. 20, for the first position PM-224, a specific sampling pattern is used to distribute the slices of the target medical image set TMIS. did Multiple in advance Defined box PM-223 (i.e. fictional area) exists. 1st For location PM-224 1st The local image descriptor may be a vector, entry teeth each , box PM-223 Within each It is the average value of the elements of the target medical image group TMIS located at (e.g., compared to the total) , each vector, regardless of the size of the box in which it is calculated entry are in the same range stipulate More details below. explanation So, this is next , robust and / or reliable similarity between descriptors decisioncan help provide

[0361] In some examples, the predefined pattern and / or the predefined boxes (e.g., the size and / or aspect ratio of each box) may be randomly or pseudo-randomly generated. In some examples, a large number of boxes PM-223, e.g., 1000 boxes, may be used to generate a PM-223. local Image descriptors can be determined, therefore , local The image descriptor can be a vector with many entries (e.g., 1000 entries). For example, referring briefly to Figure 23, For illustrative purposes, To determine a local image descriptor for a given location (not shown) in the medical image set TIMS, RIMS: Many Predefined boxes (white rectangle A set of medical images TIMS is presented, to which the TIMS technique (shown as contours of the TIMS technique) has been applied.

[0362] The local image descriptors are specified The characteristics of specified Location of PM-224 space Context Encoding can be, Therefore, , specified It is possible to provide a compact representation of the surroundings of the features of local Computing image descriptors can be relatively computationally inexpensive and fast compared to relatively dense feature representations, such as may be used in landmark detection techniques. This can be useful, for example, to enable methods to be implemented (and thus results returned) quickly. Helpful .

[0363] In some examples, the descriptor model for computing local image descriptors can be applied to "raw" medical image data. However, in other examples, the descriptor model can be applied to integral image data (e.g., Integration Area Table (also known as the integral image In the data, predetermined The value for the element is image In the data All elements with values ​​above and to the left of a given elementFor example, the target medical image set TMIS Regarding Integral image data is generated , 1st Local Image Descriptors teeth ,Target medical image group TMIS Regarding Integral imaging data Based on It can be calculated. integral The use of image data allows for faster computation of local image descriptors. In some instances, this is next , allowing the method results to be returned faster Helpful .

[0364] In the example where integral images are used, the diagonal position (x 1 ,y 1 ,z 1 ),(x 2 ,y 2 ,y 2 ) have box The sum of the element values ​​of PM-223 is the corresponding integral image I Regarding (I(x 2 ,y 2 ,z 2 )+I(x 2 ,y 2 ,z 2 )+I(x 2 ,y 2 ,z 2 )+I(x 2 ,y 2 ,z 2 )-(I(x 2 ,y 2 ,z 2 )+I(x 2 ,y 2 ,z 2 )+I(x 2 ,y 2 ,z 2 )+I(x 2 ,y 2 ,z 2 Given by In this formula, I(x i ,y j ,z k ) is the value of the element of integral image I at location x=i, y=j, and z=k, where i, j, and k are element indices. For a particular box PM-223, this sum may be divided by the total number of elements contained in box PM-223 to calculate the average element value of box PM-223. The average element value of each box is 1st The vector that makes up the descriptor In each entry Used as will be .

[0365] In some instances, other than the specific examples mentioned above local It is understood that image descriptors may be used. For example, in some examples, Haar-like descriptors may be used. And , i.e., each entry corresponds to one of a number of boxes defined in the image data. Inside represents the difference between the sum of the element values ​​of is a descriptor In some instances, local The image descriptors are, for example, entry is the number of regions of the medical image data. In each It may also be a gradient descriptor representing one or more image gradients, for example: specified The image gradient in the region is specified In some examples, the local image descriptors may be based on the variation of values ​​(e.g., intensity values) between elements within a region of each entry However, for the first position PM-224, Imaging Randomly in the data distribution Multiple elements to be Each In some examples, predetermined The descriptor for each position is entry is a set of randomly oriented rays each ray originating from a predetermined position of Each and The set of intersecting element values ​​may be: predetermined The descriptor for the location is 1st According to a predefined pattern predetermined In position On the other hand The value of the medical imaging data element at the location Represents .

[0366] Nevertheless, we have chosen to use a predefined (e.g., randomly generated) distribution pattern. Leave Multiple predefined boxes positioned relative to a given location PM-223 of respectively The value of the positioned element (e.g., average)The use of local image descriptors for a given location Although particularly fast Providing accurate positioning I'm checking What is specified is attracting attention .

[0367] In some examples, local image descriptors are calculated or otherwise obtained. 1st The location PM-224 is specified by the user. obtain For example, a representation of the target medical image set TMIS may be displayed to a user, who may select, for example, specified The characteristics of PM-226 are expressed 1st By clicking on the expression at location PM-224, Interested in The position of feature PM-226 can be specified as PM-224. User The specified position 1st This can be at position PM-224. 1st Based on location PM-224 1st Local image descriptors can be calculated or obtained in other ways.

[0368] In some instances, 1st Position PM-224 on the computer is performed by For example, the set of target medical images TMIS may be output from the method by identifying a predetermined pattern or feature PM-226 in the set of target medical images TMIS. and specified The characteristics of PM-226 are expressed 1st Output location PM-224 executed by a computer that By a method (not shown) , which has been pre-processed This output can be provided directly and / or stored in a database. keep Next, this 1st Based on location PM-224 1st Local image descriptors can be calculated or obtained in other ways.

[0369] In some instances, 1st Location PM-224 is in the database (not shown) For example, the database can be obtained from One or more features are represented in each of one or more medical image datasetsOne or more locations can be stored. A set of target medical images TMIS can be extracted from the database along with one or more locations. but ,for example, predetermined Features PM-226 reference medical image group RMIS In To determine the position to be expressed request Or based on the instruction, it can be selected as the first position PM-224. 1st Based on location PM-224 1st Local image descriptors can be calculated or obtained in other ways.

[0370] In any case, the target medical image group TMIS 1st For location PM-224 1st Local image descriptors are obtained.

[0371] Description As I did, process In PM-40, the method uses a reference medical image set RMIS Multiple candidates for the second position PM-334, PM-340 For each of the second local This includes obtaining an image descriptor.

[0372] Each second local image descriptor is a first in advance According to the defined sampling pattern, each candidate of The second local image descriptor represents the values ​​of elements PM-332 and PM-338 of the reference medical image set RMIS located at positions PM-334 and PM-340. predetermined Position (e.g., first position PM-224 or second candidate positions PM-334, PM-340) Either 1) Local image descriptors are , for a given location according to a first predefined sampling pattern Located Associated medical use Imaging Data predetermined Element Value Represents, In that sense, No. 1 For example, the first local image descriptor for the first position PM-224 may be the same as local The same method was applied to the target medical image set TMIS to generate image descriptors. localThe image descriptor model is applied to the reference medical image set RMIS to obtain a second position for each of the plurality of candidate second positions PM-334, PM-340. local Image descriptors can be generated. For example, referring to Figures 20 and 21, For each candidate second position PM-334, PM-340, a second descriptor was calculated using the Boxes PM-332, PM-338 and the location of each of these boxes is used to calculate the first descriptor for the first location PM-224. It is the same as any type of descriptor is used Even if , for the first position PM-224 in the target medical image set TMIS descriptor , and the second medical Imaging In Data 330 Multiple Each of the second candidate positions PM-334 and PM-340 descriptor is obtained.

[0373] As will be explained in more detail below, in some examples (e.g., described below with reference to FIG. 21), each candidate second of Locations PM-334 and PM-340 are Each of the previously detected features PM-226 and PM-242 is represented in the reference medical image set RMIS. In other examples (e.g., as described below with reference to Figure 22), candidate The second location PM-440 is a second medical device in a second predefined pattern. Imaging Through the data It can be a distributed location .

[0374] In any case, multiple candidate second-order images in the reference medical image set RMIS of A second local image descriptor is obtained for each of locations PM-334, PM-340, and PM-440.

[0375] As mentioned above Thus, the method process In PM-60, multiple candidates Second For each of positions PM-334, PM-340, and PM-440, No. 1 of local Image descriptors and , second local image descriptors for candidate second positions PM-334, PM-340, PM-440 Between Local Image Similarity Show similarity metric of This includes calculating.

[0376] In some examples, the local image similarity metric may include a normalized mutual information similarity between the first local image descriptor and the second local image descriptor. For example, the normalized mutual information similarity between the first local image descriptor and the second local image descriptor may be Next It can be determined as follows. 1st A histogram is formed, in which No. 1 Descriptor of Entries in teeth, 1st Minimum of local image descriptors entry Value and Max entry value and Equally sized bins between (bin) is placed inside x. 1st Histogram Counts but normalized across bin x No. 1 The probability distribution of the descriptor entries P X(x) A second histogram is formed in which the entries of the second local image descriptor are The second local image descriptor are placed in an equally sized bin y between the minimum and maximum entry values. Second Histogram Counts but normalized across bin y Second local image The probability distribution P of the descriptor entries Y(y) Get the range between the minimum and maximum values. 1st Local image descriptors and Second Local image descriptors entry Determine the joint histogram of join Each bin in the histogram is were made equal in size are two-dimensional bins, The first dimension X is From the first local image descriptor entry Compatible with death , and The second dimension Y is Associated from the second local image descriptor entry Compatible with do .for example, 1st Local image descriptors First entry is q and Second Local image descriptors First entry If is p, contains p 1st Covers the range of local image descriptor values, including q SecondCovering a range of local image descriptor values, join 2D bin x,y of the histogram receives counts It will be . join The histogram counts are normalized and span bins x,y. No. 1 and No. 2 Local image The probability distribution P of the descriptor entries XY(x,y) Then, the correlation between the first local image descriptor and the second local image descriptor is obtained. Normalization mutual information similarity I can be calculated as follows: JPEG2023274599000001.jpg16133

[0377] The higher the mutual information similarity I, the of Local image descriptors and second of Local image with local image descriptors The similarities are increasing . Normalization By using mutual information similarity, local Robust Image Similarity Right? Reliable High and / or This allows for flexible decision making. For example, mutual information Similarity is compared with a second local image descriptor. did , regardless of the relative scale differences of the entries in the first local image descriptor. For example, mutual information can be used to estimate the overall "brightness" (e.g., Imaging Even if the values ​​of data elements (intensity) differ between the target medical image set TMIS and the reference medical image set RMIS, accurate Similarity Metrics As another example, Different imaging protocols and / or modalities were used (And, for example, accordingly different value ranges have been used or achieved), or, for example, when a medical image has been transformed, e.g. Invert Mutual information can provide accurate similarity even when the image is scaled or resized. Ba 1st Medical Imaging Data and second Medical Imaging Similarity determination that is robust to non-structural variations between data provided,As a result, the method can be reliably applied to a wider range of images, and This allows for more flexibility in the types of images that the method can accommodate. can be .

[0378] As mentioned above, in some cases, the descriptor entry teeth, Related Boxes PM-223, PM-333, PM-339 During The value of the contained element Represents As mentioned before, in these examples, the local image descriptors each Each entry teeth, Each box PM-223, PM-333, PM-339 is the average of the values ​​of the elements contained in the local image descriptor entry is within a certain range that is independent of the box size. Helpful This, in turn, Predefined entries The bin in which is placed depends accordingly on the mean value of the elements in the box. death ,It does not depend on the size of the box, which makes it easy to use mutual information similarity. to .

[0379] In some examples, other similarity metrics between the first and second local image descriptors may be used, such as cosine similarity, Euclidean distance, and / or cross-correlation. (cross correlation) may alternatively or additionally be used. Nevertheless ,We use mutual information similarity metric But similarity metric Especially Robust provide reliable and / or flexible decisions; therefore , in the reference medical image group RMIS specified Especially the position where the characteristics of Robust that it can provide reliable and / or flexible decisions confirmation did.

[0380] in any case, each No. 2 of Regarding the candidate locations PM-334 and PM-340 hand , Regarding the second candidate location PM-334, PM-340 No. 1 ofLocal image descriptors and second of Shows the local image similarity between local image descriptors The similarity metric is calculated .

[0381] As mentioned above As mentioned above, the method Process In PM-40, the calculated similarity metric Based on multiple candidates No. 2 of Location PM-334, PM-340: 2nd candidate of selecting location PM-334; Process The second candidate selected in PM-50 of In the reference medical image group RMIS based on location PM-334 specified and determining the location PM-334 where feature PM-226 is expressed.

[0382] some example So, candidate Selecting the second position PM-334 allows multiple candidate Second position PM-334, PM-340 Similarity Metrics The highest among local Show image similarity Similarity Metrics have candidate Selecting the second position PM-334, e.g., the highest mutual information Similarity Metrics have candidate A second position, PM-334, may be selected. specified Determining the location where PM-226 is expressed is specified The feature PM-226 is expressed as the position in the reference medical image group RMIS. , including determining the selected candidate second position PM-334. In some cases, the position To determine is the first local image descriptor for the first location and the selected Candidate No. 2 The similarity metric between the second local image descriptor and the decision This can be answered by predetermined The location where the feature is represented in the reference medical image set RMIS is decision Only if there is some confidence in decision to ensure that help This, in turn, predetermined Reliable location of features represented in the reference medical image set RMIS decision Provide do .

[0383] As mentioned above, in some examples (e.g., as shown in FIG. 21), each candidate second of Locations PM-334, PM-340 may be locations where the respective previously detected features PM-226, PM-242 are represented in the reference medical image set RMIS. In another example (e.g., as shown in FIG. 22), candidate second locations PM-440 may be distributed throughout the second medical imaging data in a second predefined sampling pattern (also referred to as a predefined sampling pattern). These example scenarios In each of the predetermined The feature position is determined will be method but , Figure 21 and figure 22, next More details explanation will be done.

[0384] First, referring to Figure 21, Example Scenario In the reference medical image group RMIS, the first feature PM-226 and the second feature PM-242 are express In this example, these features PM-226 and PM-242 have been previously detected in the reference medical image set RMIS, e.g., by a physician or by an automated method. The respective locations PM-334 and PM-340 of these features PM-226 and PM-242 in the reference medical image set RMIS have been recorded as part of this previous detection. Thus, the respective locations PM-334 and PM-340 of these previously detected features PM-226 and PM-340 are recorded as part of this previous detection. is known However, these Which of the two locations PM-334 or PM-340? However, in the target medical image group TMIS specified Features of PM-226 but Reference Medical Image Set RMIS In Whether it is a position to be expressed I don't understand That is, these Features PM-226 or PM-224Target medical image group TMIS is expressed in It is unknown whether this corresponds to the given characteristics of PM-226. therefore ,In these examples, candidate The second position is the reference medical image group RMIS in which the previously detected features PM-226 and PM-242 are expressed. Locations PM-334 and PM-340 It is possible. Candidate No. 2 Location PM-334,PM-340 a second local image descriptor for each of First local image descriptor for first location PM-224 with A similarity metric can be calculated between example So, the highest similarity Candidate No. 2 with metric of Position PM-334 can be selected, and the second candidate is selected. of Location PM-334, specified The position PM-334 at which the feature PM-226 is represented in the reference medical image set RMIS can be determined.

[0385] Referring to FIG. 23, the reference medical image group RMIS includes a first feature PM-226 and a second feature PM-242. express However, in the example scenario of FIG. 23, the positions where these features PM-226 and PM-242 are represented in the reference medical image set RMIS are I don't understand In this example, candidate The second position, PM-440, is the second predefined sampling pattern. in Reference medical image collection distributed through RMIS This is the location In some instances, as illustrated in FIG. Predefined The sampling pattern may be a regular arrangement, and each second candidate position PM-440 (in FIG. 23, White (represented by the circle PM-440) Separated from the next in a grid pattern For example, this arrangement is for example Whole-like area spanning . this like, candidate The second location represents a spatially distributed sample of locations within the reference medical image set RMIS (and thus There is a predetermined The characteristics of It may be expressed ).

[0386] Second Predefined Sampling Pattern in distribution is doing The candidate second location, for example, determines a descriptor for each voxel of the reference medical image set RMIS. Compared to Then, the predetermined feature PM-226 is used in the reference medical image group RMIS In Allows the represented position to be estimated in a computationally efficient manner to .

[0387] In the example in Figure 23, candidate Second position PM-440 each Between the second local image descriptor and the first local image descriptor for similarity A metric may be calculated as shown in Figure 23: expression The second local image descriptor for the candidate second location PM-448, which is closest to the selected feature PM-226, has the highest similarity to the first local image descriptor. Accordingly, the candidate second location PM-448 is selected. predetermined The location where the feature PM-226 is expressed in the reference medical image set RMIS is the second selected candidate. of This can be determined based on location PM-448.

[0388] For example, in some instances, the selected candidate second position PM-448 is: predetermined Features PM-226 is the second medical Imaging data In However, in other instances, this estimation may be based on successively finer grained patterns (e.g., In Figure 23, Gray circle PM-442 pattern, followed by Black circle This can be refined by defining further candidate positions PM-442, PM-444 (based on the selected candidate second position PM-448) in the pattern (see pattern PM-444). therefore , in the reference medical image group RMIS Either Feature location too Even if it is not known, the reference medical image set RMIS specified in The exact location where PM-226 is expressed However, It is possible to provide efficient decisions.

[0389] Specifically, these example In the above, determining the position PM-446 where the predetermined feature PM-226 is represented in the reference medical image set RMIS is performed by the selected candidate second of Based on location PM-448, multiple candidates in the reference medical image set RMIS were identified. of Location PM-442 (candidate Sub-resolution For example, determining the location of the candidate The third position, PM-442, was selected candidate Second location: Within the area of ​​PM-448 (i.e., the local area) It can be defined as a position, for example: candidate The third position, PM-442, is the second selected of Location PM-448 area In , the third advance definition The sampling pattern (also known as the sub-resolution sampling pattern) say ) through the reference medical image database RMIS and distribution For example, the third predefined sampling pattern may be the same as or similar to the second predefined sampling pattern. However, in some instances (as illustrated in FIG. 23), the third predefined sampling pattern may be the same as or similar to the second predefined sampling pattern. Predefined In the sampling pattern candidate The distance between the third position PM-442 is Predefined In patterns candidate The distance between the second positions PM-440 may be smaller than the distance between the first and second positions PM-440.

[0390] Then, the method selects a plurality of candidate third images in the reference medical image set RMIS. of obtaining a third local image descriptor for each of the positions PM-442, each third local image descriptor being a candidate third local image descriptor according to a first predefined sampling pattern; of Location PM-442 Each of The value of the element of the reference medical image set RMIS located relative to RepresentsFor example, the third local image descriptor may be the same as the first and second local image descriptors described above. the same, i.e., the same Calculate by method It can be something that .

[0391] In some examples, the scale of the third pre-defined sampling pattern associated with the third descriptor is local The first predefined sampling pattern may be reduced in scale compared to the scale of the second predefined sampling pattern associated with the image descriptor. Predefined Sampling Pattern in Taking a local image descriptor based on arranged boxes (not shown in FIG. 23) as an example, the shape and spatial pattern of the boxes may be the same for the second local image descriptor and the first local image descriptor, but different for the third local image descriptor. pattern The size of the boxes and / or the size of each box for the descriptor Each position The distance between PM-442 and the second local image descriptor is calculated based on the box size and / or the distance between each box and the second local image descriptor. Each position Compared to the distance between the PM-440 Scale down This may be reduced (i.e., reduced) to the second selected candidate. of In the area of ​​location PM-448, the third local Image descriptors are based on the reference medical image set RMIS. High-definition details of Encoding Offer to do .this is, specified Features of PM-226: Reference Medical Image Group RMIS In to provide an accurate determination of the expressed position Helpful .

[0392] These examples In Then, the method selects a plurality of candidate third of For each of the positions PM-442, the first local Image descriptor and candidate 3 of The third for location PM-442 local Between image descriptors local image Similarity Show Similarity MetricsFor example, the similarity metric may be calculated in the same way as the similarity metric used between the first local image descriptor and the second local image descriptor. obtain The method then comprises: local Image descriptors and third local Image Descriptors each Between local image Similarity The calculated similarity indicates metric Based on multiple candidate From the third position PM-442 candidate 23, the third position PM-448 may be selected. Similarity Metrics have candidate Select the third position PM-448 obtain .and, This method is the third selected candidate of Based on the position PM-448, the predetermined feature PM-226 is included in the reference medical image group RMIS In This can include determining the position to be represented.

[0393] In some instances, the selected candidate The third position, PM-448, specified Features of PM-226: Reference Medical Image Group RMIS In However, in other instances, the selected candidate Third of Based on location PM-448 (e.g., in that area Leave ) Further candidate location PM-444 of Definition This estimate can be further refined by repeating the above method for these additional candidate locations. For example, as shown in FIG. Candidate location PM-444 A further set is selected candidate No. 3 of In the area of ​​location PM-448 Definition In this example, a further candidate position, PM-446, is 1st Highest similarity with local image descriptors sex a local image descriptor having a metric; therefore, can be selected. specified Features of PM-226: Reference Medical Image Group RMIS Smell The positions represented by the three levels of hierarchical structure can be determined at selected further candidate positions PM-446. particle size is shown in Figure 23, but it is possible to Many Level Hierarchy particle size but, specified that the feature PM-226 can be used to determine the location where it is represented in the reference medical image set RMIS. is natural . In any case By using such a hierarchical method, even if the reference medical image group RMIS In either Feature location Even if you don't know , allowing location to be determined accurately in a fast and efficient manner become .

[0394] candidate It should be noted that the elements (e.g., boxes) from which respective second, third or more descriptors can be calculated for each second, third or more position are not shown in FIG. 23 for clarity.

[0395] In any case, in the reference medical image group RMIS predetermined The location where feature PM-226 is expressed is determined. example wherein the method comprises: specified Features of PM-226: Reference Medical Image Group RMIS In The method can further include generating output data indicative of the determined position being represented.

[0396] For example, the output data may include coordinates or pixel / voxel indices corresponding to the determined locations ...

Claims

1. 1. A method for generating representation data for a medical image dataset, comprising the steps of: Receiving a target medical image set (TMIS) of a patient at a first time point (S10); - acquiring a region of interest (ROI) of the target medical image set (TMIS) (S20); - determining (S30) a target body region (408) represented by said target medical image set (TMIS); - selecting (S40) a reference medical imaging study (RS) from a plurality of candidate medical imaging studies (CS) based on a comparison of the target body region (408) with a plurality of candidate body regions (720); each of the plurality of candidate body regions (720) corresponds to one of the plurality of candidate medical imaging studies (CS), each of the candidate medical imaging studies (CS) including a plurality of candidate medical image sets (CMIS) of the patient at a second time point; Selecting (S50) a reference medical image set (RMIS) from the plurality of candidate medical image sets (CMIS) of the reference medical imaging study (RS) based on a degree of comparability with the target medical image set (TMIS); Performing a registration between the target medical image set (TMIS) and the reference medical image set (RMIS) (S60); - generating display data based on the registration to display a rendering of the reference medical image set (RMIS) on a display device (S70); The method includes:

2. each of the target and candidate medical imaging studies (TS, CS) is associated with one or more attributes (A) having attribute values ​​(AV) that include text strings indicative of the content of the target and candidate medical imaging studies (TS, CS), Determining the target body region (S30) Obtaining (BR-1) one or more of the text strings of the target medical imaging study (TS); inputting the obtained one or more text strings (404) into a trained machine learning model (406); the trained machine learning model (406) is trained to output a body region (408) based on an input of the one or more text strings (404); determining the body region (408) represented by the target medical imaging study (TS) by obtaining an output from the trained machine learning model (406); and / or At least one of the candidate body regions (720) Obtaining (BR-1) one or more of the text strings (716) of the candidate medical imaging study (CS); inputting the obtained one or more text strings (716) into a trained machine learning model (BR-2); the trained machine learning model is trained to output a body region (720) based on an input of the one or more text strings (716); determining the body region (720) represented by the candidate medical imaging study (CS) by obtaining output from the trained machine learning model; The method of claim 1.

3. the trained machine learning model is a trained neural network consisting of a trained character-based neural network configured to take as input individual characters of the retrieved one or more text strings; inputting the one or more retrieved text strings into the trained neural network includes inputting individual characters of the one or more retrieved text strings into the trained character-based neural network. The method of claim 2.

4. each of the object and candidate medical image sets (TMIS, CMIS) is associated with one or more attributes (A) having attribute values ​​(AV) indicative of imaging parameters used to capture the object and candidate medical image sets (TMIS, CMIS), respectively; the degree of comparability is based on determining a match of one or more attribute values ​​(AV) between the target medical image set (TMIS) and each of the plurality of candidate medical image sets (CMIS); The method of claim 1.

5. Determining the degree of comparability includes: Obtaining a first feature vector (FV) for the target medical image set (TMIS) (BR-5); obtaining (BR-6) a second feature vector (SV) for each of the candidate medical images (CMIS) included in the reference medical imaging study (RS); determining a comparability metric indicative of the degree of comparability between the first feature vector (FV) and each of the second feature vectors (SV) of the candidate medical image set (CMIS) (BR-7); The method of claim 1.

6. the first and second feature vectors (FV, SV) include indications of imaging modalities of the subject and candidate medical image sets (TMIS, CMIS); - the comparability metric comprises an imaging modality relevance score between the imaging modality of the target medical image set (TMIS) and the imaging modality of each of the candidate medical image sets (CMIS); The method according to claim 5.

7. The Reference Medical Image Set (RMIS) renders a reference image volume; The method further includes obtaining a plurality of candidate slices (S-2), each of which depicts a particular section of the reference image volume; performing the registration (S60) includes identifying at least one reference slice (REF-SLC) from the plurality of candidate slices based on a similarity between image data contained in the region of interest (ROI) and each of the candidate slices (SLS-60); The generating (S70) of the display data includes generating display data for causing the display device to display a rendering of the reference slice (REF-SLC). The method of claim 1.

8. Extracting at least an image descriptor (D-1) from the target medical image set (TMIS) (SLS-33); Extracting (SLS-34) a matching image descriptor (D-2) from each of the plurality of candidate slices (S-2); Further comprising: The similarity is based on a comparison between the extracted image descriptors (D-1) of the target medical image set (TMIS) and the matching image descriptors (D-2) of the candidate slices (S-2), respectively; The method according to claim 7.

9. Identifying (SLS-60) at least one reference slice (REF-SLC) comprises applying a trained function (TF) to the subject and reference medical images (TMIS, RMIS); The trained function (TF) is configured to determine a similarity between two-dimensional medical images; Optionally, the trained function (TF) applies a learned metric to determine similarity between the two-dimensional medical images, the trained function optionally comprising a deep metric trained network. The method according to claim 7.

10. The performing of the registration (S60) generating a first local image descriptor based on image data of the region of interest (ROI); generating (PM-20) a second local image descriptor for each of a plurality of candidate locations (PM-334, PM-340, PM-440, PM-448) in the reference medical image set (RMIS); Each of the candidate positions (PM-334, PM-340, PM-440, PM-448) is generated based on image data of the reference medical image set (RMIS) located relative to each of the candidate positions (PM-334, PM-340, PM-440, PM-448); calculating (PM-30) for each of the plurality of candidate locations (PM-334, PM-340, PM-440, PM-448) a local image similarity metric indicative of a similarity between the first local image descriptor and the second local image descriptor of the candidate location (PM-334, PM-340, PM-440, PM-448); selecting a candidate location (PM-334, PM-448) from among the plurality of candidate locations (PM-334, PM-340, PM-440, PM-448) based on the calculated local image similarity metric (PM-40); determining (PM-50) a location that matches the region of interest in the reference medical image set (RMIS) based on the selected candidate location; In the step of generating the display data (S70), the rendering is generated based on image data of the reference medical image set (RMIS) associated with the set of selected candidate locations (PM-334, PM-448); The method of claim 1.

11. Selecting (S40) at least one of the candidate medical imaging studies (CS) as the reference medical imaging study (RS) comprises: Identifying (S43) a plurality of relevant medical imaging studies from the candidate medical imaging studies (CS) based on a comparison of the body regions (408, 720); providing (S44) an indication of the plurality of related medical imaging studies to a user via a user interface (10); receiving (S45) a user selection, via the user interface (10), indicating at least one of the relevant medical imaging studies; selecting (S46) the at least one indicated related medical imaging study as the reference medical imaging study (RS). The method of claim 1.

12. - identifying (S43) the plurality of related medical imaging studies comprises: determining (S42) for each of the related medical imaging studies a degree of match with a target medical imaging study (TS) based on a comparison of each of the body regions (408, 720); providing an indication of the plurality of related medical imaging studies includes providing (S44) the measure of fit to a user via the user interface (10); the goodness of fit is optionally based on an anatomical overlap between each of the medical imaging studies based on a comparison (408, 720) of the target and candidate body regions; The method of claim 11.

13. the reference medical imaging study (RS) includes one or more annotations (RM-322 to RM-330) corresponding to the reference medical image set (RMIS); - obtaining at least a reference annotation (RM-10) related to the target medical image set (TMIS) and / or the region of interest (ROI) from the one or more annotations (RM-322 to RM-330); annotating the target medical image set (TMIS) and / or the region of interest (ROI) with the reference annotations (RM-20); Including, The reference annotation optionally includes one or more first words (A, B, C, D); The method of claim 1.

14. Obtaining a medical report (RM-504) linked to said reference medical imaging study (RS); obtaining one or more sections (F1-F3) of text of said medical report (RM-504), said sections (F1-F3) consisting of one or more second words; - for each of the one or more sections (F1-F3) and for each of the reference annotations (A, B, C, D), comparing the one or more second words with the one or more first words of the reference annotation to identify matches; - associating at least one of said reference annotations (A, B, C, D) with at least one of said sections (F1-F3) based on said identified matches; Further comprising: The method of claim 13.

15. - recognizing a user who provides the display data (S11); Obtaining one or more previous actions of the user (S12); the preceding action is directed to at least one of an examination selection action for selecting a reference medical imaging examination from a plurality of candidate medical imaging examinations, and a group selection action for selecting a group of reference medical images from a group of candidate medical images of the medical imaging examination; Further comprising: the selecting (S40) the reference medical imaging study (RS) and / or the selecting (S50) the reference medical image set (RMIS) is based on the one or more previous actions; The method of claim 1.

16. 1. A system (1) for assisting in the evaluation of a target medical image set (TMIS) of a patient acquired at a first time point, comprising: - comprising an interface unit (10) and a computing unit (30); The interface unit (10) comprises: providing a rendering of said target medical image set (TMIS) to a user; receiving user input from the user intended to specify a region of interest (ROI) in the target medical image set (TMIS) based on the rendering; The computing unit (30) Obtaining the region of interest (ROI) based on the user input (S20); determining (S30) a target body region (408) represented by said target medical image set (TMIS); selecting (S40) a reference medical imaging study (S40) from a plurality of candidate medical imaging studies (CS) based on a comparison of said candidate body region (408) with a plurality of candidate body regions (720); each of the plurality of candidate body regions (720) corresponds to one of the plurality of candidate medical imaging studies (CS), each of the candidate medical imaging studies (CS) including a plurality of candidate medical image sets (CMIS) of the patient at a second time point; selecting (S50) a reference medical image set (RMIS) from the plurality of candidate medical image sets (CMIS) of the candidate medical imaging study (CS) based on a degree of comparability with the target medical image set (TMIS); Performing registration between the target medical image set (TMIS) and the reference medical image set (RMIS) (S60); A system (1) configured to generate (S70) display data for causing the interface unit (10) to display a rendering of the reference medical image set (RMIS) based on the registration.

17. A computer program comprising a program element, A computer program comprising the program elements, when loaded into a memory of a computing unit (30) of a system (1) for supporting the evaluation of a collection of medical images, causing the computing unit to execute the steps of the method according to any one of claims 1 to 15.

18. A computer readable medium having a program element recorded thereon, A computer-readable medium, the program elements being readable and executable by a computing unit (30) of a system (1) for assisting in the evaluation of a collection of medical images such that, when executed by the computing unit, the steps of the method according to any one of claims 1 to 15 are performed.