Automatic tracking and interpretation method and system for medical image data

Abstract

A method and system for generating display data for a medical image dataset is provided, the method and system being configured to identify, for a set of target medical images of a patient at a first time point, a set of reference medical images of the patient acquired at a second time point distinct from the first time point. In one example, the selection is based on a comparison of respective depicted body regions of the patient. The method and system are also configured to generate display data for displaying a rendering of the reference medical images on a display device based on a registration between the target medical images and the reference medical images.

Description

[Technical Field]

[0001] This application relates to medical image processing, such as image processing of X-ray images, magnetic resonance images, or computed tomography images. Specifically, this application relates to processing separate medical image datasets of a patient and automatically identifying image datasets acquired at different time points for later comparison by a physician. [Background technology]

[0002] Medical imaging techniques such as MRI (magnetic resonance imaging) and CT (tomography) are crucial tools for medical diagnosis. In making clinical decisions, the temporal progression of a patient's disease can be as important, if not more so, than the current state of the disease. To aid in assessing the progression of a patient's disease, medical professionals often prefer to compare a patient's current medical images with appropriate reference medical images from one or more patients. Specifically, for example, the current medical image might be from the patient's current examination, while the reference image is from a previous examination.

[0003] There are often numerous possible reference medical images for a patient, and these may be presented as candidate medical images. For example, a patient may have multiple previous examinations performed at different points in time. Furthermore, each examination may have multiple sets of medical images, each with different characteristics. Manually evaluating the validity of previous medical images in comparison to current medical images, or in relation to current medical images, would be a waste of time and burden on healthcare professionals. In addition, medical images are often stored on storage devices away from the healthcare professional's terminal, and for healthcare professionals to collect all previous medical images of a patient for evaluation would concentrate network resources due to the large size of many of the images. [Overview of the project]

[0004] Therefore, the object of the embodiments described herein is to provide a system and method for automatically selecting a medical image (e.g., a previous or, generally speaking, a reference medical image) that is appropriate for comparison with a given medical image (e.g., a current or, generally speaking, a target medical image) or that is related to a given medical image.

[0005] For this purpose and other purposes, the display data relating to the independent form claim of This is achieved by a method of generation, a system corresponding to the method, a computer program (a product storing the computer program) corresponding to the method, and a computer-readable storage medium. The selective aspects are the subject of the cited claims and are the aspects described herein.

[0006] The following describes the technical solutions according to the present invention in relation to the claimed method and the claimed apparatus. The features, advantages, or selective embodiments described herein also apply to other claimed subjects, and vice versa. In other words, claims directed to the method of the present invention can be improved by features described or claimed in relation to the apparatus. In such cases, for example, the functional features of the method are implemented by a purpose-appropriate unit or element of the apparatus.

[0007] According to one embodiment, a method for generating display data for a medical image dataset is provided. The method comprises several steps. The first step aims to receive a set of target medical images of a patient at a first time point. The next step aims to obtain a region of interest from the set of target medical images. The next step aims to determine the target body region represented by the set of target medical images. The next step aims to select a reference medical imaging examination from a group of candidate medical imaging examinations based on a comparison between a group of candidate body regions and a group of target body regions. Each of the group of candidate body regions corresponds to one of the group of candidate medical imaging examinations, and each candidate medical imaging examination includes a group of candidate medical images of a patient at a second time point. The next step involves selecting a reference medical imaging examination from a group of candidate medical images, and comparing it with the target medical image group. Degree of comparabilityThe objective is to select a set of reference medical images based on the above. The next step is to perform registration (alignment) of the target medical image set and the reference medical image set. The next step is to generate display data and display a rendering of the reference medical image set based on the region of interest and registration on a display device.

[0008] For example, a set of medical images (target medical image set, candidate medical image set, reference medical image set) is related to one or more two-dimensional images. For example, a set of medical images is a three-dimensional image. For example, a set of medical images is a four-dimensional image with three spatial dimensions and one temporal dimension.

[0009] For example, the type of medical image set is related to the type of medical imaging device used to acquire the medical image set. For instance, a first set of X-ray images and a second set of CT images are comparable even if they were recorded by different medical imaging devices. For example, two sets of medical images are comparable if they correspond to the same anatomical region within the human body. For instance, a first X-ray image of a human lung and a second X-ray image of a human knee are not comparable, even if they relate to the same patient and were acquired with the same medical imaging modality.

[0010] For example, the type of medical image can be characterized by the modality used to generate the medical image and by the body region that the medical image depicts. Alternatively, the type of medical image can also be characterized by the parameters (of the imaging modality) used to generate the medical image (for example, a distinction between "low-dose images" and "high-dose images").

[0011] Each medical image set includes image data, for example, in the form of an array of pixels or voxels. Such arrays of pixels or voxels represent intensity, absorption, or other parameters depending on their three-dimensional position and can be obtained, for example, by appropriate processing of measurement signals obtained by a medical imaging modality.

[0012] A set of medical images may be identical to or contained within one or more image data files, specifically DICOM files. The DICOM referred to herein is understood to mean the "Digital Imaging and Communications in Medicine" (DICOM) standard, such as DICOM·PS3.1·2020c (or earlier or earlier versions of that standard).

[0013] "Reception" means, for example, that the target medical image set is acquired from a medical imaging modality. It also means, for example, that the target medical image set is acquired from an appropriate memory, such as a picture archiving and communication system (PACS) or other appropriate medical image storage equipment.

[0014] The target medical image set relates to, for example, the examination of the patient at a first point in time (the first time of implementation), while the candidate medical image set for the candidate examination relates to, for example, the examination of the patient at a second point in time (the second time of implementation) that is different from the first point in time. The second point in time could be several hours or hours before or after the first point in time, several days or several days before or several weeks before or several weeks after, several months before or several months after, or several years or several years after. Furthermore, a scan or procedure may have occurred between the first and second points in time.

[0015] For example, one or more sets of medical images are associated with a corresponding medical imaging examination. In other words, a medical imaging examination may contain multiple sets of medical images. The sets of medical images included in a medical imaging examination may differ in type, for example. For example, the sets of medical images in a medical imaging examination may be acquired using different modalities and / or using different parameters when generating the medical image sets. For example, a medical imaging examination may contain various sets of MRI medical images acquired with different MRI imaging parameters. For example, one set of medical images may be acquired with a T2 FLAIR setting, and other sets of medical images may be T1 sets.

[0016] Since the group of medical images included in a medical image examination represents the patient's body region, it is possible to assign at least one body region to the medical image examination. As an example, the group of medical images included in a medical image examination represents (although not necessarily) the (at least approximately) same body region of the patient.

[0017] The group of medical images included in a medical image examination will necessarily be acquired at slightly different times, but these groups of images can be considered to be related to the same point in time with respect to the follow-up reading workflow described here. That is, the time difference of a few minutes (or even a few hours) during the collection of the group of images of the examination belongs to the same state of the patient in a clinical sense.

[0018] As an example, the target medical image examination and the candidate medical image examination relate to one patient. According to some examples, the candidate medical image examination may also relate to a plurality of patients other than the patient of the target medical image examination. In the latter case, similar cases can be compared.

[0019] The region of interest represents a region within the target group of medical images, and this region is a specific concern to the user analyzing the target group of medical images. The region of interest usually relates to the part of the target group of medical images that the user is currently reviewing. This region relates to, for example, an image volume or an image plane. For example, the region of interest is an image slice of the target group of medical images, also called a "target slice". Also, the region of interest is, for example, a part of an image slice of the target group of medical images. Thus, the region of interest can have any shape, and preferably the region of interest is in the form of a circle or an ellipse. In any case, the region of interest can be understood as a group of pixels such as pixels or voxels within the target group of medical images.

[0020] The region of interest may be defined by the user or may be defined semi-automatically or automatically by a method executed by a computer.

[0021] Therefore, obtaining the region of interest is based on, for example, processing one or more user inputs that specify the region of interest in the target medical image group. For example, the user input may consist of scrolling to the target slice and / or defining the region of interest in the target slice.

[0022] Also, obtaining the region of interest consists of, for example, automatically identifying anatomical features in the target medical image group, and the anatomical features represent the patient's pathological condition. As an example, this includes applying a detection function configured to identify anatomical features in the medical image data to the target medical image group.

[0023] The body region is related to, for example, the patient's body part depicted in the image group. Therefore, the body region of a medical image examination represents the patient's body part depicted in the image data associated with the medical image examination. In this case, the body region is not a rigid descriptor of the patient's anatomical structure cut out by the image data in the form of, for example, an anatomical coordinate system, but rather a higher-level or coarser descriptor of the image data content of the examination. For example, the body regions of the individual medical image groups of an examination are aggregated to derive the body region for the medical image examination. According to some examples, the body region is the range of a normalized or generalized coordinate system related to the patient's anatomical structure. As an example, these normalized or generalized coordinate systems are determined along the human / patient's cranio-caudal axis or longitudinal axis.

[0024] The target and candidate body regions are determined based on, for example, the image data and non-image data associated with each of the medical image examinations. As an example, the metadata of the medical image examination indicating the content of the medical image examination can be evaluated to determine the target and candidate body regions.

[0025] The degree of comparability indicates, for example, how well a user can compare two separate sets of medical image data. For example, the degree of comparability between two sets of medical image data is based on the type of each set. According to this, the type of medical image data is characterized, as previously described, by the modality used to generate the medical image data, by one or more image parameters used to generate the medical image data, and / or by the body region to which the medical image data pertains.

[0026] Therefore, the step of selecting a reference medical image group from a candidate medical image group for a reference medical imaging examination includes, for example, determining the degree of comparability of each candidate medical image group with the target medical image group. In this case, the degree of comparability is determined, for example, by comparing the types of the target medical image group with each type of the candidate medical image group.

[0027] Performing registration between a group of target medical images and a group of reference medical images may include, for example, roughly aligning a target image (e.g., a region of interest or target slice) from the group of target medical images with a reference image (e.g., in the form of a reference slice) from the group of reference medical images.

[0028] In one example, this step is based on evaluating the similarity of image data and / or detecting the relevant locations as separately described herein. In another example, this step involves obtaining a deformation field between the target medical image set and the reference medical image set, which determines the relationship between the coordinate systems of the target medical image set and the reference medical image set such that each anatomical location in the target medical image set maps to the same anatomical location in the reference medical image set (or vice versa). That is, the deformation field includes, for example, a number of individual displacement vectors associated with pixels / voxels in the target image and the reference image, respectively.

[0029] According to one example, registration consists of rigid registration. Rigid registration consists of registration in which the coordinate system of pixels / voxels in an image is subject to rotation and translation in order to align one image to another. According to some examples, registration consists of affine registration. Affine registration consists of registration in which the coordinate system of data points in an image is subject to rotation, translation, scaling and / or shear in order to align one image to another. That is, rigid registration can be considered a specific type of affine registration. According to some examples, registration consists of non-rigid registration. Non-rigid registration provides separate displacements for each pixel / voxel of the image being registered, and can use, for example, nonlinear transformations, in which case the coordinate system of pixels / voxels in the image is subject to flexible deformation in order to align one image to another. Nonlinear transformations are determined, for example, using vector fields such as warp fields or other fields or functions that define individual displacements for each pixel / voxel in the image. For further details on image registration, US2011 / 0081066 and US2012 / 0235679 are referenced herein. Rigid image registration is highly effective when anatomical changes or deformations are not anticipated. Compared to rigid image registration, non-rigid image registration offers significantly greater flexibility and can manage local distortions (e.g., anatomical structural changes) between two image sets, but it is more complex to handle.

[0030] Providing rendering based on registration and reference medical image sets includes, as an example, identifying the image data portion of the target medical image set corresponding to the region of interest based on the registration. In this case, the rendering is based on the image data portion of the target medical image set thus identified.

[0031] In some examples, rendering is performed so that the region of interest is displayed in conjunction with the identified image data portion of the target medical image set.

[0032] The method of this embodiment enables stepwise and directional selection and representation of user-defined reference medical image data captured for easy comparison with the target medical image set. This provides, for example, efficient selection of previous medical images appropriate for comparison with the target medical image, compared to opening and evaluating all previous medical images of the patient. Furthermore, selection based on the body region of each imaging examination enables selection without the need to extract or analyze the medical imaging data itself, thus enabling, for example, efficient selection. In addition, selection based on body region and degree of comparability enables, for example, flexible selection. For example, comparing body regions is more robust and / or flexible with respect to inaccurate matches between parameters compared to, for example, attempting to directly match the imaging coordinate systems.

[0033] For example, the target medical imaging examination and each candidate medical imaging examination each include one or more attributes having attribute values ​​containing text strings (characters) that indicate the content of the medical imaging examination. Therefore, the process of determining the target body region includes, for example, obtaining one or more text strings of the target medical imaging examination, inputting the obtained one or more text strings into a trained machine learning model, and determining the body region represented by the target medical imaging examination by obtaining output from the trained machine learning model. This machine learning model is trained to output body regions based on such input of one or more text strings. Also, at least one of the candidate body regions is determined by, for example, obtaining one or more text strings of the candidate medical imaging examination, inputting the obtained one or more text strings into a trained machine learning model, and obtaining output from the trained machine learning model. Determined byThis machine learning model is trained to output body regions based on one or more text string inputs.

[0034] Generally, trained machine learning models mimic the cognitive functions that humans use to interact with the minds of other humans. For example, through training on training data, trained machine learning models can adapt to new situations and detect and infer patterns. Other terms related to trained machine learning models include, for example, trained functions, trained mapping specifications, mapping specifications with trained parameters, functions with trained parameters, artificial intelligence-based algorithms, or machine learning algorithms.

[0035] In general, the parameters of a trained machine learning model can be adapted through training. Examples include supervised training, semi-supervised training, unsupervised training, reinforcement learning, and / or active learning. Furthermore, representation learning (also known as "feature representation learning") can be used. For example, the parameters of a trained machine learning model can be iteratively adapted through several steps of training.

[0036] For example, a trained machine learning model may consist of a neural network, a support vector machine, a decision tree, and / or a Bayesian network, and / or a trained function may be based on k-means clustering, Q-learning, a genetic algorithm, and / or association rules. For example, a neural network may be a deep neural network, a convolutional neural network, or a convolutional deep neural network. Furthermore, a neural network may be an adversarial network, a deep adversarial network, and / or a generative adversarial network.

[0037] For example, a machine learning model is trained to output body regions in the form of a range of normalized or generalized coordinate systems relating to the patient's anatomical structure. For example, these normalized or generalized coordinate systems are determined along the human / patient head-to-tail axis or longitudinal axis.

[0038] For example, attributes and attribute values ​​typically relate to non-image data or metadata associated with each medical imaging examination. Attribute values ​​are stored, for example, in the image data file for each medical imaging examination, or separately. As an example, attribute values ​​may also be stored in the patient's electronic medical record.

[0039] Attributes and attribute values ​​provide additional information about the image data, such as the body part examined, the imaging modality and protocol used, and / or findings or annotations reported or created by the user. Non-image data includes information not directly related to the image data, such as patient personal information, including gender, age, weight, insurance details, and information about the treating physician.

[0040] For example, each medical imaging examination is associated with one or more DICOM image files, and one or more attributes have attribute values ​​containing text strings that indicate the content of the medical imaging data, such as the DICOM attributes "Examination Description", "Series Description", and " covered Includes one or more of the "body parts to be investigated".

[0041] Determining body regions based on text strings extracted from medical image attributes is less resource-intensive and therefore more efficient than determining body regions by extracting and analyzing the medical imaging data itself (which is relatively large in terms of bits). If the candidate medical image set is remotely stored over a network from a processing device configured to perform the method described here, determining body regions based on (relatively small) text strings has the effect of eliminating the need to transmit (relatively large) medical imaging data over the network to determine the body regions it represents, resulting in efficient use of network resources.

[0042] Determining body regions by inputting text strings into a trained machine learning model (e.g., a trained neural network) can provide a more efficient, flexible, and / or robust method of determining body regions compared to, for example, determining body regions by applying hardcoded rules to text strings. For example, hardcoded rules require text strings to precisely match the rules in order to provide body regions (thus lacking flexibility regarding text strings for which a match can be determined, and / or being inefficient due to the comprehensive nature of the set of rules that must be coded for all possible text strings that may be available). On the other hand, a trained machine learning model (e.g., a trained neural network) is derived from a training dataset for training and therefore can relatively efficiently obtain and determine appropriate body regions even for different text strings in the training dataset, and is thus relatively flexible / robust.

[0043] For example, a trained machine learning model is a trained neural network consisting of a trained character-based neural network, for example, a character-based recurrent neural network configured to take individual characters of one or more acquired text strings as input. Thus, inputting one or more acquired text strings into a trained neural network involves inputting individual characters of one or more acquired text strings into a trained character-based neural network.

[0044] The advantages of character-based language models lie in their small vocabulary and flexibility when dealing with any text string and other document structures. This is beneficial for medical applications where attribute values ​​are often not standardized across multiple healthcare organizations.

[0045] For example, a trained neural network is configured to output body regions, which represent areas of the human body in the form of one or more numerical values, and the output portion of the trained neural network is computed based on one or more input text strings.

[0046] Outputting body regions as one or more numerical values ​​allows for accurate and / or flexible determination of body regions represented by the medical imaging data of each medical imaging examination, for example. For example, the numerical values ​​are continuous and therefore allow for more accurate and flexible definition of body regions compared to, for example, the use of a limited set of predefined classes. This, in turn, allows for flexible and / or accurate comparison of body regions.

[0047] For example, a trained neural network is configured to output body regions by selecting them from a predefined list of body regions. This has the advantage of reducing the fluidity of results and providing clearer relevance when medical imaging examinations are mapped to a predefined set of body regions.

[0048] For example, a neural network is based on a training dataset containing multiple training text strings. Each training text string is derived from an attribute value of a medical image data file, representing the content of a medical imaging examination associated with that file. Each of the multiple training text strings is labeled with the corresponding body region. These labels are used as ground truth during the training of the neural network.

[0049] For example, a training dataset is generated using a graphical user interface (GUI). The GUI presents the user with one or more training text strings and a representation of a human body divided into selectable body regions. For each of the one or more presented training text strings, it receives user input to select a body region from the representation and labels the training text string with a label indicating the selected body region.

[0050] For example, each of the target and candidate image data sets is associated with one or more attributes, each having attribute values ​​that indicate the imaging parameters used to capture each of the medical imaging data sets. The degree of comparability is based on determining the agreement of one or more attribute values ​​between the target medical image set and each of the multiple candidate medical image data sets.

[0051] For example, each medical image set consists of one or more DICOM image files and each medical imaging data set. Capture It is associated with one or more attributes that have attribute values ​​containing text strings indicating the imaging parameters used for the purpose.

[0052] The above embodiments enable, for example, the selection of one or more reference medical image sets captured using the same or similar imaging parameters as the image data of the target medical image set. This provides, for example, efficient selection of previous medical images suitable for comparison with the target medical image data, compared to opening and evaluating all previous medical images of the patient. Furthermore, selection based on the attribute value AV enables selection without the need to extract or analyze the medical imaging data itself, and as a result enables efficient selection.

[0053] For example, the process of determining the degree of comparability includes obtaining a first feature vector for the target medical image group, obtaining a second feature vector for each of the candidate medical image groups constructed in the reference medical examination, and determining a comparability metric that indicates the degree of comparability between the first feature vector and each of the second feature vectors of the candidate medical image groups.

[0054] Feature vectors are typically thought of as descriptors that represent or characterize an underlying set of medical images. The feature vectors described here are based, for example, on image and non-image data associated with the medical image set. As an example, a feature vector is based on one or more attribute values.

[0055] In some cases, feature vectors are determined by encoding the image and non-image data associated with each medical image set using a specific encoding algorithm (also called a "vectorizer"). The features of the feature vector are, for example, visual features such as the visual manifestation of medical abnormalities, patterns, anatomical structures, and medical landmarks represented by each image in the medical image set. The feature vector also consists of non-image features, for example, related to the imaging modality or imaging parameters used. Feature vectors are calculated online, for example, when selecting a reference medical imaging examination. Alternatively, feature vectors are made available as (pre-)generated data items stored in a database, for example, in association with candidate medical imaging examinations.

[0056] For example, the degree of comparability can be calculated by applying a metric ("comparability metric") that represents, for instance, how similar or comparable a group of candidate medical images is to a group of target medical images. Another way to refer to the comparability metric is a similarity metric or a distance metric. For example, the comparability metric may be configured to quantify the distance between a first feature vector and a second feature vector, or the distance between the vectors or feature spaces. For example, a distance such as cosine similarity or Euclidean distance can be calculated using a predetermined mathematical function. For example, the comparability metric consists of trained metrics derived by machine learning. For example, the extraction of feature vectors and / or the evaluation of the overall similarity metric are performed by one or more trained functions.

[0057] For example, the target medical image set and the reference medical image set are associated with one or more attributes, each having attribute values ​​that indicate the imaging parameters used to capture the medical imaging data of the image files. In this case, the first feature vector is obtained based on one or more attribute values ​​that indicate the imaging parameters used to capture the target medical image set, and the second feature vector is obtained based on one or more attribute values ​​that indicate the imaging parameters used to capture each of the candidate medical image sets.

[0058] For example, the step of determining the degree of comparability includes inputting the target medical image set and the reference medical image set into a trained function configured to determine the degree of comparability between two sets of medical image data for follow-up interpretation.

[0059] As an example, the trained function consists of a neural network. This means that the first group of neural network layers are applied to extract the first and second feature vectors. These feature vectors are then sent as input to the second group of network layers. The second group of network layers functions to determine the degree of comparability based on the extracted feature vectors. However, the functions of these two neural networks can also be performed using separate, independent neural networks. That is, image analysis related to feature vector extraction can be performed with the first neural network, and classification according to comparability can be performed with the second neural network.

[0060] Selecting matching groups based on feature vectors allows for more flexible selection, for example. For instance, comparing feature vectors in a feature space is more robust and / or flexible with respect to inaccurate matches between parameters compared to, for example, directly matching imaging parameters.

[0061] For example, the first and second feature vectors include indicators of the imaging modality of the subject and candidate medical image data sets, and the comparability metric includes imaging modality relevance scores between the imaging modality of the medical image data set and the imaging modality of each of the candidate medical image data sets.

[0062] The imaging modality relevance score between two modalities represents, for example, the degree to which medical imaging data captured using one modality is relevant (i.e., appropriate or useful for comparison) to medical imaging data captured using the other modality. In some cases, regarding the imaging modality relevance score, if both of two candidate medical image sets have the same body region as the target medical image set, but one set was captured using MR and the other using CT, the set captured using MR may be preferred.

[0063] Considering the modality used when capturing each set of medical image data further improves the comparability of the selected reference medical image sets.

[0064] For example, the imaging modality relevance score incorporates a probability. This probability represents the likelihood that, given a first set of medical images associated with a specific first imaging modality, the user would select a second set of medical images associated with a specific second imaging modality in comparison to the first set of medical images. This probability is based on a statistical analysis of the user's selections recorded on previous occasions.

[0065] Imaging modality relevance scores, based on statistical analysis of actual user interactions with medical imaging data, can help ensure, for example, that a second set of medical imaging data from an appropriate modality is selected.

[0066] Furthermore, user interaction is also used, for example, for online (ad-hoc) automatic updates of system behavior.

[0067] For example, the reference medical image set renders a reference image volume (e.g., a body region represented by the reference medical image set). The method further includes the step of obtaining multiple candidate slices, each rendering a specific section of the reference image volume. The registration step includes identifying at least one reference slice from the multiple candidate slices based on the image similarity between the image data contained in the region of interest and the individual candidate slices. The display data generation step includes generating display data to display a rendering of the reference slice on a display device.

[0068] In some examples, multiple slices represent a subset of slices included in a set of reference medical images. In one example, multiple slices relate to all slices included in a set of reference medical images.

[0069] Image similarity measures, for example, how similar image data is between two medical images (in this case, a region of interest and a slice from a set of reference medical images). Image data is similar in terms of, for example, contrast, grayscale, texture, density, distortion, singularities, patterns, landmarks, masks, and visible anatomical structures.

[0070] Identifying a reference slice means, for example, evaluating an image similarity metric for each of several candidate slices in the reference medical image set, representing the image similarity between the image data of the region of interest and each slice in the reference medical image set. Based on this evaluation, for example, one or more slices with high image similarity to the image data of the region of interest are selected and identified. For example, all slices with an image similarity above a predetermined threshold can be selected. Alternatively, the slice with the highest image similarity can be selected. For example, the result of determining the similarity metric between two images is the image similarity between those images.

[0071] In other words, according to this example, one or more slices in a reference medical image set are automatically identified as corresponding to the region of interest. This automated search is based on the quantification of image similarity between the target image data (in this case, the region of interest) and the reference image data (in this case, each slice in the reference medical image set). This provides a tool that can automatically search for these slices in the reference medical image set in terms of the morphology of the region of interest. Therefore, physicians do not need to manually scroll through the reference medical image set to track down the relevant slices. This not only saves considerable time but also reduces the likelihood of errors in the process and improves reproducibility. Slice matching in multiple medical imaging examinations is based on the evaluation of similarity between individual images. Compared to other methods, this has the advantage of comparablely fast processing time. Furthermore, image similarity often yields reasonable results even with other methods, such as conventional image registration-based methods, or when anatomical landmark detection is not applicable. This may be the case when registration is impossible due to very large variability between individual slices, or when landmarks cannot be detected. Therefore, for example, it becomes possible to easily synchronize vastly different image data acquired using different imaging protocols or imaging modalities.

[0072] For example, the target medical image data set includes multiple target slices that depict the target image volume (e.g., a body region represented by the target medical image data set) and each of the target slices that depict a specific section of the target image volume. Therefore, the registration process includes, for example, identifying one candidate matching slice for each of the multiple target slices in order to determine a slice match between the target medical image data set and the reference medical image data set, with the similarity used being based on the similarity between each individual target slice and each individual candidate slice in the reference medical image data set. The process of determining at least one reference slice is based on slice matching.

[0073] In other words, this method, based on image similarity considerations, precisely matches slices in the target medical image set with matching slices in the reference medical image set. Consequently, the slices of the two different medical image sets become synchronized. Therefore, when scrolling through the target medical image set, the user is automatically provided with matching slices for each of the reference medical image sets (and vice versa). That is, the user is provided with a position where they can directly compare the target medical image set and the reference medical image set side by side. In some examples, the results of such a matching process are provided in the form of data linkage, for example, by precisely assigning each of several slices in the first medical dataset to a matching slice among several slices in the second medical image set (CMIS). Such linkage can be called slice matching (correspondence).

[0074] If the identification of a matching slice is ambiguous, this method utilizes auxiliary conditions to accurately identify the matching slice, for example. These auxiliary conditions include, for example, the slice order in the subject medical image set and / or the reference medical image set, and / or a measure of overall similarity. Thus, for example, the step of identifying one matching slice in the reference medical image set for each of several slices in the subject medical image set is additionally performed based on the slice order in the subject medical image set or the reference medical image set, and / or additionally, under conditions that maximize the overall image similarity. In this regard, the overall image similarity is, for example, the cumulative image similarity (sum of individual image similarities) between the matching slices in the subject medical image set and the reference medical image set. The slice order can be considered as a natural sequence of slices, such as that determined by the imaging process or simply by the patient's anatomical structure.

[0075] For example, the registration process includes determining an offset in the slice numbering between the target medical image set and the candidate medical image set based on slice matching. The process of determining at least one reference slice is based on that offset.

[0076] Determining the offset provides a simple method for aligning the target medical image set with the reference medical image set. This allows the user to automatically receive the corresponding matching slice from the reference medical image set when scrolling through the target medical image set (and vice versa).

[0077] For example, the registration process includes extracting at least image descriptors from the target medical image set. The process also includes extracting matching image descriptors from each of several candidate slices. Thus, image similarity is based on a comparison between the extracted image descriptors from the target medical image set and the image descriptors from the candidate slices.

[0078] In some examples, an image descriptor is a vector that represents or characterizes the underlying image data of either a region of interest or a slice of a reference medical image set. In some examples, the image descriptor is determined by encoding each slice of the region of interest or reference medical image set using a specific image descriptor encoding algorithm. For example, features of the region of interest or each slice, such as the visual manifestation of a medical anomaly, patterns, anatomical structures, or medical landmarks, are encoded in their respective image descriptors.

[0079] For example, image similarity can be calculated by applying an image similarity metric that represents the degree of similarity between individual image descriptors. In one example, the image similarity metric is configured to quantify the distance in feature space between the image descriptors of the region of interest and each image descriptor of the slices of the reference medical image set. In one example, distances such as cosine similarity or Euclidean distance can be calculated using a predetermined mathematical function. In one example, the image similarity or distance metric consists of trained metrics derived by machine learning. In one example, the extraction of image descriptors and / or the evaluation of the overall image similarity metric are performed by one or more trained functions.

[0080] The selection of matched image data based on image descriptors allows for flexible selection. For example, comparing image descriptors in feature space is more robust and / or flexible with respect to inaccurate matches between parameters compared to, for instance, attempting rigid or non-rigid image registration.

[0081] For example, the method further includes the step of extracting image descriptors from each of several target slices. In this case, the similarity is based on a comparison between the extracted image descriptors of the target and candidate slices, respectively.

[0082] For example, the process of obtaining multiple candidate slices includes resampling the candidate medical image dataset based on the target medical image dataset to determine multiple candidate slices in the reference medical image dataset.

[0083] Resampling is equally applicable to medical image sets that already have established slices, and to medical image data sets that depict some isotropic image volumes but lack established slices. In some examples, resampling involves determining the slices of the reference medical image set so that the slices have slice thicknesses corresponding to the slice thicknesses of the target medical image set. Additionally or instead, for example, the stacking direction of the slices is set or adapted to match the target medical image set. By resampling the reference medical image set, the reference medical image set can be adapted to the target medical image set. This improves the comparability of the two groups, for example, and leads to better overall results in subsequent follow-up interpretations.

[0084] For example, the step of identifying at least one reference slice involves applying a trained function to the target and reference medical image data sets. doThe process includes the following steps. In this case, the trained function is configured to determine the similarity between two-dimensional medical images. Alternatively, the trained function may further apply a learned metric to determine the similarity between two-dimensional medical images. The trained function in this case preferably consists of a deep metric trained network.

[0085] According to some examples, the trained function consists of a distance metric network, and more specifically, a deep distance metric learning network.

[0086] Distance metric learning (or simply distance learning: metric learning) is a machine learning method that aims to automatically construct task-specific distance or image similarity metrics from supervised data. These trained distance metrics can then be used to perform various tasks, such as identifying similar image data, as in this example. Compared to using preset distance metrics, trained distance metrics have advantages, such as being better adapted to specific data and tasks of interest.

[0087] Deep distance metric learning networks, for example, further transform data into a new feature space with higher discriminative power before a metric (either pre-trained or standard) is applied. In this feature space, for example, dissimilar image features are pushed apart using an appropriate distance metric, while semantically similar extracted image features are mapped to nearby locations.

[0088] In some examples, the trained function consists of a Siamese network. In one example, the trained function consists of a fully convolutional Siamese network.

[0089] A Siam network is a type of neural network that learns to compare two inputs based on a distance or similarity metric, configured such that inputs closer to each other in a given semantic receive a lower distance compared to two inputs further apart according to the same semantic. The semantics to be captured are implicitly supplied to the network during the training process. Semantics can be viewed as a vector space of image descriptors. Therefore, the semantics determine which image descriptors are extracted. Semantics are extracted, for example, using branches of subnetworks with identical structure and parameters. The extracted image descriptors are considered, for example, representations of the learned semantics. As an example, a Siam network includes a structure with two branches of subnetworks with identical structure and parameters.

[0090] Based on a fully convolutional Sham network, for example, at least one convolutional operation and at least one pooling operation are performed on, for example, a first slice of a first image dataset, resulting in the acquisition of image features (or image descriptors) for that first image slice. Similarly, for example, at least one convolutional operation and at least one pooling operation are performed on, for example, a second slice of a second image dataset, resulting in the acquisition of image features (or image descriptors) for that second image slice. The output image features (or image descriptors) after the convolutional and pooling operations are, for example, one or more feature vectors or maps of the same size. The parameters in the convolutional or pooling operations, for example, the size and number of convolutional kernels used for each convolutional layer, or each pooling layer, are pre-configured, for example, through the training process of the fully convolutional Sham network. As an example, a fully convolutional Sham network also consists of a structure having two branches of subnetworks with the same structure and parameters.

[0091] According to some examples, the trained function consists of a triplet network.

[0092] A triplet network can be thought of as an extension of a sham network, for example, as it consists of three branches of the same feedforward network. Given three samples, this network outputs median values ​​in the form of paired distances. This allows one sample to be treated as a reference (or anchor) and compared to the other samples. The median values ​​are then fed into a comparator to determine the output. Rather than directly comparing data labels, triplet networks enable learning through sample comparison, lowering the requirements for training data and making them usable as unsupervised learning models.

[0093] According to some examples, the trained function is trained using a triplet loss function.

[0094] The triplet loss function is a loss function for machine learning algorithms in which a reference (anchor) input is compared to positive (true) and negative (false) inputs. The distance from the baseline (anchor) input to the positive input is minimized, and the distance from the reference input to the negative input is maximized. In this example, for instance, the positive input would be a slice whose similarity to a given reference slice has been verified. Similarity can be verified, for example, by a user, or due to the verified adjacency of the slice to the reference slice in an image examination. The latter has the advantage that training data is relatively easy to obtain. The negative input would be a slice that is less similar to the reference slice than the positive input. For example, the negative input would be a slice that is not related to the reference slice, for example, a slice from another group or patient. By adhering to this similarity ranking, the triplet loss model embeds (i.e., extracts image descriptors) in such a way that pairs of similar slices are farther (or more similar) than dissimilar slices. One advantage of this approach is that the triplet loss function is very flexible in terms of the training data required.

[0095] As another example, other loss functions can also be used, such as a contrast loss function calculated by comparing two or more similarities of slice pairs, or a categorical cross-entropy loss function.

[0096] For example, the trained function is configured to determine the similarity between two-dimensional medical images by comparing a first candidate image with the target image, comparing a second candidate image with the target image, and determining which of the two candidate images has a higher similarity to the target image. The target image is extracted from a set of target medical images, and the candidate images are extracted from a set of reference medical images.

[0097] For example, the registration process includes determining a location in a reference medical image set that matches the region of interest. For example, this process includes generating a first local image descriptor based on image data of the region of interest; generating a second local image descriptor for each of several candidate locations in the reference medical image set (the second local image descriptor is generated based on image data of the reference medical image set located at each candidate location); calculating a local image similarity metric for each of the several candidate locations, showing the local image similarity between the first and second local image descriptors for that location; selecting a candidate location from among several candidate locations based on the calculated local image similarity metric; and determining a location in the reference medical image set that matches the region of interest based on the selected candidate location. Furthermore, in the process of generating display data, rendering is generated based on image data of the reference medical image set relating to the set selected candidate locations.

[0098] A local image descriptor is, for example, a vector that represents or characterizes the underlying image data of a region of interest in a set of medical images and image data extracted from a set of reference medical images with respect to candidate locations. In some examples, local image descriptors are determined by encoding matching images using a specific image descriptor encoding algorithm. Features of each region of interest or slice, such as visual manifestations of medical abnormalities, patterns, anatomical structures, or medical landmarks, are encoded in each local image descriptor, for example. According to some examples, a local image descriptor consists of pattern descriptors that characterize one or more patterns in the underlying image data.

[0099] In some examples, local image similarity is calculated by applying a local image similarity metric that represents how similar individual local image descriptors are. In some examples, the local image similarity metric is configured to quantify the distance in feature space between local image descriptors of a region of interest and local image descriptors extracted with respect to candidate locations in a reference set of medical images. In some examples, such distances, cosine similarity, or Euclidean distance can be calculated using a predetermined mathematical function. In one example, the local image similarity or distance metric consists of a trained metric derived by machine learning. In some examples, the extraction of local image descriptors and / or evaluation of the overall local image similarity metric are performed by one or more trained functions. In one example, the local image similarity metric is performed as a pattern similarity metric, where local image similarity is the pattern similarity between image patterns associated with each location.

[0100] As a result, a technique is provided for determining the location where a predetermined pattern or feature of a target region is represented in a reference medical image set. For example, the location where a predetermined pattern or feature is represented in a reference medical image set is determined using known locations where the predetermined pattern or feature is represented in the target medical image set (i.e., the region of interest). This reduces the burden on the user when detecting relevant locations in a reference medical image set RMIS, for example. Furthermore, this determination, based on determining local image similarity, provides fast, efficient, and / or flexible feature locations, especially when conventional image registration cannot be performed. Moreover, by determining local image similarity, a robust, consistent, and stable feature location is obtained that is less susceptible to human error.

[0101] For example, candidate locations are those distributed across the entire set of reference medical images according to a predetermined sampling pattern. Performing local image descriptor extraction based on a predetermined distribution or sampling pattern enables overall sampling of the reference medical image set.

[0102] For example, the registration process further includes determining a plurality of candidate sub-resolution locations in a reference medical image set based on selected candidate locations, and obtaining a third local image descriptor for each of the plurality of candidate sub-resolution locations in the reference medical image set. Each third local image descriptor is generated based on the image data of the reference medical image set located with respect to each of the candidate sub-resolution locations. Then, for each of the plurality of candidate sub-resolution locations, a local image similarity metric is calculated, showing the local image similarity between the first local image descriptor and the third local image descriptor for the candidate sub-resolution location. Furthermore, a candidate sub-resolution location is selected from the plurality of candidate sub-resolution locations based on the calculated local image similarity metric showing the local image similarity between each of the first and third local image descriptors. Next, a location matching the region of interest in the reference medical image set is selected based on the selected candidate sub-resolution location. Thus, the candidate sub-resolution locations are locations distributed across the entire reference medical image set for the sub-resolution sampling pattern in the region of the selected candidate location. In this case, the distance between candidate sub-resolution positions in a predetermined pattern of sub-resolutions is smaller than the distance between candidate positions in a predetermined sampling pattern.

[0103] This method of hierarchically stacking two sampling patterns with different spatial resolutions makes it possible to quickly and efficiently determine the location corresponding to the region of interest, even when the location of any feature in the reference medical image set is unknown.

[0104] For example, candidate locations are determined based on a trained policy network that executes a trained policy, and the next candidate location is iteratively determined based on at least one of the previous candidate locations in order to analyze the reference medical images for locations that match the region of interest of the reference medical images.

[0105] Furthermore, the trained policy network is trained, for example, to output a probability distribution p(a) corresponding to action a for determining the next candidate position in each iteration, and the next candidate position is determined by sampling the probability distribution p(a).

[0106] As a result, the trained policy network is configured to define one or more candidate locations based on mapping the location of the region of interest to a reference medical image dataset.

[0107] A policy network can be thought of as a trained function that follows a specific learned policy for iteratively performing multiple actions to complete a learned task. The policy of a trained policy network is defined, for example, as the ability to select the next action to select candidate locations in order to iteratively detect locations that match a region of interest in such a way that long-term rewards are favored. In other words, the manner in which individual actions are selected starting from a given state of the trained function is called the "policy". A trained policy network therefore contains a policy of actions for a manner in which to detect locations of reference medical images that match a region of interest. Thus, the iterative relocation process evolves, for example, according to the policy, to develop current estimates of match locations and ultimately identify the match locations of the reference medical images. Running a trained policy network as a policy network can therefore be viewed as one aspect of a reinforced trained function.

[0108] One consequence of breaking down the problem of detecting match locations into multiple iterative processes is that traditional learning schemes, such as supervised learning, become insufficient. This is because, typically, only the label of the final state, i.e., the actual match location, exists, and there are no labels for the intermediate states on the way to the final state. Therefore, the trained policy network must learn to take actions in the environment to maximize a portion of the concept of future cumulative rewards (which will appear when the actual match location is reached). In this regard, reinforcement learning is a technique that facilitates learning as a thorough cognitive process on a trained function, instead of a predefined methodology. That is, the trained function gains flexibility by acquiring the ability to develop strategies to deal with an uncertain environment (in this case, medical image data).

[0109] By relying on reinforcement learning, it is possible to provide a trained function that can break down complex problems into multiple individual steps. In each step, the trained function evaluates its current state based on the input data and decides on the next action to move to the next state. When applied to a landmark detection problem, for example, the current state is considered to be the current candidate position as the current estimate of the match position, and the state is sampled using the aforementioned local image similarity. The action includes, for example, moving the current candidate position in the reference medical image set to move to a new (improved) candidate position. This process continues iteratively until, for example, the action reaches outside the image space of the reference medical image set (indicating that there are no valid landmarks contained in the image data) or until convergence is reached (i.e., there are no actions remaining to further improve the candidate position). Breaking down a problem into multiple partial problems in machine learning has the advantage of improving convergence behavior, meaning that the trained function will produce faster and better results. Furthermore, the trained function becomes more flexible, for example, in the sense that it can be applied to a wide variety of medical image data. Thus, the use of reinforcement-learned and trained functions synergistically contributes to improved methods in terms of speed, accuracy, and flexibility.

[0110] For example, a trained policy network is trained as a Markov decision process using policy gradients.

[0111] Adopting a Markov decision process effectively means that individual states and actions are considered statistically independent of preceding states. This forms the basis of an approach that iteratively determines the optimal policy during training. For each state, the best action is one that maximizes the future (cumulative) reward (cumulative because the individual rewards for each future action are summed up until convergence is reached). Using policy gradients is even more beneficial when the training data is panic and / or the input to the trained function is indeterminate.

[0112] For example, a trained policy network is considered to be trained to maximize the cumulative future reward value for a sequence of actions that identify a match location, as described above, by evaluating the gradient of each cumulative future reward with respect to one or more parameters of each trained function.

[0113] The behavior of a trained policy network can be viewed, for example, as the intelligent selection of the next action to advance the candidate position in a manner that maximizes cumulative future reward. That is, the trained policy network learns to determine the most favorable repositioning sequence necessary to accurately detect a matched position. Therefore, the state of the trained policy network is modeled, for example, as the local image similarity of the current candidate position. Subsequent behavior with respect to the candidate position responds to what is observed within the state.

[0114] Determining the next action based on a probability distribution (rather than calculating a specific reward value for each action) has the advantage of making this method more easily applicable to continuous and finely discretized actions, and generally to large action spaces.

[0115] For example, the probability distribution reflects the trained policy of the trained policy network, which is learned by evaluating the gradient of each cumulative future reward for one or more parameters of each trained policy network.

[0116] For example, a trained policy network is configured to improve candidate positions until a convergence point of possible match locations is detected, or until the trajectory of improved predicted candidate positions reaches a convergence point from the image space of the reference medical image set.

[0117] For example, the step of selecting at least one of candidate medical imaging tests as a reference medical imaging test involves identifying multiple related medical imaging tests from the candidate medical imaging tests based on a comparison of body regions, and the user interface To provide users with indicators of multiple related medical imaging tests, interface This includes receiving a user selection that specifies at least one of the relevant medical imaging tests, and selecting at least one specified relevant medical imaging test as a reference medical imaging test.

[0118] Multiple related medical imaging tests can be considered, for example, automatically provided as pre-selections, from which the user can make a selection. The metrics for multiple related medical imaging tests can be provided to the user in the form of a list of tests, from which the user can make a selection. For example, the list may include further details associated with each related medical imaging test, such as the time the test was acquired, modality and other imaging parameters, patient or test clinical parameters, one or more medical records or excerpts thereof. For example, the user interface The process of providing the user with indicators for multiple relevant medical imaging tests via [a specific method] includes, for example, providing highlights for multiple relevant medical imaging tests in a set of available candidate medical imaging tests.

[0119] By providing the user with potentially relevant tests and selecting one or more reference tests based on the user's selection of one or more potentially relevant tests, a sequential human-machine interaction is achieved regarding the directional selection of imaging tests for follow-up interpretation. For example, user selections can be used / recorded as the user's previous actions, and based on this, the process of selecting the next test for the user can be optimized.

[0120] Furthermore, the step of identifying multiple related medical imaging tests includes determining the degree of fit with the target medical imaging test based on a comparison of each body region for each related medical imaging test, and the step of providing indicators for multiple related medical imaging tests includes providing the degree of fit to the user. interface This includes providing the results to the user via [a specific method]. The fit is preferably based on anatomical overlap between each medical imaging examination, based on a comparison of the subject and candidate body regions.

[0121] Goodness of fit is, for example, an objective measure of how well two imaging tests match each other. For example, goodness of fit is calculated based on the body regions of the target medical imaging test and each candidate medical imaging test. As an example, goodness of fit consists of body region overlap determined based on the body regions of the target medical imaging test and each candidate medical imaging test. Alternatively or in addition to this, for example, the modality relevance score mentioned above, or the fit of the imaging parameters of the medical image sets composed of the target medical imaging test and each candidate medical imaging test may be incorporated into the goodness of fit.

[0122] By providing a degree of fit, users can obtain an objective measure of how well two medical imaging tests match. Therefore, users are provided with further assistance in selecting the appropriate medical imaging test.

[0123] For example, the process of acquiring the region of interest of the target medical image set is performed by the user interface This includes receiving user input to specify a region of interest via [a certain method].

[0124] For example, user input includes at least one of the following: a) selection of a (target) slice from multiple slices included in a set of target medical images (the set of target medical images draws an image volume, and each of the multiple slices draws a specific section of the image volume); b) manual boundary drawing of an image region within the set of target medical images; and c) designation of a single point of interest within the set of target medical images (the region of interest around it can be determined later).

[0125] By selecting a region of interest, the user can, in a sense, set the basis for selecting a set of reference medical images, and registration is performed. Thus, a suitable set of reference medical images is specifically selected according to the relevant part of the reference medical imaging examination that the user is currently focusing on. For example, if the region of interest shows a lung lesion, the examination and group search will be directed to obtain previous images of the patient's lung, in particular, of the same part of the lung acquired with comparable imaging parameters.

[0126] For example, in the process of generating display data and displaying a rendering of a group of reference medical images on a display device, user input and especially manual boundary drawing are preferably applied to the group of reference medical images based on registration to select a matching image slice (reference slice) from the group of reference medical images, or to crop the group of reference medical images in accordance with the manual boundary drawing.

[0127] This allows the displayed data to be adapted to the region of interest. As a result, users are provided with visualizations adapted to their region of interest, improving, for example, comparative image interpretation for the user.

[0128] For example, the step of selecting at least one of the candidate image sets as the reference medical image set involves identifying multiple related medical image sets from the candidate medical image set that are related to the target medical image set, based on the degree of comparability, and the user interface To provide the user with metrics for multiple related medical image sets (these metrics may consist of the degree of comparability), interface This includes receiving user selections via (these user selections are directed towards the relevant medical image set of interest) and selecting the relevant medical image set as the reference medical image set.

[0129] Multiple sets of related medical images can be considered, for example, automatically provided as pre-selected images, from which the user can make a selection. The metrics for these multiple sets of related medical images can be provided to the user in the form of a list of selectable groups.

[0130] This enables continuous human-machine interaction regarding the directional selection of image sets for follow-up interpretation. For example, user selection is used / recorded as a preceding user action that forms the basis for optimizing the subsequent group selection process for the user.

[0131] For example, each of the subject and reference medical image sets is associated with at least one attribute having an attribute value that indicates the imaging parameters used to capture each of the medical image sets. In this case, the user interface The process of providing the user with indicators for multiple related medical image sets via [a certain method] includes providing indicators for each attribute value.

[0132] For example, a reference medical image examination includes one or more annotations that match a set of reference images. The method further includes, for example, obtaining at least one reference annotation that matches a set of medical images and / or region of interest from one or more annotations, and providing that reference annotation. The reference annotation preferably includes one or more first words.

[0133] Preferably, providing reference annotations includes annotating the target medical image group and / or region of interest using at least one reference annotation.

[0134] Obtaining at least one reference annotation includes, for example, determining the location of each annotation within the reference medical image segment to which each annotation relates; for each location, preferably based on registration, determining whether that location has a matching location in the target medical image segment or region of interest; and, if a matching location can be determined for the location of the annotation, selecting the annotation as the reference annotation.

[0135] Therefore, for example, at least one reference annotation specifically relates to a reference slice.

[0136] Annotating the target medical image set and / or region of interest includes, for example, annotating the target medical image set and / or region of interest using at least one reference annotation of the determined matching location.

[0137] As illustrated above, for example, previous annotations are automatically provided, facilitating the user's follow-up image interpretation process. By automatically annotating the target medical image set and / or region of interest, the user can directly compare the previous annotations with the image features of the target medical image set.

[0138] For example, the step of obtaining at least one reference annotation includes, for each annotation, obtaining at least one location in a set of reference medical images, and for each location, determining a matched (or target) location and / or region of interest in the set of target medical images. The matched (or target) location matches at least one location. Thus, the annotation step includes, for example, annotating the set of target medical images and / or region of interest using reference annotations for at least one target location.

[0139] Determining the matching position is based, for example, on the registration described above.

[0140] Furthermore, determining the matching (or target) location, or for each location, determining whether that location has a matching location in the target medical image set or region of interest, is based, for example, on image similarity evaluation. For example, these steps include obtaining at least one reference image patch from the reference medical image set for each annotation, the reference image patch being positioned according to the location of the annotation. These steps also include determining, for example, a local image similarity metric indicating the local image similarity between each of the multiple target image patches in the target medical image set and / or region of interest, and the local image similarity between each of the reference image patches. These steps also include selecting a matching image patch from the multiple target image patches based on the similarity metric (if possible), and determining the matching (or target) location based on that matching image patch.

[0141] Local image similarity can be calculated using the local image similarity metric, as basically described in relation to the registration process.

[0142] For example, the target patch with the highest local image similarity to the reference patch is selected as the matching image patch. Furthermore, the matching image patch is identified, for example, when its local image similarity exceeds a predetermined threshold. (Local image) Similarity If no target image patch is identified that exceeds a predetermined threshold, the associated annotation will not match the target medical image set and, therefore, will not apply to the target medical image set, for example.

[0143] For example, the method further includes obtaining a medical report linked to a reference medical imaging examination, obtaining one or more sections of the medical report text, each section containing one or more second words for each of the one or more sections, identifying matches for each reference annotation by comparing one or more second words with one or more first words in the reference annotation, and, based on the identified matches, linking at least one of the reference annotations to at least one of the first sections. Optionally, the linked first section is provided to the user. interface It may be provided to the user via [a specific method / platform].

[0144] This allows for the identification of matching sections from previous medical reports based on automatic annotations of the target medical image set, making the target medical image set suitable for interpretation. In this way, users no longer need to manually search for previous reports and relevant sections within them, and can immediately determine which findings reported in previous medical reports have changed.

[0145] The comparison of one or more words is performed, for example, by applying a machine learning-based classification to the first and second words. The machine learning-based classification is trained to map similar words to similar regions in the feature space. For example, words are encoded into vector representations using word embeddings. For example, word embeddings map words to a vector space, in this case the vector for each word can be generated by applying a word embedding model to a corpus of training texts. An example of a known word embedding model is "Word2vec," which uses a neural network to learn word embeddings from a corpus of training texts.

[0146] For example, the method further includes identifying the user providing the display data and obtaining one or more prior actions of the user. Prior actions refer to at least one of the following: a test selection action, which selects a reference medical imaging test from a group of candidate medical imaging tests; and a group selection action, which selects a group of reference medical images from a group of candidate medical images for a medical imaging test. Thus, the step of selecting a reference medical imaging test is based, for example, on at least one test selection action. Similarly, the step of selecting a group of reference medical images is based, for example, on at least one group selection action.

[0147] By taking into account the user's prior actions, the (semi)automatic selection process described here can be adapted to the user's preferences. This allows for the selection of appropriate groups and tests, and the user can choose whether to manually explore the available tests / groups. Furthermore, if the test and / or group selection is performed by a machine learning-trained function, that prior action can be used, for example, to further train the underlying machine learning-trained function.

[0148] Basing the selection process on at least one prior action includes, for example, evaluating one or more prior actions (e.g., statistically) to determine the probability that the user will select individual candidate medical imaging tests or groups. Thus, the probability provides an additional basis in, for example, the step of selecting reference medical imaging tests or groups of reference medical images.

[0149] For example, this method further includes quantifying the changes between image data of the region of interest within the target medical image set and matching image data of the reference medical image set, generating change display data, and visualizing and displaying the changes on a display device.

[0150] According to some examples, the change is determined by converting the target medical image set to the coordinate system of the reference medical image set based on registration, or vice versa, to generate a converted medical image set, and then determining the change based on this converted medical image set (for example, by comparing or subtracting the target medical image set or the reference medical image set from the converted medical image set).

[0151] The changes may relate to, for example, the patient's condition. The changes may relate to, for example, tissue changes between the target medical image set and the reference medical image set. The changes may relate to, for example, abnormal growth, reduction, appearance, or disappearance from the target medical image set to the reference medical image set (i.e., from the first time point to the second time point). Examples include the growth or reduction of nodules, and the development of new nodules and / or lesions.

[0152] Visualizing change includes, for example, rendering a set of medical images of interest (region of interest) and / or reference medical images in a way that highlights the changes. In this example, “highlighting” means, for example, visually increasing brightness, color, and / or intensity to indicate the changes. In addition to or instead of this, changes can also be highlighted using symbols. Alternatively, changes can be highlighted using labels consisting of semantic representations. Highlighting also means, for example, using a heatmap in which the amount of change is color-coded. For example, shrinking nodules can be color-coded differently from growing nodules.

[0153] As an example, generating change display data includes at least one of the following: including visualization of changes in the rendering of a set of reference medical images, preferably overlaid on the rendering of the set of reference medical images; or including visualization in the rendering of a region of interest, preferably overlaid on the rendering of the region of interest.

[0154] By providing users with renderings that highlight changes, they can instantly infer what changes have occurred and where those changes took place. This assists in guiding image interpretation, thereby enhancing the usefulness of the method and providing users with better support for guiding medical diagnoses.

[0155] For example, the target medical image set is acquired using a first medical imaging modality, and the reference medical image set is acquired using a second medical imaging modality. In this case, the first medical imaging modality is based on an imaging technique different from the imaging technique on which the second modality is based. For example, the first modality is based on X-ray imaging techniques (such as CT or X-ray projection), and the second medical image set (CMIS) is preferably based on magnetic resonance imaging techniques.

[0156] In other words, this method makes it possible to synchronize medical image datasets acquired with functionally different imaging modalities.

[0157] For example, candidate medical imaging tests and / or groups are stored in a remote storage device, and the method includes retrieving medical image data of candidate medical imaging tests and / or groups selected as reference medical imaging tests and / or groups from the remote storage device, without retrieving medical image data of candidate medical imaging tests and / or groups that have not been selected as reference medical imaging tests and / or groups.

[0158] In other words, this means that, for example, image data transfer will not occur until a candidate medical imaging test or group is selected as a reference medical imaging test or group.

[0159] For example, the step of selecting a reference medical imaging examination from multiple candidate medical imaging examinations is based on non-image data associated with the candidate medical imaging examinations. For example, the step of determining the body region of a candidate medical imaging examination is based, for example, on non-image data associated with the candidate medical imaging examination. As a result, the step of selecting a reference medical imaging examination includes retrieving non-image data of the candidate medical imaging examination from a remote storage device without retrieving the medical image data of the candidate medical imaging examination.

[0160] Similarly, for example, the step of selecting a reference medical image set from multiple candidate medical image sets is based on non-image data associated with the candidate medical image sets. For example, the step of determining the degree of comparability of the candidate medical image sets is based on non-image data associated with the candidate medical image sets. As a result, the step of selecting a reference medical image set includes, for example, searching for non-image data of the candidate medical image sets from a remote storage device without searching for the medical image data of the candidate medical image sets.

[0161] By determining the degree of body area / comparability based on (computationally relatively small) non-image data, it becomes unnecessary to transmit (computationally relatively large) medical image data over the network to determine the degree of body area / comparability represented by said data, thus enabling efficient use of network resources.

[0162] For example, the method further includes providing an anomaly detection function configured to detect one or more anatomical anomalies in medical image data, detecting anatomical anomalies in a set of medical images by applying the anomaly detection function to the set of medical images, and obtaining the location of the anatomical anomalies. In this case, the step of obtaining the region of interest is based on the location of the anomaly.

[0163] For example, an abnormality (or "abnormal structure") within a patient is an anatomical structure that distinguishes that patient from other patients. For example, an abnormality can be associated with a specific medical condition in the patient. An abnormality can be located within various organs of the patient (for example, in the patient's lungs or liver), or it can be located between organs of the patient. For example, an abnormality can be a foreign body.

[0164] For example, an abnormality is a neoplasm (also called a "tumor"), particularly a benign neoplasm, non-invasive neoplasm, malignant neoplasm, and / or an unconfirmed / unidentified neoplasm. For example, an abnormality is a nodule, particularly a pulmonary nodule. For example, an abnormality is a lesion, particularly a pulmonary lesion.

[0165] Anomaly detection functions are implemented, for example, as computer-aided detection (CADe) and computer-aided diagnosis (CADx) systems. Such systems are techniques that help radiologists interpret medical images. A common application of CAD systems is the automatic identification of suspicious areas in medical images. These suspicious areas include image patterns that indicate anomalies consisting of, for example, cancerous growth, tumors, abscesses, lacerations, calcifications, lesions, and / or other irregularities in biological tissue that, if left undetected, can cause serious medical problems. In principle, there are many functions and methods known for computer-aided detection of anomalies, all of which can be implemented in anomaly detection functions. For example, US2009 / 0092300A1, US2009 / 0067693A1, and US2016 / 0321427A1 are invoked. As an example, anomaly detection functions are based on one of the following: support vector machines, naive Bayes, neural networks, decision trees, and / or ResNet.

[0166] By acquiring regions of interest through an automated anomaly detection workflow, the user's attention can be automatically focused, for example, on the most relevant parts of a set of medical images, and based on this, appropriate reference medical images can be specifically retrieved. This further assists the user, for example, in providing medical diagnoses.

[0167] For example, a system is provided to assist in the evaluation of a set of target medical images of a patient acquired at a first time point. This system includes an interface unit and a computing unit. The interface unit is configured to provide the user with a rendering of the target medical image set and to receive user input from the user for the purpose of specifying regions of interest in the target medical image set based on the rendering of the target medical image set. The computing unit is configured to acquire regions of interest based on user input, determine the target body regions represented by the target image data set, select a reference medical imaging examination from multiple candidate medical imaging examinations based on a comparison between multiple target body regions and multiple candidate body regions, where each of the multiple candidate body regions corresponds to one of the multiple candidate medical imaging examinations, and each candidate medical imaging examination includes multiple candidate medical image sets of the patient at a second time point. The computing unit is configured to select a reference medical imaging examination from multiple candidate medical imaging examinations based on the degree of comparability with the target medical image set, perform registration of the target medical image set and the reference medical imaging examinations, and generate display data to display the rendering of the reference medical imaging examinations to the interface unit based on the registration.

[0168] The computing unit includes, for example, an examination selection module or unit configured to select a reference medical image examination as described herein. The computing unit includes, for example, a group selection module or unit configured to select a group of reference medical images as described herein. The computing unit includes, for example, an image registration unit configured to generate at least one image registration as described herein. The computing unit includes, for example, a visualization module or unit configured to generate display data based on at least a group of reference medical images as described herein.

[0169] Selectively, the computing unit includes, for example, an annotation module or unit, which is configured to annotate a target medical image set based on annotations contained in a reference medical image set RMIS (if any), and / or to match annotations associated with the target medical image set to existing sections of a patient's medical report, as described herein. Also, selectively, the computing unit includes, for example, a comparison unit that compares medical image data and determines changes based on this comparison, as described herein.

[0170] A computing unit is implemented, for example, as a data processing system or as part of a data processing system. Such data processing systems include, for example, cloud computing systems, computer networks, computers, tablet computers, smartphones, and / or other similar devices. A computing unit consists, for example, of hardware and / or software. The hardware includes, for example, one or more processors, one or more memories, and combinations thereof. One or more memories store, for example, instructions for performing each step of the method according to the present invention. The hardware is configurable and / or operable by software. Generally, all units, subunits, or modules communicate with each other, at least temporarily, for example, through network connections or their respective interfaces. That is, individual units may be located far apart from each other.

[0171] The interface unit includes, for example, an interface for exchanging data with a local server or a central web server via an internet connection to receive reference image data or tracking image data. The interface unit further includes, for example, displaying the results of processing by the computing unit to the user (e.g., a graphical user interface). interfaceIt is configured to collaborate with one or more users of the system by allowing the user to adjust parameters for image processing or visualization and / or medical imaging examinations and groups.

[0172] For example, the system is configured to perform the method according to the present invention in various embodiments to determine abnormal changes in anatomical regions of a patient. The advantages described in relation to embodiments of the method are also realized by the correspondingly configured components of the system.

[0173] For example, the interface unit is configured to further visualize the evaluation tool, in this case the evaluation tool is placed within the rendering of the target medical image set, and to receive user input regarding the placement of the evaluation tool during the rendering of the target medical image set to define the region of interest. The computing unit is also configured, for example, to place the evaluation tool in relation to the region of interest during the rendering of the target medical image set.

[0174] For example, the interface unit may be configured to display the evaluation tool as a user-configurable circular or rectangular evaluation tool, positioned as an overlay on the rendering of the target medical image set.

[0175] For example, the interface unit is configured to provide an evaluation function configured to quantify the change between image data of the region of interest and corresponding image data of a reference medical image set, and to receive user input from the user related to the selection of this evaluation function. The computing unit is configured to execute the evaluation function based on the user input, generate change display data, and display a visualization of the change on the interface.

[0176] In other words, user-friendly image evaluation / analysis tools are provided, which can be arbitrarily placed within, on top of, or overlaid on the displayed image dataset. In this way, the evaluation / analysis tools allow users to easily select a location or region of interest within the displayed image volume by directly moving the tool to that location. When placed in a region of interest, the evaluation tool is configured to provide multiple different evaluation / analysis functions for the user to select. Each evaluation function is configured to provide evaluation / analysis results for the image data within the selected location. Furthermore, medical image data, a palette of evaluation functions, and evaluation results are provided. in parallel A graphical user interface configured to visualize this is provided.

[0177] For example, an image analysis system is provided which includes the above system and a medical information system configured to acquire, store, and / or transfer at least the subject and reference medical imaging examinations and / or groups. The interface unit is configured to receive the subject and reference medical imaging examinations and / or groups from the medical information system.

[0178] According to some examples, a medical information system includes one or more archive stations that store subjects, candidates, and reference medical imaging examinations and / or groups. These archive stations may be implemented, for example, as remote storage in the form of cloud storage, or as local or distributed storage, such as a PACS (Picture Archiving and Communication System). Furthermore, a medical information system may include one or more medical imaging modalities, such as computed tomography systems, magnetic resonance imaging systems, angiography (or C-arm X-ray) systems, positron emission tomography systems, and mammography systems.

[0179] In one embodiment, the present invention also covers a computer program (recorded) containing program elements, wherein when the program elements are loaded into the memory of a system's computing unit, the computing unit is instructed to perform one or more steps according to one or more embodiments of the above method.

[0180] In one embodiment, the present invention also covers a computer-readable medium on which program elements are recorded, the program elements being read and executed by a computing unit of a system, and the computing unit executing the program elements performing one or more steps according to one or more embodiments of the above method.

[0181] The implementation of the present invention by computer program products and / or computer-readable media has the advantage that existing supply systems can be easily adapted by software updates to operate as proposed by the present invention.

[0182] A computer program product is a computer program and may include, for example, other elements that lead to such a program. Such other elements include hardware, such as a memory device storing the computer program, a hardware key for using the computer program, and / or software, such as documentation or a software key for using the computer program. A computer program product may further include, for example, development materials, a runtime system, and / or a database or library. A computer program product may be distributed, for example, between several computer instances.

[0183] The features, characteristics, and advantages of the invention described above, as well as the actions that achieve them, will be made clearer and more understandable from the following description of embodiments, which will be explained in detail with reference to the drawings. The following description is not intended to limit the invention by the embodiments contained herein. The same components, parts, or processes are given the same reference numerals in multiple drawings whenever possible. In general, the drawings are not to exact scale. [Brief explanation of the drawing]

[0184] [Figure 1] A schematic diagram illustrating an embodiment of a system that supports the evaluation of a patient's medical image set. [Figure 2] A diagram schematically showing an image file according to the embodiment. [Figure 3] A diagram schematically showing an image file according to the embodiment. [Figure 4] A schematic diagram illustrating a method for generating display data for a group of medical images according to an embodiment. [Figure 5] A schematic diagram illustrating a method for determining the body region in a medical imaging examination according to an embodiment. [Figure 6] A diagram schematically showing the data flow between components according to the embodiment. [Figure 7] A schematic diagram showing a graphical user interface (GUI) according to the embodiment. [Figure 8] A schematic diagram illustrating a method for selecting a reference medical imaging examination according to an embodiment. [Figure 9] A diagram schematically illustrating the data flow between components in a method for selecting a reference medical imaging examination according to an embodiment. [Figure 10] A schematic diagram illustrating a method for selecting a group of reference medical images according to an embodiment. [Figure 11] A diagram schematically showing the data flow between components in a method for selecting a group of reference medical images according to an embodiment. [Figure 12] A flowchart illustrating a method for registering two sets of medical images by identifying matching slices according to the embodiment. [Figure 13]A schematic diagram illustrating the data streams related to a method for registering two sets of medical images by identifying matching slices according to an embodiment. [Figure 14] A flowchart further illustrating the steps of a method based on the identification of matching slices in a group of medical images according to the embodiment. [Figure 15] A flowchart further illustrating the steps of a method based on the identification of matching slices in a group of medical images according to the embodiment. [Figure 16] A flowchart illustrating a method for selecting one or more medical image groups according to an embodiment. [Figure 17] A flowchart illustrating a method based on the identification of matching slices in a group of medical images according to an embodiment. [Figure 18] A flowchart illustrating a method for training a pre-trained function to identify matching slices in a set of medical images according to an embodiment. [Figure 19] A flowchart illustrating a method for registering two sets of medical images by identifying matching image patterns according to the embodiment. [Figure 20] A diagram schematically showing the target medical image group according to the embodiment. [Figure 21] A diagram schematically showing a set of reference medical images according to the embodiment. [Figure 22] A diagram schematically showing a set of reference medical images according to the embodiment. [Figure 23] A diagram schematically showing a set of reference medical images according to the embodiment. [Figure 24] A diagram schematically showing a set of reference medical images according to the embodiment. [Figure 25] A diagram illustrating the output data according to the embodiment. [Figure 26] A diagram illustrating the output data according to the embodiment. [Figure 27] A flowchart illustrating a method for annotating a group of medical images according to the embodiment. [Figure 28] A flowchart illustrating a method for determining annotations for a target medical image group based on a reference medical image group according to an embodiment. [Figure 29]A schematic diagram showing the image components of the morphology of image slices in the target medical image group according to the embodiment. [Figure 30] A schematic diagram showing the image components and associated annotations of image slices in a reference medical image group according to the embodiment. [Figure 31] A schematic diagram showing the data stream related to the method for annotating a group of medical images according to the embodiment. [Figure 32] A schematic diagram showing the rendering of display data according to the embodiment. [Figure 33] A schematic diagram showing the rendering of display data according to the embodiment. [Figure 34] A schematic diagram showing elements of a graphical user interface (GUI) according to the embodiment. [Figure 35] A diagram showing a system that supports the evaluation of a group of medical images according to the embodiment. [Figure 36] A diagram showing the evaluation tool according to the embodiment in a first operating state. [Figure 37] A diagram showing the evaluation tool according to the embodiment in a second operating state. [Figure 38] A diagram showing the evaluation tool according to the embodiment in a third operating state. [Figure 39] A flowchart illustrating a method for supporting the evaluation of a group of medical images according to an embodiment. [Figure 40] A diagram showing an evaluation tool for determining and visualizing reference medical image data according to an embodiment. [Modes for carrying out the invention]

[0185] Figure 1 shows the target medical image group TMIS. Reference medical imaging tests RS Identification Therefore, regarding the target medical image group TMIS Reference medical image collection RMIS Identification In order to do so, to annotate the target medical image group TMIS, and / or 、 That bass as To generate display data 、 This shows System 1. In this example System 1 is, for example, shown in Figure 2- Figure 34 See As will be discussed later In one or more examples related The method is configured to be implemented. Users of System 1 are generally medical professionals such as physicians, clinicians, technicians, radiologists, and pathologists, according to some examples. do .

[0186] System 1 is User Interface It comprises 10 (as part of the interface unit) and a processing system 20 (as part of the computing unit 30). Furthermore, system 1 includes for example, Medical information system 40 Includes or connected to this The medical information system 40 is For example, outline , reference medical imaging examination RS, target medical imaging examination TS, reference medical imaging group RMIS, and / or target medical imaging group TMIS Configured to acquire and / or store and / or transfer . in particular The medical information system 40 is For example, reference medical imaging tests RS, Target medical imaging examinations Configured to handle image data files (IF) containing TS, reference medical image sets (RMIS), and / or target medical image sets (TMIS). It will be done .for example, Medical Information System 40 is Match To store medical imaging examination and group image data files (IF), one or more archive / review stations (not shown) may be included. The archive / review station may be connected to one or more databases. implement It is possible. More specifically The archive / review station can be implemented in the form of one or more cloud storage modules. or Archive / review stations are, for example, PACS (Picture Archiving and Communication System: Medical Image Management System) as, Local or distributed storage It may be implemented as follows. Furthermore, the archive / review station will handle image data files IF Additional related to Clinical information as well It can be saved, That clinical information For example, relevant medical findings, patient case Preparations made in the preliminary survey Key image, one or more previous (structured) Medical reportPersonal information and patient records related to the patient under consideration. (Medical record) etc. include According to some embodiments, Medical Information System 40 includes a computed tomography system, a magnetic resonance imaging system, and an angiography (or C-arm X-ray) system. positron emission Tomography systems, mammography systems, systems for acquiring digital pathology images, etc. One or more medical imaging Modalities (not shown) include .

[0187] User interface 10 is, Whole unit Enter 11 unit 12 and has . User Interface 10 is, Mobile devices such as smartphones and tablet computers by It may be implemented. Furthermore, the user interface 10 is a workstation in the form of a desktop PC or laptop. It can also be implemented. The input unit 12 is connected to the display unit 11, for example, in the form of a touchscreen. Even if you incorporate it Good. As an alternative, or in addition thereto, the input unit 12 is: keyboard ,mouse 、 or digital pen, and those combinations It can include... Whole unit 11 refers to the target medical image group TMIS and / or the reference medical image group RMIS. expression It can be configured to display [something]. More specifically , Whole unit 11 can be configured to display individual slices of the target medical image group TMIS and / or the reference medical image group RMIS.

[0188] User interface 10 is further, Whole unit 11 and input unit 12 For handling at least one software component Includes an interface computing unit 13 configured to perform the following: To enable users to select target medical imaging tests (TS) to review, the target medical image set (TMIS) from the target medical imaging tests (TS) is used. In order to receive user selections, and / or In the target medical image group TMIS ROI in areas of interest Definition to、 Graphical user interface (For example, Figure 36- Figure 38 ) provides Also, the interface Computing unit 13 is For example, to receive one or more reference medical image sets RMIS for comparison with the target medical image set TMIS, or display data related to the reference medical image set RMIS, the system communicates with the medical information system 40 or the processing system 20. It is configured as follows: User, User interface via 10 Software Components It can be launched, and also, for example Internet application store By downloading from Software Components It can be obtained. For example, the software component may be a client-server computer program in the form of a web application that runs in a web browser. interface Computing unit 13 is For example, a general-purpose processor , central processing unit, control processor, graphics processing unit, digital signal Processor, 3D rendering Processor , Image processor Application-specific integrated circuits, field-programmable gate arrays, digital circuits, analog circuits, These combinations , or processing image data Other known devices .

[0189] The processing system 20 is, for example, Processor That is .this Processor teeth, For example, a general-purpose processor Central processing unit , control processor, graphics processing unit ,digital signal Processor, 3D rendering Processor , Image processor Application-specific integrated circuits, field-programmable gate arrays, digital circuits, analog circuits, and these combination , or processing image data Other known devices The processor is a single device They may be connected in series, parallel, or separately. Operation multiple deviceThe processor may also be the main processor of a computer, such as a laptop or desktop computer, or it may be a processor for handling several tasks in a larger system, such as a medical information system or a server. The process described herein by instructions, design, hardware, and / or software. of To execute It is configured. Alternatively, the processing system 20 is for example, What is commonly known as a "cluster" or "cloud," It consists of a group of real computers or a group of virtual computers. . Such server systems teeth, for example, Central server , For example, cloud server , or, For example, a local server located at a hospital or radiology department site. Furthermore, the processing system 20, For example, medical images RAM or other memory to temporarily load datasets MIDS-1 and MIDS-2 include .or, The memory is, user interface It may be included within 10.

[0190] ( for example Morphology of the target medical image group TMIS, reference medical image group RMIS, and candidate medical image group CMIS of )Here explain Medical image sets are, for example, three-dimensional image datasets acquired using computed tomography systems, magnetic resonance imaging systems, or other systems. be Image information is, for example, A three-dimensional array of m x n x p voxels Encoded . Medical image group RMIS, TMIS, CMIS medical images teeth, For example, it includes multiple image slices S-1, S-2 (which can also be simply referred to as "slice"), and these image slices are stacked in a stacking direction that extends to the image volume covered by each medical image group RMIS, TMIS, and CMIS. .

[0191] Furthermore, the medical image sets RMIS, TMIS, and CMIS are, for example, m×n pixel array Encoded Image information accompanied One or more individual two-dimensional medical images include According to some examples, these two-dimensional medical images are extracted from a set of three-dimensional medical images. It was done. According to other examples, two-dimensional medical images are X-rays. imaging Modality etc. appropriate imaging Using modalities Not really acquisition It was done. .

[0192] A collection of voxels or pixels is For example, as will be described later, each dataset will be represented as image data. Generally, ultrasound, X-ray, angiography, Fluorescence Fluoroscopy Positron emission tomography, single-photon emission computed tomography images, etc. To obtain Various types of imaging Modalities and scanners can be used. Generally speaking The medical imaging sets RMIS, TMIS, and CMIS depict various anatomical structures and organs. Consists of This shows the body parts of the patient. If we consider the chest region as a part of the body... Medical imaging systems such as RMIS, TMIS, and CMIS are used to visualize areas such as the lung lobes, thoracic cage, heart, and lymph nodes. draw For example, an appropriate image Viewer When processed by software, medical imaging The data ID is, The data represents one or more medical images rendering or Generate an expression .

[0193] Medical image sets RMIS, TMIS, and CMIS are, For example, one or more Save as an image file in IF format. It will be done An example of an image file interface is shown in Figure 2. The image file interface is: for example, Actual medical imaging Data ID In addition to preserving, medical use imaging Data ID content of point out Each of the attribute A has an attribute value AV containing a text string, further Save .

[0194] One or more attribute values ​​AV are medical imaging Data ID It is separated from and distinguished from Instead, medical imaging Data ID content of point out Includes text strings. Such attribute values ​​AV are, in some examples, metadata of an image file IF and Called In some examples, the part of the image file IF that stores attribute A and attribute value AV is the header of the image file IF and Called, the attribute A and the attribute value AV are related to the header data of the image file IF and Called .

[0195] A specific example of the image data file IF is a DICOM (Digital Imaging and Communications in Medicine) file. Examples of DICOM files This is shown in Figure 3 and will be described in detail below. . In summary , the DICOM file stores the medical Pixel data data ID as the specified data element IF-312 imaging , and further One or more other data elements IF-310, each having one or more attribute A, each having an attribute value AV containing a text string that points to the content of the medical imaging data ID. stores it. Such The One example is the "Study Description," and that attribute value IF-314 of the DICOM attribute IF-310 This is a text string that describes the examination—the examination in which the medical imaging data is a part (for example, "NERUO^HEAD" where the medical imaging data is the patient's head region), and by this is the imaging medical Point out the content data 316. Such Other examples of DICOM attributes However, these include "Series description" and "Body Part Examined," etc. are as follows.

[0196] In one example , Regarding a specified patient at a specified time , a radiologist For example, the patient in question performs imaging medical do examinations. Another test may be conducted at a different time. . A specific examination uses The opposite, a specific imaging modality (e.g., MR) on a specific body part of a patient do . In some cases, multiple examinations are performed on separate the body part of a patient Acid , and / or Different imaging modality Use (i.e., by multiple devices Capture ). Sometimes The examination results are for example, stored in one or more medical image data files IF It will be done . prescribed The examinations CS, RS, TS For example, one or more medical images include the groups TMIS, CMIS, RMIS [[ID=​​​​​​​​​​Using parameters) Captured Tests: TS, CS, RS inside Each group consists of TMIS, CMIS, and RMIS. imaging If the parameters are different too Yes. In some cases, image files (IFs) represent medical images within a specific group and within a specific examination. imaging Save the data ID. one Medical images that represent medical images, which are part of the TS, CS, and RS tests. imaging The image file interface for saving the data ID is, for example, Image file IFs that store medical imaging data IDs representing medical images that have the same unique examination identifier and are part of a single group TMIS, CMIS, RMIS are ,for example, Having the same unique group identifier . In either case , prescribed The test is, for example, at least one such image file It has at least one group .

[0197] Here explain The method and system are currently review Regarding the target medical imaging examination TS for patients undergoing this procedure, Medical Information System 40 In The objective is to automatically select the appropriate reference medical imaging test (RS) from multiple candidate medical imaging tests (CS) available for a patient. Furthermore, the methods and systems described herein are The objective is to automatically select the appropriate reference medical image group (RMIS) from multiple candidate medical image groups (CMIS) included in the selected reference medical imaging examination (RS). Let's assume Furthermore, here explain The method and system aim to automatically register a target medical image group TMIS to a reference medical image group RMIS and generate display data based on that registration. For this purpose, the processing system 20 may include a plurality of subunits 21-27 configured to process the relevant examination TS, CS, TS and group TMIS, CMIS, RMIS as appropriate.

[0198] Subunit 21 is data search Module or unit can be, Medical images inspection TS, CS, RS and / or Match Medical image group TMIS, CMIS, RMIS and / or MatchImage file IF to be processed Regarding medical information System 40 of Access and / or search configured to do so. For example, the subunit 21 is currently review regarding the target medical image group TMIS of the target medical image examination TS in progress death and Preliminary tests for potentially affected patients (i.e., candidate medical image examination CS), it can be configured to query the system 40 Medical Information Specifically, the subunit 21 can be configured to parse the relevant information and Formulate the search query send it to the medical information system 40. In particular, the subunit 21 can be configured to extract data identifiers from the available information and use them to query the medical information system 40 these

[0199] The subunit 22 is configured as an examination selection module or unit to select at least one target medical image examination TS relevant to a plurality of It can be held, medical image examinations CS candidate and reference medical image examinations RS from and especially for the region of interest ROI. For this purpose, the subunit 22 can be configured to remarkable compare body regions with each other to determine the reference medical image examination RS. By comparing body regions Determine which medical imaging tests (TS and CS) should be used to compare the body regions represented by each of them. it can be configured to Also, The subunit 22 can be configured to Thus it is derived compare body regions with each other to determine the reference medical image examination RS. When comparing body regions by it can be configured to determine the candidate medical image examination CS Each of and the target medical image examination TS degree of suitability Specifically, the subunit 22 can be configured to execute S30 and S40 here may include The process of explaining Also For example, this will be explained with reference to Figures 5-9. process In addition, the subunit 22 can be configured to execute the method 、 do to perform and / or 、 the image processing functions described in this context do to perform 、 as follows It will be done configured It will be done ​​

[0200] Subunit 23 is a group selection module or unit It can be held, The system is configured to select at least one reference medical image group RMIS from multiple candidate medical image groups CMIS included in the reference medical imaging examination RS that can be easily compared with the target medical image group TMIS. Specifically, subunit 23 can be configured to determine the degree of comparability between each candidate medical image group CMIS and the target medical image group TMIS. Then, the candidate medical image group CMIS with the highest degree of comparability can be selected as the reference medical image group RMIS. Specifically, subunit 23, for example, Here The process of explaining Configure to run S50 It will be done . Also Subunit 23 is, for example, Figure 10 and Explained with Figure 11 Method Process To perform, and / or 、 Explanation in this context do Image processing functions to perform like 、 composition It will be done .

[0201] Subunit 24 is a registration module or unit It can be captured Subunit 24 is configured to perform registration of the target medical image set TMIS and the reference medical image set RMIS. Therefore ,Here Process S to be explained It is configured to run 60. specifically Subunit 24 is, fundamentally One of the image groups image The data in the coordinate system of the other set of images fart It can be configured to calculate the coordinate transformation to be performed. The calculation results provided by subunit 24 are for example, Two-dimensional or three-dimensional transformation matrix or Deformed field Form That is Subunit 24 performs rigid image registration, affine image registration, non-rigid image registration, and Any combination of theseIt can be configured to apply one or more image registration techniques, including the following. To improve the registration results, subunit 24, Selectively, Soft tissue deformation Regarding In one or more motor models Calculation results Structured to be mathematically compatible It is also possible to do so. . Alternatively Alternatively, in addition to the registration techniques described above, the subunit 24 may be configured to correlate or synchronize slices of the target medical image group TMIS and the reference medical image group RMIS based on the image similarity between individual slices, thereby providing registration between the reference medical image group RMIS and the target medical image group TMIS. In this regard, the subunit 24 may be configured to: for example, Figure 12- This will be explained with Figure 17. method process To perform, and / or 、 Explanation in this context do Image processing functions to perform like 、 composition It will be done Furthermore, subunit 24 directly identifies similar patterns in the reference medical image group RMIS and the target medical image group TMIS. Identification death, Correlating the locations of these patterns in the reference medical image set RMIS and the target medical image set TMIS. This allows the system to be configured to provide registration. In this regard, subunit 24 is for example, Figure 19- This will be explained with reference to Figure 26. method process To perform, and / or 、 Explanation in this context do Image processing functions to perform like 、 composition It will be done .

[0202] Subunit 25 is an annotation module or unit as It can be held, Reference medical image set RMIS Notes included (if any) Based on the target medical image group TMIS Add annotations to it , and / or The annotations associated with the target medical image set TMIS are matched with existing sections F1-F3 of the patient's medical report RM-504. It can be configured. Subunit 25 is for example, Figure 27- This will be explained with reference to Figure 34. method process To perform, and / or、 Explanation in this context do Image processing functions to perform Configured in this way It will be done .

[0203] Subunit 26 is, for example, Configured as a comparison module or unit It will be done Subunit 26 is, for example, a reference medical image set RMIS. and Target medical image group TMIS Images included The system can be configured to compare data IDs with each other to determine changes between the reference medical image group RMIS and the target medical image group TMIS. Furthermore, subunit 26 compares the reference medical image group RMIS and the target medical image group TMIS in The system may be configured to correlate different representations of anomaly A. Furthermore, subunit 26 is, the It may be configured to quantify changes in abnormalities based on correlation. For this purpose, subunit 26 extracts data from the reference medical image set RMIS to the target medical image set TMIS. Up to Abnormal Size Change and / or Volume change and / or Strength change and / or Texture change It may be configured to determine changes in and / or other parameters.

[0204] Subunit 27 is a visualization module configured to generate display data based on at least the reference medical image set RMIS, It can be captured Specifically, subunit 27 is based on registration and the region of interest ROI. Match Reference medical image set RMIS Render the representation of It can be configured in this way. Also Subunit 27 renders the region of interest ROI and / or reference medical image set RMIS. Emphasize change and add annotations 、 It can be configured to provide. Such a representation is Changes and / or annotations are visually encoded. Form of auxiliary images It can be done .this is, For example, change and / or annotations are extended in their expression. , that means Specifically, subunit 27 is Changes and / or annotations from the matching rendered region of interest (ROI) and / or the matching region of the reference medical image set (RMIS) to be displayed overlaid on the matching region. Semi-transparent overlay image Render algorithm , run or execute It can be configured in this way.

[0205] individual The designations of subunits 21-27 should be interpreted illustratively, not restrictively. In other words The subunits 21-27 may be integrated to form a single unit (for example, in the form of a "computation unit 30"), or 、 Processors of processing system 20, etc. Operate method process By a computer code segment configured to execute implementation It may be done. The same thing. but Regarding the interface computing unit 13 as well Applies Each subunit 21-27 and interface computing unit 13 is a method process Other subunits of System 1 that require data exchange in order to perform this operation and / or other component They can be connected individually. For example, subunit 21 and subunit 27 is To search for medical IF, Connected to the medical information system 40 via interface 29. It is possible Similarly, interface 29 is For example, to transfer the calculation results to the user and collect user input, Interface subunits 21-27 Computing Connect to Unit 13 do .

[0206] The processing system 20 and the interface computing unit 13 can together constitute the computing unit 30. Naturally The layout of the computing unit 30, that is, the physical distribution of the interface computing unit 13 and subunits 21-27, is, in principle, arbitrary. For example, subunit 27 (or its individual elements or specific algorithm sequences) is also arbitrary, user interfaceIt can be localized to 10. The same applies to the other subunits 21-26. To put it more precisely The processing system 20 is for the user interface It may be integrated into 10. As previously mentioned Alternatively, the processing system 20 may use, for example, a cloud server, or, for example, a hospital or Radiology site to Placed Server systems such as local servers as to be implemented Such example According to the user interface 10 can be designated as the "frontend" or "client" that interacts with the user, on the other hand 、 Processing system 20 as a "backend" or server It can be captured User interface Communication between 10 and the processing system 20 is, for example, https protocol Using It will be held The system's computing power is, server and Client (i.e., user interface 10) Between distributed It is possible. Thin client In the "system", calculation ability The majority of it resides on the server. "Sick Client" The system will have more computing power, and As much as possible data , to the client exist.

[0207] Individuals of System 1 component They may be connected to each other, at least temporarily, for data transfer and / or exchange. User Interface 10 is, for example, patient data TPD, data descriptor, Alternatively, to exchange the results of calculations, it communicates with the processing system 20 via interface 29. For example, the processing system 20 is request-based Start up , this The request is from the user. interface It is transmitted by 10. Furthermore, the processing system 20, Target patients To search for cases, use a medical information system. 40 It can communicate with them. Alternatively Or in addition, user interface Unit 10 can communicate directly with the medical information system 40. Medical Information System 40 is similar, for example, Requirements-based Start up , this The request is sent to the processing system 20 and / or the user. interface By 10 To be sent The interface 29 for data exchange is a hardware or software interface, such as PCI bus, USB, or FireWire. implementation It is possible. Data transfer is for example, Using a network connection to be implemented The network is Local Area Network (LAN) For example, intranet 、 or Wide Area Network (WAN) as implementation It is possible. Network connectivity is preferred. For example, wireless methods such as wireless LAN (WLAN or Wi-Fi). Furthermore, the network is a multiple network example. combination This may include users 11, 12 and Interact for component Together, data exchange conduct Interface 29 can be considered as constituting an interface unit of system 1.

[0208] Figure 4 shows the target medical image group TMIS. against A method for obtaining a reference medical image set RMIS, and / or Based on that This shows a method for generating display data. This method involves several Process (steps) It consists of. Process The order is not necessarily Process Numbering I am not complying. , the invention Various embodiments between Things can change. Furthermore, individually Process or a series The process is, Repeat too can.

[0209] In the first step S10 The target medical image group TMIS is received. The target medical image group TMIS is, This is considered the target image data that will form the basis of the image interpretation analysis the user is attempting to perform. It is possible. This process For example, stored in the medical information system 40, MatchSelecting the target medical image group TMIS from multiple groups of target medical imaging examinations TS. may include This choice is, for example, User interface 10 Operation do In a graphical user interface By selecting the appropriate target medical image set TMIS, Users can perform this manually. Alternatively, the target medical image set TMIS can be uploaded to the computing unit 30. twist The user can provide the target medical image group TMIS to the computing unit 30. Process S10 is In either the user interface 10 or the processing system 20, It can be implemented at least partially. If necessary, The exchange of related data is included in this process. .

[0210] Process S20 In The ROI of the target medical image group TMIS is Acquired ROI in areas of interest Obtaining teeth, for example, User interface Manual operation of the area performed manually by the user via 10 Definition User input including Including obtaining Alternatively, the ROI of the area of ​​interest acquisition teeth, for example, Automatically or semi-automatically It can be advanced According to some embodiments, Figure 35- As explained with Figure 39 , in other words, the evaluation tool ML-40 Using The region of interest (ROI) can be obtained. Manual and semi-automatic ROI analysis is possible. The definition teeth, For example, a graphical user interface (preferably user interface 10) Through this, the representation of the target medical image group TMIS is display to include The user, for example, sees Representation Images As an overlay on RI 、 User against Visualized area of ​​interest tool ML-40 arrangement It is possible. Selectively , The tool The size and / or shape Further adjustments could follow. . Alternatively Or, in addition to that, the ROI of the area of ​​interest Definition teeth, For example, this includes freehand selection of a region of interest ROI by tracing its contour using any appropriate input unit 12.. Also ROI in areas of interest Definition teeth, for example, Based on identifying features in the target medical image group TMIS zuku .this is, for example, Anatomical features indicating the patient's pathological condition That is . In addition In other words, the location, size, and shape of the region of interest (ROI) are, Preferably, Abnormal or atypical features present in the target medical image group TMIS Based on The unusual features are, For example, localized increases or decreases in tissue density, cysts, calcifications, etc., corresponding to displacements of anatomical structures, organs, or tissues such as the lungs, heart, blood vessels, and brain. . Atypical anatomical features are therefore, patient pathological condition It represents. Furthermore, Its characteristics are, Characteristics of pathological images or related to pathological features Possible . this The features are, for example, Visually examined or identified by a radiologist / pathologist. Alternatively, the result of the feature extraction and / or target detection process, which is selectively included in process S20. . Process S20 is Either the user interface 10 or the processing system 20 at least partially execution It is possible. If necessary, The exchange of related data is included in this process. .

[0211] Process S30 In Target medical image group TMIS Expressed by Body region 408 is determined. decision teeth, for example, Based on the target medical image group TMIS zukuka , or , matching medical image group TMIS Target medical images inspection TS Based on To determine the body region 408, the target medical image group TMIS and / or target medical image examination TS image Data can be evaluated. For example, in medical image data anatomical structure An image recognition algorithm configured to recognize [specific elements] can be applied to the medical image data ID of the target medical imaging examination TS to determine one or more anatomical structures represented by the target medical imaging examination TS. Then, the Based on anatomical structure, body region 408 can be determined. Alternatively Or, in addition, medical imaging tests included in the target medical imaging tests TS imaging Data ID content By evaluating the attribute value AV that indicates this, the body region 408 can be derived. Body region of medical imaging examination and / or medical image group decision Further details regarding this will be explained in relation to Figures 5-7. Process S30 is mainly the processing system 20 in It can be done. If necessary, The exchange of related data is included in this process. .

[0212] Process S40 In Then, at least one reference medical imaging test RS is selected from multiple candidate medical imaging tests CS. As mentioned above, the reference test RS is compared with the target medical imaging test TS that the user currently has open. Reasonable Possible It is . Specifically, this The choice is, For example, process Based on body region 408 determined in S30 zuku Regarding this point, Process S40 determines candidate body regions 408 for each candidate medical imaging examination CS. Selective sub-processes aimed at achieving this S41 can be included. This , candidate body region 408 decision teeth , basically a process Related to S30 It can be done in an explanatory manner. . Selective sub-processes S42 In , target medical imaging examination TS and Each of the candidate physician imaging tests CS and but Are they comparable? To determine this, the target medical imaging test TS Regarding The determined body region 408 is compared individually with each of the candidate body regions 408. Specifically, For example, Two anatomical structure Based , Each of the candidate physician imaging tests CS How well does it match the target medical imaging test TS? As a scale For each candidate medical imaging test (CS), the goodness of fit is calculated. do Based on a comparison of body domain 408 (goodness of fit), one or more reference medical imaging tests (RS) are selected from candidate imaging tests (CS). It will be done Body domain 408 Further details regarding the decision, subsequent comparison of body regions, and the identification of the reference test RS based thereon, can be found at: Figure 8 and Figure 9 It is provided in connection with this.

[0213] moreover, Process S40 is Selective sub-processes In S43, from among several medical imaging test candidates... 、 Based on a comparison of 408 body regions, Multiple potentially related Image examination Identification This can include doing this. Potentially related Imaging examinations are, for example, Other candidate image examinations CS A better degree of fit with higher comparability. Candidate image inspection CS That is Next Selective sub-processes In S44, Multiple potentially related The indicators for imaging examinations are for example, User interface Provided to users via 10 do . In response to that , user selection but , For example, selective sub-processes In S45, the user interface Received via 10 So , This user selection is at least the relevant image data files It targets one specific item. Next, Process In S46, user selection Specify done A specific item is selected, for example, as a reference image inspection (RS). .

[0214] As mentioned above, One of the following is selected Reference medical imaging tests (RS) almost always include multiple medical image sets, and these but Candidate medical image group is called CMIS. In order to automatically provide the user with at least one related medical image group of the selected reference medical image examination RS, Process S50 is used to select images from the candidate medical image group CMIS for user-guided interpretation. For the purpose A good foundation is provided by at least one reference medical image set RMIS. Identification do The purpose is to For this purpose, In a selective sub-process S51, for example, the degree of comparability is determined for each of the candidate medical image groups CMIS, and this degree of comparability serves as a measure of how well each of the candidate medical image groups CMIS can be compared to the target medical image group TMIS. . This , Medical image sets TMIS and CMIS Included image Characteristics of Data ID of , The Further metadata for TMIS and CMIS, for example, each set of medical images Capture Used for done Image parameters , togetherIt can be evaluated. specifically , This process teeth, For example, linked to the medical image sets TMIS and CMIS respectively Evaluating the attribute value AV include . Process Further details regarding the group matching process of S50 can be found in Figure 10 and Figure 11 It is provided in connection with this.

[0215] moreover, Process The S50 is a good choice. for example, User interaction for selecting at least one reference medical image set (RMIS) include .in particular, Process S50 is Selective sub-processes In S52, for example, From the candidate medical image group CMIS Multiple potentially related Image group Identification to include . This process , For example, process S51 Based on the determined degree of comparability . Potentially related The image set is for example, Other candidate physician image groups CMIS A higher degree of comparability This indicates The candidate medical image group is CMIS. Next Selective sub-processes In 1978, Multiple potentially related Metrics of the image group but , for example, User interface Provided to users via 10 It will be done . Following this, User Selection but , For example, selective sub-processes In S54, the user interface Received via 10 So , This user selection is at least of the related image set One specific of Target Next, Process In S55, the user selection was shown Certain things Selected as reference medical image set (RMIS). It will be done .

[0216] Process S60 is In the registration process Registration of the reference medical image set RMIS and the target medical image set TMIS is provided. It will be doneThere are various ways to achieve this type of registration. In this regard, the available options include: Rigid body Registration, Affine Registration, Non-rigid body Registration, non-affine (non-affine) Registration, and all related matters combination This includes the coordinate systems of the target medical image group TMIS and the reference medical image group RMIS. Link You can obtain registration. In other words, On the other hand Medical image sets TMIS, RMIS each Each image data The aforementioned group of images A transformation can be calculated that allows conversion to the RMIS and TIMS coordinate systems. For this purpose, at least a portion of the target medical image set TMIS but , at least a portion of the reference medical image set RMIS and It will be registered. Basically, this process is Two image sets This includes identifying matching data points. Such Match By identifying data points, This makes it possible to calculate the local offset between these matching points, providing an index of the local shift in the coordinate system between the two image sets. This is the foundation and It is Multiple images that are well distributed within the image set Matching data points to Regarding What to do is, each A good indicator of displacement and deformation between image groups Soon provide This will be the case. Apart from such formal registration using coordinate transformation, there are also medical image sets TMIS, RMIS Matching Slice or image pattern Identification Registration can be performed based on the actions taken. in this case Furthermore, slices in the target medical image set TMIS, particularly slices of the region of interest (ROI), can be processed in the reference medical image set RMIS without the need to compute a complete coordinate transformation. Match Image slice (also specified as the reference slice REF-SLC, described later) obtain ) Assign to It is possible. Process S60 Further feasible options are presented in relation to Figures 12 to 26. .

[0217] ProcessS70 generates display data, User Interface 10. Render the reference medical image set RMIS. The purpose is to display it on a display device. . specifically , rendering is, For example, the registration determined in step S60 means that the image data ID extracted from the reference medical image set RMIS matches the image data ID of the target medical image set TMIS that the user is currently reviewing. . especially , rendering For example, see Based on image slice REF-SLC zuku .

[0218] Referring to Figure 5, Process S30 and Process In S41 Executed , medical data saved in the first image file imaging Data expression A computer that determines the body area to be affected. execution The method is illustrated.

[0219] In short The method shown in Figure 5 is Next step include: In process BR-1, for example Match By evaluating the image file interface, Image examination TS, CS Retrieve one or more text strings AF,IF-314; Process In BR-2, Acquired One or more text strings AF,IF-314 are input into a trained machine learning model (see, for example, trained neural network 406 in Figure 6), The body region 408, represented by the medical imaging data ID, is determined by obtaining the output from the trained machine learning model 406. The trained machine learning model 406 is trained to output a body region based on one or more text string inputs.

[0220] The system determines body regions based on such text string input. trained Machine learning model 406 (for example, a neural network) is used for the target medical imaging examination TS. connected By entering one or more text strings AV,IF-314, medical imaging By data ID expression Determining the body region 408 to be affected is subject and / or medical imaging data included in candidate image examinations TS,CS Expressed by This can provide efficient and / or flexible decision-making for the body domain 408.

[0221] For example, the text string of a file IF (These are relatively small in terms of bits) Determining body regions based on this is, for example, medical imaging data That thing (bit In terms of Body regions are determined by extracting and analyzing (relatively large) areas. Compared to , low resource intensity, and therefore efficient be . The image file is saved remotely via the network from the processing device. In some cases, by determining the body region based on a (relatively small) text string, (relatively large) medical imaging The data The data is represented Network to determine body area through send This will save me from having it happen. , therefore, This enables the efficient use of network resources.

[0222] Another example is a trained machine learning model (for example, trained Neural Network Determining a body region by entering a text string is, for example, in the text string Hardcoded rules Compared to determining the body region by applying the method, the efficient method of determining the body region ,flexible, and / or can provide robust decisions. For example, hardcoded rules, body area In order to provide this, the rule must precisely match the text string (therefore, there is no flexibility regarding the text string that can determine the match, and / or use Available A set of rules that must be coded for every text string Comprehensive properties (This is inefficient in this respect). On the other hand, trained machine learning models (e.g., trained Neural Network )teeth, Train that Training data set from It is induction. Therefore, training data set of Even if the text string is different from the text string, the appropriate body area can be obtained relatively efficiently and It can be decided, and therefore, relatively Flexible / Robust That is the case.

[0223] From the above , medical imaging Data expression Efficient and / or efficient of the body area being treated flexible Automatic decision-making may be provided.

[0224] According to some examples, trained machine learning models It can complement rule-based systems (or be complemented by rule-based systems), and it enables induction of rule-based systems when rule-based systems are insufficient. Rule-based systems are for example, One or more selections for selecting the reference medical imaging test (RS). rule of Execute .choice rule For example, Regulatory requirements and / or derived from medical guidelines jiru .

[0225] In some cases, the imaging data ID of the target medical imaging examination TS is used Expressed Body domain 408 decision Next, the medical imaging tests included in the target medical imaging examination TS image Candidate medical imaging test CS related to data ID (e.g., suitable for comparison) Reference image Automatic data selection Can promote For example, the medical file of the current examination imaging Data Expressed Determining body region 408 is medically based on previous examinations of the same body region in the patient. imaging This can be used to select data. This will be explained in more detail below with reference to Figures 6 and 7.

[0226] As mentioned above, in some examples, the medical image data file IF is a DICOM file IF (i.e., for example DICOM standard “NEMA PS3 / ISO 12052, Digital Imaging and Communications in Medicine (DICOM) Standard, National Electrical Manufacturers Association, Rosslyn, VA, USA) follow DICOM file Image file formatIF) is possible. Referring again to Figure 3, the DICOM file IF includes, in more detail, a header IF-302 and a dataset IF-308. The header IF-302 includes a 128-byte preamble IF-304 (all bytes set to zero if not used) and a 4-byte prefix 306 containing the string "DICM". The dataset IF-308 contains data elements IF-310 and IF-312. Each data element IF-310 and IF-312 is: Includes tags , tag Identified by: Each tag is in the format (xxxx,xxxx), where each "x" is a hexadecimal number. DICOM file 300 contains a unique identifier for the inspection to which the file is part in the "Inspection ID" data element (not shown). It can be saved (That is, identified by the DICOM tag (0020,0010)) 、 And the file A group that is part of it The unique identifier is in the "Group ID" data element (not shown). It can be saved (That is, identified by the DICOM tag (0020,000E)) 。

[0227] The DICOM file IF stores the medical image data ID (in this case, pixel data) in the "Pixel Data" data element IF-312 of the DICOM file IF (i.e., identified by the DICOM tag (7FE0,0010)). The DICOM file IF further stores the medical imaging Data ID content Store one or more attributes IF-310 (provided by one or more other data elements 310), each having an attribute value IF-314 containing a text string indicating the following. For example, One of the attributes is "Examination Description" (i.e., identified by "DICOM tag (0008,1030)"), and its attribute value IF-314 is a text string that describes the examination in which the medical imaging data is part (for example, "NERUO^HEAD" where the medical imaging data is the patient's head region, or "PELVIS^PROSTATE" where the medical imaging data is the patient's pelvic region), and therefore indicates the content of the medical imaging data ID. In some examples, attribute IF-310 and attribute value IF-314 are In that way It is not included in header IF-302, but rather in dataset IF-308. Obtain , attribute IF-310 and attribute value IF-314 (medical imaging (Does not include the data element IF-312 which stores the data ID itself) Medical imagingBecause it is data related to medical image data IDs rather than the data ID itself, it is sometimes called the header data of a DICOM file interface. obtain .

[0228] medical use imaging data content Text indicating string Other examples of attribute values ​​that include of DICOM attributes It can also be used For example, such Other examples DICOM attributes are medical imaging Groups where the data is partial Describe the data (for example, "ax t1 whole pelvis" indicating that the medical imaging data is of the whole pelvis, captured in the axial direction using a T1-type MRI), " Series Description (i.e., identified by the DICOM tag (0008,103E)). For example, other such The example DICOM attributes are, "Body Part Examined" (i.e., DICOM tags (0018,0015)) (Identified by) this is, investigation body parts This indicates (for example, "PELVIS" indicating that the medical imaging data is of the pelvis). Other such appropriate It will be recognized that DICOM attributes exist and can be used. In some examples, the DICOM attribute " Reason for the Requested Procedure (i.e., identified by the DICOM tag (0040,1002)) may also be used.

[0229] moreover, some In the example, it will be recognized that such image files other than DICOM files may be used. For example, in some examples, the image file It can be saved as a Portable Network Graphics (PNG) file. , this is, One of those chunks is medical imaging Save data And in another chunk, medical use imaging data content One or more attributes, each having an attribute value containing a text string indicating included metadata Save .

[0230] In some examples, one of such text strings is used to train a machine learning model (e.g., trained Neural Network) can be entered into. However, in other examples, medical image data files related to the target medical imaging examination and / or the target medical image group TMIS. Multiple attributes of IF From multiple attribute values Multiple of the above The text string is First Medical files imaging Data expression To determine which body part to target, a trained machine learning model (e.g., trained Neural Network ) may be entered together. For example, this is a more precise and / or Robust It can provide a judgment. For example, a trained machine learning model (for example, trained Neural Network The accuracy with which it determines the body region is The basis for that decision Input data ( 1 From the file The above This can be improved by increasing the number of text strings. Another example is when one of the text strings happens to be unreliable and / or missing (e.g., by a radiologist). According to regulations or To Because it is not filled in, For example, this can be seen in the case of "body parts under investigation". ), inputting multiple text strings mitigates this, Even so , It should be decided Trustworthy in the physical domain decision This makes it possible.

[0231] Figure 6 shows an example. related The flow between the components of the method described above is illustrated with reference to Figures 1 to 5. Figure 6 Referring to the above, as mentioned earlier, Target medical One or more text strings 404 corresponding to the image inspection TS are input to the trained machine learning model 406, and the trained machine learning model 406 outputs the body region 408.

[0232] In some examples As shown in the figure, one or more text strings 404 can be extracted from the image file IF corresponding to the target medical imaging examination TS. Specifically, one or more text strings 404 are extracted from the image file IF related to the target medical imaging examination TS. Without extracting the medical imaging data,It can be extracted. For example, taking a DICOM file as an example, the image file identifies the attribute that has the attribute value to be extracted. 1 The above predefined tags (for example, "inspection") Description "attribute In the case of DICOM tag (0008,1030), Series Description " In the case of DICOM tag (0008,103E)) locate Analyzed for this purpose obtain If a tag is found, the attribute values ​​(i.e., text strings 404) of one or more attributes identified by one or more tags may be extracted from the image file IF associated with the medical imaging examination TS. The extracted text strings 404 are, for example, the medical imaging examination TS or its medical image data identifier for to Images can be saved in association with each other. In some cases, images related to the target medical imaging examination TS. file IF text string 404 teeth Extracted from the image file IF related to the target medical imaging examination TS. Are you , or Otherwise, this method It is obtained before it is executed. We For example, image files related to the target medical imaging examination TS Identifier of the IF or its medical image data ID Saved in relation to doing In either case, one or more text strings 404 are obtained and input into the trained machine learning model 406.

[0233] In some examples, only one text string It can be used However, in some cases, multiple text strings 404 from the image file IF associated with the target medical imaging examination TS or candidate medical imaging examination CS can be obtained and input into a trained machine learning model 406. For example, the attribute "Examination" in the DICOM file IF. Description " Series Description ", and " Body part to be investigated By obtaining the text strings of the attribute values ​​for all attributes of "[filename]" and inputting them into the trained machine learning model 406, the trained machine learning model 406 uses all of these input text strings to determine the medical status of that file. imaging Data expression The body that is subjected region It is possible to output the following. For example, text strings can be concatenated and input together into the trained machine learning model 406. In any case, an output 408 is obtained from the trained machine learning model 406, thereby for medical imaging Data expression The body region 408 to be affected is determined.

[0234] As mentioned above, machine learning model 406 is, One or more of the above It is trained to output body regions based on text string input. In other words, the machine learning model is trained to output one or more attribute values ​​of one or more attributes of an image file IF for medical imaging data content Based on the input of one or more text strings, the medical imaging data of the TS and CS images are displayed. Expressed by It is trained to output from the body's domain.

[0235] In some examples, the trained machine learning model 406 may be a trained neural network 406. In fact, in the examples described below, the trained neural network 406 To mention However, in other examples, the system outputs a body region based on the input of one or more text strings. Trained to do so Other types of machine learning models are being used. , It will be recognized that, for example, in another example, a trained machine learning model is trained Random Forest It can take the form of an algorithm, for example, random forest algorithms. It consists of an ensemble of decision trees trained with training data that includes training text strings labeled with relevant body regions (determined, for example, by experts). In other words, to minimize the error in predicting body regions by comparing them with labeled body regions for the training text string, Decision tree ensemble It is possible to construct this. However, As already mentioned, below, Example of a trained neural network 406 to mention The use of the pre-trained neural network 406 is explained in more detail below. As stated above , some It has advantages do .

[0236] In some examples, neural network 406, deep It may be a neural network (i.e., having one or more hidden layers). In some examples, the neural network is: With a teacher It can be trained using learning. For example, neural network 406 can be trained using multiple training text strings (in fact, Hundreds or thousands of training text strings The training may be conducted using a training dataset that includes (and may also include) an attribute value from an image file attribute, and each training text string is derived from the attribute value of an attribute in the image file. It is Medical data stored in the image file imaging data content This shows that, for example, the training text strings are attribute values ​​of the appropriate attributes of the DICOM image file (e.g., "Inspection Description"), series Description, Subject of investigation Text strings extracted from "body parts," etc. That is In some cases, as mentioned above... In the same way as above Each training text string is a multiple appropriate attribute of the image file. Each of the above It can represent the concatenation of text strings. In any case, each training text string is a training text string Matching Labeled in the body region obtain For example, in some cases the training text string is Occur Medical files imaging Data expression The actual body area may be labeled, and the actual body area may be determined, for example, by a specialist. In some cases (see Figure 7 for more details, which will be discussed later), the actual body area may be determined by a specialist. It will be done , whether it corresponds to the text string itself The text string itself by expression The body area that was determined to be in , training text string teeth Labeling It will be done In either case, the body region labels for each text string are used to train the neural network. Teacher signalIt can be used as such.

[0237] In some embodiments, the trained neural network 406 is In this body region classification method, for one or more input text strings, the classifier portion of a trained neural network 406 is selected from among multiple body region classifications. It can be configured to output body regions 408. For example, each classification may be a standardized word representing a body region, such as "ABDOMEN", "PELVIS", "CHEST", etc. In such an example, each training text string is labeled with the classification to which it belongs. This , for training neural networks teacher signal Used as obtain For example, training is consisting of deep learning For example, training involves the classification predicted by the classifier for each training text string and the actual classification of each training text string defined by its respective label. 、 This may involve updating the weights of the connections between layers of neurons in the neural network to minimize the error between them.

[0238] In some embodiments, the trained neural network 406 uses one or more numerical values ​​representing regions of the human body. It can be configured to output the body region 408 in a specific format. 406 trained neural networks Regressor The part is calculated for one or more input text strings 404. doing For example, one or more numbers are: Body Ruler ", in other words, It can be a ruler or scale marking defined for the human body, for example, The value 0.0 Human toes This represents a value of 1.0. The top of a person's head To represent death , in this case, 0 to 1 between The value Toes and the top of the head between Various regions of the human body It represents. In some examples, two such numbers can be used to represent body regions of the human body. For example, The The two values ​​define the position where the body region is defined. Can be demonstrated . for example Body Ruler values ​​for the bladder [0.5, 0.55] They are assignedIn such an example, each training text string is labeled with one (or more, e.g., two) numbers that represent the body region it corresponds to. So This is for training neural network 406 teacher signal Used as obtain For example, training is deep learning For example, training is performed for each of the training text strings. Regressor The one (or more) numerical value predicted by and the actual one (or more) numerical value of each training text string defined by each label 、 This may involve updating the weights of the connections between layers of neurons in a neural network to minimize the error between them.

[0239] Outputting body regions as one or more numerical values ​​is a medical application of the CS and TS tests. imaging Data expression This can enable accurate and / or flexible determination of the body area being measured. For example, the numerical values ​​are continuous, and therefore, For example, in advance Compared to using a limited set of defined classes 、 Body regions can be defined more accurately and flexibly. This can then be explained in more detail below, for example, with reference to Figure 8. First medical use imaging When selecting medical imaging RS as the data-related item, the body region Flexible And / or it may enable accurate comparisons.

[0240] some example Then, the trained neural network 406 processes the individual characters of one or more text strings 404 that it has obtained. Take as input The trained character-based neural network 406 may be configured as follows. In these examples, inputting one or more text strings 404 into the trained neural network 406 includes inputting each character of the one or more text strings into the trained neural network 406. Can seeFor example, neural network 406 processes each character of the input text string. Encode into vector configured encoder It can include, for example, one hot for a vocabulary of characters including the alphabet, numbers 1 through 9, and special characters. encoding This is done using character embeddings like this. obtain These vectors can be used by neural network 406 as the basis for determining body regions. It is also available. .

[0241] For example, several example So, neural network 406 is, It consists of character-based recurrent neural networks (RNNs), such as bidirectional long-short-term memory (LSTM) RNNs. A vector for each character of the text string. but Continuous input to the RNN So Subsequently, the RNN maintains a certain internal state (for example, when a vector for the last character of a text string is input). of (A vector representing the value of that neuron) It will happen Next, this internal state is processed by the neural network 406. Regressor Alternatively, the data can be passed to a classifier, in which case the internal state can be mapped onto the body region 408.

[0242] As another example, neural network 406 is a character Base folding Neural Network (CNN) Consists of In these examples, we can assemble a matrix by arranging vectors of consecutive characters from a text string. Then, Folding and Pooling process Apply this to a matrix to represent the features present in the text string. Condensed characteristics Vector (condensed feature vector) This can be determined. The feature vector is from the neural network 406. Regressor Or pass to the classifier So , these are Next, the feature vectors can be mapped onto the body region. of Precise mapping Possible Features is, itselfThis can be learned during the training of the neural network 406.

[0243] In some examples, other neural networks, for example, word base A neural network can be used. however , character-based neural network Consists of Neural network 406 used a portion of the training data. Abbreviations, misspellings, and other words that were not present Regarding Robust The physical domain decision This can provide a similar representation. For example, "ABDOMEN" can be abbreviated as "ABD," but because the first few letters are the same, character-based neural networks can represent similar positions in the vector space. It is possible to generate vectors for these two words. (Therefore, the appropriate body area is determined It is possible ). On the other hand, word-based neural networks can process "ABD" Deviating from vocabulary This may be the reason for the decision. This is because words outside the vocabulary may reduce accuracy. Later, It may help in making accurate decisions.

[0244] As mentioned above, each training text string is a training text string Match The body region may be labeled, and in some cases, the training text string is expert By decision , into text string Does it match? Alternatively, it may be labeled with a body region determined to be represented by a text string. See Figure 7 for details. Then In such cases, Training dataset ( Training text string (and their labels) Graphical User Interface (GUI)502 may be generated using GUI502. GUI502 is training The GUI can be configured to present one or more text strings 508 and a representation of a human body 510 divided into body regions 511 that can be selected by the user. For each of the one or more presented training text strings 508, The system receives user input to select a body region 511 from representation 510.The system may be configured to label the training text string 508 with a label 509 that indicates the selected body region 511. In this way, a training dataset can be generated that contains training text strings labeled with body region labels.

[0245] For example, referring to a specific example in Figure 7, GUI 502 includes a progress bar 504 that shows the user a specific pane of the currently displayed GUI. The title of the specific pane is also 506 It is displayed as, In this case, Annotate Key Words It will display " doing In this example, training text string 508 is medical imaging data keep These are keywords extracted from the attribute values ​​of the image file's attributes. In the illustrated example, the keywords include "aaa", "ab", "abdpel", "abdroutine", "aquired", "aif", "angiogram", and "ascities". for , specific keywords The following was presented. ( In this example, "aaa" In bold ), and, This corresponds human body Display of expression 510 body The user is asked to select one of the 511 domains. The user is, for example, a professional and a medical professional. imaging Knowing that "aaa" in the context is an abbreviation for "abdominal aortic aneurysm" Ori , displayed human body Select the abdominal area of ​​representation 511, and accordingly, the body region label "ABDOMEN" will be added to the text string "aaa". Labeling Other keywords presented but, "ab" - "ABDOMEN", "abdpel" - "ABDOMEN", "abdroutine" - "ABDOMEN", "aif" - "HEAD", "angiogram" - "CHEST", and " ascities "-"ABDOMEN" and, Similar selections can be made. The text string "acquired" is not associated with a specific body region, This means that the body region is not labeled and therefore will not be part of the training dataset.Please note the following: GUI502 also has selectable button 512. In other words Includes "Uterus", "Prostate", "Unknown", and "None". Uterus "and" Prostate The " " buttons are for providing body area labels for "PROSTATE" and "UTERUS," respectively. The "Unknown" button allows the user to... body I don't know the region labels, but for example, body The "None" button is for when you think you can assign a region label, and it's for when you know the user doesn't know there's a body region that can be assigned to a text string (for example, the above " Accurate (Like in the case of ").

[0246] GUI502 generates multiple body region labels in a simple and efficient manner (for example, hundreds )of training Includes text string training This makes the dataset possible. The labels are, human body Expression (e.g., stylized) figure ) obtained using, This expression It is visual and easy for users (including medical professionals, not necessarily programming experts) to interact with. This simple and efficient GUI for users Dialogue Next is the training dataset. (That is, the text string to be labeled) This makes it possible to generate efficiently.

[0247] In the above embodiment, the neural network 406 is trained to output a body region 408 based on an input text string. example The neural network 406 may then be further trained to output the left-right difference of the output body region 408 based on the input of one or more text strings. The left-right difference is the difference in the body Which On the side (i.e., "left" or "right"), medical imaging Data expression Where the body region being targeted is located say For example, the left-right difference in output is either "left" or "right" as appropriate. In these examples, the method shown in Figure 5 Process 104 is based on the output obtained from the trained neural network 406, medical imaging Data expression This can further include determining the left-right difference in the body region being treated. As one example, This process It outputs left-right differences based on one or more text string inputs. Trained This may also be done by providing a dedicated left-right differential portion for the neural network 406. For example, This process is a text string Matching Due to differences in the left and right sides of the body Label each Training can be performed using a training dataset consisting of the trained text strings. As another example, This process The input text string is Neural network 406 Mapping Set of body region classifications To expand on this, the " left Include versions of " and "Right" by .for example, This process This is the training text string Matching Body domain and Left-right difference It can be trained using a training dataset containing training text strings each labeled according to classification. Determining the left-right difference in body regions allows for more precise determination of body regions. This Next, as will be described in more detail later, the second related medical imaging This could enable a more accurate selection of data.

[0248] As mentioned above, in some examples, subject By examination TS or target medical image group TMIS expression The determined body region 408 is, subject A second medical image related to the examination TS or the target medical image group TMIS (for example, for appropriate comparison) imaging It can be used to select data. For example, body region 408 is prescribed patient RegardingBased on the current examination image file IF, the decision is made for the target medical image group TMIS. So This refers to the current target medical image group TMIS, namely subject Suitable for comparison with test TS (e.g., the same or similar body area). , prescribed patient Regarding It can be used to select one or more image datasets (for example, stored in one or more image files IF) from one or more previous examinations (reference medical imaging examination RS).

[0249] Two medical imaging tests TS, CS Degree of comparability or degree of fit This may be calculated based on each body region. For this purpose, the relationship between each body region Overlap may be determined For example, the body region is the anatomical structure of the patient. against Normalized or generalized coordinates Using the range If provided, between the two body regions Overlap These can be calculated based on their respective ranges. Furthermore, Degree of comparability or degree of fit This can be determined by calculating the Dice score between each body region to measure the similarity between the two output values.

[0250] Referring to Figure 8, TMIS / Target inspection At least one reference related to TS inspection The method for selecting RS is illustrated.

[0251] In this method, in process BR-3, subject The first medical image stored in the examination TS / target medical image group TMIS imaging Data expression It will be done subject Body region 408, multiple candidate medical imaging examinations CS Each This method involves comparing each of several candidate body regions represented by Process In BR-4, this includes selecting one or more candidate medical imaging tests (CS) related to the target medical imaging group TMIS based on a comparison of 408 body regions.

[0252] each Imaging examination for prospective physicians CS includes multiple (candidate) medical image sets CMIS of the same type as the target medical image set TMIS. Can see Similarly, each candidate medical imaging examination (CS) and its associated image set (CMIS) are stored in one or more image files (IF). It will be done In other words, each candidate medical image group CMIS and each candidate medical image examination CS are further stored in the corresponding image file IF. sometimes , candidate physician imaging examination CS content It has an attribute value AV that contains a text string indicating 1 This is associated with attribute A above. Target body Region 408 contains one or more text strings of the target medical imaging examination TS, in the manner described above, as shown in Figure 5. Process By applying BR-1 and BR-2 、 Decision made doing Alternatively or additionally, at least one (and in some cases all) of the candidate body regions 408, At least one (and possibly all) Imaging examination for candidate physicians CS Each of One or more text strings, as described above, refer to Figure 5. Process By applying BR-1 and BR-2 、 It is possible to make a decision.

[0253] Figure 9 shows One example Referring to Figure 8, the flow between the components of the method described above is illustrated. Referring to Figure 9, in this example, as described above with reference to Figure 6, one or more text strings 404 from the target medical image set TMIS or the target medical imaging examination TS image file IF are input to the trained neural network 406, and the trained neural network 406, Target body Output region 408. Target body Area 408 is one or more image files IF or separate files to, identifier In relation to It may be stored.

[0254] Furthermore, in this example, medical care Information Systems 40 search Candidate medical imaging test CS multiple Image data file IF Each from, One or more Text string 716 is multiple Extracted. For example, multiple candidates inspection Each of them is a patient Previous separate tests Related to Possible . One or more text strings 716 multiple Each 、 The data is input into the trained neural network 720. next, Multiple candidate body regions 720 Each Obtain. Each of the candidate body regions 720 is one or more image files IF or separate files to, identifier Save in association with It is possible. Next, Target body Region 408 is analyzed by comparator 710 for multiple candidates body Compared to each of the 720 regions, subject and candidates body For each pair of regions degree of suitability It can be decided, this Candidates based on comparison inspection One or more selections 712 can be made. Specifically, one or more Reference Test RS is a candidate inspection From CS In this way choice It is possible .

[0255] In some cases, the selection is based solely on comparison. zuku For other examples, see below in more detail. explanation Therefore, the selection is based on further criteria. zuku In some cases, the selection is based on the body area. Saved in association This may involve selecting an identifier for the candidate medical image data file. The selected identifier is: Inquire with the medical information system 40 Medical Information System 40 (For example, online or nearline DICOM archiving devices) From the relevant candidate medical image data files Search (e.g., prefetch) It can be used for that purpose.

[0256] In some examples, Medical image data or, One or more selected image data set The image file IF of those tests (e.g., only those tests) including one of them is: medical care Information Systems 40 search These files may be, for example, selected medical imaging This can be determined by matching the "check identifier" attribute of the file containing the data. In some examples, as will be discussed later, Medical image data or, Second medical use imaging Image files of groups (e.g., only those groups) that contain one or more selected sets of data, medical care Information Systems 40 search These files can be used, for example, for selected medical imaging This can be determined by matching the "group identifier" attribute of the file containing the data.

[0257] In any case, the second medical imaging data Searched sets One or more renderings of Display unit 11 It may be displayed.

[0258] In some cases, the target medical image set TMIS is the patient's current medical image or a series of medical images. express Multiple candidate medical image sets (CMIS) are selected from the patient's previous medical images or a series of medical images. expressable For example, the target medical image set TMIS corresponds to the patient's current examination. death Each of the candidate physician image sets CMIS represents a patient Previous separate Inspection Possible . medical care Experts say, for example, two inspection To assess the progression of the disease during the current period, inspection One or more images of TS in the patient's previous inspection CS Our One related inspection RS One or more images Sometimes we want to make comparisons. An important criterion that enables a valid comparison is, Current examination TS and previous inspection CS (that is, Current and previous versions of these tests The medical images are of the same or similar body parts (e.g., head, abdomen, legs). Is that so? That is the case. subject Medical image data and candidate medical use imaging Based on the comparison of body regions determined for the data reference medical use imaging By automatically selecting data, this method allows medical professionals to, for example, subject Medical images of the TS examination, related Previously Medical images (e.g., previous examination CS of the same or similar body region) Medical images ) only Making it possible to compare, This teeth, The user It is necessary to compare it with all past medical images. Having It is more efficient in comparison.

[0259] moreover, some In this example, candidate physician imaging examination CS Multiple sets teeth, Medical As part of Information System 40 be Remote storage device to keep It will be done In these cases, refer to the medical imaging test RS. One or more selected sets (or individual image data IDs of the reference test RS) are candidate physician image data. Multiple sets of Other things search Without doing so, from the remote storage device search (for example, medical care Part of Information System 40 That is DICOM Archive device Prefetch ) Obtain. The body region is determined based on the text string of the image file's attributes. Therefore , candidates for those files medical use Image data teeth , When selected as relevant remote storage from Just search Related to or not selected as something to search for candidate medical use Image data teeth , Search at all It's not necessary. In this way , Network Resources efficient Potential to be utilized .

[0260] In some cases, comparisons of body regions are used in medical applications. imagingBody region classification for data (e.g., "ABDOMEN", "CHEST") is used for candidate medical imaging data. classification This may include determining whether it is the same as, match If there is a case (for example, both have the same body region classification "ABDOMEN"), candidate physician imaging The data is for the target medical imaging It can be selected as something related to data.

[0261] some example So, the comparison of the 408,720 body domains is as follows: Numerical values ​​defining body regions for target and candidate medical imaging tests TS and CS. This can include comparing the following. For example, this is Target medical imaging The numerical values ​​for the body region in the Data are Imaging for candidate physicians The numerical value of the data is the same as, or similar to, or Overlap This may include determining whether or not the target medical image inspection If the TS value is 0.5, then the candidate medical image has a value of 0.45. inspection CS Set / , similar (for example, The difference is less than the predetermined amount. ) can be selected as another example, subject If the TS value for medical imaging tests is [0.5,0.55], numerical values [0.5,0.55] candidate Medical imaging examination CS Assuming the set is the same choice So , numerical values [0.4,0.55] candidate medical use imaging data Assuming the sets overlap choice It is possible .

[0262] As mentioned above, the target medical image group TMIS to , especially in terms of ROI in areas of interest One or more reference medical imaging tests (RS) are relevant (e.g., prescribed (Patient's previous examinations) To choose is Based on comparing those body regions It can However, in some cases, the choice is more detailed below. explanation This can be based on further factors or criteria.

[0263] Showing the same body part Current examination TS and Previous inspection Besides RS, another important criterion that enables effective comparison of those images is, TMIS and CMIS of the reference medical imaging test RS , identical, similar, or comparable modalities (i.e., the image) Capture Used for Ta medical use imaging It must be of a mode or type (e.g., CT, MRI, X-ray).

[0264] therefore some example In, This method For each of the multiple groups of candidate medical imaging data (i.e., multiple candidate medical image groups CMIS), the first of the target medical image group TMIS imaging Modality and the second set of candidate medical images for the selected medical imaging test RS, CMIS. imaging Between modalities imaging Modality Relevance This could include determining the score. imaging The modality relevance score is, Is it included in the similarity? , or , can be used as a similarity factor .

[0265] In these examples, the target medical imaging As something related to data Imaging data for candidate physicians Selecting one or more groups CMIS is determined imaging It may also be based on modality relevance scores. For example, higher imaging One of the CMIS groups of reference medical imaging tests (RS) with modality association scores. but , lower imaging The group of reference medical imaging tests (RS) with modality association scores (CMIS) Other things Prioritize the target medical imaging Data ROI can be selected for comparison with TMIS. In some cases, the target medical image set has the same body region as the TMIS. AndThe candidate medical image group CMIS with the highest imaging modality relevance score can be selected as the reference medical image group RMIS. In some examples, candidate medical image groups CMIS having the same or similar body regions as the target medical image group TMIS may be pre-selected, for example, by referring to Figure 8 and following the method described above, and then, based on the imaging modality relevance score, one or more candidate medical image groups CMIS may be selected from the pre-selected set as the reference medical image group RMIS.

[0266] In some cases, medical files imaging data Capture Used to imaging Modality is medical imaging Image file IF where data is stored imaging The modality can be determined from the attribute value AV of modality attribute A. For example, the image modality can be directly obtained from the attribute value of the "Modality" attribute in the DICOM file IF (e.g., identified by DICOM tags 0008,0060). It will be done The "modality" attribute value of DICOM is medical imaging Identify the type of device used to acquire the data. The DICOM standard specifies multiple types of devices. imaging Values ​​that can represent modality But They are defined. For example, "CT" stands for "Computed Tomography," and "MR" stands for "Magnetic Resonance Imaging." To represent, etc. It is defined as follows. Furthermore, in some examples, imaging Modality attribute values ​​are automatically set to appropriate values ​​by the software used with the device to generate image files. obtain .therefore, imaging The modality can be reliably obtained directly from the "modality" attribute of the file.

[0267] The imaging modality association score between two modalities is calculated using one modality. Capture Medical imaging data but , using a different modality Capture Medical imaging Data and related do (that is, For comparison Appropriate or It can be a value that represents the degree of usefulness. In some examples, imaging The modality relevance score is, imaging Modality transition This can be determined using a matrix. For example, each element of the matrix is ​​a specific imaging Modality and another specific imaging Modality and one between imaging It may also correspond to modality relevance scores, i.e., the elements s of the matrix ij teeth, imaging Objects having modality i (for example, current ) Medical imaging data and, imaging Between candidate (e.g., prior) medical imaging data with modality j imaging This could be a modality association score. For example, i=MR and j=MR (i.e., s MRMR ) and between imaging The modality relevance score is 0.6. can be , on the other hand, i=MR and j=CT (i.e., s MRCT ) and between imaging The modality relevance score was 0.22. be In this case, for example, subject When medical image sets (TMIS) are captured using MR, Two candidates Medical image group CMIS but both candidate Having the same body area as the CMIS medical image group There are but, One set is being captured using MR, , The other set was captured using CT. In this case, sets captured using MR can be selected as the preferred option.

[0268] In some examples, imaging Modality Relevance score (i.e., transition Each element of the matrix s ij ) is a specific first imaging Modality i Related medical imaging data RegardingA user (for example, a medical professional) can identify a specific second imaging Modality j possess reference medical use imaging data Target This can represent the probability of selecting a medical image for comparison. For example, this probability can be expressed as a medical image. imaging Data and record User interaction ( Preliminary Action ) can be determined based on statistical analysis. For example, data logs are prescribed session In a predetermined by medical professionals prescribed About the patient search Medical imaging Files can be recorded. Statistical processing may be applied to these logs to determine probabilities. For example, current medical data where the modality is MR. imaging When reviewing a file, If medical professionals decide to review previous medical imaging files of patients with a 60% probability of having used MRI as their imaging modality, while reviewing previous medical imaging files of patients with a 22% probability of having used CT as their imaging modality, , s MRMR It was determined to be 0.6, s MRCT This can be determined as 0.22. This , transition The Matrix Add In order to, imaging This is performed for all combinations of modalities. obtain . Imaging modalities Relevance score (i.e., transition Each element s of the matrix ij ) is medical imaging data against This is based on statistical analysis of actual user interactions and the appropriate modality for the second medical treatment. imaging Data is selected reliably Useful for .

[0269] In the example above, to the specified patient Regarding Medical imaging data of the candidate medical image group CMIS teeth , for example Body area and / or imaging Based on the modality, for the designated patient Regarding Medical image group TMIS imaging Data-related (for example, appropriate for comparison) things It can be chosen. As mentioned above, prescribedDuring an examination, there may be multiple groups of medical images. For example, prescribed Inspection (especially) prescribed Modality ,for example MR , inside ) at Multiple groups, different Image parameters (for example, MR: Echo time, flip angle, echotrain length, patient orientation, etc. Regarding Medical data obtained using imaging May contain data. By medical professionals. the current And another important criterion that enables effective comparison of previous medical images is, the current and previous medical images Capture Used when Ta medical use imaging The parameters are the same. mosquito or similar That is the case. . therefore In some cases, the second medical imaging The selection of data is Alternatively or additionally, This can be done based on a comparison of image parameters. For example, prescribed patient Regarding Previous inspection Candidate group CMIS within RS candidate medical use imaging The data consists of images from previous CMIS and current group TMIS. Capture Used for Ta medical use imaging Based on parameter comparison, the target medical treatment for the current TMIS group imaging Data related (For example, it is appropriate for comparison.) It can be selected as an object.

[0270] Refer to Figure 10. Then , imaging Relevant medical information based on parameters imaging The method for selecting data is illustrated.

[0271] In these examples, each of the image file IFs related to the CMIS of the reference medical imaging examination RS contains the medical imaging data of the image file. Capture Used for ta o The attribute value AV that indicates the parameter EachStore one or more attributes A that the object possesses.

[0272] For example, the above Similarly Each image file IF is, It is a DICOM file interface, That medical imaging data Capture Used for ta o parameters This indicates attribute value AV One or more first attributes A are , Includes one or more of the following DICOM attributes: “Image Direction Patient” (identified by DICOM tags (0020,0037), That value is medical information regarding the patient. imaging Data 1 line and the first row Azimuth cosine Specify death , For example, the value [1,0,0, 0, 1,0] ) ;" series Description (DICOM tag) (0020,0037) Identified by 、 The value includes a description of the group, for example The value indicates the axis direction, "axt1 total pelvis". ) ;" Echo Time (DICOM tag (0018,0081) Identified by 、 The value is, Excitation of echoes generated by MR imaging Center and peak of the pulse and time in milliseconds between Identify , for example The value is "4.2". ) ; 「 Repeat time 」( Identified by DICOM tags (0018,0080) 、 that value teeth MR imaging The start of the pulse sequence and the start of the subsequent pulse sequence and The time in milliseconds between identification death, For example, the value is "8"; "Flip Angle" (DICOM tag (0018,1IF-314) ) Identified by 、 that value teeth MR imaging Primary place Flips from the magnetic vector For magnetic vectors Steady state angle Identify , For example, the value is "90". ); "Ecotrain leader"(DICOM attribute Tags (0018,0091) Identified by , that value Excitation per image each The k-space obtained (An array of numbers representing spatial frequencies in MR images) in Identify the number of lines , For example, the value is "1"); " scanning Sequence (identified by DICOM tag (0018,0020), and its value is, capture Types of MR data Show , For example, the value "SE" indicates spin echo type MR. ); "Sequence name" (DICOM Identified by tags (0018,0024), that value teeth MR imaging in scanning User-defined names for combinations of sequences and sequence variants Identify , for example The value is "spcir_242". flavor "protocol name" (DICOM tag Identified by (0018,1030), and its value teeth, Name of the CT protocol Identify , for example The value is "T2W_TSE SENSE". some In this example, it will be understood that other such attributes and, in fact, other types of image files can be used.

[0273] Process S50 related For selecting a reference medical image set (RMIS). One example of a method This is shown in Figure 10. The corresponding data stream is shown in Figure 11.

[0274] In the embodiment shown in Figure 10, This method teeth, Process In BR-5, the target medical image group is TMIS. About Obtain the first vector FV Includes , The first vector FV is, Target medical images inspection The target medical image group TMIS for TS Capture Used for ta o Generated based on one or more attribute values ​​AV that indicate parameters. It is being done . This method teeth, Process In BR-6, the reference medical imaging test RS is included. multiple For each of the candidate medical image sets in CMIS, multiple second of This includes obtaining a vector SV, where the second of medical use imaging For each set of data, the candidate medical image group CMIS for the reference medical imaging examination RS is selected. Capture Used for ta o Based on one or more attribute values ​​AV that indicate the parameter, the second of A vector SV is being generated.

[0275] This method teeth, Process In BR-7, for each of the multiple second vector SVs, the target medical image group TMIS and the candidate medical image group CMIS are used. Each To determine the degree of comparability, we determine a similarity metric that shows the similarity between the first vector FV and the second vector SV. The process In BR-8, Based on the determined degree of comparability, Target medical imaging As something related to data (And consequently, as reference medical image sets RMIS) Select one or more candidate medical image sets (CMIS). thing, This includes, for example, the highest among multiple candidate medical image sets (CMIS) in some cases. similarity Candidate medical image sets (CMIS) with a degree of metric / comparability can be selected as reference medical image sets (RMIS). In some cases, the two or more highest similarity levels among multiple candidate medical image sets (CMIS) are selected. metric Comparability degree Two or more candidate medical image groups CMIS set This can be selected as the reference medical image set RMIS.

[0276] In some cases, the selection of reference medical image data is 、 Determined similarity Sexual metrics / Degree of comparability Based on , that is, the above body area and / or imaging Comparison of Modality Relevance Scores In additionIt may be based on this. For example, in some cases, Previous examinations (e.g., including multiple candidate physician image sets, including CMIS) Medical imaging The data includes the aforementioned body regions and / or imaging Based on a comparison of modalities, current medical imaging Related to the data, patient Previous examinations regarding It can be selected in advance from the following: 。 And, pre-selected inspection From among multiple groups, the reference medical image group RMIS determines the similarity. metric Based on the degree of comparability, the target medical imaging It can be selected as something related to data. In this case, it is shown to the user as display data. reference Image data is used for comparing body regions. imaging Modality relevance score and determined similarity metric Based on this, it can be selected as something relevant to the area of ​​interest.

[0277] The method described with reference to Figure 10 involves the same or similar image data or region of interest of the target medical image group TMIS. imaging Using parameters Capture It is possible to automatically select one or more reference medical image sets (RMIS). I'll make it This is, for example, a patient Regarding Compared to opening and evaluating all previous medical images to This provides efficient selection of previous medical images suitable for comparison with the target medical image. do Furthermore, by selecting based on the AV attribute value of the image file IF (relatively small size), medical imaging This allows for selection without having to extract or analyze the data itself (which is relatively large in size), enabling more efficient selection. This will be .moreover, Generated based on attribute value AV Vector FV ,SV By making selections based on this, flexible choices are possible. It becomes For example, comparing vectors FV and SV in the feature space is, for example, imaging Compared to attempts to directly match parameters, this approach is more effective in terms of inaccurate matches between parameters. Robust And / or it may be flexible.

[0278] Figure 11 shows One example The flow between the components of the method described in Figure 10 is illustrated. The image file IF of the target medical image group TMIS is the target medical imaging Data, such as region of interest (ROI) and target medical image set (TMIS), Capture Used for ta o Store attribute A which has attribute value AV that indicates the parameter. of Extract the attribute value AV, Vectorizer Provided to 906, Vectorizer 906 is the first of Outputs vector FV. Multiple candidate medical image sets CMIS are medical care It may be part of information system 40 storage It is stored in [location]. In some cases, each of the multiple candidate medical image sets CMIS is a different group within the reference medical image set RS. Each of It may be from one or more. Second Multiple attribute values ​​AV set (The candidate medical image group CMIS stored in each medical image data file IF that contains them) Capture Used for ta o Parameters Each of them shows ) are extracted. These sets of attribute values ​​AV are, next , Provided to the vectorizer 906, the vectorizer 906 each has multiple The second vector SV (one of each of the input sets of one or more attribute values ​​AV) Output The first vector FV and / or the second vector SV can be saved in association with the file of the target medical image set TMIS or candidate medical image set CMIS (or its identifier) ​​from which it was generated. Each of the first vector FV and the multiple second vector SVs is, The input is passed to comparator 910, and comparator 910, Determine the similarity metric between the first vector FV and the second vector SV. death, Target medical image group TMIS and candidate Medical images group CMIS and The degree of comparability between them is determined. The comparator 910 determines the degree of comparability (for example, the highest comparability) degree(It is possible to select one or more that have the above), and output one or more selections of target medical image group TMIS or candidate medical image group CMIS (or their identifiers). Next, the selected one or more candidate medical image group CMIS set of, storage It can be obtained from.

[0279] some In this example, as already mentioned, multiple set The candidate medical image group CMIS Remote storage These can be saved. example In this method, Select one or more sets of candidate medical image sets (CMIS) from remote storage (or a set of medical image data including the second set of medical image data from the selected set), multiple set Candidate medical image group other CMIS set It can include searching without having to search for anything. This provides efficient use of network resources. do .

[0280] some example In, This method It generates display data, Display devices (For example, 11 of (See reference) 、 ROI in areas of interest Rendering and , one or more selected candidate medical image groups CMIS image Based on data Rendering and displaying This can include, medical care Experts, patients Regarding Area of ​​Interest ROI (for example) Target or current ) and reference (for example, previous) medical imaging It provides a way to visually evaluate the difference between the data and the actual data. do The candidate medical image group CMIS is for the target medical imaging It is appropriate for comparison with data / area of ​​interest ROI. do Since it has been selected, the user can choose the modality and imaging parameters unrelated to diseases such as The difference is not due to factors, Image group Differences due to disease progression as represented by By It can be focused.

[0281] In some examples, This method is the first vector FV and / or One or more (for example, all) This may include generating a second vector SV.

[0282] In examples where one or more of the first attribute value AV or the second attribute value AV contain a text string, generating the first vector FV or the second vector SV respectively means converting the text string into a vector representation. encoding This may include: For example, a text string is Word embedding Encode using vector representation It will be done Words can be included. For example, word embeddings are a way to store words from a dictionary in a vector space. Mapping to Here, the words in the dictionary and the vectors for each word can be generated by applying a word embedding model to a corpus of training texts. A known example of a word embedding model is to use a neural network to generate vectors from a corpus of training texts. word This is "Word2vec" which learns embeddings. In some examples, for instance, a large corpus of general-purpose text. in Pre-trained That is , advance Trained word embeddings may be used. In some examples, the training text is a text string of attribute values ​​from radiology reports, medical literature, and / or training image files. etc. It can include medical texts. Medical field Words and abbreviations specific to semantic Meaning (within the context of the training text) Learning Making possible do If the text string consists of multiple words, each word Word embedding related to By combining vectors from, for example, each word in a text string Word embedding related to The vector join By doing so, or by taking the average, First Vector (or First It is possible to generate (part of) a vector. Other methods may also be used.

[0283] In an example where one or more of the first attribute value AV or the second attribute value AV includes a numerical value, generating the first vector FV or the second vector SV respectively can include formatting the numerical value into a vector representation. For example, one or more of the attribute values AV may be numerical values. For example, as described above, One example the DICOM attribute " Echo Time " of Example the value is "4.2". In such an example, the attribute value may be used as an element of the first Vector FV or the second vector SV as necessary. In some examples, the attribute value is normalized 、 before including it as an element of the first or second of vector as necessary You may . For example, the echo time attribute value can be divided by 10000 before being included as an element of the vectors FV, SV. As another example, one or more of the attribute values may include a plurality of numerical values, for example, a series of values. In such an example, the series of numerical values can be formatted into Column vectors one numerical value per element. For example, as described above, One example an example of the attribute value of the DICOM attribute "Image Orientation Patient" is "[1,0,0,0,1,0]". Format this Column vectors and use it as the first Vector FV or the second vector SV (or a part thereof) as necessary.

[0284] In some examples, generating the first vector FV or the second vector SV includes the first or second attribute value AV Each, , generating a third vector based on the first or second attribute value AV each , and combining the third vector to generate the first vector FV or the second vector SV to do . For example, the plurality of attribute values can be " each ", "echo Series Description ", and "Image Orientation Patient". In this case, the third vector is time For each of these three attribute values, for example, v SD ,v ET as described above,v IOP Next, these three The third Connect each of the vectors, as needed First Vector FV or Second It is possible to generate a vector SV. 、 For example, [v SD ,v ET ,v IOP ].

[0285] some In this example, similarity Determining the degree of metric / comparability is the second step for each. of Regarding vector SV, 1 of Vector FV and the second of This may include determining the cosine similarity between two vectors. As is known, cosine similarity is the dot product between two vectors and represents the similarity between them. For example, the cosine similarity of 1 is that two vectors are identical or very Similar vectors (therefore identical or very (corresponding to similar imaging parameters), the cosine similarity of 0 is the orthogonal vector (and therefore, The exact opposite or They differ greatly. This indicates the imaging parameters. In some examples, the first of the target medical image group TMIS / region of interest ROI is shown. of Vector FV and high cosine similarity offal 2nd of The vector SV candidate Medical image group CMIS but , second cosine similarity of The vector SV candidate It is preferred over the medical image set CMIS. In some examples, other similarities such as the Euclidean distance between the first vector FV and the second vector SV are also considered. metric This can be used. In these examples, the comparator 910 uses the cosine similarity (or other similarity) between the first vector and the second vector. Similar Calculation to calculate the similarity scale unit It may include (not shown)

[0286] some example In this case, the comparator 910, based on the input of two vectors FV and SV, determines, for example, the degree of comparability. Style Then, we train the model to output a value that shows the similarity between two vectors, FV and SV. done It may include a neural network (not shown). In some examples, Similarity Metrics To determine this, input the first vector FV and the second vector SV into the trained neural network for each of the second vector SVs. to do By thereby obtaining the output from the trained neural network, the relationship between the first vector FV and the second vector SV similarity This may include determining the degree of comparability as a value indicating the degree of comparability. For example, a neural network may have one or more hidden layers of neurons between the initial and final layers. deep A neural network would also be acceptable. initial The layer can be configured to take a first vector FV and a second vector SV of fixed size as inputs. The neural network is configured to map the vector representation from the final layer of the neural network to a value (e.g., between 0 and 1) that indicates the similarity between the two input vectors. Regressor (The following may be included.)

[0287] In some examples, neural networks are trained based on training datasets. So The training dataset is, Multiple pairs of vectors as described above (For example, each vector in the pair has the same format as the first vector FV and the second vector SV.) Multiple pairs of such vectors ) including each versus Training similarity value( training (Degree of comparability) Leveling And so, The training similarity value is, During neural network training teacher Provide a signal do In reality, training is a counter to such labeled training hundreds or thousands It may include. For example, training is Deep LearningIt can consist of: For example, training updates the weights of the connections between layers of neurons in a neural network for each of the training pairs of vectors. Regressor Predicted by similarity The value and each actual training pair defined by its respective label similarity This may include minimizing the error between the value and the actual value.

[0288] In some examples, training similarity The value labels are for each training pair of vectors. This represents probability, and this probability is, A first medical having a specific first attribute value represented in one vector of the training pair imaging data Assumed case to , user but , For comparison with the first medical imaging data, A second medical having a specific second attribute value expressed in the other vector of the pair imaging data They would choose probability That is For example, this probability is medical imaging This can be determined based on statistical analysis of recorded user interactions with the data. For example, data logs are: prescribed During the session prescribed by medical professionals prescribed About the patient search Medical imaging Files can be recorded. Statistical processing may be applied to these logs to determine probabilities. For example, for a set of medical images (TMIS) where the imaging parameter is X. Medical professionals, imaging The parameters Y Review of previous reference medical image sets (RMIS). What is the probability of doing so (for example, 60%)? , and Medical professionals, imaging The parameter is Z That is Review of previous reference medical image sets (RMIS). What is the probability of doing so (for example, 20%)? This can be determined. In this case, two training pairs can be generated, one of which has vectors representing X and Y. And 0.6 training similarity One has value labels, and the other has vectors representing X and Z. And 0.2 training similarity It has value labels. This is used to generate the training dataset. imaging Numerous parameters combination It is done against obtain The training dataset is medical imaging This is based on a statistical analysis of actual user interactions with the data, and is appropriate. imaging Using parameters Capture The second medical imaging Data is selected reliably Helps to make it so .

[0289] Figure 12 shows the embodiment Process As a method of performing registration in accordance with S60, the medical image group TMIS and RMIS Match slice Identification This shows how to do it. Corresponding data stream Figure 13 shows 。 This method has several Process There is. Process The order is not necessarily Process Corresponding to the numbering It is not necessary This may vary between different embodiments of the present invention. Furthermore, individual Process or a series Process This can be repeated.

[0290] The target medical image set TMIS and the reference medical image set RMIS relate to three-dimensional image data that depicts parts of a patient's body. Specifically, the target medical image set TMIS is: subject Image volume specific Each section Multiple image slices S-1 (hereafter, "1 of It is written as "Slice S-1". ) The reference medical image set RMIS also includes the reference image volume specific Each section Multiple image slices S-2 (hereafter, "2 of It is written as "Slice S-2". ) It may include.

[0291] In some examples, the target medical image group TMIS and the reference medical image group RMIS relate to the same anatomical structure of the same patient but were acquired at different times. In some examples, the target medical image group TMIS relates to the current imaging examination that the user is expected to analyze, while the reference medical image group RMIS relates to a previous examination at an earlier time, showing the same body part. By analyzing the target medical image group TMIS together with the reference medical image group RMIS, the user can therefore infer how the disease progressed or whether a particular treatment was successful. To assist the user in this regard, the method in Figure 12 is: In that matching slice CS is identified in the target medical image group TMIS and the reference medical image group RMIS, the target medical image group TMIS and the reference medical image group RMIS are synchronized or registered. .

[0292] Process SLS-10 in The system receives the target medical image set (TMIS). Process The SLS-10 has a user interface 10. The user uses the target medical imaging examination. Manually selecting TS, and / or 、 Medical Information System 50 Target medical image group TMIS search This includes nothing .moreover, Process The SLS-10 is subject Medical image set TMIS, My current cases and tasks This may include automatically selecting based on the following. For this purpose, the medical information system 40 may be queried for the appropriate group using a data identifier such as a patient or trial identifier. Process SLS-10 can be performed, at least partially, on either the user interface 10 or the processing system 20. Corresponding data exchange is performed as needed. but this Process .

[0293] Process SLS-20 in This allows for the acquisition of a reference medical image set (RMIS). This The process is shown in Figure 8. From Figure 9 This will be outlined in relation to Process to continuationThis may include selecting an appropriate reference medical imaging test (RS) from multiple candidate medical imaging tests (CS), and selecting a reference medical imaging group (RMIS) from multiple candidate medical imaging groups (CMIS) associated with the reference medical imaging test (RS).

[0294] Process In SLS-30, the slice between the target medical image group TMIS and the reference medical image group RMIS Match Establish a CS (Critical System) and, if possible, transfer a separate slice S-1 of the target medical image group TMIS to slice S-2 of the reference medical image group RMIS. Link That is to say, Process SLS-30 performs the following for each slice S-1 of the target medical image group TMIS: Match Slice CS Only one To attempt identification. Therefore, the target medical image group TMIS of RMIS (Remote Medical Image Set) for a given slice S-1 Match Find slice CS It is possible that one thing may not be possible, and vice versa. This is a single medical image set called TMIS. ,RMIS to included Anatomical location of slices S-1 and S-2 However, the other Medical image sets not covered by RMIS, TMIS It is possible In this case, according to some examples, Match slice association Ha-gyo Unbreakable . Process SLS-30 (and all optional Process Image processing of ) is mainly performed by processing system 20 in It is possible.

[0295] Below are optional Lower process SLS-31 from The SLS-35 is Process SLS-30 Match Identification of slice CS How will it be implemented? some Examples of implementations are provided. These provide some examples in this regard, This should not be considered to limit the scope of the SLS-30 process. How can the similarity between image slices S-1 and S-2 be determined? Another Eliminate the possibility It is not something to do.For example, image descriptors D-1, D-2 Clear extraction and their subsequent comparisons of As an alternative, we can use a Fourier-based analysis scheme. execution It is possible. in this case Each image or image region will be transformed into a Fourier space, and similarity will be calculated by applying a mathematical convolution function.

[0296] Optional Process The SLS-31 is The objective is to resample the reference medical image group RMIS based on the target medical image group TMIS, and to optimize this for slice-by-slice registration of the following two medical image groups. This involves using multiple slices S-2 in the reference medical image set RMIS with appropriate slice thickness and stacking direction. Definition This may include doing so. This process Furthermore, In the reference medical image set RMIS, slice S-2 already exists, They have the appropriate slice thickness and direction To have, resampling This includes nothing In particular, slice S-2 of the reference medical image group RMIS is similar to slice S-1 of the target medical image group TMIS in terms of thickness, spacing, slice direction, etc. It is comparable Therefore, the reference medical image set RMIS may be resampled. To put it further , This process The target medical image set TMIS is compatible with the medical image set RMIS. Let , It can also be done in the opposite way. .moreover, Process SLS-31 is an additional image processing technique to improve the comparability between the reference medical image set RMIS and the target medical image set TMIS. Process It is possible. This process The target medical image group TMIS Image data included (For example, in the metadata file of the target medical image set TMIS) Encoded ) is performed against Ta Image processing Process Reading the images and applying the same image processing to the reference medical image set RMIS. Process This may include applying the following:

[0297] Next any Process SLS-32 provides a pre-trained function TF configured to determine, and in particular quantify, the image similarity between two-dimensional medical images. The purpose is toIn particular, the trained function is Subsequent processes SLS-33 to SLS-35 To carry out It can be configured in this way. However, Subsequent processes SLS-33 to SLS-35 teeth Without using a pre-trained function, that is, for example, image To select features from the data and determine similarity based on them, 1 The above deterministic rules execution It has a hardcoded function. Image analysis function By IAF 、 It should be understood that it is possible to do so.

[0298] Process In SLS-33, image descriptors D-1 are extracted from each slice S-1 of the target medical image group TMIS. Image descriptor D-1 is in the form of an image feature vector and represents a typical slice S-1. or characterizing It may contain features. The target medical image group TMIS is, usually , Since it consists of image data and is associated with non-image data, the image descriptor D-1 is similar to image features Signature and non-image features It could be based on Image feature signatures are present in any image data included in the target medical image set TMIS. Includes identification, analysis, and / or measurement of objects, local structures and / or overall structures and / or textures. The generated image features can be generated by image analysis methods. Signature For example, the presence of landmarks. 、 or organ The size, or the structure, texture, and / or of the identified tissue or organ. Image features may include anatomical features and / or structures such as density. Signature Similarly, the colors and / or colors present in the analyzed image. gray Scale scheme or contrast characteristics or local gray scale gradient It may include parameters that characterize the image. Image features Signature Preferably, Not just one feature, but multiple features that characterize the image as a whole. Includes. Non-medical images included in the target medical image group TMIS. image data from The extracted non-image features include metadata related to the image data. Can see Furthermore, these include electronic health records and clinical test data. etc. Features extracted from such as to provide further status information regarding the target patients. image Data related to data independent of data Possible .

[0299] Process In SLS-34, matching image descriptor D-2 is extracted from the reference medical image set RMIS. The image descriptor D-2 of the reference medical image set RMIS is the same as the image descriptor D-1 of the target medical image set TMIS. Using the same method Generate Possible . some According to this example, the image descriptor D-2 of the reference medical image RMIS has already been generated and together with the reference medical image RMIS medical care Stored in Information System 40 It will be done .

[0300] Process In SLS-35, the extracted image descriptors D-1 and D-2 are used in the target medical image group TMIS and the reference medical image group RMIS. Match Slice CS Identification To do so, we will compare. The comparison is shown in Figure 10 and figure Introduced in relation to 11 It will be done Similarity or distance metric It can be performed again by applying it. The comparison is Comparison That is sufficient. In other words, each slice S-1 of the target medical image group TMIS is compared with each slice S-2 of the reference medical image group RMIS. Alternatively, each slice has two more similar slices that are kept for the next comparison. other Medical image datasets MIDS-1, MIDS-2 Each of those Comparison in a triplet, compared to two slices. It is possible to For each slice S-1 and S-2 in the medical image group, in order to provide a clear relationship between slices S-1 and S-2 in two medical image groups, other Medical image datasets MIDS-1, MIDS-2 each of Match Slice CS Identify only one Or not at all Do not identifyIf the slices of one medical image set are outside the imaging area of ​​each other medical image set, or if the slice resolution of a medical image set is, for example, that of a medical image set Compare with either slice S-1 or S-2. If it's too coarse to provide a match (If this is not corrected in any of the SLS-31 processes) , Match Slices I can't find it. Reference medical image set RMIS and between slices S-1 and S-2 of the target medical image group TMIS The identified assignment is To ensure that the ranking of each slice is saved, and, Ambiguity Resolved To this end, additional constraints or auxiliary conditions that the final result must satisfy may be provided. Such constraints may be provided by the order of slices and / or the overall similarity (which should be maximized) in the reference medical image set RMIS and the target medical image set TMIS.

[0301] The results of process SLS-30 are based on the reference medical image group RMIS. and the target medical image group TMIS Assignments that explicitly assign slices S-1 and S-2 to each other. Can be considered This result is "slice Match "CS and It can also represent In this way, slice Match CS is a form of registration between the target medical image group TMIS and the reference medical image group RMIS. Can be considered .

[0302] In the SLS-40 process, this result is used by the user or subsequent processing. Process It may be provided to the following. Process SLS-40 can be performed at least in part on either the user interface 10 or the processing system 20. Corresponding data exchange as needed. but this Process Include Mare .

[0303] Figure 14 shows several embodiments. to be concerned, Medical image collection Match Any method that can be performed based on the method for identifying slices is shown. In particular, any selection shown in Figure 6 of method The process , Process According to SLS-40 Can . Process The order is not necessarily Process This corresponds to the numbering system. no This may vary between different embodiments of the present invention. The method shown by reference numbers such as Figure 12. Process This refers to the method introduced and explained in relation to the embodiment shown in Figure 12. Process It corresponds to.

[0304] In particular, Figure 14 shows the process SLS Slice determined at -30 If This explains how CS can be used to synchronize views of two sets of medical images. An example is shown in Figure 14. Examples of use This occurs when the user has opened the target medical image set TMIS and is viewing a specific slice S-1 or region of interest ROI, and the reference medical image set RMIS is Comparison When obtained as a reference , Here, see Figure 12 and figure Similar slice searches introduced in relation to 13 allow for the analysis of the target medical image set TMIS. either Slice S-1 and reference medical image set RMIS Match Slice S-2 and automatically It can be matched If the user selects one slice, for example, for viewing, Each match This provides the benefit of automatically providing slice CS. Slice S-2 of the reference medical image set RMIS corresponding to the region of interest ROI is designated as the "reference slice" REF-SLC. It is also represented .

[0305] in particular, Process In SLS-50, user input is received indicating the selection of slices S-1 and S-2 in either the target medical image group TMIS or the reference medical image group RMIS. User input may be entered into system 1, for example, via user interface 10. The user input is: subject Scroll through one of the medical image sets (TMIS) or reference medical image sets (RMIS). This will allow you to reach the desired slices S-1 and S-2. Scrolling behavior It can be a form For example, such scrolling could be done using a scroll wheel or jog dial or wheel, or a suitable graphical user interface. Action button or slider , gestures, voice commands Using etc Input is possible. Furthermore, such user input allows the user to directly indicate the desired slice they want to see, for example, by clicking the z-axis bar that shows the z-position in the respective stacks of slices S-1 and S-2, or by directly entering the slice number. In particular, slice S-1 of the target medical image set TMIS can be implicitly selected by defining the region of interest in the target medical image set TMIS. Process SLS-50 can be performed, at least partially, on either the user interface 10 or the processing system 20. Corresponding data exchange is performed as needed. but this Process Include Mare .

[0306] Process In the SLS-60, the user's currently selected slices S-1 and S-2 are... Each match Slice CS Identification This method is used for this purpose. Process Slices determined by SLS-30 Match CS return Next, slice Match Based on CS, System 1 is a group of other medical images. Each Which slice is currently selected (viewed) by the user? Match It is possible to calculate whether or not it will happen. reference In the case of RMIS (Remote Image Scheme), this is used to identify the reference slice (REF-SLC), which corresponds to the region of interest (ROI). Connect . Process SLS-60 can be run primarily on the processing system 20.

[0307] Process In SLS-70, the identified Match Slice CS and / or reference slice REF-SLC are provided. In this case, "provided" means: Identification done Match Slice CS and / or reference slice REF-SLC ( Identification done MatchThis means displaying the data to the user in an appropriate graphical user interface by generating corresponding display data based on the slices. Possible According to some examples, synchronized medical image sets Identification done Match The slice CS / reference slice REF-SLC may be displayed along with the currently selected one / region of interest ROI. for example The two slices REF-SLC and S-1 can be displayed side-by-side in a suitable graphical user interface. Process SLS-70 has a user interface 1 0 or more Processing System 2 0 This can be implemented in either way, or at least partially. Corresponding data exchange as needed. but this Process Include Mare .

[0308] Process The SLS-80 is Process SLS-50 from SLS-70, i.e., receiving user input. (SLS-50) And Contrary to the agreement Slice CS Identification (SLS-60) , Match Output of slice CS (SLS-70) This demonstrates that it is possible to repeat this multiple times. Process Therefore, Match Slice CS can be dynamically adapted to slice S-1, which is currently selected by the user. In other words, User When a new slice is selected in a set of medical images, Match The slice CS is updated. This allows the user to scroll through two image sets simultaneously. Process The SLS-80 is, at least in part, User Interface This can be performed on either system 10 or processing system 20. Corresponding data exchange is performed as needed. but this Process Include Mare .

[0309] Figure 15 shows an embodiment of the present invention. to be concerned,Medical image collection Match This describes any method that can be performed based on the method for identifying slice CS. In particular, an optional method shown in Figure 15. Process teeth, Process SLS-40 continuation , Process Additional steps to determine the reference medical image set RMIS in S50 scale It can be used as such. Process The order is not necessarily Process This corresponds to the numbering system. no This may vary between different embodiments of the present invention. The method shown by reference number as in Figure 12. Process The method introduced and described in relation to the embodiment in Figure 12 Process It supports individual Process or Process This sequence can be repeated.

[0310] In particular, Figure 15 was explained in relation to Figure 4. similar Slice search allows for comparison of two sets of medical images. There This section explains how it can be additionally used to infer whether it is suitable for comparative interpretation by users. Specifically, slices Match Using CS, the imaging volumes of the target medical image group TMIS and the candidate medical image group CMIS are compared. Overlap To decide this. Overlap The larger the value, the more the two medical image sets Good comparison This means it is possible. .

[0311] therefore , Process The SLS-90 is Process SLS-40 in Slices provided Match Based on CS, Match Anatomical images of TMIS and CMIS medical imaging Overlap to decide The purpose is to . anatomical Overlap This involves two medical imaging sets, TMIS and RMIS. at each other Match do Each Number of slices Decision madeIt is possible. Further examples include anatomical Overlap This involves two medical imaging systems, TMIS and CMIS. Overlap Mark the end of the region. Match It can be determined from the image pair. Anatomical Overlap This involves two medical image data sets, TMIS and CMIS. Okeru Measurements can be taken in any unit along the stacking direction of slices S-1 and S-2. Process SLS-90 can be run primarily on processing system 20.

[0312] Process The SLS-100 uses two medical imaging systems called TMIS. and The degree of comparability of CMIS is anatomical Overlap It is calculated based on the following. According to some examples, the degree of comparability is, for example, the anatomical Overlap This is possible. Further examples show that additional information may be included as a factor. Such additional information could be, for example, Acquired Two medical imaging datasets: TMIS and Between CMIS Time span ( Low comparability Two image sets teeth Two medical image sets (TMIS), where a longer time may have elapsed. and Used to obtain CMIS Ta Each image modality type (data acquired with similar modalities is more Highly comparable (Sometimes), two medical imaging systems TMIS and CMIS included This includes metadata, etc. Process SLS-100 can be run primarily on processing system 20.

[0313] Process In SLS-110, the degree of comparability is provided. "Provided" means the determined degree of comparability. Make it so that it attracts the user's attention This means Possible If the two image sets are difficult to compare, the user will receive a warning message. receive It is possible to compare Image Interpretation For another thingYou can choose this option. Furthermore, "provided" means that the degree of comparability of the two medical image sets, TMIS and CMIS, will be recorded, i.e., for later use. medical care This means that it will be stored in the information system 40. Possible .moreover, "Provide" The degree of comparability is such that the candidate medical image group CMIS is automatically selected as the reference medical image group RMIS. Process This means it will be used in S50. Possible . Process SLS-110 has a user interface 1 0 or more Processing System 2 0 This can be implemented in either way, or at least partially. Corresponding data exchange as needed. but this Process Include Mare .

[0314] Figure 16 shows the reference medical image group RMIS and one candidate medical image group CMIS. About This paper presents a method for determining the degree of comparability of medical image data sets. This method involves several Process It consists of. Process The order is not necessarily Process This corresponds to the numbering system. no This may vary between different embodiments of the present invention. Figure 12 and / or figure Methods indicated by similar reference numbers, such as 15 Process The methods introduced and described in relation to each embodiment Process It corresponds to individual Process or a series Process This can be repeated.

[0315] The embodiment shown in Figure 16 is, Problem From multiple candidate medical image sets (CMIS) of Comparative Image Interpretation About The above embodiment provides a method for automatically evaluating and / or selecting a reference medical image set RMIS. It will proceed based on .

[0316] First stepThe SLS-120 receives the target medical image set TMIS. The purpose is to The target medical image set TMIS is used by the user. Trying to interpret the image For patients undergoing this procedure, consider the current examination as the appropriate course of action. can Separately, Process SLS-120 is Process It is equivalent to the SLS-10.

[0317] Process SLS-130 searches for multiple candidate medical image sets (CMIS) that pose problems for comparative interpretation, in addition to the target medical image set (TMIS). The purpose is to This includes, Process Related to S40 Explained Reference medical imaging test RS Identification This includes obtain Alternatively, the candidate medical image set CMIS is, Using another method, For example, CS imaging for candidate physicians Each test Related to do Regarding appropriate candidate medical image sets (CMIS), medical care By directly querying the information system 40 、 It is also possible to obtain such information. For example, appropriate case or patient identifiers, such as patient names or IDs, can be extracted from the target medical image set (TMIS) and used as search terms. Process The SLS-130 has a user interface 10 、 or processing system 20 、 or medical information system 40 of This can be implemented in either way, or at least partially. Corresponding data exchange as needed. but this Process Include Mare .

[0318] Next, for each of the CMIS candidate medical image sets that were searched, Process SLS-30, SLS-100, and SLS-110 are repeatedly used. Process SLS-140 via Repeat. These Process This can be mainly executed on the processing system 20. As mentioned above, Process SLS-30 is the target medical image group TMIS. andReference medical image set RMIS each slice between Match CS Identification do. This , Process SLS-30 will be repeated, in particular, for each candidate medical image group CMIS. Lower process Includes SLS-31, SLS-34 and / or SLS-35 Can see . Process Candidate medical image sets such as SLS-32 and / or SLS-33 are not related to CMIS. Other optional steps S130' Preferably, the candidate physician image group CMIS each The process is performed only once, not repeatedly, using the method shown in Figure 16. Processes SLS-100 and SLS-110 are shown in relation to Figure 15. Process It is identical to, This indicates the degree of comparability for each candidate medical image group (CMIS) to the target medical image group (TMIS), specifically the comparability of each candidate medical image group (CMIS). .

[0319] Process In SLS-150, the degree of comparability is provided to the user or for further processing. For example, the degree of comparability is provided for the candidate medical image group CMIS. Each It may be displayed to the user along with it. candidate Which of the CMIS medical image sets is suitable for follow-up interpretation? Overview We can provide this. therefore The user is appropriate candidate The CMIS medical image set can be intentionally selected as the RMIS medical image set. To be placed in a position . Process The SLS-150 can be performed, at least partially, using either the user interface 10 or the processing system 20. Corresponding data exchange is performed as needed. but this Process Include Mare .

[0320] Furthermore, the degree of comparability is, Process In SLS-160, it can be used to automatically or pre-select an appropriate candidate medical image group CMIS as the reference medical image group RMIS. For example, System 1 can select a degree of comparability greater than a predetermined threshold (anatomical Overlap )of candidate Medical image collection CMIS Everything It can be selected. candidate The CMIS medical image set then follows: Process Provided in S50 It will be done Therefore, offer " means, Presenting them to the user via user interface 10 for review and further selection and / or loading them into temporary memory of system 1. , means Possible . Process The SLS-160 has a user interface 10 、 or processing system 20 、 or medical information system 40 This can be implemented in either way, or at least partially. Corresponding data exchange as needed. but this Process Include Mare .

[0321] Figure 17 shows one or more of the above embodiments similar Using slice search, the temporal analysis of one or more lesions development Any method for determining Process This demonstrates several methods. Process It consists of. Process The order is not necessarily Process This corresponds to the numbering system. no This may vary between different embodiments of the present invention. The method shown with the same reference numerals as in Figure 12. Process The methods introduced and described in relation to each embodiment Process It corresponds to individual Process or a series Process This can be repeated.

[0322] Process The SLS-170 uses one or more related slices in the target medical image set TMIS. The purpose is to identify The relevant slice is the patient's lung. or One or more lesions in a patient's body, such as lesions in liver tissue drawing It is characterized in that it does so. In addition, or as an alternative form, connection A slice is, for example, in that slice. One or more Presence of lesions point out , user One or more Previous annotations may include Characterized by points can The relevant slices are identified, for example, by applying an appropriate computer-aided detection algorithm configured to detect lesions in medical imaging data. It is possible Furthermore, related slices are indicated by the user. One or more Related to ROI in areas of interest Possible . Process SLS-170 can be run primarily on processing system 20.

[0323] Process The SLS-180 detects one or more lesions in one or more related slices, for example, by using the computer-aided detection algorithm. The purpose is to . By detecting a slice with a lesion at the first location If the relevant slice has been identified, Process The SLS-180 is Process S510 Included obtain. Process SLS-180 can primarily be run on processing system 20.

[0324] Process In the SLS-190, Match Slices in the reference medical image set RMIS Identification And so, Match A slice corresponds to a related slice. In the reference medical image set RMIS. Match slice Identification teeth, Process Calculated in SLS-30, as shown in Figure 12. Process Slices provided in SLS-40 Match It will be conducted based on CS. Process SLS-190 can be run primarily on processing system 20.

[0325] Process In SLS-200, one or more corresponding lesions Process Identified in SLS-190 Match It is detected in the slice. This is also, Match slice imageThis can be done by applying computer-aided detection algorithms to the data. Process SLS-200 can be run primarily on processing system 20.

[0326] Process Lesions detected in the target medical image group TMIS using SLS-210. but , corresponding lesions and Matching In other words, lesions in the target medical image group TMIS are clearly assigned to corresponding lesions in the reference medical image group RMIS, and as a result, Lesion and corresponding lesions The pair Obtainable In particular Yes. Lesions that cannot be assigned in that way are not paired and, in some cases, new or disappearing lesions are designated. Such new or disappearing lesions are, for example, Process Displayed on S70 image data In By outputting the appropriate markings, As an option This can draw the user's attention. The lesions are located in each slice, relative to each other. or relative to other anatomical structures By evaluating their relative positions, shapes, characteristics (degree of calcification, structure at the boundary, etc.), size, optical properties, etc. matching It is possible. Process Lesions in SLS-210 of matching Regarding A specified computer-assisted lesion matching algorithm can be applied. Process SLS-210 can be run primarily on processing system 20.

[0327] Process The SLS-220 is Process SLS-210 matching Based on this, the temporal analysis of one or more lesions detected in the reference medical image group RMIS and the target medical image group TMIS development to decide The purpose is to For this purpose, one or more measurements were extracted for each lesion, MatchIt can be compared to the lesion. For example, such measurements relate to geometric values ​​such as the size, diameter, and volume of the lesion. Possible . Temporal development is, This shows how a particular lesion progressed. Results can be obtained Furthermore, the mean value / time of various lesion pairs development You may also calculate this. Process SLS-220 can be run primarily on processing system 20.

[0328] Process In SLS-220, the time was determined in this way. development This will be provided. Specifically, in terms of time development This can be, for example, in the form of a trend graph, or the user can place it on a lesion. When you hover over something (for example, the mouse cursor) Open Pop-up window It can also be provided to the user as such. Furthermore, temporal development teeth , By relating to individual lesions Potentially available In addition, or as an alternative form, temporal Development is medical Information Systems 40 to Archived In a structured medical report To be provided, that is, to put in It is possible. Process The SLS-220 has a user interface 1. 0 or more Processing System 2 0 This can be implemented in either way, or at least partially. Corresponding data exchange as needed. but this Process Include Mare .

[0329] If you add The above is applicable not only to two sets of medical images, One or more temporal developments from, Two or more point in time It shows the progression of individual lesions across the area. Furthermore, one or more lesions Regarding temporal development The automatic extraction is for follow-up image interpretation according to the above-mentioned example. correct Automatic selection and matching of reference medical image sets (RMIS). Specifically, first, sufficient anatomical information is obtained from the target medical image set (TMIS). Overlap One or more reference medical image sets (RMIS) having the following characteristics are determined: next , The target medical image group included in TMIS Temporal aspect of the lesion Development and The RMIS reference medical image set identified in this way speculation It can be considered that this is the case.

[0330] As already described in detail in relation to other examples, in the medical image sets TMIS, CMIS, RMIS Match Processing to identify slices Process This can be done by a pre-trained function TF configured to determine the similarity between two-dimensional medical images.

[0331] Generally speaking Such a trained function TF is trained task According to image Suitable for classifying data Intelligent Agent or related to the classifier do These are a pair of two-dimensional medical images. image It is possible to predict how similar the data are. certain Related to methods or apparatus Possible This definition is not limited to, but applies to support vector machines, decision trees, naive Bayes or ( Folding ) Including data mining tools and technologies such as neural networks. Specifically, Examples According to this, the trained function TF is Folding A neural network may be included. In one embodiment, the trained function TF arrangement is complete Folding It is a neural network. Alternative network arrangement For example, 3D Very Deep Convolutional Network (3D-VGGNet) can be used. this VGGNet is The maximum pooling layer is followed by many layers, including narrow convolutional layers, which are stacked on top of each other. . Examples According to this, skip connection , or shortcuts that skip over several layers, Use residual Neural Network (ResNet) It can be used .

[0332] Folding A neural network is defined as a series of consecutive layers. "continuous" This is used to show the general flow of how feature values ​​output from one layer are input to the next layer. Information from the next layer the It is supplied to the next layer and continues to the final output layer. The layers are, Sometimes it's feedforward only, and sometimes it's a lateral approach that includes some feedback to previous layers. There are also layers. weighting Furthermore, each layer is generally weighted. too Many node This includes... Basically, each node is a mathematical that maps one or more input values ​​to output values. Calculation It can be considered that it is performing the following: Immediately before It can connect to all or only a subset of the nodes in the subsequent layers. The two nodes are If inputs and / or outputs are connected, "connection" It is done. node The input value for is Each area of ​​interest The ROI or the image element values ​​of slices S-1 and S-2, preferably pixel values. The last layer is an output layer that outputs the similarity between the input image data. The output is a continuously changing value indicating the similarity. form It is possible. According to other examples, the output (similarity) is a binary format indicating whether the two images are similar. possible Numerous hidden layers exist between the input and output layers. The first group of neural network layers can be applied to extract features from an image. In this case, medical images, In other words , images individual Image elements Each Grayscale and / or color values ​​serve as input values ​​for the neural network. Thus extracted 、 Contrast, gradient, texture , density etc. Features such as , The second group of the network layer (Also known as a classifier) ​​is given as an input value, These network layers, Further assign an object and / or feature to at least one of the extracted features present in the image. It works in this way Convolution, pooling (for example, maximum value pooling or averaging value Pooling, upsampling, Reverse convolution , Fully connectedVarious types of layers may be used, such as other types of layers. Folding The layer receives input Folding , the result 、 By moving the image filter kernel onto the input, the next layer move . pooling The layers carry over the output of the node cluster in one layer to the next layer. 1 on the node combine As a result, the data size Upsampling reduces the amount of data used, thereby making the underlying calculations more efficient. and deconvolution Layers are abstract transformation From a level perspective, Convolution and pooling Layers action To reverse the situation. Fully connected Layers connect all nodes in one layer to all nodes in another layer, so that essentially all features receive "votes". One example According to this, skip connections can be used, and as a result, the layer is It is possible to output to layers other than the next layer in succession. 1 The above residuals Blocks or layers can be introduced. Such a configuration is also called ResNet. Using residual blocks, this 、 Known from very deep neural networks Gradient disappearance This mitigates the problem, thus improving the ability to train deeper networks. You can obtain .

[0333] According to some examples, the trained function TF extracts image descriptors from a two-dimensional image. (also called encoding task) Then, compare the image descriptors of these images. (Also called decoding) , evaluate the similarity between two images. (That is, the task of determining similarity) It can be configured to perform this action. therefore The trained function is at least Encoder branch and decoder branch Includes Can see .

[0334] Following some examples, the trained function uses multiple encoders. branch and decoder branch It may include each encoderBranch can process two-dimensional images and extract image descriptors D-1 and D-2 from them. Decoder branch teeth, Encoder branch By aggregating the image descriptors D-1 and D-2 obtained from, merged Processes latent image descriptor data structures. Processes multiple two-dimensional images. Encoder branch These are copies of each other that share the same parameters. Encoders branch is convolution layer It can include a CNN backbone like ResNet or Custom design Encoder can be used. branch The weight of the encoder branch It can be shared between layers, enabling efficient learning and processing at each layer of the network. This is how multiple layers process individual two-dimensional images. Encoder branch The ability to share the same parameters, that is, all encoders branch The same as weight This means that it is possible to use this. weight By sharing and making changes During training It can be implemented. Multiple Encoder branch The concept of sharing parameters between them is, Siam Called a copy Sometimes Specifically, two Encoder branch A network that includes Siam It can be called a network, while the three Encoder branch A network that includes Triplet It can be called a network. Encoder branch In the feature space of image descriptors, the image descriptors D-1 and D-2 of similar image pairs are more... Dissimilar Image descriptors D-1 and D-2 of an image pair The gap widens further. It can be configured in this way.

[0335] In the decoding branch Processing image descriptors D-1 and D-2 involves learning or preset distance or similarity. metric However, this means that it is applied to evaluate the similarity between two image descriptors D-1 and D-2. Possible . Similarity MetricsThis can be configured so that the distance between two image descriptors D-1 and D-2 is appropriately quantified in the feature space of image descriptors D-1 and D-2. Distance or similarity metric This can take the form of a mathematical function, for example, by outputting the cosine similarity or L1-norm for each image descriptor D-1, D-2. Furthermore, the similarity metric one that has appropriately fitted weights or parameters The above Network layer implementation It is possible.

[0336] Generally speaking The trained function TF in this embodiment is based on the training data, with each layer and node of weight or Weighting parameters By making it compatible Learn. Potential of similar medical images sign Instead of pre-programming, we define the architecture of a pre-trained function TF and learn these patterns at different levels of abstraction based on the input data. The pre-trained function TF is With a teacher Learning follow It is desirable to be trained using a method. The backpropagation method is well established and is the present invention. Examples It can be applied to the following: During training, the trained function TF is applied to the training input values, and the corresponding output values ​​whose target values ​​are known are obtained. Generate The difference between the generated output value and the target output value (for example, in the form of the mean squared error (MSE) of the difference between the generated value and the target value) is calculated to what extent. Good or bad The trained function TF is Function As a measure, it can be used to introduce a cost or loss function. The goal of training is to find the (locally) minimum loss function by iteratively adjusting the weights of the trained function TF so that the trained function eventually produces acceptable results over a (sufficiently) large cohort of training data. This optimization problem can be performed using stochastic gradient descent or other approaches known in the art.

[0337] As a general rule, one or more Encoder branchand Decoder branch The trained function TF, which includes the following, encoding (i.e., extraction of image descriptors) or Decoding It can be trained by adapting either (i.e., quantification of image similarity) or both. For example, one or more decoder Branches are exceptionally It can be adapted so that meaningful image descriptors D-1 and D-2 are extracted. Furthermore, the decoder branch Appropriate similarity metric It may be trained to be learned and / or applied.

[0338] According to some examples, Loss function is a triplet Loss function It can include triplets. loss This is when the baseline (anchor) input is a positive (similar) input. and Negative ( dissimilar ) Compared to the input loss This is a function that minimizes the distance from the baseline (anchor) input to the positive input and maximizes the distance from the baseline input to the negative input.

[0339] According to the examples, a typical training scheme can take the following form: First, a pre-trained function TF is received. The pre-trained function is already trained in advance. dolphin They were not trained at all. do .next, training We can provide the dataset. Training dataset teeth, Multiple images and training Image datasets determine the similarity between images. This indicates predetermined similarity and This includes, in particular, the images included in the training image dataset. 1 person Extracted from the above patient medical image set It could be Next, the trained function TF will be used. training Images from an image dataset versus To determine the degree of similarity that indicates the similarity between them, Images included in the training image dataset are input to the trained function TF.The similarity scores thus determined can be compared with pre-determined similarity scores. Finally, based on this comparison, the trained function TF is adjusted, and this adjusted trained function TF can be provided for further training or development.

[0340] In addition, or as an alternative, Figure 18 shows a specific method for training a trained function TF to determine the similarity between two-dimensional medical images. Process This indicates. Process The order is not necessarily Process This corresponds to the numbering system. no This may vary between different embodiments of the present invention.

[0341] First step T10 is Pre-trained or not trained Receive a pre-trained function TF The purpose is to achieve this. The trained function TF is received from memory (for example, in the form of a library of KI models) 、 In temporary memory keep It is possible.

[0342] What happened after that? Process T20 aims to provide a training image dataset containing at least three two-dimensional medical images. Each two-dimensional medical image represents an image volume of a patient's body part. drawing The images are extracted from one or more medical image sets such as RMIS, CMIS, and TMIS. The medical images are then expanded. and Easily learned Pre-trained function It is desirable that the slices S-1 and S-2 processed by TF be of the same type. therefore , Medical images teeth ,each Similarly, it is possible to show cross-sections of parts of a patient's body and draw multiple anatomical structures and organs. As mentioned above medical use imaging Acquired using one of the modalities doing According to some examples, two-dimensional medical images are extracted from the same set of medical images. Two-dimensional images are, 2nd The medical images , 1st For medical images Third The medical images contain similarityRather expensive Characterized by having similarities (It will be selected in that way) That is, the first of Medical images are anchors It was perceived as such. , 2nd of Medical images are positive (Positive) This is an image, and the third of Medical images are negative (Negative) This is an image.

[0343] To be continued Process In T30, the first, second, and third medical images are input to the trained function TF. Process In T40, The trained function TF is, The first similarity between the first medical image and the second medical image, and the second similarity between the first medical image and the third medical image are determined. Specifically, the trained function TF uses the first, second, and third medical images to determine the first similarity between the first, second, and third medical images. each Image descriptors D-1 and D-2 can be extracted. In other words, the trained function TF can extract the first, second, and third medical images. Each is encoded It is possible to do so. Subsequently, the trained function extracts the image descriptor each By comparing the first and second similarities, It is possible to infer In this regard, the trained function TF is, encoding space in Image descriptor each It is possible to output the distance. Similarity Metrics Apply Possible .

[0344] Next, the subsequent Process In T50, the trained function TF is the first of Similarity is second of It is adjusted to be greater than the similarity score. This adjustment is, encoding , that is, the process of extracting image descriptors, and / or Decoding , in other words, appropriate similarity metric The process of applying to adapt It can be a quantity. However, according to some examples, similarity metric For example, Euclid Fixed as distance or cosine similarity. heldIt may be done. In that case, the trained function All of the following are compatible. , One or more encoding branches Focus It becomes a thing. .

[0345] Figure 19 shows the embodiment Process As a method for performing registration according to S60, a method for determining the position in which a given feature is represented in medical image data is shown. This method can be used. One example medical use imaging data expression Figure 20 ~Figure This is shown in 23.

[0346] Figure 20 shows the representation of the target medical image group TMIS. Figure 21 ~Figure Figure 21 shows the representation of the reference medical image group RMIS. ~Figure As can be seen from 23, the target medical image group TMIS and the reference medical image group RMIS may be different. Specifically, in this example... The group , Regarding the patient, there is a difference in the location of the area where imaging data was captured... This area is offset to the right in the reference medical image set RMIS in Figures 21-23 compared to the target medical image set TMIS in Figure 20. Furthermore, the slices of the target medical image group TMIS and the reference medical image group RMIS match. Not , or slightly different body The area is covered It is In that sense, Figure 20 ~Figure On 23 image planes Regarding The target medical image group TMIS and the reference medical image group RMIS are vertically misaligned. In that respect, there is a difference between the two. Nevertheless, Features PM-226, PM-242 specific pattern but It is represented in both the target medical image group TMIS and the reference medical image group RMIS. doing In this example, medical imaging The predetermined characteristic PM-226 represented in the data is the PM-226 lesion in the patient's right lung. However, In one example, Features: Medical imaging data In Expressed Some kind of Features, for example, imaging target be It can be a specific part (for example, including an internal cavity). That is only natural. .

[0347] Usage example Now (for the purpose of explanation) The physician reviews the rendering of the target medical image set TMIS (as shown in Figure 20). doing Specifically, physicians review the ROI of their area of ​​interest. doing The physician compared the reference medical image set RMIS, prescribed Evaluate the characteristics, for example, the progression of the PM-226 lesion. I am trying to In the target medical image group TMIS prescribed Pattern PM-226 is expressed It is being done Location PM-224 is known However, prescribed Pattern PM-226 is represented in the reference medical image set RMIS. It is being done Location Not known The doctor must confirm it by visual inspection only. teeth Difficult or burdensome That is The method shown in Figure 20 is prescribed Features PM-226 is a reference medical image group RMIS In expression It is being done Determine the position.

[0348] Referring again to Figure 20, this method is overview This includes: Process PM-10 In , to obtain a first local image descriptor for a first position PM-224 in the target medical image group TMIS, the first position PM-224 is a position in the target medical image group TMIS where a predetermined pattern or feature PM-226 is represented. can be The first local image descriptor is the value of element PM-222 (i.e., pixel or voxel) of the target medical image group TMIS located at the first position 224 according to a first predefined sampling pattern. Represents ; Process In PM-20, multiple candidate second-in-commands were identified in the reference medical image set RMIS. of Each of the following locations: PM-334, PM-340, PM-440, PM-448 About Obtain the second local image descriptor. thing,Each second local image descriptor corresponds to the first predefined sampling pattern. Therefore Candidate #2 of Position PM-334, PM-340, PM-440, PM-448 each Represents the values ​​of elements PM-332 and PM-338 (i.e., pixels or voxels) of the reference medical image set RMIS located relative to [the specified location]; PM-30 In multiple candidates Second For each of the locations PM-334, PM-340, PM-440, and PM-448, 1 descriptor and Candidate #2 Locations PM-334, PM-340, PM-440, PM-448 The second Between the descriptors Local image similarity This indicates Local image Calculate the similarity metric thing ; Process Local images calculated in PM-40 Similarity Metrics Based Multiple candidate second positions PM-334, PM-340, PM-440, PM-448 Candidate position 2 from among them PM-334, PM-448 of Select thing ; Process In PM-50, the specified feature PM-226 is found in the reference medical image group RMIS. expression The positions PM-334 and PM-446 are selected as the second candidate. of Determined based on locations PM-334 and PM-448. thing .

[0349] Therefore, if the predetermined pattern or characteristics of PM-226 are medical imaging Data TMIS , in RMIS Techniques for determining the represented positions PM-334 and PM-446 but It will be provided. Specifically, prescribed The pattern or feature 226 is represented in the first medical image group TMIS. doing Using known locations, prescribed A pattern or feature is a second, for example Previous , Medical image sets, patient reference medical image sets, RMISDetermine the position to be represented. This is, for example, the reference medical image set RMIS. In a predetermined This reduces the burden on physicians when finding the location where the feature PM-226 is expressed. The reference medical image set RMIS for the selected candidate location... image The data is Process This can be used as the basis for generating display data in S70.

[0350] Furthermore, this decision This method is based on determining the similarity between local image descriptors for known locations where patterns or features PM-226 are represented in the reference medical image set RMIS, and for each of several candidate locations in the target medical image set TMIS. This can provide fast, efficient, and / or flexible feature locations.

[0351] A predetermined pattern or feature PM-226 can be obtained based on the region of interest (ROI). In some examples, prescribed The pattern or feature PM-226 is included in the region of interest (ROI). image This can be obtained from the data. In some examples, prescribed Pattern or feature PM-226 is in the region of interest ROI. Match .

[0352] For example, known 1 The location of PM-224 and Candidate #2 Location PM-334 PM-340 to Local Determining the similarity between image descriptors The thing By optimizing the cost function hand Compared to image registration techniques where every pixel / voxel in one image is mapped to every pixel / voxel in another image, this can be computationally considerably less expensive. Therefore, for a given computational budget, This disclosure The method is significantly faster than image registration-based methods. Ku This can provide results, which in turn could enable real-time or near-real-time interaction with image data.

[0353] A predetermined distribution or sampling pattern in An element located relative to a predetermined position PM-224 (for example, Pixel or voxel )value Based on descriptors relating to As a result, the surrounding and spatial context of feature PM-226, local In the image descriptor encoding This becomes possible. prescribed Pattern or features PM-226 The position represented in the reference medical image set RMIS is , determined in a reliable, efficient, and / or flexible manner that ,provide.

[0354] For example, such a local image descriptor is: According to image registration technology All pixels of a single image of Rather than attempting to map to pixels of another image, we map around the pattern or feature and candidate locations of interest. Encode . therefore , Target medical image group TMIS and reference medical image group RMIS of The image Relatively different (For example, the bodies they depict whole area In the case of Even if , prescribed The location where the pattern or feature is represented in the reference medical image RMIS can nevertheless be reliably determined (for example, by attempting to map all pixels between images, typically, Relatively Limited to similar images. Conventional (Compared to image registration technology). Therefore, This disclosure The technology provides accurate results for a wider range of image sets and can therefore be applied more flexibly.

[0355] As another example, at the known first location PM-224 Local Image descriptor and candidate Second location PM-334 PM-340 Between local Determining image similarity That is, Landmark detection-based methods Like , Classifier To detect within the image TrainedThere is no need to rely on the presence of "landmarks" in the medical image sets TMIS and RMIS. therefore , This disclosure The method is, method It may be more flexible regarding the types of medical image sets to which it can be effectively applied. Furthermore, the positioning local By doing so based on the similarity between image descriptors, This disclosure The technology is, for example, landmark detection. bass method Like classification vessel training It is being done Not a landmark, either It can be applied to specific characteristics. Therefore , This disclosure The technology provides accurate results for a wider range of features. therefore , can be applied more flexibly. By determining the similarity between descriptors, medical imaging data In a predetermined Location where the feature is expressed teeth Landmark detection bass technology Like Trained Classification vessel Decision made without using I became able to do it Therefore, classification vessel The time and effort involved in preparing the training dataset, as well as the classification vessel The computational load of training savings It is possible. therefore , This disclosure The method is medical imaging data In the prescribed Features of PM-226 The position in which it is represented can be determined in an efficient manner. .

[0356] As mentioned above As described above, this method is Process In PM-20, the target medical image group TMIS The first in Location PM-224 The first concerning This includes obtaining a local image descriptor. First Location PM-224 is the location within the target medical image group TMIS where a predetermined feature PM-226 is represented. The first local image descriptor, according to a first predefined sampling pattern, obtains the value of element PM-222 of the target medical image group TMIS located relative to the first location PM-224. represent.

[0357] In some examples, First Local image descriptors are, First The descriptor model applied to the target medical image set TMIS for position PM-224 can be output from the descriptor model. The descriptor model is the first In advance According to the defined sampling pattern, prescribed Location PM-224 Regarding Based on the value of the element in position, prescribed It may be configured to calculate the local image descriptor for position PM-224.

[0358] In some examples, First Local image descriptors may be obtained from a database (not shown). For example, First The local image descriptor for position PM-224 has already been calculated (for example, by applying a descriptor model), for example, First Stored in the database in relation to location PM-224 doing For example, the database is determined by the first local image descriptor. The basis for medical use imaging A plurality of first local image descriptors are associated with each corresponding first location in the data. keep It is possible. therefore some example In this case, the method may include selecting a first position PM-224 from among several and extracting a first descriptor associated with the selected first position PM-224.

[0359] In either case, relative to the designated position PM-224 local Image descriptors are multiple entry It may be a vector containing each entry There is one The above element set value Represents , one The above element set teeth, According to the first predefined sampling pattern For the designated position PM-224 location For example, each entryThis involves a plurality of predefined boxes PM-223 (i.e.,) positioned relative to a given position PM-224 according to a first predefined pattern. rectangle (domain) In each of the one or more The value of the element at the position can be represented . medical use imaging The data is in three spatial dimensions. End In this specification, it will be recognized that the term "box" may mean a cubic area or volume.

[0360] In some examples, each Descriptor entry This is a predefined set of multiple boxes PM-223 Each of them The value of the element located at can be represented .for example, Each local Image descriptor entry This is a set of multiple predefined boxes Each of the PM-223 It may also be the average of the values ​​of the elements at each position. entry This is the sum of the values ​​of the elements located within the specific box PM-223. in , divided by the number of elements contained in box PM-223. For example, as illustrated in Figure 20, for the first position PM-224, a specific sampling pattern is distributed within the slice of the target medical image group TMIS. did multiple In advance Defined box PM-223 (i.e., fictional A region exists. First Position PM-224 First A local image descriptor may be a vector, and entry teeth each , box PM-223 Each of them This is the average value of the elements of the target medical image group TMIS located at [location]. Using the average value means (For example, when compared to the total) Regardless of the size of the box in which it is calculated, each vector entry Being within the same range stipulate This will be helpful. More details below. explanation So this is next Robust and / or reliable similarity between descriptors decisionThis could be helpful in providing...

[0361] In some examples, predefined patterns and / or predefined boxes (e.g., the size and / or aspect ratio of each box) may be generated randomly or pseudo-randomly. In some examples, many boxes PM-223, e.g., 1000 boxes are used. local The image descriptor can be determined. therefore , local An image descriptor can be a vector with many entries (e.g., 1000 entries). For example, referring briefly to Figure 23, For the purpose of explanation, In order to determine the local image descriptor for a predetermined location (not shown) within the medical image set TIMS, RIMS, numerous Predefined box (white rectangle A set of medical images (TIMS) to which the contour (shown as the outline) has been applied is presented.

[0362] Local image descriptors are, prescribed The characteristics are expressed. prescribed Location PM-224 space Context encoding It is possible to Therefore, next , prescribed This can provide a compact representation of the features surrounding the characteristic. local The computation of image descriptors can be relatively computationally inexpensive and fast compared to relatively dense feature representations, such as those used in landmark detection techniques. This allows, for example, a method to be implemented quickly (and therefore results to be returned). Useful .

[0363] In some cases, descriptor models that compute local image descriptors can be applied to "raw" medical image data. However, in other cases, the descriptor model can compute the integrated image data of the target medical image set TMIS ( Calculation Area Table It can be applied to integrals (also known as integrals). image In the data, prescribed The values ​​for the elements are: image In the data All elements of the values ​​above and to the left of a given elementThis is the total. For example, the target medical image group TMIS Regarding Integrated image data A new generation is generated. , First Local image descriptor teeth Target medical image group TMIS Regarding Integrated imaging data Based It can be calculated. integral The use of image data enables faster computation of local image descriptors. In some examples, this is possible. next , enabling faster return of the results of the method Helpful .

[0364] In examples where integrated images are used, the diagonal position (x 1 ,y 1 ,z 1 ),(x 2 ,y 2 ,y 2 ) has box The sum of the element values ​​of PM-223 corresponds to the integral image I Regarding (I(x 2 ,y 2 ,z 2 ) + I(x 2 ,y 2 ,z 2 ) + I(x 2 ,y 2 ,z 2 ) + I(x 2 ,y 2 ,z 2 )-(I(x 2 ,y 2 ,z 2 ) + I(x 2 ,y 2 ,z 2 ) + I(x 2 ,y 2 ,z 2 ) + I(x 2 ,y 2 ,z 2 ))) is given In this equation, I(x i ,y j ,z k ) are the element values ​​of the integrated image I at positions x=i, y=j, and z=k. Here, i, j, and k are element indices. For a particular box PM-223, the average element value of box PM-223 may be calculated by dividing this sum by the total number of elements contained within box PM-223. The average element value for each box is: First Vectors that make up the descriptor In each entry Used It will be done .

[0365] In some cases, other than the specific examples mentioned above, local It is understood that image descriptors may be used. For example, in some examples, Haar-like descriptors are used. So That is, each entry is one of the multiple boxes defined within the image data. Inside This represents the difference between the sums of the element values. Descriptor In some cases, local Image descriptors are, for example, each entry multiple regions of medical image data In each This may also be a gradient descriptor representing one or more image gradients. For example, prescribed The image gradient in the region is prescribed This can be based on changes in values ​​(e.g., intensity values) between elements within a region. In some examples, the local image descriptor is based on each entry However, for the first position PM-224, medical imaging randomly in the data distribution Multiple elements Each It may also be something that is a value. In some examples, prescribed The descriptor for a location is each entry However, multiple randomly oriented rays and each of them is a ray originating from a predetermined position of Each and It could be something like a set of values ​​for intersecting elements. In each case, prescribed The descriptor for a location is, First According to a predefined pattern prescribed Location In contrast Values ​​of elements in the medical imaging data at location Represents .

[0366] Nevertheless, the inventors have found that the distribution patterns can be defined in advance (for example, randomly generated). at Multiple predefined boxes positioned relative to a given location PM-223 of Each Value of the element at the position Each entry represents (for example, the average).The use of a local image descriptor for a given location Especially fast, but To provide accurate positioning Checking Identifying This is attracting attention. .

[0367] In some cases, local image descriptors are computed or otherwise obtained. First Location PM-224 is specified by the user. obtain For example, the representation of the target medical image group TMIS can be displayed to the user, and the user can, for example, prescribed The characteristics of PM-226 are expressed. First By clicking on the representation at location PM-224, The prescribed The characteristic PM-226's position can be specified as PM-224. Next, this User The specified position First The position can be set to PM-224. Next, this First Based on location PM-224, First Local image descriptors can be calculated or obtained by other means.

[0368] In some examples, First Location PM-224 is a computer Executed by The method can output the following: For example, the target medical image group TMIS identifies a predetermined pattern or feature PM-226 within the target medical image group TMIS. and The characteristics of PM-226 are expressed. First Output position PM-224 Executed by the computer By method (not shown) , pre-processed This output is provided directly and / or to a database. keep It is possible. Next, this First Based on location PM-224, First Local image descriptors can be calculated or obtained by other means.

[0369] In some examples, First Location PM-224 is in the database (not shown) It can be obtained from. For example, a database is One or more medical image datasets each represent one or more features.One or more locations can be saved. The target medical image set TMIS can be extracted from the database along with one or more locations. A specific one of the locations but ,for example, prescribed Features PM-226 is a reference medical image group RMIS In To determine the position to be represented request Alternatively, based on instructions, the first position PM-224 may be selected. First Based on location PM-224, First Local image descriptors can be calculated or obtained by other means.

[0370] In any case, among the target medical image group TMIS First Position PM-224 First A local image descriptor is obtained.

[0371] Description As done, Process In PM-40, this method is used with the reference medical image group RMIS. Multiple candidate second positions in PM-334, PM-340 For each of the following, the second local This includes obtaining an image descriptor.

[0372] Each second local image descriptor is the first In advance According to the defined sampling pattern, each candidate second of This represents the values ​​of elements PM-332 and PM-338 of the reference medical image group RMIS, which are positioned relative to locations PM-334 and PM-340. The second local image descriptor is: prescribed Location (for example, first location PM-224 or second candidate locations PM-334, PM-340) either For one, Local image descriptor , According to a first predefined sampling pattern, at a predetermined position location Associated medical use imaging data prescribed Element values Representing, In that sense, 1 It may be the same as the local image descriptor. For example, the first for position PM-224 local The same medical image set TMIS applied to generate image descriptors. localThe image descriptor model is applied to the reference medical image set RMIS, and for each of the multiple candidate second locations PM-334 and PM-340, the second... local Image descriptors can be generated. For example, referring to Figures 20 and 21, The second descriptor for each of the candidate second positions PM-334 and PM-340 was used to calculate the second descriptor for each of the candidate second positions PM-334 and PM-340. Box PM-332, PM-338 And the position of each of these boxes is the box and position of each box relative to the first position PM-224 used to calculate the first descriptor relating to the first position PM-224. It is the same as. descriptor is used Even in , relative to the first position PM-224 in the target medical image group TMIS descriptor , and the second medical imaging In data 330 multiple The second candidate locations PM-334 and PM-340, respectively descriptor This can be obtained.

[0373] As will be explained in more detail below, in some examples (see, for example, Figure 21 below), each candidate 2 of Locations PM-334 and PM-340 are, The previously detected features PM-226 and PM-242 are represented in the reference medical image set RMIS. It can be a position. In other examples (see, for example, Figure 22 and described below), candidate The second position PM-440 is the second medical in the second predefined pattern imaging Through data It could be a distributed position .

[0374] In any case, multiple candidate 2 in the reference medical image set RMIS of A second local image descriptor is obtained for each of the positions PM-334, PM-340, and PM-440.

[0375] As mentioned above As shown above, this method is Process In PM-60, multiple candidates Second For each of the locations PM-334, PM-340, and PM-440, 1 of local Image descriptor and , second local image descriptors relating to candidate second locations PM-334, PM-340, PM-440 Between Local image similarity This indicates similarity metric of This includes calculations.

[0376] In some examples, the local image similarity metric may include normalized cross-information similarity between a first local image descriptor and a second local image descriptor. For example, the normalized cross-information similarity between a first local image descriptor and a second local image descriptor may be: Next It can be decided in this way. First A histogram is formed, and within it, 1 descriptor entries in teeth, First Minimum local image descriptor entry Value and Maximum entry value and Bins of equal size between (bin) It is placed inside x. First histogram count but Normalized, the entire bin x 1 Probability distribution P of entries for descriptors X(x) This is obtained. A second histogram is formed, in which the entries for the second local image descriptor are, Second local image descriptor It is placed in bin y of equal size between the minimum entry value and the maximum entry value. Second histogram count but Normalized, the entire bin y Second local image Probability distribution P of descriptor entries Y(y) Obtain the values ​​within the range between the minimum and maximum values. First Local image descriptor and Second Local image descriptor entry Determine the combined histogram. join Each bin in the histogram represents: Equal size It is a two-dimensional bin, The first dimension X is From the first local image descriptor entry Compatible death , and The second dimension Y is Related from the second local image descriptor entry Compatible do .for example, First Local image descriptor First entry is q, and, Second Local image descriptor First entry If p, then including p First Covers the range of local image descriptor values, including q. SecondCovers the range of local image descriptor values, join The 2D bins x,y of the histogram receive the count. This will be the case. . join The histogram counts are normalized and spread across bins x and y. 1 and 2nd local image Probability distribution P of descriptor entries XY(x,y) Obtain the following: Then, between the first local image descriptor and the second local image descriptor. Normalization mutual information similarity I can be calculated as follows: JPEG0007851965000001.jpg16133

[0377] The higher the mutual information similarity I, the better the first of Local image descriptor and second of Local image with local image descriptor Similarity increases . Normalization By using mutual information similarity, local Robustness of image similarity Well, Reliability high, and / or This allows for more flexible decision-making. For example, mutual information Similarity is compared with the second local image descriptor. did This is independent of differences in the relative scale of entries in the first local image descriptor. For example, using mutual information, the overall "luminance" (e.g., medical imaging Even if the intensity (the value of a data element) differs between the target medical image group TMIS and the reference medical image group RMIS, accurate Similarity Metrics It can be determined. Another example is medical Different imaging protocols and / or modalities were used. (And, for example, accordingly different If a value range is used or achieved, or if, for example, a medical image is transformed, Invert Even if the images are scaled up or down, the mutual information can still provide accurate similarity. Therefore, by using mutual information similarity in this way, even if Ba First Medical imaging Data and second Medical imaging Robust similarity determination against unstructured variations between data. ProvidedAs a result, this method can be reliably applied to a wider range of images. and This method offers greater flexibility in the types of images that can be used. can .

[0378] As mentioned earlier, in some examples, descriptor entry teeth, Related Boxes PM-223, PM-333, PM-339 inside Values ​​of the elements included Represents As mentioned above, in these examples, local image descriptors each Each entry teeth, Each box: PM-223, PM-333, PM-339 The average of the values ​​of the elements contained within, within the local image descriptor. entry However, to guarantee that it is within a certain range independent of the box size Useful Next, Prescribed entry The bin in which it is placed depends on the average value of the elements in the box accordingly. death Since it is independent of the box size, it facilitates the use of mutual information similarity. I'll make it .

[0379] In some examples, other similarity metrics may be used between the first and second local image descriptors. For example, cosine similarity, Euclidean distance, and / or cross-correlation. (Cross-correlation) These may be used as alternatives or additionally. Even so, The inventors have identified mutual information similarity metric However, similarity metric In particular Robust Therefore, it provides reliable and / or flexible decision-making. therefore In the reference medical image set RMIS prescribed The position in which the characteristics are expressed Robust Therefore, it can provide reliable and / or flexible decision-making. confirmation did.

[0380] in any case, each 2nd of Regarding candidate locations PM-334 and PM-340 hand , Regarding candidate location 2 PM-334, PM-340 1 ofLocal image descriptor and second of This shows the local image similarity between local image descriptors. The similarity metric is calculated. .

[0381] As mentioned above As described above, this method is Process Similarity calculated in PM-40 metric Based on this, multiple candidates 2 of Candidate No. 2 from positions PM-334 and PM-340 of Select position PM-334, Process In PM-50, the second selected candidate of Based on position PM-334, in the reference medical image group RMIS prescribed This includes determining the position PM-334 where the characteristic PM-226 is represented.

[0382] some example So, candidate Selecting the second position PM-334 is possible for multiple candidate Second position PM-334, PM-340 Similarity Metrics The highest local Shows image similarity Similarity Metrics has candidate This may include selecting a second location PM-334, for example, the highest mutual information. Similarity Metrics has candidate A second position, PM-334, can be selected. In some examples, prescribed Determining the position where the characteristic PM-226 is represented is prescribed The characteristic PM-226 is represented as the position in the reference medical image group RMIS. This includes determining the selected candidate second location PM-334. In some examples, the position To decide This includes a first local image descriptor for a first position and a selected Candidate #2 The similarity metric between the second local image descriptor and the first local image descriptor is above the threshold. decision It can respond to this. prescribed The position in which the feature is represented within the reference medical image set RMIS is, decision Only if there is a certain degree of trust decision To ensure that it is done help Next, prescribed The reliable location where features are represented in the reference medical image set RMIS. decision Provide do .

[0383] As mentioned above, in several examples (for example, as shown in Figure 21), each candidate 2 of Locations PM-334 and PM-340 may be the locations where previously detected features PM-226 and PM-242, respectively, are represented in the reference medical image set RMIS. In another example (for example, as shown in Figure 22), candidate second location PM-440 may be distributed through second medical imaging data in a second predefined sampling pattern (also called a predefined sampling pattern). These are exemplary scenarios In each of these, prescribed The location of the feature is determined. It will be done method but , Figure 21 and figure See 22. next More details explanation It will be done.

[0384] First, referring to Figure 21, this Example Scenario In this context, the reference medical image group RMIS includes the first feature PM-226 and the second feature PM-242. express In this example, these features PM-226 and PM-242 have been detected in the reference medical image RMIS, for example, previously by a physician or by an automated method. The respective positions PM-334 and PM-340 of these features PM-226 and PM-242 in the reference medical image RMIS are recorded as part of this previous detection. Therefore, the respective positions PM-334 and PM-340 of these previously detected features PM-226 and PM-340 are recorded. is known However, these Which of the two positions, PM-334 or PM-340? However, among the target medical image group TMIS prescribed Features of PM-226 but Reference medical image set RMIS In Whether it is a position that is expressed They don't understand In other words, these Which of the two features, PM-226 or PM-224?Target medical image group TMIS Expressed in It is unclear whether it corresponds to the specified characteristic PM-226. therefore In these examples, candidate The second position is in the reference medical image set RMIS where previously detected features PM-226 and PM-242 are represented. Positions PM-334, PM-340 It is possible. Candidate No. 2 Location PM-334,PM-340 A second local image descriptor for each of and First local image descriptor for the first location PM-224 to The similarity metric between them can be calculated. example So, the highest similarity Candidate #2 with a metric of Position PM-334 can be selected, and the second selected candidate is of Position PM-334, prescribed The characteristic PM-226 can be determined as position PM-334 in the reference medical image group RMIS.

[0385] Referring to Figure 23, the reference medical image group RMIS has the first feature PM-226 and the second feature PM-242. express However, in the illustrative scenario in Figure 23, the positions in which these features PM-226 and PM-242 are represented within the reference medical image set RMIS are They don't understand In this example, candidate The second position PM-440 is t...

Claims

1. A method for generating display data for a medical image dataset, - Receiving the patient's target medical image set (TMIS) at the first time point (S10), - Determining the target body region (408) represented by the aforementioned target medical image group (TMIS) (S30), - Based on a comparison between the target body region (408) and multiple candidate body regions (720), a reference medical imaging test (RS) is selected from multiple candidate medical imaging tests (CS) (S40). Each of the plurality of candidate body regions (720) corresponds to one of the plurality of candidate medical imaging tests (CS), and each of the candidate medical imaging tests (CS) includes a plurality of candidate medical imaging images (CMIS) of the patient at a second time point. Each of the aforementioned subject and candidate medical image sets (TMIS, CMIS) is associated with one or more attributes (A) having attribute values ​​(AV) that indicate the imaging parameters used to capture each of the aforementioned subject and candidate medical image sets (TMIS, CMIS), - Select a reference medical image group (RMIS) from the multiple candidate medical image groups (CMIS) of the reference medical image examination (RS) based on the degree of comparability with the target medical image group (TMIS) (S50). The degree of comparability is determined at least by determining the matching of attribute values ​​(AV) indicating the imaging parameters between the target medical image group (TMIS) and each of the multiple candidate medical image groups (CMIS), - Perform registration between the target medical image group (TMIS) and the reference medical image group (RMIS) (S60). - Based on the registration, generate display data to display the rendering of the reference medical image group (RMIS) on a display device (S70). A method that includes this.

2. A method for generating display data for a medical image dataset, - Receiving the patient's target medical image set (TMIS) at the first time point (S10), - Determining the target body region (408) represented by the aforementioned target medical image group (TMIS) (S30), - Based on a comparison between the target body region (408) and multiple candidate body regions (720), a reference medical imaging test (RS) is selected from multiple candidate medical imaging tests (CS) (S40). Each of the plurality of candidate body regions (720) corresponds to one of the plurality of candidate medical imaging tests (CS), and each of the candidate medical imaging tests (CS) includes a plurality of candidate medical imaging images (CMIS) of the patient at a second time point. - Select a reference medical image group (RMIS) from the multiple candidate medical image groups (CMIS) of the reference medical image examination (RS) based on the degree of comparability with the target medical image group (TMIS) (S50). To determine the degree of comparability, For the aforementioned group of medical images (TMIS), a first feature vector (FV) is obtained (BR-5). For each of the candidate medical images (CMIS) included in the aforementioned reference medical image examination (RS), a second feature vector (SV) is obtained (BR-6). This includes at least determining a comparability metric (BR-7) that indicates the degree of comparability between the first feature vector (FV) and the second feature vector (SV) of the candidate medical image group (CMIS), The first and second feature vectors (FV, SV) include indices of the imaging modality of the subject and candidate medical image groups (TMIS, CMIS), The comparability metric includes an imaging modality correlation score between the imaging modality of the target medical image group (TMIS) and the imaging modality of each of the candidate medical image groups (CMIS). - Perform registration between the target medical image group (TMIS) and the reference medical image group (RMIS) (S60). - Based on the registration, generate display data to display the rendering of the reference medical image group (RMIS) on a display device (S70). A method that includes this.

3. The subject medical image group (TMIS) is included in the subject medical imaging examination (TS), Each of the target medical imaging examination (TS) and the candidate medical imaging examination (CS) is associated with one or more attributes (A) having attribute values ​​(AV) that include text strings (404, 716) indicating the content of the target and candidate medical imaging examinations (TS, CS), - Determining the target body region (408) (S30) To obtain one or more of the text strings (404) of the target medical imaging examination (TS) (BR-1), Inputting one or more of the text strings (404) of the acquired target medical image examination (TS) into a trained machine learning model (406) (BR-2), The trained machine learning model (406) is trained to output body regions based on the input of one or more text strings (404) of the target medical image examination (TS), This includes determining the target body region (408) represented by the target medical image set (TMIS) by obtaining the output from the trained machine learning model (406), and / or, - At least one of the candidate body regions (720) is To obtain one or more of the aforementioned candidate medical imaging tests (CS) (BR-1), Inputting one or more of the text strings (716) of the acquired candidate medical image examinations (CS) into the trained machine learning model (406) (BR-2), The trained machine learning model (406) is trained to output body regions based on the input of one or more of the text strings (716) of the candidate medical imaging examination (CS), The candidate body region (720) represented by the candidate medical image set (CMIS) is determined by obtaining the output from the trained machine learning model (406). The method according to claim 1 or 2.

4. - The trained machine learning model (406) is a trained neural network consisting of a trained character-based neural network configured to take individual characters of one or more acquired text strings as input, Inputting the acquired text strings into the trained neural network includes inputting each character of the acquired text strings into the trained character-based neural network. The method according to claim 3.

5. - The aforementioned reference medical image set (RMIS) draws the reference image volume, This method, To obtain the region of interest (ROI) of the aforementioned target medical image group (TMIS), The method further includes obtaining a plurality of candidate slices (S-2) each drawing a specific section of the aforementioned reference image volume, - Performing the registration (S60) includes identifying at least one reference slice (REF-SLC) (SLS-60) from the plurality of candidate slices based on the similarity between the image data contained in the region of interest (ROI) and each of the candidate slices, - Generating the display data (S70) includes generating display data to display the rendering of the reference slice (REF-SLC) on the display device. The method according to claim 1 or 2.

6. - Extract at least image descriptors (D-1) from the aforementioned medical image group (TMIS) (SLS-33), - Extract a matching image descriptor (D-2) from each of the multiple candidate slices (S-2) (SLS-34), It further includes, - The similarity is based on a comparison between the extracted image descriptor (D-1) of the target medical image group (TMIS) and the matching image descriptor (D-2) of the candidate slice (S-2). The method according to claim 5.

7. Identifying the at least one reference slice (REF-SLC) (SLS-60) includes applying the trained function (TF) to the subject and reference medical image sets (TMIS, RMIS), The pre-trained function (TF) is configured to determine the similarity between two-dimensional medical images. The method according to claim 5.

8. - Performing the registration (S60) is, Based on the image data of the region of interest (ROI) acquired with respect to the aforementioned target medical image group (TMIS), a first local image descriptor is generated (PM-10). To generate a second local image descriptor (PM-20) for each of the multiple candidate locations (PM-334, PM-340, PM-440, PM-448) in the aforementioned reference medical image set (RMIS), Each of the second local image descriptors is generated based on image data of the reference medical image set (RMIS) located in relation to each of the candidate locations (PM-334, PM-340, PM-440, PM-448), For each of the plurality of candidate locations (PM-334, PM-340, PM-440, PM-448), calculate a local image similarity metric (PM-30) that shows the similarity between the first local image descriptor and the second local image descriptor of the candidate location (PM-334, PM-340, PM-440, PM-448). Based on the calculated local image similarity metric, select candidate positions (PM-334, PM-448) from among the multiple candidate positions (PM-334, PM-340, PM-440, PM-448) (PM-40). This includes determining a position in the reference medical image group (RMIS) that matches the region of interest based on the selected candidate position (PM-50), - In generating the display data (S70), the rendering is generated based on the image data of the reference medical image group (RMIS) associated with the set of selected candidate positions (PM-334, PM-448). The method according to claim 1 or 2.

9. - Selecting at least one of the candidate medical imaging tests (CS) as the reference medical imaging test (RS) (S40) From the candidate medical imaging examinations (CS), multiple related medical imaging examinations are identified based on a comparison of the subject and candidate body regions (408, 720) (S43). The user is provided with indicators for the plurality of related medical image examinations via the user interface (10) (S44). The user receives a user selection indicating at least one of the related medical imaging examinations via the user interface (10) (S45), The selection of at least one of the associated medical imaging tests as the reference medical imaging test (RS) (S46), The method according to claim 1 or 2.

10. The subject medical image group (TMIS) is included in the subject medical imaging examination (TS), Identifying the multiple related medical imaging tests (S43) includes determining the degree of fit with the target medical imaging test (TS) for each of the related medical imaging tests based on a comparison of the respective body regions (408, 720) (S42). Providing the indicators for the plurality of related medical imaging tests includes providing the degree of fit to the user via the user interface (10) (S44), - The degree of fit is based on the anatomical overlap between each of the medical imaging tests, based on the comparison of the subject and candidate body regions (408, 720). The method according to claim 9.

11. - The aforementioned reference medical image examination (RS) includes one or more annotations (RM-322 to RM-330) corresponding to the aforementioned reference medical image set (RMIS), - Obtain at least one reference annotation (RM-10) from the one or more annotations (RM-322 to RM-330) relating to the subject medical image group (TMIS) and / or the region of interest (ROI) relating to the subject medical image group (TMIS). - Annotate the aforementioned medical image group (TMIS) and / or the aforementioned region of interest (ROI) with the aforementioned reference annotation (RM-20). Includes, - The above reference footnote contains one or more first words (A, B, C, D), The method according to claim 1 or 2.

12. - Obtain a medical report (RM-504) linked to the aforementioned reference medical imaging examination (RS). - Obtain one or more sections (F1-F3) of the text of the aforementioned medical report (RM-504), each section (F1-F3) consisting of one or more second words. - For each of the one or more sections (F1 to F3) and each of the reference notes (A, B, C, D), compare the one or more second words with the one or more first words in the reference note and determine if they match. - Based on the confirmed match, link at least one of the reference annotations (A, B, C, D) to at least one of the sections (F1 to F3), Further including, The method according to claim 11.

13. A system (1) that supports the evaluation of the patient's target medical image group (TMIS) acquired at a first point in time, - Includes an interface unit (10) and a computing unit (30), The interface unit (10) is The rendering of the aforementioned target medical image group (TMIS) is provided to the user. It is configured to receive user input from the user intended to specify a region of interest (ROI) of the target medical image group (TMIS) based on the rendering, The computing unit (30) is The target body region (408) represented by the aforementioned target medical image group (TMIS) is determined (S30), From among multiple candidate medical imaging tests (CS), a reference medical imaging test (RS) is selected based on a comparison between the target body region (408) and multiple candidate body regions (720) (S40). Each of the plurality of candidate body regions (720) corresponds to one of the plurality of candidate medical imaging tests (CS), each of the candidate medical imaging tests (CS) includes a plurality of candidate medical imaging images (CMIS) of the patient at a second time point. Each of the aforementioned subject and candidate medical image sets (TMIS, CMIS) is associated with one or more attributes (A) having attribute values ​​(AV) that indicate the imaging parameters used to capture each of the aforementioned subject and candidate medical image sets (TMIS, CMIS), From the multiple candidate medical image groups (CMIS) of the reference medical imaging examination (RS), a reference medical image group (RMIS) is selected based on the degree of comparability with the target medical image group (TMIS) (S50). The degree of comparability is determined at least by determining the matching of attribute values ​​(AV) indicating the imaging parameters between the target medical image group (TMIS) and each of the multiple candidate medical image groups (CMIS), Registration is performed between the target medical image group (TMIS) and the reference medical image group (RMIS) (S60), The system (1) is configured to generate display data (S70) to display the rendering of the reference medical image group (RMIS) on the interface unit (10) based on the registration.

14. A system (1) that supports the evaluation of the patient's target medical image group (TMIS) acquired at a first point in time, - Includes an interface unit (10) and a computing unit (30), The interface unit (10) is The rendering of the aforementioned target medical image group (TMIS) is provided to the user. It is configured to receive user input from the user intended to specify a region of interest (ROI) of the target medical image group (TMIS) based on the rendering, The computing unit (30) is The target body region (408) represented by the aforementioned target medical image group (TMIS) is determined (S30), From among multiple candidate medical imaging tests (CS), a reference medical imaging test (RS) is selected based on a comparison between the target body region (408) and multiple candidate body regions (720) (S40). Each of the plurality of candidate body regions (720) corresponds to one of the plurality of candidate medical imaging tests (CS), each of the candidate medical imaging tests (CS) includes a plurality of candidate medical imaging images (CMIS) of the patient at a second time point. From the multiple candidate medical image groups (CMIS) of the reference medical imaging examination (RS), a reference medical image group (RMIS) is selected based on the degree of comparability with the target medical image group (TMIS) (S50). To determine the degree of comparability, For the aforementioned group of medical images (TMIS), a first feature vector (FV) is obtained (BR-5). For each of the candidate medical images (CMIS) included in the aforementioned reference medical image examination (RS), a second feature vector (SV) is obtained (BR-6). At least the following steps are performed: determine a comparability metric (BR-7) that indicates the degree of comparability between the first feature vector (FV) and the second feature vector (SV) of the candidate medical image group (CMIS), The first and second feature vectors (FV, SV) include indices of the imaging modality of the subject and candidate medical image groups (TMIS, CMIS), The comparability metric includes an imaging modality correlation score between the imaging modality of the target medical image group (TMIS) and the imaging modality of each of the candidate medical image groups (CMIS). Registration is performed between the target medical image group (TMIS) and the reference medical image group (RMIS) (S60), The system (1) is configured to generate display data (S70) to display the rendering of the reference medical image group (RMIS) on the interface unit (10) based on the registration.

15. A computer program that includes program elements, The program element is a computer program that, when executed by a computing unit (30) of a system (1) that supports the evaluation of a group of medical images, causes the computing unit (30) to perform the steps according to the method described in claim 1 or 2.

16. A computer-readable medium on which program elements are recorded, The program element is a computer-readable medium that is readable and executable by the computing unit (30) of a system (1) that supports the evaluation of a group of medical images, such that when executed by the computing unit (30), the steps of the method according to claim 1 or 2 are performed.

Images