Computer-implemented method for determining at least one feature of a predetermined pixel of an image
Patent Information
- Application Number
- CN202610414713.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2025-04-02
- Filing Date
- 2026-03-31
- Publication Date
- 2026-10-16
Smart Images
Figure CN122928694A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a computer-implemented method for determining at least one feature of predetermined pixels in an image. In a preferred example, the image is a medical image. Furthermore, this invention relates to a method for determining at least one feature of predetermined pixels in an image. Additionally, this invention relates to a data processing system and a computer program product for performing such a computer-implemented method. Background Technology
[0002] A treatment or examination may rely on accurate information obtained from an image to locate a target and / or key anatomical structure within that image. This is, for example, in the case of particularly high-precision radiotherapy or image-guided interventions. An image comprises multiple pixels, allowing that accurate information to be determined individually for each pixel. However, this determination should be rapid for use in intraoperative medical tasks, while simultaneously maintaining accuracy, meaning a balance must be struck between the accuracy and speed of the determination. Summary of the Invention
[0003] The object of the present invention is to provide at least one piece of information about at least one predetermined pixel of an image in a fast and highly accurate manner.
[0004] The independent claims address this objective.
[0005] A first aspect of the invention relates to a computer-implemented method for determining at least one feature of a predetermined pixel in an image. In a preferred example, the image is a medical image. The medical image can be acquired by magnetic resonance imaging (MRI), computed tomography (CT), positron emission tomography (PET), ultrasound (US) imaging, or another medical imaging technique. A data processing system can execute the computer-implemented method. The data processing system may be included by a medical imaging device, and / or may be an external data processing system that can receive data, particularly first image information, from the medical imaging device to execute the computer-implemented method.
[0006] A computer-implemented method includes receiving first image information. The first image information includes at least one image, particularly at least one medical image. The at least one image includes predetermined pixels, and at least one feature is determined for the predetermined pixels. In one example, the first image information may include a single image. An image may be referred to as the first image. In another example, the first image information may include a pair of images or more than two images, wherein all images included by the first image information at least partially describe the same object. If the first image information includes more than one image, all of these images may have the same resolution.
[0007] A computer-implemented method includes receiving at least one second image information. The second image information corresponds to first image information but has a different resolution. This means that the second image information includes at least one other image. This other image differs in resolution from at least one image included by the first image information. Apart from resolution, the images match each other, meaning they describe the same object only at different resolutions. In a preferred example, multiple second image information pieces are received, each having a different resolution compared to the first image information. The resolution of the second image information can be smaller than the resolution of the first image information. In this example, the first image information is downsampled or reduced to the second image information in terms of image resolution. This means that at least one image included by the second image information is coarser than at least one image included by the first image information. If the second image information includes more than one image, all of these images can have the same resolution. Alternatively or additionally, at least one second image information can have a higher resolution than the first image information. The computer-implemented method may include determining at least one second image information based on the first image information.
[0008] The computer-implemented method comprises three steps, each performed individually for a first image and at least one second image. The first step involves determining multiple features by applying a first trained machine learning model to each image. The second step involves selecting pixel features to be assigned to predetermined pixels by applying a pixel feature selection algorithm to the multiple features determined for each image. The third step involves determining interpolated features for each image by applying an interpolation algorithm to the selected pixel features. At the end of these three steps, interpolated features for the first image and interpolated features for at least one second image are provided.
[0009] In other words, the first trained machine learning model is used to determine features, which can be understood as information describing details or specific information of the first image information or at least one second image information. When the various image information comprises more than one image, features are determined considering these multiple images, particularly without distinguishing between the individual images included in the various image information. The first trained machine learning model may include at least one neural network module, particularly at least one encoder for feature extraction.
[0010] The selected pixel features are features from a plurality of defined features, chosen because they are assigned to a predetermined pixel, for which at least one feature is determined. The pixel feature selection algorithm may include at least one predetermined rule, meaning it can be a rule-based algorithm. Alternatively or additionally, the pixel feature selection algorithm may be another trained machine learning model different from the first trained machine learning model. It may include at least one neural network or at least a portion of a neural network. In this example, the pixel feature selection algorithm is trained during training to identify features assigned to the predetermined pixels.
[0011] The interpolation algorithm may include at least one predetermined rule, and therefore may be a rule-based algorithm. The interpolation algorithm is configured to interpolate between selected features. Here, it is assumed that multiple features are selected and assigned to predetermined pixels. However, not all of the multiple selected pixel features are subsequently considered; instead, intermediate values between at least some of the selected features are calculated and treated as interpolated features.
[0012] The computer-implemented method includes determining pixel information. The pixel information describes at least one feature of a predetermined pixel. The pixel information is determined by applying a second trained machine learning model to the determined interpolation features. It is applied to all determined interpolation features. The second trained machine learning model differs from a first trained machine learning model. The second trained machine learning model may include at least one neural network that receives interpolation features and determines the pixel information based on the received interpolation features.
[0013] The described computer-implemented method can be applied rapidly and reliably determines at least one feature for a predetermined pixel by using combinations of the different machine learning models and algorithms described. In particular, the computer-implemented method can be executed simultaneously for multiple pixels of the first image information, resulting in the rapid determination of pixel information for various pixels of the medical image included in the first image information.
[0014] One embodiment includes applying a first trained machine learning model to result in the separate determination of hash tables for first image information and at least one second image information. The hash table includes multiple features assigned to each image information. The hash table can be understood as a multi-resolution hash grid comprising multiple features. This means that the first trained machine learning model determines and provides multiple features in the multi-resolution hash table. Therefore, the pixel feature selection algorithm can be a hash function configured to select a single entry included in the hash table and thus a feature. In the context of this invention, a hash table can be a data structure implementing an associative array. An entry in this array can be a single feature of a determined set of features. Generally, hash tables are efficient compared to other data structures, such as lookup table structures. Therefore, using a hash structure to organize a determined set of features is particularly advantageous, making the features easily accessible to the pixel feature selection algorithm.
[0015] According to another embodiment, the first trained machine learning model includes multiple resolution-specific encoder networks. Encoder networks specific only to the resolution of each image information are applied to that image information. This means that there can be a first encoder network specific to the resolution of a first image information, and at least one second encoder network specific to the resolution of at least one second image information. The encoder networks are pre-trained specifically for the resolution of the image information to which they are applied. This explains how, when applying the first trained machine learning model, multiple features can be determined individually and precisely for the first image information and at least one second image information.
[0016] Another embodiment includes applying a pixel feature selection algorithm to determine a bounding box, wherein predetermined pixels are located in various image information. If the various image information includes multiple separate images, the bounding box can be determined for the predetermined pixels in a first image among the multiple images of the various image information. In particular, the bounding box can be determined for the predetermined pixels only in the first image among the multiple images of the various image information.
[0017] Applying a pixel feature selection algorithm involves selecting features assigned to at least one corner of a defined bounding box. The features assigned to at least one corner are the selected pixel features. In other words, the bounding box is arranged or positioned around a pixel of interest, where the pixel of interest is a predetermined pixel, and features determined by a first trained machine learning model for the pixel positions of at least one corner of the bounding box are considered pixel features. In a preferred example, features assigned to all corners of the bounding box are selected as pixel features. These corners are described, for example, by coordinates, which can be used to find the feature assigned to at least one corner among all determined features. If a feature is assigned to multiple coordinates and these coordinates describe more than one corner, then the feature can be assigned to multiple corners. In this case, the pixel feature selection algorithm can select features that trace back to multiple corners instead of just one. The described pixel feature selection is particularly fast and requires very little computation time because only the corners of the bounding box are considered.
[0018] According to another embodiment, the determined bounding boxes are resolution-specific. This means that for first image information and at least one second image information, different bounding boxes that can be different from each other can be determined. The smaller the resolution, the larger or smaller the bounding box can be. Therefore, the bounding box takes into account the size of the pixels of the image information for which it selects pixel features, in order to provide a particularly customized selection of pixel features.
[0019] In the alternative example, the bounding box is the same for all considered image information. Determining the bounding box may include: the size of the bounding box is predetermined, particularly dependent on the resolution, such that only the position of the bounding box and therefore the coordinates of its corners depend on the individual image information, and are thus determined when applying the pixel feature selection algorithm.
[0020] Another embodiment includes an interpolation algorithm interpolating between selected pixel features assigned to at least one corner of the bounding box. For example, if features assigned to all corners of the bounding box are selected, taking into account resolution and the location of predetermined pixels, these features are not all fed individually to the second trained machine learning model, but are first combined through interpolation. For example, this may lead to determining interpolated features of a predetermined data type, such as integers instead of decimals. This can be particularly useful if the second trained machine learning model requires a specific data type and can result in appropriate preprocessing of the selected pixel features.
[0021] Another embodiment includes determining a feature vector comprising the determined interpolation features after determining the interpolation features. A second trained machine learning model is applied to the feature vector to determine pixel information. In a preferred example, all interpolation features are combined to determine a single feature vector. This means that the interpolated features are not randomly fed into the second machine learning module, but are first combined to form a feature vector. The feature vector is a multi-resolution vector comprising multiple interpolation features, specifically all the determined interpolation features. The feature vector can be understood as a data structure that combines interpolation features used for first image information but also for at least one second image information, such that the second trained machine learning model receives a specific input and can identify which interpolation features were determined for which image information. The positions within the feature vector can depend on the source of the interpolation features, meaning they depend on whether they were determined based on first image information or based on at least one second image information. This can accelerate the computation performed by the second trained machine learning model, as it receives the interpolation features in a predetermined, specifically optimized structure.
[0022] According to a preferred embodiment, the second trained machine learning model is a multilayer perceptron. A multilayer perceptron is a type of feedforward neural network. Multilayer perceptrons are organized in layers and can distinguish non-linearly separable data. Multilayer perceptrons are easily trained neural networks used on images, allowing computer-implemented methods to benefit from the advantages of deep learning techniques.
[0023] Another embodiment includes determining pixel information for a predetermined set of pixels representing a region of interest in at least one image of the first image information, or for all pixels in at least one image of the first image information. Therefore, the predetermined pixels targeted by the computer-implemented method can be pixels in the predetermined pixel set, or pixels in all pixels of the image. Preferably, before executing the computer-implemented method, it is determined which pixels require pixel information. This can be determined based on user input, or it can be determined at least partially automatically, for example, by applying a pixel selection algorithm that can determine the demand for pixel information in the region of interest or for all pixels. The pixel selection algorithm can be a rule-based algorithm and / or can include at least one trained machine learning model. The computer-implemented method can be executed simultaneously for at least a portion of the pixels in the region of interest or for all pixels in at least one image, particularly for the region of interest or for all pixels in at least one image. Alternatively, it can be executed sequentially, individually for each pixel or for groups of pixels. By determining pixel information only for the region of interest, it may be possible to reduce computation time and / or improve the accuracy of the determined pixel information without changing the computation time and / or the computational resources used, compared to determining pixel information for all pixels in at least one image. However, if pixel information is required or desired for all pixels of at least one image, it is still possible to determine the pixel information for all pixels of at least one image.
[0024] According to another embodiment, the first image information includes an image pair. The image pair includes a first image and a second image that are different from each other but at least partially describe the same object. The first image and the second image can be different image types, meaning they can be received from different imaging devices using different imaging techniques. For example, the first image can be a CT image, and the second image can be an MRI image, or vice versa. Other image types are possible. Furthermore, the computer-implemented method includes at least one second image information comprising an image pair, the image pair including another first image describing the first image at a different resolution and another second image describing the second image at a different resolution. This means that at least one second image information also includes an image pair that differs from the image pair of the first image information in terms of resolution.
[0025] Multiple features, pixel features, and interpolation features are each determined for each image pair, meaning for the image pairs included in the first image information and for the image pairs included in at least one second image resolution. The predetermined pixels considered here are predetermined pixels of the first image included in the first image information. A second trained machine learning model performs displacement determination such that the determined pixel information describes the placement between the predetermined pixels of the first image and the corresponding pixels of the second image. This means determining the movement of the predetermined pixels between the first and second images. The corresponding pixels describe the portion of the object in the second image described by the predetermined pixels in the first image.
[0026] Displacement determination can be understood as image registration to estimate the transformation between a first image and a second image. The first and second images can alternatively be referred to as the moving image and the stationary image. When a second trained machine learning model is applied, it produces a registration result that matches the stationary image (meaning the second image) based on the moving image (i.e., the first image). In clinical settings, image registration has varying levels of requirements for accuracy and speed to facilitate different intraoperative tasks, while also adapting to field systems that typically have limited computational resources. These drawbacks are overcome by computer-implemented methods, as they can provide an instance-independent implicit neural representation (INR) framework, which, in preferred examples, incorporates neural graph primitives (NGPs) via efficient multi-resolution hash encoding with inference hash tables.
[0027] In an alternative embodiment, a second trained machine learning model performs segmentation such that the determined pixel information describes an object category. Here, it is not required to consider image pairs as described above for displacement determination. However, in the example, the first image information may include multiple images. The second trained machine learning model determines, for predetermined pixels of at least one image included in the first image information, which object category within a predetermined object category group the predetermined pixels belong to. The predetermined group may distinguish specific organs or other body parts, tissues, background, etc. By performing segmentation, the objects described by the predetermined pixels are identified, thereby determining and providing useful features for further processing of the first image information.
[0028] In this embodiment, the first trained machine learning model may also be segmentation-specific, meaning it can be trained to determine multiple features that are relevant and / or useful for the segmentation task for the second trained machine learning model.
[0029] Alternatively or additionally, for segmentation, a second trained machine learning model performs denoising such that the determined pixel information describes another pixel corresponding to a predetermined pixel but less affected by image noise compared to the predetermined pixel. Denoising means noise reduction, meaning that for the predetermined pixel, unwanted or random variations in pixel values are at least reduced. Image noise can degrade image quality and sharpness, making pixel-by-pixel denoising particularly useful.
[0030] In this embodiment, the first trained machine learning model may be denoising-specific, meaning it can be trained to determine multiple features that are relevant and / or useful for the denoising or noise reduction task for the second trained machine learning model.
[0031] Alternatively or additionally, for segmentation or denoising, a second trained machine learning model performs image inpainting such that determined pixel information describes another pixel corresponding to the reconstruction of predetermined pixels. Image inpainting means reconstructing missing regions in an image and / or reconstructing regions where at least one object may have been removed from the image. Therefore, image inpainting can include image restoration, image manipulation, image relocation, image compositing, and / or image-based rendering. Reconstruction is performed pixel-by-pixel.
[0032] In this embodiment, the first trained machine learning model may be image inpainting-specific, meaning it can be trained to determine multiple features relevant to and / or useful for the image inpainting task for the second trained machine learning model.
[0033] Alternatively or additionally, for segmentation, denoising, or image inpainting, a second trained machine learning model performs image enhancement such that the determined pixel information describes another pixel corresponding to a predetermined pixel but having at least one increased image quality parameter compared to the predetermined pixel. In other words, image enhancement filtering, particularly image super-resolution, can be performed. Image super-resolution refers to the task of enhancing an image resolution from low to high resolution, performed pixel-by-pixel. An image including multiple other pixels based on multiple determined pixel information can allow for particularly precise detection of objects of interest, such as medical targets.
[0034] In this embodiment, the first trained machine learning model may be image enhancement-specific, meaning it can be trained to determine multiple features relevant to and / or useful for the image enhancement task for the second trained machine learning model.
[0035] In another embodiment, at least one image included in the first image information is of a first image type. In an example, this could be a CT image. A second trained machine learning model performs image synthesis from the first image type to a different second image type, such that determined pixel information describes another pixel in the second image type corresponding to a predetermined pixel. This other pixel could be a pixel of another image included in another image information of the second image type, such as an MR image. This means that when a computer-implemented method is applied, at least partially, a pixel-by-pixel transformation of at least one image included in the first image information is performed to another image of the second image type. Image synthesis can at least perform changes in pixel intensity and / or other pixel characteristics depending on the first and second image types. By performing image synthesis, the image type can be changed quickly and reliably if, for example, further processing and / or analysis of the first image information can benefit from the change in image type.
[0036] In this embodiment, the first trained machine learning model may be image synthesis-specific, meaning it can be trained to determine image synthesis task-relevant and / or useful features for the second trained machine learning model.
[0037] According to another embodiment, the second trained machine learning model performs translation determination and / or segmentation and / or image synthesis. Alternatively or additionally, the first trained machine learning model may be specific to translation determination and / or segmentation and / or image synthesis.
[0038] Segmentation, denoising, image inpainting, image enhancement, and / or image synthesis may require at least one input image, meaning that first image information including at least one image and potentially including multiple images is required. The determined pixel information can be a single value, specifically a real-valued scalar value. This scalar value can be mapped to a physical image intensity value, which may be, for example, the Huntsfield unit used in computed tomography.
[0039] Another aspect of the present invention relates to a method for determining at least one feature of predetermined pixels in an image. The method includes acquiring first image information via a medical imaging device and performing the computer-implemented method described above. Furthermore, the method includes outputting the determined pixel information via an output device. The output device may include at least one screen. The medical imaging device may be a magnetic resonance imaging (MRI) system, a computed tomography (CT) system, a positron emission tomography (PET) system, an ultrasound (US) imaging system, and / or another medical imaging system or device.
[0040] Specifically, the present invention may relate to a method for determining the displacement between a first pixel in a first image and a corresponding second pixel in a second image, the second image being different from the first image but at least partially describing the same object. The method includes: acquiring a first image using a first medical imaging device; acquiring a second image using the first medical imaging device at different time points or using a second medical imaging device different from the first medical imaging device; and performing the above-described computer-implemented method while taking into account first image information including the acquired first and second images. Furthermore, the method includes outputting determined pixel information describing the displacement between the first pixel and the second pixel via an output device.
[0041] Another aspect of the invention relates to a data processing system configured to perform the computer-implemented method described above. The data processing system may be included in a medical imaging device.
[0042] Unless otherwise stated, all steps of the computer-implemented method can be performed by a data processing system, which includes at least one data processing device. In particular, at least one data processing device is configured or adapted to perform the steps of the computer-implemented method. For this purpose, at least one data processing device may, for example, store a computer program comprising instructions that, when executed by at least one data processing device, cause at least one data processing device to perform the computer-implemented method. The expressions "data processing system" and "at least one data processing device" are used interchangeably herein and hereinafter. This also applies to corresponding expressions derived therefrom.
[0043] In cases where at least one data processing device comprises two or more data processing devices, certain steps performed by at least one data processing device can also be understood as different data processing devices performing different steps or different parts of steps. Specifically, it is not required that each data processing device fully executes the steps. In other words, the execution of steps can be distributed among two or more data processing devices.
[0044] From each implementation of the computer-implemented method, various implementations of the method that are not purely computer-implemented are obtained by including steps of acquiring and / or re-acquiring first image information.
[0045] Another aspect of the present invention relates to a medical imaging apparatus, which includes a scanner and a data processing system. The medical imaging apparatus is configured to perform the above-described method.
[0046] According to a further aspect of the present invention, a computer program comprising instructions is provided. When executed by a data processing system, the instructions cause the data processing system to perform a computer-implemented method according to the present invention.
[0047] Instructions may be provided, for example, as program code. Program code may be provided, for example, as binary code or assembler and / or as source code of a programming language (e.g., C) and / or as a program script, such as Python.
[0048] According to another aspect of the invention, a computer-readable storage medium is provided, in particular, a tangible and / or non-transitory computer-readable storage medium storing a computer program provided according to the invention.
[0049] Computer programs and computer-readable storage media are individual computer program products that include instructions.
[0050] Receiving information within the scope of this invention can, for example, involve receiving or obtaining information, specifically, receiving or obtaining information from a sending entity by a data processing system, or reading information from a data storage device, or receiving an information stream containing information, or extracting information from an information stream. Specifically, wired or wireless data transmission can be used for this purpose. Specifically, information transmission can occur between the hardware and / or software interface of the sending entity and the hardware and / or software interface of the data processing system. Information within the scope of this invention can be understood as, or may include, data or portions of data.
[0051] Generally speaking, trained machine learning models can mimic human cognitive functions and other aspects of human thought. Specifically, by training on training data, machine learning models can adapt to new environments and detect and reason about patterns. Another term used for trained machine learning models is "trained function."
[0052] Typically, the parameters of a machine learning model can be adjusted or updated through training. Specifically, supervised training, semi-supervised training, unsupervised training, reinforcement learning, and / or active learning can be used. Furthermore, representation learning, also known as feature learning, can be used. In particular, the parameters of a machine learning model can be iteratively adjusted through multiple training steps. Specifically, during training, a loss function, also known as a cost function, can be minimized. Specifically, the backpropagation algorithm can be used within the training of artificial neural networks. Artificial neural networks are neural networks.
[0053] Specifically, the trained machine learning model may include artificial neural networks, support vector machines, decision trees, and / or Bayesian networks, and / or the trained machine learning model may be based on k-means clustering, Q-learning, genetic algorithms, and / or association rules. Specifically, the neural network may be or include deep neural networks, convolutional neural networks, or convolutional deep neural networks. Furthermore, the neural network may be an adversarial network, a deep adversarial network, and / or a generative adversarial network (GAN).
[0054] Regardless of the use of grammatical terms, individuals with male or female identities are included in this term.
[0055] Further features of the invention will be apparent from the claims, drawings, and description of the drawings. Features and combinations of features mentioned above in the description, as well as features and combinations of features mentioned below in the description of the drawings and / or shown in the drawings, may be included in the invention not only in the aforementioned combinations but also in other combinations. In particular, the invention may also include embodiments and combinations of features that do not have all the features of the originally claimed claims. Furthermore, the invention may also include embodiments and combinations of features that go beyond or deviate from the combinations of features set forth in the claims.
[0056] The invention will be explained in detail below with reference to specific exemplary embodiments and various schematic diagrams. In the drawings, the same or functionally identical elements may be denoted by the same reference numerals. Descriptions of the same or functionally identical elements are not necessarily repeated in different drawings. Attached Figure Description
[0057] The attached diagram shows:
[0058] Figure 1 Schematic representation of a medical imaging system
[0059] Figure 2 A schematic representation of a computer-implemented method for determining at least one feature of a predetermined pixel.
[0060] Figure 3 Determine in application based on Figure 2 When considering the method, a schematic representation of which predetermined pixel step should be taken into account, and
[0061] Figure 4 A schematic representation of a multilayer perceptron. Detailed Implementation
[0062] Figure 1 An exemplary embodiment of a medical imaging system is schematically illustrated, comprising a first medical imaging device 1, such as a magnetic resonance imaging (MRI) system. The first medical imaging device 1 includes a medical imaging scanner 2. The medical imaging system, particularly the first medical imaging device 1, includes a data processing system 3 for controlling the scanner 2. Furthermore, the medical imaging system may include an output device 4 coupled to the data processing system 3 and / or the scanner 2. The medical imaging system may include a storage unit (not shown) storing a computer program according to the invention. The data processing system 3 can execute according to... Figure 2 The computer implementation method.
[0063] In this example, the medical imaging system may include a second medical imaging device 5, which may include a scanner 6. The second medical imaging device 5 may use a data processing system 3 to control the scanner 6. The second medical imaging device 5 may be another MRI system, or, for example, a computed tomography (CT) system. MRI and CT systems are merely examples. Other types of medical imaging devices 1 and 5 are possible. However, the first medical imaging device 1 and the second medical imaging device 5 may be configured to acquire images of different or the same image types.
[0064] Figure 2 A computer-implemented method for determining at least one feature of a predetermined pixel 22 of an image is shown. The computer-implemented method may include receiving first image information 10. First image information 10 may include at least one image, such as an image having the predetermined pixel 22. In the depicted example, first image information 10 includes an image pair comprising a first image 11 and a second image 12. First image 11 and second image 12 are different from each other but at least partially describe the same object. First image 11 and second image 12 may have been acquired by the same medical imaging devices 1, 5 but at different times, or by different medical imaging devices 1, 5. Alternatively, first image information 10 may include one image or more than two images. Hereinafter, it is assumed that first image information 10 includes an image pair.
[0065] The computer-implemented method includes receiving at least one second image information 13, which corresponds to first image information 10 but has a different resolution compared to the first image information 10. In the depicted example, the second image information 13 includes image pairs corresponding to the image pairs included in the first image information 10 but having a different resolution than the image pairs included in the first image information 10. Here, the resolution reduction means that the first image information 10 is downsampled to determine the second image information 13. This means that at least one second image information 13 includes image pairs, which include another first image 14 describing the first image 11 at a different resolution compared to the first image 11, and another second image 15 describing the second image 12 at a different resolution. All images included in the second image information 13 may have the same resolution or may have different resolutions. If the first image information 10 includes only one image, then at least one second image information 13 includes only one image having a different resolution compared to the image included in the first image information 10.
[0066] The computer-implemented method includes three steps, each performed for a first image information 10 and at least one second image information 13. The first step may include determining multiple features 18 by applying a first trained machine learning model 16 to each image information 10, 13. In a preferred example, the first trained machine learning model 16 separately determines a hash table 19 for each image information 10 and at least one second image information 13, the hash table 19 including multiple features 18 assigned to each image information 10, 13. This can result in multiple hash tables T. i , where i is an integer between 0 and L. The first trained machine learning model 16 may include multiple resolution-specific encoder networks 17. Encoder networks 17 specific only to the resolution of each image information 10, 13 are applied to that image information 10, 13. Here, encoder networks 17 are distinguished as g0, g1, etc., up to g L , where L is the number of distinct second image information 13. Here L=2, because two second image information 13 are determined and considered as an example. Alternatively, more or fewer second image information 13 can be determined and considered.
[0067] The second of these three steps may include selecting pixel features 20 assigned to predetermined pixels 22 of each image information 10, 13 by applying pixel feature selection algorithm 21 to a plurality of features 18 determined for each image information 10, 13. The pixel feature selection algorithm can be understood as a rule-based algorithm and / or another trained machine learning model.
[0068] The pixel feature selection algorithm 21 may include determining a bounding box 23, in which predetermined pixels 22 are located in the respective image information 10, 13. The predetermined pixels 22 are here marked as... The pixel feature selection algorithm 21 may include selecting feature 18 as selected pixel feature 20, and feature 18 is assigned to at least one corner point 24 of bounding box 23. This is schematically depicted here for the three considered image information 10, 13. The determined bounding box 23 may be resolution-specific, as depicted here. The pixel feature selection algorithm 21 may include separate sub-algorithms depending on the resolution of image information 10, 13. The sub-algorithms are depicted here as h0, h1, etc., up to h L They can be understood as hash functions.
[0069] The third of these three steps may include determining interpolation features 25 for each of the image information 10, 13 by applying interpolation algorithm 26 to selected pixel features 20. Interpolation algorithm 26 may interpolate between selected pixel features 20 assigned to the corner points 24 of the bounding box 23. After determining the interpolation features 25, the method may include determining a feature vector 27, which includes all determined interpolation features 25. Feature vector 27 is depicted herein as... .
[0070] The method may include applying a second trained machine learning model 28 to the interpolation features 25, and particularly to the feature vector 27. The second trained machine learning model 28 may be a multilayer perceptron, which is described herein as... Multilayer perceptrons in Figure 4 It is described in more detail.
[0071] The second trained machine learning model 28 can determine pixel information 29, which describes at least one feature of the predetermined pixel 22. In the example depicted here, pixel information 29 describes a displacement 30 between the predetermined pixel 22 of the first image 11 and the corresponding pixel of the second image 12. The displacement 30 is depicted here as... Alternatively or additionally, the second trained machine learning model 28 can perform segmentation such that the determined pixel information 29 describes the object category 31 of the predetermined pixel 22. Alternatively or additionally, the second trained machine learning model 28 can perform image synthesis 32 such that the determined pixel information 29 describes another pixel corresponding to the predetermined pixel 22, however, this other pixel belongs to a second image type 34, while the predetermined pixel 22 belongs to a first image type 33. The first image type 33 and the second image type 34 are different from each other. In the example, the first image type 33 could be MRI data, and the second image type 34 could be CT data. Other image types 33 and 34 are possible.
[0072] Alternatively or additionally, the second trained machine learning model 28 can perform denoising, such that the determined pixel information 29 describes another pixel corresponding to the predetermined pixel 22 but less affected by image noise compared to the predetermined pixel 22. Alternatively or additionally, the second trained machine learning model 28 can perform image inpainting, such that the determined pixel information 29 describes another pixel reconstructed corresponding to the predetermined pixel 22. Alternatively or additionally, the second trained machine learning model 28 can perform image enhancement, such that the determined pixel information 29 describes another pixel corresponding to the predetermined pixel 22 but having at least one increased image quality parameter compared to the predetermined pixel 22.
[0073] The computer-implemented method can be understood as a method comprising three consecutive stages or blocks A, B, and C. These blocks are also labeled here.
[0074] In other words, the computer-implemented method includes the following. The first stage A of the computer-implemented method includes receiving paired images (first image information 10) as input and their respective downsampled images (second image information 13), in a configurable... L At high resolution as well as (For moving and stationary images respectively). A set of trainable encoder networks (here, encoder network 17). With each trainable parameter ,Will and Encoding as a multi-resolution hash table set 19 means Resolution level i Hash table at the location T i It has a configurable size ,storage eigenvectors Each vector has Dimensions.
[0075] Phase B via a hash table Multi-resolution hash grid execution query coordinates exist The position encoding in the hash table consists of stage A and a hash function. (This means that pixel feature selection algorithm 21) provides that the hash function can be predefined or trainable and has parameters. .set up A multi-resolution hash grid set, predefined by configurable resolutions, with configurable resolution ratios... Coarser. In each middle, Surrounding Corner point 24 will be marked as ,in The dimension of the input image (for two-dimensional images) 2, for 3D images (3), and . Then by Hash to T i The stored feature vector is extracted from the index. As shown in equation (1) below. Note that in Less than Given the total number of corner points, hash collisions will be implicitly handled through gradient updates of trainable parameters. It was then used for linear interpolation (represented as) To calculate resolution levels i Place The feature vector at that point. The results from each level are concatenated (represented as...). ), thus generating ,Right now The multi-resolution latent feature vector at the location is shown in equation (2): (1) (2)
[0076] The final stage C is composed of a multilayer perceptron ( It consists of ( ) and has trainable parameters As an implicit neural representation of the estimated displacement vector field (DVF), it will Decoded as The displacement vector at that point, i.e. .
[0077] In the example, training the proposed framework means that at least one of the first trained machine learning model 16 and the second trained machine learning model 28 can be trained in a supervised, unsupervised, weakly supervised, or adversarial manner using large-scale data, which can be real, simulated, unimodal, multimodal, between-patient, and / or within-patient. However, if trained in an instance-independent manner on a large-scale training dataset, each model needs to have a sufficient number of parameters to allow the model to learn possible deformation spaces. A model means... and (and if trainable,) They can be jointly trained in an end-to-end manner. The loss function is computed with respect to, for example, the output displacement vector, and the gradient is propagated to... and (and if it exists,) Alternatively or additionally, by utilizing implicit neural representations that are continuous functions and therefore... The dimension is determined solely by the registered coordinates of the query. Given the established fact, advanced training strategies can be used, allowing the queried registration coordinates to be sampled arbitrarily. This means that during training, Both inference and loss calculation can be performed at different resolutions than the input image, can be performed locally, and can be sampled strategically, providing options for super-resolution, region localization, and multi-resolution pyramid training.
[0078] Figure 3A flowchart is shown illustrating how to determine which predetermined pixels 22 can be used to determine pixel information 29. For example, pixel information 29 can be determined for each pixel in a predetermined pixel set of the region of interest or for all pixels of the first image information 10.
[0079] Once fully trained, a computer-based approach can be used. Starting from step 40, given a pair of input images... and In step 42, extract the coordinate set to be registered for the given pair. . It can cover the entire moving image (first image information 10) or only the region of interest, sampled at any desired resolution. Utilizing , and In step 41, the corresponding displacement vector is estimated. This produces a one-time registration result. These results are determined by performing stages B and C as described above.
[0080] Based on these results, a decision can be made in step 43 as to whether it is worthwhile to spend more time on potential improvements in accuracy. This can be decided by the user and / or with the aid of other registration evaluation algorithms, such as the registration evaluation algorithm described in "A neural network approach for fast, automated quantification of DIR performance" published by Neylon, J. et al. (2017, Medical Physics, Vol. 44). If the initial results are considered ideal, then one-time registration will be used in step 44. Otherwise, a subsequent instance optimization phase can be initiated in step 45, where the end user has additional options to further decide whether to make improvements in specific regions, which can speed up the optimization process. If no specific region is given in step 47, a choice can be made between higher accuracy or faster speed in step 48. Assuming that during instance optimization, These are the main parameters that receive gradient updates in step 50, and and In step 48, the algorithm can be optionally optimized for better accuracy or frozen to reduce computational cost. Therefore, to improve speed, step 49 is not performed; instead, the process proceeds directly from step 48 to step 50. In step 50... Updated.
[0081] The instance optimization in step 47 can follow existing methods, namely, using implicit neural representations in deformable registration, such as the publication "Implicit Neural Representations for DeformableImage Registration" by Wolterink, JM et al. (2022, Proceedings of Machine Learning Research, Vol. 172). In this publication, the loss functions (such as similarity and dice) and regularization terms (such as Jacobi norm and inverse consistency) are related to... The calculation is performed, and the gradient is propagated to... (optional) and And it is updated by optimizers such as Adam and stochastic gradient descent (SGD).
[0082] Alternatively or additionally, the computer-implemented method can be applied to image segmentation and synthesis tasks by simply modifying the number of input channels of the encoding network and then changing the output of the second trained machine learning model 28 from the displacement vector to the segmentation label and pixel intensity, respectively. The training loss and instance optimization loss are also adjusted according to the task.
[0083] Figure 4 An embodiment of a neural network 800 is shown, which is designed, for example, as a multilayer perceptron. Therefore, the neural network 800 can be an example of a second trained machine learning model 28. The neural network 800 includes nodes 820, ..., 832 and edges 840, ..., 842, where each edge 840, ..., 842 is a directed connection from a first node 820, ..., 832 to a second node 820, ..., 832. Typically, the first nodes 820, ..., 832 and the second nodes 820, ..., 832 are different nodes 820, ..., 832. However, it is also possible that the first nodes 820, ..., 832 and the second nodes 820, ..., 832 are the same. For example, in... Figure 4 In the diagram, edge 840 is a directed connection from node 820 to node 823, and edge 842 is a directed connection from node 830 to node 832. The edges 840, ..., 842 from the first node 820, ..., 832 to the second node 820, ..., 832 are also called the incoming edges for the second node 820, ..., 832 and the outgoing edges for the first node 820, ..., 832.
[0084] In this example, nodes 820, ..., 832 of the artificial neural network 800 can be arranged in layers 810, ..., 813, where each layer may include an inherent order introduced by edges 840, ..., 842 between nodes 820, ..., 832. Specifically, edges 840, ..., 842 may exist only between adjacent node layers. In the example shown, there is an input layer 810, consisting only of nodes 820, ..., 822 with no incoming edges; an output layer 813, consisting only of nodes 831, 832 with no outgoing edges; and hidden layers 811, 812 between the input layer 810 and the output layer 813. Typically, the number of hidden layers 811, 812 can be chosen arbitrarily. In an MLP, this number is at least one. The number of nodes 820, ..., 822 in the input layer 810 is usually related to the number of input values of the artificial neural network 800, and the number of nodes 831, 832 in the output layer 813 is usually related to the number of output values of the artificial neural network 800.
[0085] Specifically, each node 820, ..., 832 of the artificial neural network 800 can be assigned a real number as a value. Here, x (n) i This represents the value of the i-th node 820, ..., 832 in the nth layer 810, ..., 813. The values of nodes 820, ..., 822 in the input layer 810 are equal to the input values of the artificial neural network 800. The values of nodes 831, 832 in the output layer 813 are equal to the output values of the artificial neural network 800. Furthermore, each edge 840, ..., 842 may include a weight, which is a real number. Specifically, the weight is a real number within the interval [-1, 1] or the interval [0, 1]. Here, w... (m,n) i,j This represents the weight of the edge between the i-th node (820, ..., 832) of layer m (810, ..., 813) and the j-th node (820, ..., 832) of layer n (810, ..., 813). Furthermore, for the weight w... (m ,n+1) i,j Define the abbreviation w (n) i,j Specifically, in order to calculate the output value of neural network 800, the input value is propagated through neural network 800. Specifically, the values of nodes 820, ..., 832 of layer n+1 810, ..., 813 can be calculated based on the values of nodes 820, ..., 832 of layer n 810, ..., 813 using the following formula:
[0086] Here, the function f is called the transfer function or activation function. Known transfer functions are step functions, sigmoid functions, such as logistic functions, generalized logistic functions, hyperbolic tangent functions, arctangent functions, error functions, smoothed step functions, or rectified functions. Transfer functions are used, for example, for normalization purposes. Specifically, values are propagated layer by layer through neural network 800, where the values of input layer 810 are given by the inputs of neural network 800, where the values of the first hidden layer 811 can be computed based on the values of input layer 810 of neural network 800, where the values of the second hidden layer 812 can be computed based on the values of the first hidden layer 811, and so on.
[0087] To set edge w (m,n) i,j The value of must be obtained by training the neural network 800 using training data. Specifically, the training data includes the training input data and the training output data (denoted as t). i For the training step, the neural network 800 is applied to the training input data to generate computed output data. Specifically, the training data and the computed output data include multiple values, the number of which is equal to the number of nodes in the output layer. Specifically, the weights within the neural network 800 are recursively adjusted using a comparison between the computed output data and the training data (backpropagation algorithm). Specifically, the weights are changed according to the following formula: Where γ is a predefined learning rate, and δ is a numerical value. (n) j It can be based on δ (n+1) j The recursive calculation is as follows: If the (n+1)th layer is not an output layer 813, and If the (n+1)th layer is an output layer 813, where f' is the first derivative of the activation function, and t (n+1) j It is the comparison training value of the j-th node of the output layer 813.
[0088] In summary, the computer-implemented method involves an algorithm that can be called HEMI-INR, which is a framework for deformable image registration using instance-independent implicit neural representations (INR) via hash encoding with multi-resolution reasoning (HEMI).
[0089] HEMI-INR has at least the following advantages: 1.) Improved encoding of image features, specifically, a high spatial frequency structure compared to existing INR-independent methods. This is possible because HEMI-INR: - This is achieved using a multi-resolution coding approach, which manipulates images at multiple resolution levels through multiple coding networks. In existing instance-agnostic methods, such as NePhi (described in the 2023 publication "NePhi: Neural Deformation Fields for Approximately Diffeomorphic Medical Image Registration" by Tian, L. et al.), low-resolution registration is performed via a separate frame and serves as a priori for high-resolution INR, without extracting additional latent features from the low-resolution image. - By encoding features via hash table 19, it exhibits superior performance in representing high-frequency content compared to the dense latent spatial feature grid proposed in existing instance-independent INR. 2.) By using a multi-resolution hash grid, memory usage (logarithmically linear with resolution) and computation time are reduced: - This avoids voxel-by-voxel convolutional decoding, which would consume a lot of memory due to intermediate outputs and gradients. - Significantly reduces the number of floating-point operations and memory accesses. - Allows the use of smaller multilayer perceptrons to decode features, which is faster and consumes less memory. 3.) This leverages the efficient and flexible training options inherent in the continuous nature of INR, which has not been fully utilized in previous instance-independent INR methods. This is achieved through strategically sampling and registering query coordinates. This disclosure can achieve the following objectives: - By sampling at a resolution higher than the input image Super-resolution training is performed. This enables the network to infer high-frequency features from low-resolution images, thus saving memory in the encoding network. - By sampling more frequently from organs of interest typically used in registration tasks Perform region-oriented training. This will effectively train the network to focus on anatomically important features. - By sparsely sampling at the start of training Furthermore, its density is gradually increased to effectively capture features from all frequencies, enabling multi-resolution pyramid training. 4.) Deeply customizing the balance between performance and computational cost allows HEMI-INR to be used in different clinical applications without redesign and retraining, which have different priorities in terms of speed and accuracy, and different clinical systems with varying amounts of available computational resources. While previously proposed INR-based methods have not addressed this issue, according to Figure 3 The method allows for detailed control over accuracy-speed balance in the following ways: - Strategic extraction of registration coordinates. Depending on the clinical task, not all voxels in a moving image require the same registration accuracy, such as air and patient bed compared to prostate and bladder. Flexible options for selecting registration coordinates provide the option to trade accuracy for faster inference and less memory consumption in task-independent regions while still achieving the desired results in the region of interest. - Flexibility in instance optimization objectives. Not all regions following a single registration require the same level of additional improvement from instance optimization. By excluding sufficiently good and / or less important registration regions for a given clinical task, the approach proposed in this disclosure can achieve faster instance optimization in other regions that truly require improvement. - Flexibility in instance optimization parameters. While instance optimization of all parameters in HEMI-INR provides the ability to achieve the most accurate results, this option may have diminishing returns or be limited by available computational resources. The framework provided in this disclosure, which allows end users to purposefully control the balance between accuracy and speed during instance optimization to suit clinical tasks and available clinical systems, offers the option to select the parameters that receive gradient updates.
Claims
1. A computer-implemented method for determining at least one feature of a predetermined pixel (22) of an image, wherein the computer-implemented method comprises: Receive first image information (10), the first image information (10) includes at least one image, the at least one image includes the predetermined pixel (22); Receive at least one second image information (13), the at least one second image information (13) corresponding to the first image information (10) but having a different resolution than the first image information (10); Individually for the first image information (10) and the at least one second image information (13): Multiple features (18) are determined by applying the first trained machine learning model (16) to the respective image information (10, 13). The pixel features (20) assigned to the predetermined pixel (22) are selected by applying the pixel feature selection algorithm (21) to the plurality of features (18) determined for the respective image information (10, 13). Interpolation features (25) for each of the image information (10, 13) are determined by applying an interpolation algorithm (26) to the selected pixel features (20). Pixel information (29) describing at least one feature of the predetermined pixel (22) is determined by applying a second trained machine learning model (28) to the determined interpolation features (25).
2. The computer-implemented method of claim 1, wherein applying the first trained machine learning model (16) results in the separate determination of a hash table (19) for the first image information (10) and the at least one second image information (13), the hash table (19) including the plurality of features (18) assigned to the respective image information (10, 13).
3. The computer-implemented method according to any one of the preceding claims, wherein the first trained machine learning model (16) comprises a plurality of resolution-specific encoder networks (17), wherein the encoder network (17) specific to the resolution of the respective image information (10, 13) is applied to the image information (10, 13).
4. The computer-implemented method according to any one of the preceding claims, wherein applying the pixel feature selection algorithm (21) includes determining a bounding box (23) and selecting features of the pixel feature (20) as a selection of at least one corner point (24) assigned to the bounding box (23), wherein the predetermined pixel (22) is located in at least one image of the respective image information (10, 13) within the bounding box (23).
5. The computer-implemented method according to claim 4, wherein the determined bounding box (23) is resolution-specific.
6. The computer-implemented method according to claim 4 or 5, wherein the interpolation algorithm (26) interpolates between selected pixel features (20) of the at least one corner point (24) assigned to the bounding box (23).
7. The computer-implemented method according to any one of the preceding claims, wherein after determining the interpolation feature (25), a feature vector (27) is determined, the feature vector (27) including the determined interpolation feature (25), and the second trained machine learning model (28) is applied to the feature vector (27) to determine the pixel information (29).
8. The computer-implemented method according to any one of the preceding claims, wherein the second trained machine learning model (28) is a multilayer perceptron.
9. The computer-implemented method according to any one of the preceding claims, wherein the pixel information (29) is determined for a predetermined set of pixels of a region of interest in at least one image of the first image information (10) or for all pixels of the at least one image of the first image information (10).
10. A computer-implemented method according to any one of the preceding claims, wherein: The first image information (10) includes an image pair, which includes a first image (11) and a second image (12), the first image (11) and the second image (12) being different from each other but at least partially describing the same object; The at least one second image information (13) includes an image pair, the image pair including another first image (14) describing the first image (11) at the different resolution and another second image (15) describing the second image (12) at the different resolution. The plurality of features (18), the selected pixel feature (20), and the interpolation feature (25) are determined for each image pair, wherein the predetermined pixel (22) is a predetermined pixel (22) of the first image (11); and The second trained machine learning model (28) performs displacement determination such that the determined pixel information (29) describes the displacement (30) between the predetermined pixel (22) of the first image (11) and the corresponding pixel of the second image (12).
11. A computer-implemented method according to any one of the preceding claims, wherein the second trained machine learning model (28) performs: Segmentation, such that the determined pixel information (29) describes the object category (31) for the predetermined pixel (22); and / or Denoising such that the determined pixel information (29) describes another pixel corresponding to the predetermined pixel (22) but less affected by image noise compared to the predetermined pixel (22); and / or Image inpainting, such that the determined pixel information (29) describes the reconstruction of another pixel corresponding to the predetermined pixel (22); and / or Image enhancement such that the determined pixel information (29) describes another pixel corresponding to the predetermined pixel (22) but having at least one increased image quality parameter compared to the predetermined pixel (22).
12. The computer-implemented method according to any one of the preceding claims, wherein... The at least one image included in the first image information (10) belongs to the first image type (33); and The second trained machine learning model (28) performs image synthesis (32) from the first image type (33) to a different second image type (34), such that the determined pixel information (29) describes another pixel in the second image type (34) corresponding to the predetermined pixel (22).
13. A method for determining at least one feature of a predetermined pixel (22) of an image, wherein the method comprises: First image information (10) is acquired using medical imaging equipment (1, 5); Perform the computer-implemented method according to any one of the preceding claims; The determined pixel information (29) is output through the output device (4).
14. A data processing system (3) configured to perform a computer-implemented method according to any one of claims 1 to 12.
15. A computer program product comprising instructions that, when executed by a data processing system (3), cause the data processing system (3) to perform a computer-implemented method according to any one of claims 1 to 12.