System and procedure for identifying landmarks in an image

Abstract

The invention relates to a system (7a) for identifying landmarks (2) in an image (1) comprising multiple agents (7), each agent (7) being a functional module designed to search for a predefined main landmark (2) and a number of other landmarks (2) in a landmark area (4) around a position of the agent (7), wherein each agent (7) is designed to: a) Defining or receiving its starting position (3), b) Estimating a principal dislocation (5) of its principal landmark (2), c) Estimating election dislocations (6) of other landmarks (2) in the landmark area (4), d) Selecting an updated starting position (3a) determined based on the estimated main dislocation (5) and the choice dislocations (6) estimated by other agents (7) for this main landmark (2), e) Repeat steps b) to d) with the updated starting position (3a) as starting position (3) until a termination condition is met. Furthermore, the invention describes a method, a control unit and a medical imaging system.

Description

[0001] The invention describes a system and a method for identifying landmarks in an image, a control unit for a medical imaging system, and a medical imaging system.

[0002] Landmark localization is crucial for automated image processing. Numerous well-known landmark identification procedures exist, some of which provide automation tools for various steps. One example of a tool that automatically classifies pixels or voxels in an image to determine whether they contain a specific landmark is Adaboost. A module that can be used for automatic landmark detection is ALPHA Technology and Voting. This landmark algorithm consists of several cascaded Adaboost models that classify voxels in an image as containing a particular landmark, ranging from coarse to fine. After Adaboost detects multiple landmarks, a voting model based on the spatial correlation of landmark groups is used to remove outliers and interpolate missing landmarks.Optionally, the adaboost and tuning steps can be repeated after aligning the image to a canonical space, improving the robustness and speed of landmark detection. Training the adaboost and tuning models requires at least 50–100 annotations per landmark. Once the classifier is trained, landmarks can be found by scanning an entire image, which takes about half a second per landmark.

[0003] Another example of an algorithm for automatically finding landmarks is "BodyGPS." This algorithm is based on a self-supervised methodology for estimating normalized landmark positions using a regression network. This method generalizes many types of landmarks without requiring explicit supervised training. However, because there is no supervision, the estimates are not precise and can be off by 10 mm in 90% of cases. Nevertheless, the runtime is fast, at approximately 50 ms with a single search agent.

[0004] Another example of an algorithm for automatically finding landmarks is "MedLSAM". It is based on a method that could localize organs based on predefined templates. It uses similar regression methods by generating a baseline of truth based on relative dislocations in the images.

[0005] On the other hand, the relationships between the target structures are not taken into account. Thus, individual landmarks do not benefit from other positions found.

[0006] Noothout et al., in their paper “Deep learning-based regression and classification for automatic landmark localization in medical images” (IEEE Transactions on Medical Imaging, 39(12), 4011-4022; 2020), demonstrated that recent research has reported on several algorithms for estimating landmark positions that employ a global-local strategy using convolutional neural networks. The global network estimates all landmark positions simultaneously using a combination of a classifier and a regressor. Subsequently, the landmark-specific model refines the proposed position. They report fast processing times with good localization accuracy. However, all areas of the image are still used for the localization estimation, which is unnecessary. Processing times are reduced in GPU hardware, which is a limitation in a production environment.

[0007] In their paper “Multi-scale deep reinforcement learning for real-time 3D-landmark detection in CT scans” (IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(1), 176-189; 2017), Ghesu et al. demonstrate the existence of supervised landmark search algorithms for individual agents. Within their environment, the agent updates its state using a limited number of actions to locate the desired position. A reinforcement algorithm is used to train the agent's strategy.

[0008] However, there is still no way to identify landmarks in a robust, accurate, yet fast manner.

[0009] The object of the present invention is to improve known systems and methods and to provide a system and method for identifying landmarks in an image, a control unit for a medical imaging system, and a medical imaging system itself, in order to overcome the problems described above. In particular, it is an object of the present invention to provide a multi-agent landmark search.

[0010] This task is accomplished by a system according to claim 1, a method according to claim 8, a control unit according to claim 12 and a medical imaging system according to claim 13.

[0011] The following describes a method for searching for landmarks and simultaneously using the voting strategy at each step of the search, which improves both speed and robustness. It should be noted that landmark finding is not the core of the invention. Many known models exist that can identify landmarks. However, the invention delivers accurate results quickly by utilizing a specific architecture of these known models.

[0012] A system according to the invention serves to identify landmarks in an image. This system comprises several, preferably parallel, agents, each agent being a functional module designed to search for a predefined main landmark and a number of other landmarks in a landmark area around a position of the agent.

[0013] Each agent of the system is designed to: a) Defining or receiving its starting position, b) Estimating a principal dislocation of its principal landmark, c) Estimating election dislocations of other landmarks in the landmark area, d) Selecting an updated starting position, determined based on the estimated main dislocation and the choice dislocations estimated by other agents for that main landmark, e) Repeat steps b) to d) with the updated starting position as the starting position until a termination condition is met.

[0014] It should be noted that the system could also include other models, even other models that could be referred to as "agents" used for other purposes. The term "each agent" is therefore to be understood as "each agent designed to perform the task of locating landmarks" in the manner according to the invention.

[0015] The landmark identification system therefore comprises multiple agents (preferably operating in parallel to increase speed) to identify several landmarks in an image. Each agent is a functional module designed to search for a predefined primary landmark and a number of additional predefined landmarks within a specific landmark area (this can be the entire image or a custom space for each agent). An agent's primary landmark is the landmark that the agent is assigned to search for.

[0016] The agents could be visualized as units moving across the image surface toward their primary landmark. However, this is merely one way to convey a better understanding. In reality, each agent is a computational model (e.g., based on Adaboost) that moves a variable position (start position / updated start position) toward the primary landmark position of its primary landmark. The identification of the (unknown) primary landmark position is thus achieved through this approximation process. The most recently updated start position could then be interpreted as the primary landmark position of the primary landmark.

[0017] Regarding the terms "position" and "dislocation," it might seem difficult to distinguish between them at first glance. A "position" refers to a coordinate within an image. Since an image can be two- or three-dimensional (e.g., an MRI or CT scan), the position can also be two- or three-dimensional. A dislocation can be a position, a vector, or a direction and distance from the starting position. The term "dislocation" implies that the starting position is not yet the primary position of the main landmark and that there is a dislocation relative to this primary position. There is also a dislocation from the starting position to the estimated positions of the other landmarks. For example, for an election dislocation, a "voting position," a "voting vector," or a "voting direction / distance" can be estimated.

[0018] The starting position is the position where the agent is "placed" on the image. As mentioned earlier, there is no true positioning; rather, each agent chooses or calculates its starting position. There is an initial starting position (preferably unique to each agent) where every agent begins, and an "updated starting position," which is the new starting position for the next iteration. In the figurative example of agents moving across the image, the starting position and the updated starting position could be interpreted as agent positions. Furthermore, the term "agent position" is used later to refer to the current starting position of another agent. An agent can receive its initial starting position as a predetermined position or choose it randomly. It might also be preferred that all agents start from the same point (e.g.,Agents start either in the center of the image or in the center of their assigned landmark area. To better understand this, imagine that each agent has a landmark area, with feature spaces distributed across the entire image and overlapping each other, and each agent choosing their starting position within their own landmark area. However, since the landmarks are initially unknown, one could also imagine that the landmark area for at least some of the agents is the entire image.

[0019] Each agent is designed to estimate the dislocation of its assigned primary landmark from its current starting position. This dislocation is referred to as the "primary dislocation" to indicate that it relates to the agent's primary landmark.

[0020] This principal dislocation can be based on an estimated principal landmark position or a principal landmark vector of the principal landmark, or on an estimate of the direction to the principal landmark and an estimated distance to the principal landmark.

[0021] Additionally, each Agent A should i the dislocation of other landmarks within its landmark range, i.e., the main landmarks of other agents from the current starting position of agent A iThis dislocation is estimated. This dislocation is called the "choice dislocation" to indicate that it relates to the main landmark of other agents and is later used to "choose" the new starting positions. This choice dislocation can be based on an estimated position of the other landmark, or on an estimated direction to the other landmark and an estimated distance to the other landmark, or even a vector from the starting position of agent A. i proceed to the other landmark.

[0022] To locate a dislocation (both a primary and a secondary dislocation), several methods are known in engineering. As mentioned earlier, the initial location of landmarks is not the core of the invention, but rather the robust and rapid identification of a precise landmark position.

[0023] For better understanding, one could visualize the dislocations as arrows from each starting position to the estimated positions of the landmarks, where the arrow representing the primary dislocation could be a different color than the arrows representing the alternative dislocations. In reality, the dislocations could be data sets of vectors.

[0024] To estimate dislocations, the agents can use machine learning algorithms trained for landmark localization. These algorithms analyze features and patterns within the image to determine the most probable position of a landmark.

[0025] Now there is a main landmark for Agent A i There are probably several possible positions. One is the position resulting from Agent A's starting position. iand the main dislocation, and the other positions are the positions that result from the starting positions of other agents and the choice dislocations of these other agents for this main landmark of Agent A. i derivable. It should be noted that each dislocation could be the position or a vector to that position.

[0026] Each agent attempts to approximate the (real) "primary landmark position" of its primary landmark (unknown) by choosing an updated starting position that is closer to its primary landmark. This updated starting position is a new starting position based on the estimated primary dislocation and the choice dislocations predicted by other agents. The updated starting position is preferably determined using a regression mechanism that combines the agent's and other agents' estimates of the primary landmark. This determination could be performed by a central instance of the system or by each agent individually, although it is preferable for each agent to update its starting position individually, as this is much faster than calculating all updated starting positions centrally.

[0027] This process is then repeated with the updated starting positions; that is, new dislocations are estimated from these new starting positions, and new updated starting positions are derived from them. The agents interact with each other by exchanging their estimated choice dislocations, thereby updating their starting positions and improving the accuracy of their landmark identification. This process is repeated until a termination condition is met, preferably when all agents converge to a consistent set of landmarks (e.g., when the distance between old starting positions and updated starting positions is less than a predefined minimum threshold) or when a predetermined number of iterations have been completed.

[0028] A method according to the invention for identifying landmarks in an image using a system according to the invention comprises the following steps: - Providing image data, - Forwarding image data sets to the agents, with each agent receiving at least its landmark area of ​​the image data as a data set, - Processing, preferably in parallel, the data records by the agents, each agent: a) defines or obtains its starting position, b) estimates a major dislocation of its major landmark, c) Estimates election dislocations of other landmarks in the landmark area, d) selects an updated starting position, determined based on the estimated main dislocation and the choice dislocations estimated by other agents for that main landmark, e) repeat steps b) to d) with the updated starting position as the starting position until a termination condition is met.

[0029] Most of the procedure has already been explained above through the description of the agents' function. However, it will be briefly summarized below.

[0030] First, image data is provided. This is data from a digital image that defines, in particular, the color or grayscale values ​​of the pixels of voxels. The invention is very advantageous for processing medical images, since landmarks are often required for these images for automated procedures such as organ identification or image registration. For better understanding, one could imagine that the image data is a medical image in which organs or bones are visible, and that landmarks are to be inserted at specific points in the image.

[0031] The image data is then distributed to the agents. Each agent can receive all or part of the image data. Since the landmark area is essential for an agent, it is only necessary to forward the image data sets containing the landmark area to that agent. However, each agent can also choose its own landmark area from within the image data, meaning that the entire image data could be sent to every agent. It's important to note that the landmark area can be a predefined subset of the image data or a predefined sparse-sampling descriptor for an agent's neural network. This part of the data could contain information about a current starting position, which initially might be far from the landmark area.

[0032] The data records received by the agents are now processed. This is preferably done through parallel processing, as this is significantly faster than sequential processing. The system architecture is designed so that each agent can be processed in parallel with the other agents.

[0033] During the process, each agent defines or receives its (initial) starting position (predefined and / or randomly chosen) and estimates a primary dislocation of its primary landmark, as well as choice dislocations of other landmarks within that agent's landmark area. This procedure has already been described above. Landmark discovery is state of the art. The agents then exchange information, so that each agent receives choice dislocations for its primary landmark from the other agents. From the primary dislocation and the received choice dislocations, an updated starting position is derived (e.g., by regression). From this, each agent derives its starting position for the next iteration (i.e., the updated starting position).

[0034] In the next iteration step, each agent determines or receives its starting position again, which is the updated starting position, and the process of determining a new updated starting position from estimated dislocations is restarted again and again until a termination condition is met.

[0035] The method could be advantageously integrated into a BodyGPS regression algorithm. However, other neural regression networks are equally applicable to the invention with multiple agents of the method. Partial volumes could also be used instead of sparse-sampling descriptors. The advantage of BodyGPS is its fast computation for operation. Beyond BodyGPS, the proposed solutions use multiple agents and multiple regressions simultaneously. This would offer the advantage of implicit voting, which would contribute to robustness.

[0036] In practice, a single agent could comprise a network that estimates the relative distances of all landmarks (at least within the landmark area) to the agent's position. This network would then be trained using randomly sampled points and monitored landmark positions. Each agent in this environment would then search for its specified landmark while voting on the positions of the other landmarks, preferably using a multi-regression header. The position of that agent could then be updated by a weighted average of the votes of the other agents and the estimate of the agent responsible. As the agent gets closer, it will have a more precise estimate, while the votes of the other agents will ensure the robustness of the image.

[0037] An agent can be viewed as an abstraction of a position within the image and an associated main landmark. The algorithm computes the relative dislocations to reach all possible landmarks at the agent's position. Regression is preferably performed by computing the descriptor or subvolume at the agent's position and applying a trained neural network to it. The preferred neural network includes a residual network that takes an input descriptor and preferably outputs landmark positions in a world coordinate system. The advantage of a residual network is that it allows for a better flow of input data and thus a better model. An initial descriptor is preferably projected into a lower dimension using a linear projection layer to reduce computational overhead in later layers.After the initial projection, several layers with a residual connection are preferably applied. Each layer contains linear weights, normalization functions, and nonlinear Swish activation functions. The final layer projects a hidden state into dislocation vectors for each landmark in the 3-dimensional coordinate system.

[0038] However, since multiple agents are used, multiple landmarks would be considered in each image. Furthermore, each agent will estimate choice dislocations to the primary landmarks of other agents. In a multi-agent environment, each agent is associated with a specific primary landmark and uses dislocation vectors to approach its primary landmark. Dislocation estimations can be performed in parallel, allowing for faster computation.

[0039] A control unit according to the invention is suitable for a medical imaging system. It comprises a system according to the invention and / or is designed to carry out a method according to the invention.

[0040] A medical imaging system according to the invention comprises the control unit according to the invention.

[0041] Some of the aforementioned units or modules of the invention can be implemented wholly or partially as software modules that run on a processor of a computer system. Implementation largely in the form of software modules can have the advantage that applications already installed on an existing computer system can be updated with relatively little effort to install and execute these units of the present patent application. The object of the invention is also achieved by a computer program product comprising a computer program that can be loaded directly into the memory of a computer system and that includes program units for carrying out the steps of the methods, at least those steps that could be executed by a computer when the program is run by the computer system.In addition to the computer program, such a computer program product may also include other parts such as documentation and / or additional components, including hardware components such as a hardware key (dongle, etc.) to facilitate access to the software.

[0042] A computer-readable medium, such as a memory stick, a hard drive, or other portable or permanently installed storage device, can be used to transport and / or store the executable parts of the computer program product so that they can be read by a processor unit of a computer system. A processor unit can comprise one or more microprocessors or their equivalents.

[0043] Particularly advantageous embodiments and features of the invention will become apparent from the dependent claims, as disclosed in the following description. Features of different claim categories can be combined in a suitable manner to give further embodiments not described here.

[0044] According to a preferred system, each agent comprises a machine learning network trained to estimate the principal dislocation of its principal landmark and the choice dislocations of other landmarks within the landmark area. Such a machine learning network could be used to search for landmarks and identify their positions in an image. However, it should be noted that the position may not be exact, so the choice dislocations of other agents are very valuable for robust landmark identification. Preferably, the machine learning network incorporates a ResNet architecture. Alternatively or additionally, it is preferred that the machine learning network be trained to compute relative dislocations to all landmarks within the landmark area and / or be trained using randomly selected points and monitored landmark positions.

[0045] It is important to note that it is not strictly necessary to locate the other landmarks in the image to estimate the choice dislocations. If the relative positions of the landmarks are known, the choice dislocations can also be derived from an estimated position of the main landmark and the known relative positions. By using relative dislocations, an accurate estimate of the main landmark can guide another estimate without an explicit estimate by a neural network.

[0046] A preferred system is designed such that each agent receives the choice dislocation of its primary landmark, as estimated by other agents. Thus, the updated starting position is not determined by a central authority, but by each agent independently. Each agent is designed to determine its updated starting position based on its own estimate of the primary dislocation and the choice dislocations received from other agents.

[0047] Preferably, the system is also configured such that each agent receives the current agent positions of at least the other agents providing choice dislocations (i.e., the current starting positions of these other agents). In this case, it is preferred that each agent is configured to determine the updated starting position based on a weighted determination, with choice dislocations from closer agents having a greater weight than choice dislocations from more distant agents. With this weighting, agents that are closer and therefore have a better "view" of another agent's main landmark could be favored by that agent. It should be noted that this embodiment is advantageous when distance negatively affects the estimation of choice dislocations, for example, when estimating relative dislocations.

[0048] Preferably, the system, and in particular each agent, is also designed to determine the updated starting position based on a weighted determination, where choice dislocations that are further away than a predefined threshold from the current starting position or primary dislocation have a lower weight (e.g., "0") than closer choice dislocations. This weighting would allow choice dislocations that are far away (and therefore untrustworthy) to be ignored.

[0049] A preferred system comprises multiple independent computing units (e.g., processors and memory units), with different agents being processed by different computing units. Preferably, each agent is processed by its own computing unit, so that all agents are processed in parallel. This significantly speeds up the process.

[0050] According to a preferred system, a number of agents, preferably each agent, comprise multiple regression heads designed to determine an updated starting position based on a primary dislocation and choice dislocations. These heads could be used for voting on different landmarks.

[0051] A preferred system, preferably each agent, is designed to select updated starting positions by determining an average or mean position from an estimated primary dislocation and choice dislocations estimated by other agents. Preferably, an agent's updated starting position is the determined mean position or a position between that agent's current starting position and the determined mean position.

[0052] Generally, it is preferred that each agent determines a primary landmark position for its primary landmark (first estimating, then regression), and that this specific primary landmark position is the updated starting position. Thus, assuming the agents move across the image, each agent determines a position for its primary landmark, moves to that position, and checks in the next iteration whether this position is correct or not.

[0053] According to a preferred system, each agent comprises a residual network designed to receive an input descriptor of the agent's current starting position in the image, and where the residual network is trained to output choice dislocations of landmarks in a world coordinate system. It is preferred that the agent be designed to project the input descriptor into a lower dimension using a linear projection layer to reduce the computational overhead in later layers. Alternatively or additionally, it is preferred that the agent be designed to apply multiple layers of the residual network with residual connection after the initial projection. Here, a preferred neural network has several layers, and the first layer of the neural network is the initial projection.

[0054] According to a preferred method, the system, and in particular each agent, determines the current agent positions (i.e., their current starting positions) of at least the other agents providing choice dislocations, and each agent determines the updated starting position based on a weighted determination, where choice dislocations from closer agents have a greater weight than choice dislocations from more distant agents. Alternatively or additionally, choice dislocations pointing to positions farther from the agent's current starting position have a lower weight (e.g., "0" upon exceeding a predefined threshold) than choice dislocations pointing to positions closer to the agent's current starting position.

[0055] According to a preferred method, the updated starting position is determined based on a linear regression of the estimated main dislocation and the positions of this main landmark estimated by other agents, in particular a weighted linear regression.

[0056] According to a preferred method, the use of dislocation vectors is formulated as a weighted average of the voting estimate and the dislocation of the assigned agents, preferably based on the formula: d1 = λ - d1 + (1 - λ)median{d i} N i=1 In this equation, the agent's dislocation estimate d1 is replaced by an average estimate of other agents (the dislocation estimates d). iupdated by N other agents) using the scaling factor λ, where λ is preferably updated during the process to give more weight to an agent's own estimate as that agent gets closer to its associated landmark.

[0057] For the method according to the invention, the use of AI-based methods (AI: “artificial intelligence”) is preferred. Artificial intelligence is based on the principle of machine learning and is generally implemented with an adaptive algorithm that has been trained accordingly. The term “machine learning” is often used for machine-based learning, which also includes the principle of “deep learning”.

[0058] The methods can also incorporate elements of cloud computing. In the technical field of cloud computing, an IT infrastructure, storage space or computing power, and / or application software are provided via a data network. Communication between the user and the cloud takes place via data interfaces and / or data transmission protocols. In a preferred embodiment of the methods according to the invention, data is provided to a cloud via a data channel (for example, a data network). This cloud comprises a (remote) computing system, e.g., a computer cluster, which typically does not include the user's local computer. It is particularly preferred that the cloud service provides both computing power and application software.

[0059] Further features and functions of the present invention will become apparent from the following detailed descriptions in conjunction with the accompanying drawings. It should be understood, however, that the drawings serve only for illustration and not as a definition of the limits of the invention. Fig. Figure 1 shows a system according to the invention and predictions of an agent. Fig. Figure 2 shows the operation of a system according to the invention. Fig. Figure 3 shows estimates from multiple agents to update the positions of individual agents. Fig. Figure 4 shows the process carried out with independently operating agents.

[0060] Fig. Figure 1 shows a system 7a for identifying landmarks 2 in an image 1 and making predictions by an agent. The system 7a comprises several agents 7 and consists of multiple interacting agents 7 operating in parallel. Each agent 7 is a functional module designed to search for a predefined main landmark 2 and a number of other landmarks 2 within a landmark area 4 around the agent 7's position. The illustration depicts the case where an agent 7 searches for landmarks (indicated by the arrows) in an X-ray image 1 of a foot from a starting position 3. The position of the landmarks is initially unknown.

[0061] Fig. Figure 2 shows the function of system 7a from Fig. 1. Each agent 7 is configured to define or receive its starting position 3, estimate a primary dislocation 5 (solid arrows) of its primary landmark 2, estimate choice dislocations 6 (dashed arrows) of other landmarks 2 within landmark area 4, choose an updated starting position 3a, determined based on the estimated primary dislocation 5 and the choice dislocations 6 estimated by other agents 7 for that primary landmark 2, and repeat these steps with the updated starting position 3a as the starting position 3 in the form of multiple iterations. Note that the landmarks 2 are unknown.

[0062] Fig. Figure 3 shows estimates for multiple agents to update the positions of individual agents 7. A network 10, which can be part of each agent, is depicted. The network 10 comprises a descriptor 11, a projection layer 12, normalization layers 13, linearization layers 14, and an output layer 15. An image 1 is input into the initial descriptor 11 and projected into a lower dimension using the linear projection layer 12 to reduce computational overhead in later layers. Following the initial projection, several layers, the normalization layers 13 and the linearization layers 14, are applied with a residual connection. Each layer contains linear weights, normalization functions, and nonlinear Swish activation functions.The last layer projects a hidden state into dislocation vectors for each landmark in the 3-dimensional coordinate system (estimating the dislocation to reach a landmark from the current position).

[0063] Fig.Figure 4 shows the process carried out with independently operating agents 7. First, image data in the form of an X-ray image 1 of a foot is provided and distributed to the agents 7 of system 7a. Each agent 7 receives all the image data and selects its own landmark area 4, which is the same for all agents. In practice, each agent 7 likely has its own landmark area. The data sets received by the agents 7 are processed (indicated by arrows from the agents into image 1). This is preferably done by parallel processing, as this is significantly faster than sequential processing. During the process, each agent defines its initial starting position 3 (shown here as hatched circles) and estimates a principal dislocation 5 of its principal landmark as well as choice dislocations 6 of other landmarks within the landmark area 4. This is shown in the upper part of the figure.

[0064] In the middle part, each agent now receives 7 choice dislocations 6 from the other agents 7, and from the main dislocation 5 and the received choice dislocations 6, an updated starting position 3a is derived by regression.

[0065] In the lower section, each agent derives its new starting position 3 for the next iteration from the updated starting position 3a (hatched circles). For comparison, the dashed circles show the initial starting positions.

[0066] In the next iteration step (see again the upper part), each agent determines or receives its starting position 3 as updated starting position 3a, and the process of determining a new updated starting position 3a from estimated dislocations is restarted again and again until a termination condition is met.

[0067] Although the present invention has been disclosed in the form of preferred embodiments and variations thereof, numerous additional modifications and variations can be made without departing from the scope of protection of the invention. For the sake of clarity, it should be noted that the use of "a" or "an" in this application does not preclude multiple uses, and that "comprising" does not preclude other steps or elements. The expression "a number of" means "at least one." The mention of a "unit" or a "device" does not preclude the use of more than one unit or device. Regardless of the grammatical term used, persons of male, female, or other gender identities are included under the term. Reference symbol list 1 image 2 Landmark 3 Starting position 3a Starting position (updated) 4 Landmark Area 5 main dislocations 6 Election Dislocation Agent 7 7a System 10 Network 11 Descriptor 12 Projection layer 13 Normalization layer 14 Linearization layer 15 Output layer QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited non-patent literature

[0000] Deep learning-based regression and classification for automatic landmark localization in medical images” (IEEE Transactions on Medical Imaging, 39(12), 4011-4022; 2020

[0006] Ghesu et. al. zeigen in ihrem Papier „Multi-scale deep reinforcement learning for real-time 3D-landmark detection in CT scans“ (IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(1), 176-189; 2017

[0007]

Claims

[1] System (7a) for identifying landmarks (2) in an image (1) comprising several, preferably parallel, agents (7), wherein each agent (7) is a functional module designed to search for a predefined main landmark (2) and a number of other landmarks (2) in a landmark area (4) around a position of the agent (7), wherein each agent (7) is designed to: a) Defining or receiving its starting position (3), b) Estimating a principal dislocation (5) of its principal landmark (2), c) Estimating election dislocations (6) of other landmarks (2) in the landmark area (4), d) Selecting an updated starting position (3a) determined based on the estimated main dislocation (5) and the choice dislocations (6) estimated by other agents (7) for this main landmark (2), e) Repeat steps b) to d) with the updated starting position (3a) as starting position (3) until a termination condition is met. [2] System according to claim 1, wherein each agent (7) comprises a machine learning network (10) trained to estimate the principal dislocation (5) of its principal landmark (2) and the choice dislocations (6) of other landmarks (2) in the landmark area (4), wherein preferably the machine learning network (10) comprises a Resnet architecture, wherein preferably the machine learning network (10) was trained to compute relative dislocations to all landmarks (2) in the landmark area (4), and / or was trained with randomly sampled points and monitored landmark positions (2). [3] System according to any of the preceding claims, wherein the system (7a) is designed such that each agent (7) receives the choice dislocation (6) of its main landmark (2) estimated by other agents (7), wherein each agent (7) is designed to determine the updated starting position (3a) based on its estimate of the main dislocation (5) and the choice dislocations (6) received from other agents (7), wherein the system (7a) is preferably also designed such that each agent (7) receives the current agent positions (3) of at least the other agents (7) providing choice dislocations (6), and each agent (7) is designed to determine the updated starting position (3a) based on a weighted determination, wherein choice dislocations (6) from closer agents (7) have a greater weight than choice dislocations (6) from more distant agents (7), and / or wherein preferably the system (7a), in particular each agent (7), is also designed to determine the updated starting position (3a) based on a weighted determination, In the case of choice dislocations (6) that are a distance greater than a predefined threshold from the current starting position (3) or main dislocation (5), choice dislocations (6) have a lower weight than closer choice dislocations. [4] System according to one of the preceding claims, wherein the system (7a) comprises several independent computing units and wherein different agents (7) are processed with different computing units, preferably each agent (7) being processed with its own computing unit, so that all agents (7) are processed in parallel. [5] System according to one of the preceding claims, wherein the system, in particular a number of agents (7), preferably each agent (7), comprises multiple regression heads designed to determine an updated starting position (3a) based on a principal dislocation (5) and choice dislocations (6). [6] System according to any of the preceding claims, wherein the system (7a), preferably each agent (7), is designed to select updated starting positions (3a) based on a specific average position or a mean position based on an estimated main dislocation (5) and choice dislocations (6) estimated by other agents (7), wherein preferably the updated starting position (3a) of an agent (7) is the specified average position or a position between the current starting position of that agent (7) and the specified average position. [7] System according to any one of the preceding claims, wherein each agent (7) comprises a residual network (10) designed to receive an input descriptor (11) of a current starting position (3) of an agent (7) in the image (1), and wherein the residual network (10) is trained to output choice dislocations (6) of landmarks (2) in a world coordinate system (7a), wherein the agent (7) is preferably designed to project the input descriptor (11) into a lower dimension using a linear projection layer, and wherein the agent (7) is preferably designed to apply several layers of the residual network (10) with residual connection after the initial projection. [8] Method for identifying landmarks (2) in an image (1) using a system (7a) according to any one of the preceding claims, comprising the steps: - Providing image data (1), - Forwarding data sets of the image data (1) to the agents (7), wherein each agent (7) receives at least its landmark area (4) of the image data (1) as a data set, - Processing, preferably in parallel, the data records by the agents (7), wherein each agent (7): a) defines or obtains its starting position (3), b) estimates a principal dislocation (5) of its principal landmark (2), c) estimates election dislocations (6) of further landmarks (2) in the landmark area (4), d) selects an updated starting position (3a) determined based on the estimated main dislocation (5) and the choice dislocations (6) estimated by other agents (7) for this main landmark (2), e) repeat steps b) to d) with the updated starting position (3a) as starting position (3) until a termination condition is met. [9] Method according to claim 8, wherein the system (7a), in particular each agent (7), also determines the current agent positions (3) of at least the other agents (7) providing choice dislocations (6), and each agent (7) determines the updated starting position (3a) based on a weighted determination, wherein choice dislocations (6) of closer agents (7) have a greater weight than choice dislocations (6) of more distant agents (7) and / or choice dislocations (6) that point to positions farther away from the current starting position (3) of the agent (7) have a lesser weight than choice dislocations (6) that point to positions closer to the current starting position (3) of the agent (7). [10] Method according to one of claims 8 or 9, wherein the updated starting position (3a) is determined based on a linear regression of the estimated main dislocation (5) and the positions of this main landmark (2) estimated by other agents (7), in particular a weighted linear regression. [11] Method according to any one of claims 8 to 10, wherein the use of dislocation vectors is formulated as a weighted average of the voting estimate and the dislocation of the assigned agents (7), preferably based on the formula: d1 = λ · d1 + (1 - λ)me dian{di}Ni=1, wherein in this equation the dislocation estimate d1 of agent (7) is updated by an average estimate of other agents (7) using the scaling factor λ, and wherein λ is preferably updated during the process to give more weight to an agent's own estimate as soon as that agent (7) approaches its assigned landmark (2). [12] Control unit for a medical imaging system comprising a system (7a) according to any one of claims 1 to 7 and / or designed to perform a method according to any one of claims 8 to 11. [13] Medical imaging system comprising the control unit according to claim 12. [14] Computer program product comprising instructions which, when executed by a computer, cause the computer to perform the steps of the method according to any one of claims 8 to 11. [15] Computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to perform the steps of the method according to any one of claims 8 to 11.

Images