Image acquisition method

Abstract

A medical imaging and segmentation method comprising the step of initializing a segmentation model for a subsequent 3D frame using segmentation results obtained for a previous 3D frame in the same imaging session, which accelerates and stabilizes the segmentation results, particularly in the context of scanning procedures where segmentation is performed in real time (on the fly) for each acquired frame.

Description

[Technical Field]

[0001] The present invention relates to the field of medical image acquisition, and in particular to a method for user-guided image acquisition. [Background technology]

[0002] Within the context of medical imaging, it is often necessary to acquire a standardized set of image views of specific anatomy. For example, a standard echocardiography protocol involves acquisition of the following views: parasternal long axis, parasternal short axis, apical four-chamber, subxiphoid (subcostal), and inferior vena cava views. Fetal ultrasound imaging also involves acquisition of a sequence of views to capture standard fetal measurements, such as biparietal diameter (BPD), head circumference (HC), abdominal circumference (AC), and femur length (FL).

[0003] One approach to doing this is to acquire one or more 3D images and extract the required planes from the 3D images. This has the advantage that the user does not need to precisely position the ultrasound imaging probe for each required planar view. Instead, a more limited set of one or more 3D images can be acquired and the required view planes can be extracted as 2D images from the 3D image data.

[0004] Methods exist for automatically extracting a set of one or more 2D planes from a 3D ultrasound image. At least a subset of these methods utilize the use of anatomical segmentation algorithms to identify anatomical landmarks and then identify target planes through the imaged volume with reference to the anatomical landmarks. Model-based segmentation can be used. Alternatively, machine learning-based segmentation algorithms can be used. Summary of the Invention [Problem to be solved by the invention]

[0005] In at least one implementation approach, planes are extracted in real time and displayed to the user on the fly. This provides useful guidance and can lead to shorter scan durations and more stable extraction of 2D planes. The latter effect also results in more reliable derived measurements. Real-time plane extraction requires rapid processing, which imposes constraints on the computational complexity of the segmentation algorithm used.

[0006] However, during a scan, there is both motion of the moving scanhead (e.g., ultrasound transducer probe) and motion of the patient (e.g., motion related to breathing). As a result, the anatomical plane position changes over time. Therefore, the stability of plane extraction and measurements is a challenge.

[0007] Therefore, a tension exists between the need for real-time plane extraction and the need for stable and accurate plane extraction. A computer-implemented method that provides improved optimization of these two requirements would be advantageous.

[0008] See Goldberger J et al: "Context-Based Segmentation of Image Sequences", IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, IEEE COMPUTER SOCIETY, USA, vol. 28, no. 3, March 20, 2006 (2006-03-20), pages 463-468, which discloses a method for segmentation of video image sequences. [Means for solving the problem]

[0009] The invention is defined by the claims.

[0010] According to an example according to one aspect of the present invention, there is provided a computer-implemented method comprising the steps of receiving, during an imaging session, a medical image data stream comprising a series of medical image data frames of an anatomical object; deriving a segmentation model comprising one or more anatomical segmentation algorithms, the segmentation model being configured to receive the medical image data frames as input and to generate segmentation results as output; and performing a segmentation operation for each of the series of medical image data frames.

[0011] The segmentation operation may include applying an initialization procedure to initialize a segmentation model, applying the segmentation model to the current medical image data frame to obtain a segmentation result for the current medical image data frame, and accessing a segmentation data store and storing the segmentation result for the current medical image data frame in the segmentation data store.

[0012] With regard to the initialization procedure, this may include the steps of accessing a segmentation data store, identifying in the data store stored segmentation results for at least one previous medical image data frame of the current imaging session, and initializing a segmentation model using the segmentation results for the at least one previous medical image data frame of the current imaging session recorded in the segmentation data store. Initializing the segmentation model may include initializing one or more of the segmentation algorithms of the segmentation model.

[0013] An embodiment of the proposed invention is based on the insight that segmentation results obtained for previous image frames during an imaging session can be used to initialize the segmentation model used to segment the current image frame, accelerating and stabilizing the sequence of segmentation results.

[0014] The segmentation model may be retrieved from a data store or memory that stores the model.

[0015] The medical image data stream may in some embodiments be an ultrasound image data stream comprising ultrasound image data frames, hi some embodiments the anatomical object is a heart or a portion thereof.

[0016] In some embodiments, a method, e.g., an initialization procedure, may include checking whether a segmentation data store includes segmentation results for at least one previous medical imaging data frame of the current imaging session. In response to a data store including segmentation results for at least one previous medical image data frame of the current imaging session, the initialization procedure may include initializing a segmentation model using the segmentation results for at least one previous medical image data frame of the current imaging session recorded in the segmentation data store. In response to a data store not including segmentation results for at least one previous medical imaging data frame of the current imaging session, the initialization procedure may include initializing the segmentation model using a general initialization state. The general initialization state may be stored in a memory or data store. It may, for example, be stored in the aforementioned segmentation data store. Alternatively, it may be included as a native part of the segmentation model.

[0017] In this way, the method can initialize the model differently depending on whether previous segmentation results are available, for example, at the start of an imaging session, previous segmentation results may not yet be available.

[0018] Optionally, in some embodiments, the segmentation operation further includes configuring a computational complexity level of the segmentation model before applying the segmentation model. For example, in some embodiments, it is proposed to configure the computational complexity level depending at least in part on whether the data store includes segmentation results for at least one previous medical image data frame of the current imaging session. For example, the inventors recognize that if the segmentation model can be initialized using previous segmentation results, a segmentation model with less computational complexity can be used without reducing the accuracy of the results. This helps to improve temporal resolution and reduce processing demands. Therefore, in some embodiments, if the segmentation data store does not include segmentation results for at least one previous medical image data frame of the current imaging session, the computational complexity level can be configured at a lower level than if the segmentation data store does not include such segmentation results. For example, in some embodiments, it is proposed to use a higher computational complexity for the first segmentation operation (where previous segmentation results are not yet available) and a lower computational complexity for subsequent segmentation operations where previous segmentation results are available.

[0019] For the avoidance of doubt, the term "computational complexity" is a term of art in the field of computer science and its meaning will be understood by those skilled in the art.

[0020] In some embodiments, the method includes, after performing the step of initializing the segmentation model using segmentation results for at least one previous medical imaging data frame of the current imaging session recorded in the segmentation data store, reducing the computational complexity level. For example, reducing the computational complexity level means for configuring the segmentation model to have a computational complexity level that is less than the computational complexity level of the model used to generate the stored segmentation result for the at least one previous medical imaging data frame. Thus, in some embodiments, the method includes applying a segmentation model configured with a first level of computational complexity to at least a first medical image frame, storing the segmentation result to obtain a stored segmentation result, and applying a segmentation model configured with a second, lower level of computational complexity to at least further medical image frames, wherein the segmentation model is initialized using the stored segmentation result for at least the first image frame to apply the segmentation model to the further image frames.

[0021] In some embodiments, configuring the computational complexity level of the segmentation model includes switching the model between different computational complexity modes, in each mode a different respective set of one or more segmentation algorithms is used for anatomical segmentation. In other words, in the different modes, a different respective set of one or more algorithms having different computational complexities is used. Thus, the segmentation model can have different processing channels or streams, each subject to a different set of one or more algorithms that process the image data with a different computational complexity.

[0022] In some embodiments, the segmentation model is configurable in at least a first mode and one or more further modes. In the first mode, the segmentation model may be configured at a first computational complexity level. It may be initialized during use using a common initialization state. In each of the one or more further modes, the segmentation model may be configured at a respective further computational complexity level lower than the first computational complexity level, which is initialized during use using segmentation results for previous image data frames stored in a data store. Thus, as briefly alluded to above, in some embodiments, it is proposed to use segmentation algorithms of different computational complexity depending on whether it is possible to initialize the segmentation using previous segmentation results.

[0023] In some embodiments, the one or more further modes include at least two further modes, each having a different respective computational complexity.

[0024] In some embodiments, for a first frame (or the first one or more frames) of a medical image data stream, the computational complexity may be configured at a first level, and for one or more (e.g., all) subsequent frames in the image data stream, the computational complexity may be configured at one or more different, lower levels.

[0025] One way to implement the above proposed feature may be to configure the computational complexity level depending on whether the segmentation data store contains segmentation results for at least one previous medical image data frame of the current imaging session. For example, in some embodiments, the computational complexity is configured at a first level if the segmentation model is initialized using a previous medical image data frame of the current imaging session, and at a different, higher level if the segmentation model is initialized using a general initialization state.

[0026] It has been proposed above that in some embodiments the computational complexity of the segmentation can be adapted depending on whether the segmentation model can be initialized with a previous segmentation result. According to at least one set of embodiments, it is proposed to additionally or alternatively analyze any available previous segmentation results to assess their quality and to configure the segmentation operation differently depending on the quality analysis. If the segmentation quality of the previous segmentation result (used to initialize the model) is lower and the segmentation quality of the previous segmentation result is higher, the segmentation model can be configured with a greater computational complexity.

[0027] One proposed way to do this is to run an algorithm that generates a quality classification associated with the input segmentation result. The quality classification may be, for example, a pass / fail classification for the input segmentation result. The quality classification may be a binary value or a numeric score along a scale. When used to initialize a segmentation model, the quality classification may refer to a prediction regarding the extent to which the input pre-segmentation result will yield a segmentation result of quality above a threshold.

[0028] For example, in some embodiments, the method may include deriving a quality assessment algorithm configured to receive the segmentation result as an input and determine a quality classification of the segmentation result as an output. The method may further include applying the quality assessment algorithm to the at least one segmentation result of the previous medical image data frame to obtain a quality classification. The method may further comprise configuring a computational complexity level of the segmentation model based on the quality classification.

[0029] The quality assessment algorithm may be configured to receive only the previous segmentation result as input. In this case, the classification is based on inherent characteristics of the segmentation itself. The quality assessment algorithm may be configured to receive the previous segmentation result as input in addition to the current imaging frame to be segmented using the previous result as initialization. The quality assessment algorithm may be configured to receive the previous segmentation result as input in addition to the previous imaging frame from which the segmentation was derived. The quality assessment algorithm may be retrieved, for example, from a data store.

[0030] As an example, the quality assessment algorithm may be a machine learning algorithm that is trained in a supervised manner to generate a prediction as to whether the combination of the current 3D image volume (frame) and the associated pre-segmentation result will result in a reasonable 2D anatomical plane extraction. An example of one suitable algorithm for performing quality classification is described in detail in document EP 4080449 A1.

[0031] In some embodiments, the method may include a first stage and a second stage. The first stage includes applying a segmentation model in a first mode for each received frame to obtain a segmentation result, applying a quality assessment algorithm to each segmentation result, and continuing in the first stage until a quality classification of at least one frame matches a predefined success criterion. For example, if the quality classification is a score, the success criterion may be a threshold value for the score. The second stage may include applying the segmentation model in one or more of the further modes for each received frame. With regard to the quality assessment algorithm, this may be a quality assessment algorithm configured to receive the segmentation result as input and determine / provide a quality classification of the segmentation result as output. The first stage may include retrieving the quality assessment algorithm, for example, from a data store.

[0032] As mentioned above, in some embodiments, the quality assessment algorithm is a machine learning algorithm.

[0033] In some embodiments, each image data frame contains 3D image data, and the quality classification of the segmentation result for each frame indicates the predicted success in using the segmentation result to extract one or more 2D planes from each frame.

[0034] In some embodiments, one or more algorithms applied by the segmentation model perform steps including fitting a mesh to the medical imaging data, and initializing the model includes configuring an initial state for the mesh.

[0035] In some embodiments, the anatomical object is a heart or a portion thereof.

[0036] In some embodiments, the method further comprises generating a control signal for controlling a user interface to display a visualization of each segmentation result, which helps guide the user in performing the test in an efficient manner.

[0037] In some embodiments, the method further comprises controlling a user interface to display a representation of the quality classification of each segmentation result.

[0038] By way of example, a suitable segmentation model for use with one or more embodiments is described in European Patent Application Publication No. 2994053B1. By way of further example, another suitable segmentation model for use with one or more embodiments is described in Ecabert, O. et al.: "Automatic Model-based Segmentation of Heart in CT Images", IEEE Transactions on Medical Imaging, Vol. 27(9), pp. 1189-1291, 2008.

[0039] The present invention may also be implemented in software form. Thus, another aspect of the present invention is a computer program product comprising computer program code configured, when executed on a processor, to cause the processor to perform a method according to any embodiment described herein or according to any claim of the present application.

[0040] The present invention may also be implemented in hardware form.

[0041] One aspect of the present invention is a processing device having an input / output section, wherein during an imaging session, one or more processors are configured to receive at the input / output a medical image data stream comprising a series of medical image data frames of an anatomical object, derive a segmentation model comprising one or more anatomical segmentation algorithms, the segmentation model being configured to receive the medical image data frames as input and to generate segmentation results as output, and perform a segmentation operation for each medical image data frame in the series.

[0042] The segmentation operation may include applying an initialization procedure to initialize a segmentation model, applying the segmentation model to a current medical image data frame to obtain a segmentation result for the current medical image data frame, and accessing a segmentation data store and storing the segmentation result for the current medical image data frame in the segmentation data store. The initialization procedure may include accessing the segmentation data store, identifying stored segmentation results in the data store for at least one previous medical image data frame of the current imaging session, and initializing the segmentation model using the segmentation results for the at least one previous medical image data frame of the current imaging session recorded in the segmentation data store.

[0043] Preferably the one or more processors are further adapted to generate a data output representing the segmentation results of the series of images, preferably adapted to route the data output to the input / output, preferably adapted to export the data output to a further device such as a memory or a data store or a server.

[0044] Retrieving a segmentation model may include communicating, via the input / output, with a data store or memory that stores the model.

[0045] In some embodiments, the one or more processors may be configured to check whether the segmentation data store includes segmentation results for at least one previous medical image data frame of the current imaging session, and perform the step of initializing the segmentation model using the segmentation results for the at least one previous medical imaging data frame of the current imaging session only if such segmentation results for the previous image data frame are present. Optionally, in some embodiments, if no such prior segmentation results are present, the initialization procedure may include the step of initializing the segmentation model using a general initialization state.

[0046] In some embodiments, the segmentation operation may further include configuring a computational complexity level of the segmentation model before applying the segmentation model.

[0047] In some embodiments, the computational complexity level may be configured depending at least in part on whether the segmentation data store includes segmentation results for at least one previous medical image data frame of the current imaging session. For example, the computational complexity level may be configured at a first level or levels if the segmentation data store does not include segmentation results for at least one previous image data frame of the current imaging session, and at a different, higher level or levels if the segmentation data store does not include segmentation results for at least one previous image data frame of the current imaging session.

[0048] Another aspect of the invention is a system comprising a medical imaging device and a processing device as described above operatively coupled to the medical imaging device for receiving the medical image data stream.

[0049] These and other aspects of the invention will be apparent from and elucidated with reference to the embodiments described hereinafter.

[0050] For a better understanding of the present invention and to show more clearly how it may be carried into effect, reference will now be made, by way of example only, to the accompanying drawings in which: [Brief explanation of the drawings]

[0051] [Figure 1] 1 illustrates an overview of exemplary method steps in accordance with one or more embodiments of the present invention. [Figure 2] FIG. 1 is a block diagram of an exemplary processing device and system in accordance with one or more embodiments of the present invention. [Figure 3] 1 outlines steps of an exemplary method according to at least one preferred set of embodiments. [Figure 4] 1 illustrates components of an exemplary ultrasound imaging device. DETAILED DESCRIPTION OF THE INVENTION

[0052] The present invention will now be described with reference to the drawings.

[0053] It should be understood that the detailed description and specific examples, while indicating exemplary embodiments of the devices, systems, and methods, are for purposes of illustration only and are not intended to limit the scope of the invention. These and other features, aspects, and advantages of the devices, systems, and methods of the present invention will become better understood from the following description, appended claims, and accompanying drawings. It should be understood that the figures are schematic only and are not drawn to scale. It should also be understood that the same reference numerals are used throughout the figures to indicate the same or similar parts.

[0054] Embodiments of the present invention provide a medical imaging and segmentation method that utilizes segmentation results obtained for previous 3D frames in the same imaging session to initialize a segmentation model for a subsequent 3D frame, which accelerates and stabilizes the segmentation results, especially in the context of scanning procedures where segmentation is performed in real time (on the fly) for each acquired frame.

[0055] Embodiments of the present invention may be applied in the context of a method for obtaining a set of target planar views through an anatomical structure of interest through processing of 3D ultrasound image data, which may include processing received 3D image frames in real time to extract target planes representing target views of an anatomical object.

[0056] Although the description and examples are presented below specifically with respect to ultrasound imaging systems, it should be noted that the principles of the present invention have more general application to any modality of medical imaging. Other exemplary imaging modalities to which embodiments of the present invention may be advantageously applied include, by way of non-limiting example, MRI, optical coherence tomography, and CT imaging. The present method may find particularly advantageous application in non-ionizing imaging modalities that allow for real-time image acquisition.

[0057] To aid in understanding, the motivation behind the production of the present invention can be understood, at least in part, from the following.

[0058] During a medical imaging examination, such as an ultrasound examination, a user or operator may typically need to record images representing multiple different anatomical views. These different images may be referred to as anatomical planes. Recording each plane can be time-consuming, especially for less experienced operators. Also, there is potential for error or inaccuracy in acquiring the planes, especially for operators with lower levels of experience.

[0059] To aid in the acquisition of standard anatomical planes, algorithms exist in the art that are configured for automatic plane extraction from recorded 3D image volumes. At least one example of such an algorithm is described in European Patent Application Publication No. 2994053B1 (also published as WO 2014 / 162232).

[0060] Such algorithms facilitate real-time processing of 3D image sequences for acquisition. In one possible scenario, the operator can continuously scan while the system automatically processes acquired 3D image frames to extract target anatomical planes and displays the extracted planes to the user via a user interface. This allows the user to focus on different anatomical structures while continuing to scan.

[0061] In some examples, the image data is ultrasound image data. In some examples, the anatomical structure of interest is a heart. In some examples, at least one 3D frame is acquired per cardiac cycle. The frame may be segmented using anatomical segmentation to identify anatomical features or landmarks, and then one or more target anatomical planes are sliced ​​from the 3D frame based on the locations of the anatomical features or landmarks. The segmentation may be model-based segmentation. For example, see the following document, which details one exemplary method for model-based segmentation of the heart: Ecabert, O et al. Automatic Model-Based Segmentation of the Heart in CT Images Medical Imaging, IEEE Transactions on, 2008, 27, pp. 1189-1201. See also document EP2994053B1, which details a method for extracting image slices from a 3D image frame based on segmentation.

[0062] As described above, once the segmentation is calculated for a given frame, anatomical landmarks and anatomical 2D planes can be identified. Other parameters of the image can also be configured based on other parameters of the image, such as the field of view zoom level, beam steering, and / or Doppler gate location (e.g., placing the Doppler gate at the center of an artery).

[0063] The continuous scanning scenario described above allows for continuous updating of the extracted 2D planes and identified anatomical landmarks. Segmentation can be performed frame-by-frame or at fixed frame intervals. The extracted anatomical planes may be displayed to the user on-the-fly, which allows for shorter examination durations, more stable acquisition of 2D slices, and improved reliability of the acquired data for deriving anatomical measurements. This can improve workflow and diagnostic reliability.

[0064] According to an embodiment of the present invention, it is proposed to perform time series segmentation, also known as sequence segmentation, which means performing the segmentation of image frames sequentially in real time as the frames are being acquired.

[0065] In such continuous scan scenarios, anatomical plane positions can change over time due to both transducer motion and patient breathing. A key challenge in sequence segmentation is capturing such relevant plane changes in a way that does not result in unstable segmentation, leading to unstable plane extraction, for example, resulting from unclear vertex positions. Unstable segmentation refers to scenarios in which the segmentation results in differences between frames for substantially the same physical anatomical structure. For example, model-based segmentation involves fitting a surface mesh to the imaged anatomical structure, which often involves some degree of estimation and interpolation based on incomplete image data. While this typically results in a sensible segmentation for each frame, there may be small jumps in the segmentation between successive frames, and therefore in the resulting extracted planes. This results in non-smooth (unstable) segmentation in continuous acquisition scenarios.

[0066] Frame-by-frame model-based segmentation (MBS) approaches known in the art suffer from high latency, which is counter to the objective of providing a smooth user experience and allowing real-time user guidance for continuous execution of the scan.

[0067] To overcome one or more of the above-mentioned problems, according to one or more embodiments of the present invention, it is proposed to dynamically utilize previous segmentation results from the same imaging session to accelerate and stabilize segmentation. In particular, it is proposed to use segmentation results obtained for previous image frames during scanning to initialize the segmentation algorithm for the subsequent (current) image frame. This achieves the twin effects of improved inter-frame segmentation stability and increased segmentation speed.

[0068] In some embodiments, the proposed method is applied to ultrasound image data acquired in the context of an ultrasound imaging procedure.

[0069] For example, at least one advantageous set of embodiments provides a method that may include one or more of the following steps: One step may include acquiring a current 3D image frame of a volumetric region from an imaging system (e.g., an ultrasound acquisition system) during a scan of the patient; Another step may include checking whether segmentation results for a previous 3D image frame acquired by the imaging system during the same scan of the patient are available, and if so, initializing a segmentation model based on the segmentation results for the previous image frame. For example, this may include setting a starting state for the mesh, where the segmentation model is configured to fit the mesh to data in the acquired image frame; Another step may include segmenting the current 3D image frame using the segmentation model based on the initialized segmentation model.

[0070] In one particular set of embodiments, the proposed method may further comprise one or more of the following steps, performed before the segmentation step: One step may comprise using a quality assessment algorithm (which may, for example, utilize a trained AI algorithm) to determine a quality state (e.g., a pass / fail state) of the segmentation result for said previous 3D image frame; Another step may comprise setting the computational complexity or accuracy level of a segmentation model for segmenting the current 3D image frame based on the quality state determined for the previous segmentation result.

[0071] Therefore, the segmentation workflow is complemented herein by a quality assessment procedure, which may be referred to as a fault detection module in the following description. Such a module can advantageously address two potential problems. First, it may help avoid error propagation by avoiding the use of low-quality pre-segmentation results to initialize a new segmentation. The fault check allows for dynamic deactivation of tracking to prevent errors from propagating over time throughout the test. In this case, the next segmentation may be performed "from scratch," for example, using a general initialization step for the segmentation model.

[0072] Additionally, the results of the fault detection module can be communicated to the user via controls in the user interface. In this way, a warning can be given to the user if the segmentation is found to be of poor quality. This may indicate, for example, that none of the target planes can be extracted from the respective 3D image. This failure may be due to an error in the segmentation itself (e.g., the mesh was snapped to an incorrect position) or to insufficient image acquisition (e.g., due to motion artifacts caused by the patient's breathing, or an incorrectly aligned field of view, or poorly configured contrast settings). This information allows the user to dynamically adjust the scan performance to try to avoid the same problem continuing for future frames.

[0073] In one particular set of embodiments, the method may further include one or more of the following steps: if segmentation results are not available for any previous 3D image frames from the current imaging session, initializing the segmentation model with a (non-patient-specific) average generic model, and selecting a high computational complexity / accuracy level for the segmentation model to segment the current 3D image frame.

[0074] 1 outlines in block diagram form the steps of an exemplary method according to one or more embodiments, the steps being summarized before being further described in the form of an exemplary embodiment.

[0075] Method 10 is a computer-implemented method. The method includes receiving (14) a medical image data stream including a series of medical image data frames of an anatomical object during an imaging session. The imaging data frames may be 3D image frames. For example, in some embodiments, ultrasound imaging data frames may be received.

[0076] The method further includes retrieving (16) a segmentation model including one or more anatomical segmentation algorithms, the segmentation model configured to receive the medical image data frames as input and to generate segmentation results as output.

[0077] The method further includes performing, for each medical image data frame in the series, a segmentation operation 20 including at least the following steps: The segmentation operation 20 includes applying an initialization procedure 22 to initialize a segmentation model; The segmentation operation further includes applying (24) the segmentation model to the current medical image data frame to obtain a segmentation result for the current medical image data frame. For example, this may involve fitting a mesh to an anatomical structure depicted in the medical imaging data frame. In some embodiments, the initialization includes configuring an initial state of the mesh, e.g., configuring an initial form, shape, or position of the mesh; The segmentation operation may further include accessing a segmentation data store 28 and storing (26) the segmentation result for the current medical image data frame in the segmentation data store. The segmentation data store may be provided by local memory or may be a remote data store accessible, for example, via a network link.

[0078] It is proposed that the aforementioned initialization procedure 22 comprises at least the following steps: accessing the segmentation data store 28; identifying stored segmentation results for at least one previous medical image data frame of the current imaging session in the data store; and initializing a segmentation model using the segmentation results for at least one previous medical image data frame of the current imaging session recorded in the segmentation data store.

[0079] Preferably, the initialization procedure includes a process of checking whether the segmentation data store contains segmentation results for at least one previous medical image data frame of the current imaging session. If so, the method includes a step of initializing the segmentation model using the segmentation results for the at least one previous medical imaging data frame of the current imaging session recorded in the segmentation data store. If not, the method includes a step of initializing the segmentation model using a general initialization state.

[0080] As mentioned above, the method may also be implemented in the form of hardware, for example in the form of a processing device configured to perform the method according to any example or embodiment described in this document or according to any claim of the present application.

[0081] To further aid in understanding, Figure 2 presents a schematic diagram of an exemplary processing device 32 configured to perform methods according to one or more embodiments of the present invention. The processing device is shown in the context of a system 30 that includes the processing device. The processing device alone represents an aspect of the present invention. The system 30 is another aspect of the present invention. A provided system need not include all of the illustrated hardware components, but may include only a subset.

[0082] The processing device 32 comprises one or more processors 36 configured to carry out the methods outlined above or according to any embodiment of the methods set forth in any claim of this document or application. In the illustrated example, the processing device further comprises an input / output unit 34 or communication interface.

[0083] In the illustrated example of Figure 2, system 30 further includes a user interface 52. Features of the user interface are described in more detail below. In the illustrated example of Figure 2, system 30 further includes an ultrasound acquisition or imaging device 54 for acquiring 3D ultrasound imaging data 44. An example of an ultrasound imaging device is described below with reference to Figure 4. The aforementioned data store 28 for storing segmentation results for each processed imaging frame is also shown schematically in Figure 2.

[0084] The system 30 may further comprise a memory 38 for storing computer program code (i.e., computer-executable code) configured to cause one or more processors 36 of the processing unit 32 to perform the method outlined above, or a method according to any embodiment or claim described in this disclosure.

[0085] As mentioned above, the present invention may also be implemented in software form. Thus, another aspect of the present invention is a computer program product comprising computer program code configured, when executed on a processor, to cause the processor to perform a method according to any example or embodiment of the invention described herein or according to any claim of the present patent application.

[0086] It should also be noted that full compatibility of features between different aspects (systems, methods, computer program products) of the present invention is intended, and that features or options recited in the context of one aspect may be equally applied to any other aspect.

[0087] The one or more algorithms applied by the segmentation model may perform steps including fitting a mesh to the medical image data, and initializing the model may include configuring an initial state for the mesh.

[0088] In a preferred embodiment, the segmentation model may be applicable with adjustable computational complexity, in other words, the segmentation operation further comprises the step of configuring the computational complexity level of the segmentation model before applying the segmentation model.

[0089] The term "computational complexity," when applied to an algorithm, is a term of art and refers to the amount of computing resources required to execute the algorithm. Computational complexity is usually measured in terms of the number of elementary operations that are performed during the execution of the instructions of the algorithm.

[0090] Therefore, the computational complexity roughly approximates the level of intensity at which the input is processed to obtain the output. Therefore, it is of course a general advantage that the computational complexity of the algorithm is minimized for the same reliability or accuracy of the output. In general, the more reliable, accurate, or high-resolution the input or initialization, the lower the required computational complexity of the algorithm to obtain an output of a given accuracy or reliability. The lower the computational complexity, the faster the algorithm and the lower the processing cost. Therefore, the inventors have recognized that it would be beneficial to be able to adjust the computational complexity of the applied segmentation algorithm depending on the assumed accuracy or reliability of the input or model initialization.

[0091] In a preferred embodiment, the method includes a process for checking the reliability or quality of the previous segmentation result to be used to initialize the segmentation model for the current segmentation operation.

[0092] In a preferred embodiment, the computational complexity at which the segmentation model is implemented may be adjusted depending on the determined reliability or quality of a previous segmentation result to be used to initialize the segmentation model for the current segmentation operation. If the reliability or quality is found to be lower, a higher computational complexity may be used. If the reliability or quality is found to be higher, a lower computational complexity may be used. This aims to provide an optimal balance between the segmentation accuracy of the current frame and the processing cost of performing the segmentation.

[0093] Therefore, to state this more explicitly, in some embodiments, the segmentation operation may further include retrieving a quality assessment algorithm configured to receive the segmentation result as an input and determine a quality classification of the segmentation result as an output, applying the quality assessment algorithm to the at least one segmentation result of a previous medical image data frame to obtain a quality classification, and configuring a computational complexity level of the segmentation model based on the quality classification. The steps may be applied in a different order. For example, the quality assessment algorithm may be applied to the segmentation result before it is stored in the segmentation data store, and the method includes retrieving the segmentation result together with its quality classification from the data store and configuring a computational complexity level of the segmentation model based on the quality classification.

[0094] Regarding the quality assessment algorithm, one possible implementation could be as follows:

[0095] In some embodiments, the quality assessment algorithm may be configured to generate a quality classification of the segmentation result for each frame indicating the predicted success in using the segmentation result to extract one or more 2D planes from the respective frame.

[0096] In some embodiments, the quality assessment algorithm may be a machine learning algorithm, for example an artificial neural network.

[0097] As described above, in a preferred embodiment, the system 30 includes a user interface having a controllable display. The display may be controlled to present to the user the results of the segmentation operation for one or more of the image data frames. In some embodiments, the display may be controlled to display one or more 2D planes extracted from one or more of the received 3D image data frames. Thus, the user interface can provide real-time feedback to the user as they engage in the ongoing procedure of performing a scan. Thus, the user can modify the performance of the scan in a continuous machine-human interaction process based on the information provided on the display. For example, the user can view the extracted planes and use them to adjust the positioning of the ultrasound probe. As a further example, the user can review the results of the quality assessment algorithm applied to the segmentation of the image frames and see if the segmentation results are currently poor. In response, the user can adjust the positioning or movement of the ultrasound probe or adjust acquisition settings (e.g., contrast or field of view settings). In some embodiments, the user interface may be controlled to allow the user to input user control commands for configuring various aspects of the segmentation operation described above. For example, the user may be provided with the option to re-trigger segmentation "from scratch", in which a general initialization is used instead of initialization based on prior segmentation results.

[0098] In some embodiments, the received imaging data frames are 3D image data frames. In some embodiments, the method may further include using a segmentation of at least one of the received 3D image data frames to extract one or more 2D planes through a 3D volume covered by the 3D image frames. In particular, in some embodiments, based on the anatomical context provided by the segmentation, the method may further include determining the position of a defined set of 2D views within the 3D volume. For example, the position of a standard view plane may be encoded in terms of a set of segmentable anatomical elements located within the view plane. The segmentation result provides a set of segmentable elements labeled with respect to their position in the 3D frame volume. A linear regression may then be performed through the segmented elements to determine the closest fitting plane through the 3D frame volume that matches the standard view plane. Reference EP 2994053 B1 describes a method for segmenting a 3D image and extracting one or more 2D planes using the segmentation result.

[0099] In a further embodiment, based on the segmentation results for a given 3D frame, an anatomical coordinate system can be fitted to the 3D frame, where a set of anatomical elements having known relative positioning to one another are identified in the frame, and a coordinate system is fitted to the frame based on the positions of the anatomical elements in the frame. The positions of a set of target planes can then be identified based on the known coordinate ranges of the planes and based on the fitted coordinate system.

[0100] The set of available planes depends on the imaging modality and the region being scanned (eg, TTE probe vs. TEE probe, or left ventricle vs. right ventricle focus).

[0101] Once the target plane has been identified, in some embodiments the method may further comprise extracting the plane from the 3D frame by slicing the 3D frame. In this regard, reference is again made to document EP 2994053 B1.

[0102] Referring again to the optional user interface 52, this may be controlled to provide a display output including the extracted 2D plane or planes.

[0103] In some embodiments, the user interface 52 may be controlled to provide a visual representation of the extracted 2D anatomical plane for each 3D image frame.

[0104] In some embodiments, the user interface 52 may be controlled to provide a visual representation of orthogonal slices of the original 3D Cartesian image volume in addition to the anatomical 2D plane extracted from the 3D volume (e.g., in a second display window presented on the user interface display).

[0105] In some embodiments, the user interface 52 may be controlled to provide visual representations of anatomical 2D planes extracted from multiple different 3D image frames. For example, in some embodiments, a set of 2D planes extracted from a single previous 3D frame may be shown in one provided display window, which remains static. Additionally, a second display window may be provided that displays 2D planes extracted from the current 3D frame. In some embodiments, the second display window showing the 2D planes extracted from the current frame may be presented only if the mesh distance between the previous frame segmentation result and the current frame segmentation result exceeds a predetermined threshold. In some embodiments, the user interface may have configurable display settings that, for example, allow the user to optionally set the second display window (showing the 2D planes extracted from the current 3D frame) as the main default view. A control option may exist that allows these planes to be temporarily locked to a static view, preventing further updates, thereby providing the user with the opportunity to inspect the anatomical planes without any planar movement.

[0106] In some embodiments, the first and second display windows described above may be presented simultaneously in a side-by-side format, or alternatively, they may be presented in a switchable format that allows the user to switch views between the main view and the secondary view.

[0107] In some embodiments, for each frame, the user interface may be controlled to provide a quality classification generated by the quality assessment algorithm for each segmentation result. These may be presented, for example, in the form of a graphical display, for example, in the form of a color-coded graphical display. In some embodiments, the quality classification may be a binary pass / fail or success / fail classification. In some embodiments, the quality classification may be a graded score.

[0108] In some embodiments, the user interface may be configured to allow a user to manually adapt one or more settings or parameters of the performed method. For example, the user interface may allow a user to adjust parameters of a quality assessment algorithm, e.g., adjust a threshold of success criteria against which the quality assessment algorithm's quality classification is compared. In some embodiments, the user interface may be controlled to provide a user with a control element that allows the user to manually trigger re-segmentation of a given 3D frame. In some examples, the quality assessment algorithm is a machine learning model. In some embodiments, manual user retriggering of the segmentation algorithm may be used as information to train the model. Such manual adaptation aids in the continuous learning of the quality assessment algorithm.

[0109] Exemplary implementations of methods according to a particular set of embodiments will now be described as illustrations of the above-described concepts of the present invention. It will be understood that not all features of this particular set of embodiments are essential to the inventive concepts, but are set forth to aid understanding and provide examples to illustrate the inventive concepts.

[0110] According to this particular set of embodiments, the segmentation model includes three different segmentation algorithms that can be selectively applied depending on the situation.

[0111] In particular, the segmentation model is selectively operable in different modes, in which the model performs segmentation at different levels of computational complexity, each mode using a different respective set of one or more segmentation algorithms for anatomical segmentation. Specifically, according to this set of embodiments, the segmentation model includes three different types of segmentation algorithms, called Alg. 1, Alg. 2, and Alg. 3: The segmentation model can be run in each of at least two different modes: In the first mode, segmentation algorithm Alg. 1 is applied; in the second mode, segmentation algorithm Alg. 2 or Alg. 3 is applied; and the segmentation algorithms differ in computational complexity.

[0112] Regarding Alg. 1, this segmentation algorithm is configured to be initialized with a general initialization state and has the first level of computational complexity. Because it does not require initialization with a prior segmentation result, it can be applied to, for example, the initial first 3D frame. Of the three segmentation algorithms, this has the highest computational complexity and therefore provides the most robust segmentation. Therefore, it is also the slowest algorithm.

[0113] Regarding Alg. 2, the algorithm is designed to be initialized with a previous segmentation result, which defines the initial shape for the mesh to be fitted to the 3D image frame. However, Alg. 2 also includes a shape finder function to determine the approximate shape of the mesh, which can correct any substantial errors in the initialized mesh shape. Alg. 2 also increases the number of adaptation iterations compared to Alg. 3, making it more robust. However, using a previous segmentation result for initialization allows the algorithm to have fewer adaptation iterations than Alg. 1, since the mesh typically requires fewer adjustments compared to initialization using a general mesh shape.

[0114] Regarding Alg.3, this segmentation algorithm has the lowest computational complexity of the three algorithms. This algorithm is also designed to be initialized with previous segmentation results. For example, initialization involves setting the initial state of the mesh before fitting it to image data from a 3D image data frame. Therefore, this algorithm is the fastest of the three algorithms. This algorithm is sometimes referred to as a fast "follow-up" segmentation algorithm. This algorithm requires fewer adaptive iterations than Alg.2. Furthermore, this algorithm does not include shape finder functionality and therefore assumes that the initialized mesh shape is substantially accurate and requires only minor modifications.

[0115] During operation, the computational complexity level of the segmentation model can be configured by switching the model between different computational complexity modes.

[0116] FIG. 3 outlines the steps of an exemplary workflow according to one set of embodiments.

[0117] The method can be understood as including a first stage 82 and a second stage 84. The first stage includes configuring a segmentation model in a first mode, where the segmentation model utilizes a first segmentation algorithm, Alg. 1. In this mode, the segmentation model operates at its highest computational complexity. The method also includes applying an initialization procedure (22) to initialize the segmentation model thus configured. In this example of the initialization procedure, the segmentation model is initialized with a general initialization state. The segmentation model is applied to a first received frame in the first mode (24) to obtain a segmentation result. A quality assessment algorithm (already described above) is then applied to the segmentation result (92) to derive a quality classification of the segmentation result. This, when used to guide 2D plane extraction in a subsequent operation, can indicate the predicted success / failure of the segmentation. If the quality classification passes the predefined criteria or success criteria, the method may proceed to a second stage 84 after storing 26 the segmentation results for the first frame in the segmentation data store 28. If the quality classification does not meet the predefined success criteria, the method remains in the first stage, the segmentation model is configured in the first (high complexity) mode, and the steps of the first stage are repeated for new 3D frames until the success criteria are met.

[0118] In a second stage 84, the segmentation model may be applied in a second mode using the second segmentation algorithm Alg.2 or the third segmentation algorithm Alg.3.

[0119] Segmentation algorithms Alg.2 and Alg.3 are each configured to be initialized with segmentation results from at least one previous medical imaging data frame of the current imaging session. Alg.2 has lower computational complexity than Alg.1. Alg.3 has lower computational complexity than Alg.1 and Alg.2. Thus, in stage 2, a segmentation model using either Alg.2 or Alg.3 is initialized with segmentation results for at least one previous medical image data frame stored in segmentation data store 28 (22). The segmentation model thus initialized is then applied to the current medical image data frame (24). A quality assessment algorithm is then applied to the segmentation results derived for the current frame (92). In some embodiments, a resulting quality classification for the segmentation result is recorded in segmentation data store 28 in combination with the segmentation result 26 itself. In some embodiments, the segmentation result for the current frame is recorded in the segmentation data store only if the segmentation result meets predetermined success criteria.

[0120] In a preferred embodiment, the method includes, in a second stage 84, a step of checking the quality classification of previous segmentation results in a data store used to initialize the model and select the computational complexity of the model based on the quality classification (by choosing between Alg. 2 and Alg. 3). In particular, it is proposed that the data store records a quality classification for each recorded segmentation result. If the quality classification does not meet the criteria, a segmentation algorithm with higher computational complexity (e.g., Alg. 2) may be applied, and if the quality classification meets the criteria, a segmentation algorithm with lower computational complexity (e.g., Alg. 3) may be applied.

[0121] The method steps of stage 2 (84) may be repeated for each of the series of frames until the imaging procedure is complete.

[0122] Therefore, the basic operational flow of the method according to FIG. 3 can be summarized as follows: In a first step 82, an initial segmentation of the first frame is performed using Alg. 1 (i.e., using the segmentation model of the first mode). This is designed to be initialized with a general initialization state, e.g., a general mesh for the anatomical object. Optionally, this step is repeated for one or more subsequent frames until the quality assessment algorithm indicates a quality classification that meets the success criteria for the first time. Subsequently (step 2 84), segmentation algorithms Alg. 2 and / or Alg. 3 are used. These are designed to be initialized using segmentation results from at least one previous frame, e.g., the segmentation result for the first frame calculated using Alg. 1 (which has the highest computational complexity). Therefore, for the initialization of Alg. 2 or Alg. 3, the general or average mesh is replaced by the previous successful segmentation result. Thus, the patient's individual shape for the anatomical object can be utilized by Alg. 2 and Alg. 3. Each subsequent view is segmented using Alg.2 or Alg.3 depending on the previous quality classification.

[0123] In particular, if the previous segmentation result is classified as meeting the success criteria, the fast follow-up segmentation model Alg. 3 is applied. In contrast to the other models, Alg. 3 does not have a shape finder and has a reduced number of adaptive iterations, i.e., this model is tuned to adapt the mesh to only minor frame-to-frame changes. If the previous segmentation is classified as not meeting the success criteria, the intermediate "from scratch" segmentation model Alg. 2 can be applied. Alg. 2 includes shape finder functionality and has an increased number of adaptive iterations, but still uses the patient-specific mesh shape obtained in the pre-segmentation result as a starting point. Thus, both Alg. 2 and Alg. 3 use the mesh shape found in the previous segmentation result (preferably found in Stage 1 using Alg. 1), but their respective computational complexities differ. Alg. 2 has a shape finder and therefore can more substantially recalculate the mesh shape / morphology compared to the initialized shape, and also has additional adaptive iterations compared to Alg. 3. Alg.3 does not have a shape finder and can only make minor changes to the mesh shape compared to Alg.2.

[0124] The complexity and runtime requirements of model-based segmentation generally depend on the number of adaptation iterations.

[0125] It will be recognized that there is a trade-off between model complexity and segmentation accuracy. The segmentation algorithms Alg. 1, Alg. 2, and Alg. 3 are tuned to enable real-time processing while decreasing computational complexity from Alg. 1 to Alg. 3. The slow and robust "initial" segmentation model Alg. 1 is the most complex model because knowledge transfer from previous image frames is not provided during segmentation, i.e., there is no initialization using segmentation results from previous frames. However, in preferred embodiments, this algorithm may be applied only once to set an initial starting point for follow-up segmentation. In some embodiments, the quality assessment algorithm can be repeated for one or more additional frames until it first indicates a quality classification that meets the success criteria.

[0126] With regard to the quality assessment algorithm, it may in some embodiments be adapted to generate a binary (Boolean) classification, e.g., pass / fail or pass / fail. In some embodiments, the quality assessment algorithm is configured to generate a numerical score as the output classification. This score may be compared, for example, to a threshold value to assess whether the classification meets a success criterion. In some embodiments, the input to the quality assessment algorithm may be the segmentation result alone. In further embodiments, the input to the quality assessment algorithm is the segmentation result combined with the current 3D image frame. In further embodiments, the quality assessment algorithm may be configured to receive as input a previous segmentation result in addition to the previous imaging frame from which the segmentation was derived.

[0127] The quality assessment algorithm may in some embodiments be a trained machine learning algorithm, such as an artificial neural network, such as a convolutional neural network.

[0128] The quality assessment algorithm can be trained in a supervised manner to generate an output prediction or classification of the quality of the segmentation, referred to herein as a quality classification. The training of the algorithm can be performed using training data including segmentation results annotated with manually assessed quality classifications (e.g., binary pass / fail classifications or success / fail scores). In some embodiments, each input training data entry can include both a segmentation result and an ultrasound image frame, annotated with a manually assessed classification regarding the degree to which the segmentation correctly represents the associated anatomical structures depicted in the image frame. Thus, the quality assessment algorithm can be trained to generate a decision indicating whether an input combination of a previous 3D ultrasound image frame and an associated previous segmentation result (mesh) derived for the image frame leads to significant 2D anatomical plane extraction. Such a module can be trained on manually annotated data (e.g., a mixture of real and artificially corrupted segmentations). Alternatively, existing 2D view classifiers can be used to automatically determine the quality of the extracted planes, as described, for example, in document EP 4080449 A1.

[0129] In some embodiments, the quality assessment algorithm may include an initial stage that involves extracting one or more region features of the segmentation result, which form input to the machine learning algorithm. For example, the one or more region features may be features that, for a given region of the segmentation mesh, characterize the degree of support for the mesh region in the underlying image from which it was generated and / or the degree of interpolation used in forming the mesh region. The machine learning algorithm may be trained in a supervised manner to map these region features to a quality classification.

[0130] As previously discussed, the method may include comparing the output quality classification produced by the quality assessment algorithm with a predefined success criterion. In some embodiments, the success criterion may be a threshold value for the quality classification. In some embodiments, the success criterion may be determined in part based on user input. In other words, the success criterion may be user-configurable. This effectively allows a user to adjust the level of confidence in the segmentation for the current frame, and therefore the level of stability of the inter-frame segmentation.

[0131] In some embodiments, the initialization procedure may include initializing the segmentation model using segmentation results from multiple previous imaging frames. For example, in some embodiments, an anatomical structure motion trajectory may be estimated based on segmentation results for image frames in a time series preceding the current image frame to generate an estimated mesh shape and position for the current frame, and the estimated mesh shape and position for the current frame may be used to initialize the current frame.

[0132] In some embodiments, the method includes storing each segmentation result acquired for each frame along with a quality classification of the segmentation result. In some embodiments, the method can include an acquisition guidance function, where guidance information is presented to a user via a user interface to guide the user to re-acquire at least one previous image frame in which a stored segmentation result has an associated quality classification that fails to meet a predefined criterion. For example, the guidance information can be information to guide a user in moving a probe to a suitable position for acquiring the associated image frame.

[0133] Embodiments of the present invention may find particularly advantageous application in, for example, ultrasound imaging of the human heart. However, the principles of the present invention are in no way limited to this field. Embodiments of the present invention are generally applicable to any non-ionizing imaging modality that allows real-time acquisition of images, for example, using a fixed geometric configuration, e.g., optical coherence tomography (OCT). Furthermore, the target anatomical object of interest may be any part of the human body and is not limited to the heart. Other examples include, for example, fetal scanning, pathology, or veterinary applications.

[0134] Embodiments of the present invention may be used in the context of an image-guided acquisition system or a 3D-based stress echo examination system in some examples.

[0135] The application of segmentation establishes an anatomical context from the 3D volume. For example, the segmentation can employ the use of model-based segmentation. For at least one suitable exemplary method of model-based segmentation, see the article Ecabert, O et al. Automatic Model-Based Segmentation of Heart in CT Images, IEEE Transactions on Medical Imaging, 2008, 27, pp. 1189-1201.

[0136] As a further example, the segmentation can use a machine learning-based segmentation algorithm, for example, using a trained convolutional neural network to perform the segmentation. At least one suitable machine learning model for performing the segmentation is described in LI, Yuanwei, et al. "Standard plane detection in 3D fetal ultrasound using an iterative transformation network." In: International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, Cham, 2018. S. 392-400.

[0137] As mentioned above, certain embodiments may include an ultrasound acquisition device 54 and / or means for processing ultrasound echo data to derive further data.

[0138] The general operation of an exemplary ultrasound acquisition device will now be described in further detail with reference to FIG.

[0139] The system includes an array transducer probe 104 having a transducer array 106 for transmitting ultrasound waves and receiving echo information. The transducer array 106 may include CMUT transducers, piezoelectric transducers formed from materials such as PZT or PVDF, or any other suitable transducer technology. In this example, the transducer array 106 is a two-dimensional array of transducers 108 capable of scanning either a 2D plane or a three-dimensional volume of a region of interest. In another example, the transducer array may be a 1D array.

[0140] The transducer array 106 is coupled to a microbeamformer 112 that controls the reception of signals by the transducer elements. The microbeamformer is capable of at least partial beamforming of signals received by subarrays of transducers (commonly referred to as "groups" or "patches"), as described in U.S. Patent Nos. 5,997,479 (Savord et al.), 6,013,032 (Savord), and 6,623,432 (Powers et al.).

[0141] It should be noted that the microbeamformer is generally entirely optional. Additionally, the system includes a transmit / receive (T / R) switch 116, to which the microbeamformer 112 may be coupled to switch the array between transmit and receive modes and to protect the main beamformer 120 from high-energy transmit signals when the microbeamformer is not used and the transducer array is operated directly by the main system beamformer. The transmission of ultrasound beams from the transducer array 106 is directed by a transducer controller 118 coupled to the microbeamformer by the T / R switch 116 and a main transmit beamformer (not shown), which may receive input from user operation of a user interface or control panel 138. The controller 118 may include transmit circuitry configured to drive the transducer elements of the array 106 (either directly or via the microbeamformer) during transmit mode.

[0142] The functionality of the control panel 138 in this exemplary system may be facilitated by an ultrasound controller unit according to one embodiment of the present invention.

[0143] In a typical line-by-line imaging sequence, the beamforming system in the probe may operate as follows: During transmit, the beamformer (which may be a microbeamformer or a main system beamformer, depending on the implementation) activates the transducer array or a subaperture of the transducer array. A subaperture may be a one-dimensional line of transducers or a two-dimensional patch of transducers within a larger array. In transmit mode, the focus and steering of the ultrasound beam generated by the array or a subaperture of the array are controlled as described below.

[0144] Upon receiving backscattered echo signals from the subject, the received signals undergo receive beamforming (as described below) to align the received signals, and if subapertures are used, the subapertures are shifted, for example, by one transducer element. The shifted subapertures are then activated, and the process is repeated until all of the transducer elements in the transducer array have been activated.

[0145] For each line (or subaperture), the total receive signal used to form the associated line of the final ultrasound image is the sum of the voltage signals measured by the transducer elements of the given subaperture during the receive period. The resulting line signal, following the beamforming process, is commonly referred to as radio frequency (RF) data. Each line signal (RF data set) generated by the various subapertures then undergoes additional processing to generate a line of the final ultrasound image. The change in amplitude of the line signal over time contributes to the change in brightness of the ultrasound image with depth, with high amplitude peaks corresponding to bright pixels (or groups of pixels) in the final image. Peaks appearing near the beginning of the line signal represent echoes from shallow structures, while peaks appearing gradually later in the line signal represent echoes from structures deeper within the subject.

[0146] One of the functions controlled by the transducer controller 118 is the direction in which the beam is steered and focused. The beam may be steered straight ahead (orthogonal) from the transducer array, or at a different angle for a wider field of view. The steering and focusing of the transmit beam may be controlled as a function of the activation time of the transducer elements.

[0147] Two methods can be distinguished in general ultrasound data acquisition: plane wave imaging and "beam steering" imaging. The two methods are distinguished by the presence of beamforming in the transmit ("beam steering" imaging) and / or receive modes (plane wave imaging and "beam steering" imaging).

[0148] Looking first at the focusing function, by simultaneously activating all of the transducer elements, the transducer array generates a plane wave that diverges as it travels through the subject. In this case, the ultrasound beam remains unfocused. By introducing position-dependent time delays in the transducer activation, the wavefront of the beam can be focused to a desired point, called a focal zone. A focal zone is defined as a point where the lateral beamwidth is less than half the transmit beamwidth. In this way, the lateral resolution of the final ultrasound image is improved.

[0149] For example, if a time delay is applied to sequentially activate the transducer elements, starting with the outermost elements of the transducer array and finishing with the central elements, a focal zone is formed along the central elements at a predetermined distance from the probe. The distance of the focal zone from the probe varies depending on the time delay between each subsequent round of transducer element activation. After the beam passes through the focal zone, it begins to diverge, forming a far-field imaging region. Note that with a focal zone located close to the transducer array, the ultrasound beam rapidly diverges in the far field, leading to beamwidth artifacts in the final image. Typically, the near-field located between the transducer array and the focal zone shows little detail due to the large overlap of the ultrasound beams. Therefore, varying the position of the focal zone can result in significant changes in the quality of the final image.

[0150] Note that in transmit mode, only one focal point may be defined unless the ultrasound image is divided into multiple focal zones (each of which may have a different transmit focal point).

[0151] Also, upon receiving echo signals from within the subject, the above process can be reversed to achieve receive focusing. In other words, the input signals can be received by the transducer elements and subjected to electronic time delays before being passed to the system for signal processing. The simplest example of this is called delay-and-sum beamforming. It is possible to dynamically adjust the receive focus of the transducer array as a function of time.

[0152] Looking now at the function of beam steering, through the proper application of time delays to the transducer elements, it is possible to impart a desired angle to the ultrasound beam as it exits the transducer array. For example, by activating transducers on one side of the transducer array and then activating the remaining transducers in a sequence ending on the opposite side of the array, the wavefront of the beam is angled toward the second side. The magnitude of the steering angle relative to the normal to the transducer array depends on the magnitude of the time delay between the activation of subsequent transducer elements.

[0153] Furthermore, it is possible for the total time delay applied to each transducer element to focus the steering beam by the sum of both the focusing time delay and the steering time delay, in which case the transducer array is called a phased array.

[0154] For CMUT transducers that require a DC bias voltage for activation, the transducer controller 118 can be coupled to control a DC bias control 145 for the transducer array, which sets the DC bias voltage applied to the CMUT transducer elements.

[0155] For each transducer element in the transducer array, an analog ultrasound signal, typically referred to as channel data, enters the system via a receive channel. In the receive channel, a partially beamformed signal is generated from the channel data by the microbeamformer 112 and then passed to the main receive beamformer 120, where the partially beamformed signals from the individual transducer patches are combined into a fully beamformed signal, referred to as radio frequency (RF) data. The beamforming performed at each stage may be performed as described above or may include additional functions. For example, the main beamformer 120 may have 128 channels, each of which receives partially beamformed signals from patches of tens or hundreds of transducer elements. In this way, signals received by thousands of transducers in the transducer array can efficiently contribute to a single beamformed signal.

[0156] The beamformed received signals are coupled to a signal processor 122. The signal processor 122 can process the received echo signals in various ways, such as bandpass filtering, decimation, I and Q component separation, and harmonic signal separation, which acts to separate linear and nonlinear signals to enable identification of nonlinear (harmonic of the fundamental frequency) echo signals returned from tissue and microbubbles. The signal processor can also perform additional signal enhancements, such as speckle reduction, signal combining, and noise removal. The bandpass filters in the signal processor can be tracking filters whose passbands slide from higher to lower frequency bands as echo signals are received from increasing depths, thereby eliminating noise at higher frequencies from greater depths that typically lack anatomical information.

[0157] The beamformers for transmit and receive can be implemented with different hardware and have different functions. Of course, the receiver beamformer is designed to take into account the characteristics of the transmit beamformer. In Figure 4, for simplicity, only the receiver beamformers 112, 120 are shown. In a complete system, there will also be a transmit chain with a transmit microbeamformer and a main transmit beamformer.

[0158] The function of the microbeamformer 112 is to provide an initial combination of signals to reduce the number of analog signal paths, which is typically performed in the analog domain.

[0159] Final beamforming occurs in the main beamformer 120, typically after digitization.

[0160] The transmit and receive channels use the same transducer array 106 with a fixed frequency band. However, the bandwidth occupied by the transmit pulses can vary depending on the transmit beamforming used. The receive channels can capture the entire transducer bandwidth (classical approach) or use bandpass processing to extract only the bandwidth containing the desired information (e.g., harmonics of the main harmonic).

[0161] The RE signals may then be coupled to a B-mode (i.e., intensity mode, or 2D imaging mode) processor 126 and a Doppler processor 128. The B-mode processor 126 performs amplitude detection on the received ultrasound signals for imaging structures within the body, such as organ tissues and blood vessels. In line-by-line imaging, each line (beam) uses the associated RE signal amplitude to generate an intensity value to be assigned to a pixel in the B-mode image. The exact location of a pixel in the image is determined by the location of the amplitude measurement associated with the RE signal and the number of lines (beams) of the RF signal. B-mode images of such structures can be formed in harmonic or fundamental imaging modes, or a combination of both, as described in U.S. Patents 6,283,919 (Roundhill et al.) and 6,458,083 (Jago et al.). The Doppler processor 128 processes temporally distinct signals resulting from tissue motion and blood flow for detection of moving objects, such as flowing blood cells, within the image field. The Doppler processor 128 typically includes a wall filter with parameters set to pass or reject echoes returned from selected types of material within the body.

[0162] The structural and motion signals generated by the B-mode and Doppler processors are coupled to a scan converter 132 and a multiplanar reformatter 144. The scan converter 132 arranges the echo signals in the spatial relationship from which they were received and in the desired image format. In other words, the scan converter acts to convert the RF data from a cylindrical coordinate system to a Cartesian coordinate system suitable for displaying ultrasound images on the image display 140. For a B-mode image, the brightness of a pixel at a given coordinate is proportional to the amplitude of the RF signal received from that location. For example, the scan converter can arrange the echo signals in a two-dimensional (2D) sector format or a pyramidal three-dimensional (3D) image. The scan converter can overlay a B-mode structural image with a color corresponding to the motion at a point within the image field, where the Doppler estimated velocity generates a given color. As described in U.S. Patent 6,443,896 (Detmer), the combined B-mode structural image and color Doppler image depict tissue motion and blood flow within the structural image field. The multiplanar reformatter converts echoes received from points in a common plane within a volumetric region of the body into an ultrasound image of that plane. The volume renderer 142 converts the echo signals of the 3D data set into a projected 3D image viewed from a given reference point as described in U.S. Patent 6,530,885 (Entrekin et al.).

[0163] The 2D or 3D images are coupled from the scan converter 132, multiplanar reformatter 144, and volume renderer 142 to the image processing unit 130 for further enhancement, buffering, and temporary storage for optional display on the image display 140. The imaging processor may be adapted to remove certain imaging artifacts from the final ultrasound image, such as acoustic shadowing caused by strong attenuators or refraction, back enhancement caused by weak attenuators, reverberation artifacts where highly reflective tissue interfaces are located nearby, etc. Additionally, the image processor may be adapted to process certain speckle reduction functions to improve the contrast of the final ultrasound image.

[0164] In addition to being used for the images, the blood flow values ​​produced by the Doppler processor 128 and the tissue structure information produced by the B-mode processor 126 are coupled to a quantification processor 134. The quantification processor produces measurements of different flow conditions, such as volumetric velocity of blood flow, as well as structural measurements, such as organ size and gestational age. The quantification processor can receive input from a user control panel 138, such as the points within the anatomy of the image at which measurements are to be taken.

[0165] Output data from the quantification processor is coupled to a graphics processor 136 for reproducing measurement graphics and values ​​with images on a display 140 and for audio output from the display 140. The graphics processor 136 can also generate graphic overlays for display with the ultrasound images. These graphic overlays can include standard identifying information such as the patient name, the date and time of the image, and imaging parameters. For these purposes, the graphics processor receives inputs such as the patient name from a user interface 138. The user interface is also coupled to a transmit controller 118 for controlling the generation of ultrasound signals from the transducer array 106 and, therefore, the generation of images produced by the transducer array and the ultrasound system. The transmit control function of the controller 118 is only one of the functions performed. The controller 118 also takes into account the operating mode (provided by the user) and the corresponding required transmitter and bandpass settings in the receiver analog-to-digital converter. The controller 118 can be a state machine with fixed states.

[0166] The user interface is also coupled to a multiplanar reformatter 144 for selection and control of multiple multiplanar reformat (MPR) image planes that can be used to perform quantified measurements in the image field of the MPR image.

[0167] The ultrasound system described above may be operatively coupled to the processing unit 32. For example, the ultrasound system described above may, in some instances, be used to implement the ultrasound acquisition unit 54 of the system 30 shown in FIG.

[0168] The above-described embodiments of the present invention employ a processing device. The processing device may generally comprise a single processor or multiple processors. It may be located within a single containing device, structure, or unit, or may be distributed among several different devices, structures, or units. Thus, reference to a processing device adapted or configured to perform a particular step or task may correspond to that step or task being performed by any one or more of several processing components, alone or in combination. Those skilled in the art will understand how such a distributed processing device may be implemented. The processing device may include a communication module or input / output for receiving data and outputting data to further components.

[0169] The one or more processors of a processing unit can be implemented in a number of ways using software and / or hardware to perform the various functions required. A processor typically uses one or more microprocessors that can be programmed using software (e.g., microcode) to perform the necessary functions. A processor can also be implemented as a combination of dedicated hardware to perform some functions and one or more programmed microprocessors and associated circuitry to perform other functions.

[0170] Examples of circuitry that may be used in various embodiments of the present disclosure include, but are not limited to, conventional microprocessors, application specific integrated circuits (ASICs), and field programmable gate arrays (FPGAs).

[0171] In various implementations, a processor may be associated with one or more storage media, such as volatile and non-volatile computer memory, including RAM, PROM, EPROM, and EEPROM. The storage media may be encoded with one or more programs that, when executed on the one or more processors and / or controllers, perform the necessary functions. The various storage media may be fixed within the processor or controller, or may be portable, such that the one or more programs stored thereon can be loaded into the processor.

[0172] Variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality.

[0173] A single processor or other unit may fulfill the functions of several items recited in the claims.

[0174] The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

[0175] The computer program may be stored / distributed on a suitable medium, such as an optical storage medium or a solid-state medium, supplied together with or as part of other hardware, but may also be distributed in other forms, such as over the Internet or other wired or wireless telecommunications systems.

[0176] It should be noted that when the term "adapted for" is used in the claims or description, the term "adapted for" is intended to be equivalent to the term "configured for."

[0177] Any reference signs in the claims should not be construed as limiting the scope.

Claims

1. 1. A computer-implemented method comprising: receiving, during an imaging session, a medical imaging data stream having a series of medical imaging data frames of an anatomical object; retrieving a segmentation model having one or more anatomical segmentation algorithms, the segmentation model configured to receive the medical imaging data stream as an input and to generate a segmentation result as an output; performing a segmentation operation on each medical imaging data frame of the series, applying an initialization procedure to initialize the segmentation model; applying the segmentation model to a current frame of medical imaging data to obtain a segmentation result for the current frame of medical imaging data; accessing a segmentation data store and storing segmentation results for the current medical imaging data frame in the segmentation data store; and The initialization procedure includes: accessing the segmentation data store; checking whether the segmentation data store contains segmentation results for at least one previous medical imaging data frame of the current imaging session; If the segmentation result is included, identifying the stored segmentation results for at least one previous medical imaging data frame of a current imaging session in the data store; initializing the segmentation model using segmentation results for at least one previous medical imaging data frame of the current imaging session recorded in the segmentation data store; and The segmentation operation further comprises configuring a computational complexity level of the segmentation model before applying the segmentation model; the computational complexity level is configured at least in part dependent on whether the segmentation data store includes segmentation results for at least one previous medical imaging data frame of the current imaging session, the computational complexity level being configured at a lower level when the segmentation data store includes segmentation results for at least one previous medical imaging data frame of the current imaging session than when the segmentation data store does not include segmentation results for at least one previous medical imaging data frame of the current imaging session. Computer-implemented methods.

2. The initialization procedure includes: if the segmentation data store does not contain segmentation results for at least one previous medical imaging data frame of the current imaging session, initializing the segmentation model using a general initialization state.

2. The method of claim 1, comprising:

3. 3. The method of claim 1, wherein configuring the computational complexity level of the segmentation model comprises switching the model between different computational complexity modes, in each mode a different respective set of one or more segmentation algorithms is used for the anatomical segmentation.

4. the segmentation model is configurable in at least a first mode and one or more further modes; In the first mode, the segmentation model is configured at a first computational complexity level and initialized using a common initialization state; in each of the one or more further modes, the segmentation model is configured at a respective further computational complexity level lower than the first computational complexity level and is initialized using segmentation results for previous imaging data frames stored in the data store. The method of claim 3.

5. The method of claim 4 , wherein the one or more further modes include at least two further modes, each having a different respective computational complexity.

6. The segmentation operation further comprises: retrieving a quality assessment algorithm configured to receive the segmentation result as an input and to determine a quality classification for the segmentation result as an output; applying the quality assessment algorithm to the at least one segmentation result for a previous medical imaging data frame to obtain a quality classification; configuring a computational complexity level of the segmentation model based on the quality classification; 6. The method according to claim 1, comprising:

7. The method comprises a first step and a second step, The first stage comprises: retrieving a quality assessment algorithm configured to receive the segmentation result as an input and to determine a quality classification for the segmentation result as an output; applying the segmentation model in the first mode to each received frame to obtain a segmentation result; applying said quality assessment algorithm to each segmentation result; continuing in the first stage until a quality classification of the at least one frame meets a predefined success criterion; and The second stage comprises: applying the segmentation model in one or more of the further modes to each received frame.

6. The method of claim 4 or 5, or claim 6 when dependent on claim 4 or 5, comprising:

8. 8. The method of claim 1, wherein one or more algorithms applied by the segmentation model perform a step of fitting a mesh to the medical imaging data, and wherein initializing the model comprises configuring an initial state of the mesh.

9. The method of claim 1 , wherein the anatomical object is the heart or a part thereof.

10. The method according to any one of claims 1 to 9, wherein the method further comprises generating a control signal for controlling a user interface to display a visualization of each segmentation result.

11. The method of claim 10 , further comprising controlling the user interface to display a representation of a quality classification of each segmentation result.

12. A computer program product comprising computer program code configured, when executed on a processor, to cause the processor to perform a method according to any one of claims 1 to 11.

13. A processing device, an input / output unit; one or more processors, receiving, at said input / output during an imaging session, a medical imaging data stream comprising a series of medical imaging data frames of an anatomical object; retrieving a segmentation model having one or more anatomical segmentation algorithms, the segmentation model configured to receive medical imaging data frames as input and generate segmentation results as output; For each medical imaging data frame in said series: applying an initialization procedure to initialize the segmentation model; applying the segmentation model to the current frame of medical imaging data to obtain a segmentation result for the current frame of medical imaging data; accessing a segmentation data store and storing segmentation results for the current medical imaging data frame in the segmentation data store; performing a segmentation operation having the following steps: The initialization procedure includes: accessing the segmentation data store; checking whether the segmentation data store contains segmentation results for at least one previous medical imaging data frame of the current imaging session; If it contains identifying the stored segmentation results for at least one previous medical imaging data frame of a current imaging session in the segmentation data store; initializing the segmentation model using segmentation results for at least one previous medical imaging data frame of the current imaging session recorded in the segmentation data store; The segmentation operation further comprises configuring a computational complexity level of the segmentation model before applying the segmentation model; the computational complexity level is configured at least in part dependent on whether the segmentation data store includes segmentation results for at least one previous medical imaging data frame of the current imaging session, the computational complexity level being configured at a lower level when the segmentation data store includes segmentation results for at least one previous medical imaging data frame of the current imaging session than when the segmentation data store does not include segmentation results for at least one previous medical imaging data frame of the current imaging session. Processing equipment.

14. a medical imaging device; a processing device according to claim 13 operatively coupled to the medical imaging device to receive the medical imaging data stream; A system having:

Images