System and method for extended view medical imaging analysis

Abstract

The present disclosure provides a system for generating a report that provides an extended view 3D model based on medical images of a subject acquired over at least two partial views. The system includes a processor that is configured to access first and second sets of partial view medical images of the subject and segment the first and second sets of partial view medical images to generate first and second partial 3D models. The processor is further configured to identify an overlapping region in the first and second sets of partial view images, determine a spatial transformation, and apply the spatial transformation to produce an extended view 3D model that aligns the first and second partial 3D models.

Description

SYSTEM AND METHOD FOR EXTENDED VIEW MEDICAL IMAGING ANALYSISCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application is based on, claims priority to, and incorporates herein by reference in its entirety, US Provisional Application Serial No. 63 / 534,962, Filed August 28, 2023, and entitled “SYSTEM AND METHOD FOR EXTENDED VIEW MEDICAL IMAGING ANALYSIS.”BACKGROUND

[0002] Most magnetic resonance imaging (MRI) scans use small fields of view (FOV) to focus on imaging specific organs, such as the brain, heart, spinal cord, knee, and so forth. However, imaging the full musculoskeletal system requires much larger imaging volumes. For example, it may be desirable to characterize the muscle and fat of the full shoulder region or both upper and lower extremities. Most MRI systems are not capable of imaging such large FOVs in a single scan. Thus, methods are needed to provide complete analysis of large anatomical regions.SUMMARY OF THE DISCLOSURE

[0003] The foregoing and other aspects and advantages of the present disclosure will appear from the following description. In the description, reference is made to the accompanying drawings that form a part hereof, and in which there is shown by way of illustration a preferred embodiment. This embodiment does not necessarily represent the full scope of the invention, however, and reference is therefore made to the claims and herein for interpreting the scope of the invention. Like reference numerals will be used to refer to like parts from Figure to Figure in the following description.

[0004] In one configuration, the present disclosure provide a system for generating a report that provides an extended view 3D model based on medical images of a subject acquired with a medical imaging system over at least two partial views. The system includes a processor that is configured to access first and second sets of partial view medical images of the subject. The processor is configured to segment the first and second sets of partial view medical images to generate first and second partial 3D models, respectively. The processor is further configured to identify an overlapping region in the first and second sets of partial view images using at least one of the first plurality of anatomical structures or the second plurality of anatomicalstructures, determine a spatial transformation using the overlapping region, and apply the spatial transformation to produce an extended view 3D model that aligns the first and second partial 3D models. The processor can generate a report that includes the extended view 3D model.

[0005] In accordance with another aspect of the present disclosure a method is provided for generating a report providing an extended view 3D model based on medical images of a subject acquired with a medical imaging system over at least two partial views. The method includes accessing at least a first set of partial view medical images and a second set of partial view medical images of the subject and segmenting the first set of partial view medical images to generate a first partial 3D model, wherein the first partial 3D model comprises delineations of a first plurality of anatomical structures. The method also includes segmenting the second set of partial view medical images to generate a second partial 3D model, wherein the second partial 3D model comprises delineations of a second plurality of anatomical structures. The method further includes identifying an overlapping region in the first and second sets of partial view images that is common to the first and second sets of partial view images using at least one of the first plurality of anatomical structures or the second plurality of anatomical structures and determining a spatial transformation using the overlapping region. The method also includes applying the spatial transformation to produce an extended view 3D model that aligns the first partial 3D model with the second partial 3D model and generating a report comprising the extended view 3D modelBRIEF DESCRIPTION OF THE DRAWINGS

[0006] FIG. 1 is a block diagram of a non-limiting example MRI system that can implement the methods described in accordance with the present disclosure.

[0007] FIG. 2A is a flowchart of non-limiting example steps for a method to provide extended view images and analysis.

[0008] FIG. 2B is a flowchart of non-limiting example steps for a method to provide extended view images and analysis.

[0009] FIG. 3A illustrates an example application of a method to provide extended view images of the upper body.

[0010] FIG. 3B illustrates a further example application of a method to provide extended view images of the full body.

[0011] FIG. 3C illustrates example images produced using a method to provideextended view analysis of the full body.

[0012] FIG. 4A illustrates a first example report that may be generated in accordance with the present disclosure.

[0013] FIG. 4B illustrates a second example report that may be generated in accordance with the present disclosure.

[0014] FIG. 4C illustrates a third example report that may be generated in accordance with the present disclosure.

[0015] FIG. 4D illustrates a fourth example report that may be generated in accordance with the present disclosure.

[0016] FIG. 5 is a block diagram of a system for automatically segmenting bone, fat, and / or tissue using image data in accordance with some implementations of the disclosed subject matter.

[0017] FIG. 6 is a block diagram showing an example of hardware that can be used to implement the system of FIG. 5 in accordance with some implementations of the disclosed subject matter.DETAILED DESCRIPTION

[0018] The present disclosure provides systems and methods to combine medical images, such as magnetic resonance images (MRI), acquired at adjacent fields of view (FOVs) to create images and analysis of an extended view of the musculoskeletal system. Providing extended images and analysis can include generating a report for a desired contiguous volume of interest that is larger than a single acquired FOV. As non-limiting examples, such contiguous volume of interest may include the full shoulder region, the full right and left thorax regions, the combined upper and lower extremities, or a full-body region. Providing such extended or comprehensive analyses may advantageously allow for comparison between various anatomical regions and convenient interpretation of large anatomical regions. For example, the report can provide a comprehensive analysis of the shoulders and upper extremities that allows for comparison between the right arm and shoulders with the left arm and shoulders. Additionally, the extended views can allow for comprehensive analysis of athletic performance, injury recovery, sarcopenia associated with aging, and disease characterization of the entire musculoskeletal system, rather than a limited region. As a non-limiting example, the extended analysis could help a care team identify deficits in a patient’s ankle that are causing hip pain, where standard imaging and analysis may be limited to the hip region. Asanother example, the extended model could provide a comprehensive view of the kinematic chain of an athlete, such as a pitcher who uses their arms to throw a ball while simultaneously twisting their trunk and planting their leg.

[0019] The process can be fully automated to circumvent the need for error-prone and time-consuming manual input, providing standardized and fast results. The report may contain images or analysis of various parts of the musculoskeletal system, which may include muscle, fat, bone, tendon, cartilage, ligaments, or other connective tissue.

[0020] Referring particularly now to FIG. 1, an example of an MRI system 100 that can implement the methods described here is illustrated. The MRI system 100 includes an operator workstation 102 that may include a display 104, one or more input devices 106 (e.g., a keyboard, a mouse), and a processor 108. The processor 108 may include a commercially available programmable machine running a commercially available operating system. The operator workstation 102 provides an operator interface that facilitates entering scan parameters into the MRI system 100. The operator workstation 102 may be coupled to different servers, including, for example, a pulse sequence server 110, a data acquisition server 112, a data processing server 114, and a data store server 116. The operator workstation 102 and the servers 110, 112, 114, and 116 may be connected via a communication system 140, which may include wired or wireless network connections.

[0021] The pulse sequence server 110 functions in response to instructions provided by the operator workstation 102 to operate a gradient system 118 and a radiofrequency (“RF”) system 120. Gradient waveforms for performing a prescribed scan are produced and applied to the gradient system 118, which then excites gradient coils in an assembly 122 to produce the magnetic field gradients Gx, G .. , and G7that are used for spatially encoding magnetic resonance signals. The gradient coil assembly 122 forms part of a magnet assembly 124 that includes a polarizing magnet 126 and a whole-body RF coil 128.

[0022] RF waveforms are applied by the RF system 120 to the RF coil 128, or a separate local coil to perform the prescribed magnetic resonance pulse sequence. Responsive magnetic resonance signals detected by the RF coil 128, or a separate local coil, are received by the RF system 120. The responsive magnetic resonance signals may be amplified, demodulated, filtered, and digitized under direction of commands produced by the pulse sequence server 110. The RF system 120 includes an RF transmitter for producing a wide variety of RF pulses used in MRI pulse sequences. The RF transmitter is responsive to the prescribed scan and direction from the pulse sequence server 110 to produce RF pulses of the desired frequency, phase, andpulse amplitude waveform. The generated RF pulses may be applied to the whole-body RF coil 128 or to one or more local coils or coil arrays.

[0023] The RF system 120 also includes one or more RF receiver channels. An RF receiver channel includes an RF preamplifier that amplifies the magnetic resonance signal received by the coil 128 to which it is connected, and a detector that detects and digitizes the I and Q quadrature components of the received magnetic resonance signal. The magnitude of the received magnetic resonance signal may, therefore, be determined at a sampled point by the square root of the sum of the squares of the I and Q components:(i);

[0024] and the phase of the received magnetic resonance signal may also be determined according to the following relationship:(2).

[0025] The pulse sequence server 110 may receive patient data from a physiological acquisition controller 130. By way of example, the physiological acquisition controller 130 may receive signals from a number of different sensors connected to the patient, including electrocardiograph (“ECG”) signals from electrodes, or respiratory signals from a respiratory bellows or other respiratory monitoring devices. These signals may be used by the pulse sequence server 110 to synchronize, or “gate,” the performance of the scan with the subject’s heartbeat or respiration.

[0026] The pulse sequence server 110 may also connect to a scan room interface circuit 132 that receives signals from various sensors associated with the condition of the patient and the magnet system. Through the scan room interface circuit 132, a patient positioning system 134 can receive commands to move the patient to desired positions during the scan.

[0027] The digitized magnetic resonance signal samples produced by the RF system 120 are received by the data acquisition server 112. The data acquisition server 112 operates in response to instructions downloaded from the operator workstation 102 to receive the realtime magnetic resonance data and provide buffer storage, so that data is not lost by data overrun. In some scans, the data acquisition server 112 passes the acquired magnetic resonance data to the data processor server 114. In scans that require information derived from acquired magnetic resonance data to control the further performance of the scan, the data acquisition server 112 may be programmed to produce such information and convey it to die pulse sequence server 110. For example, during pre-scans, magnetic resonance data may be acquiredand used to calibrate the pulse sequence performed by the pulse sequence server 110. As another example, navigator signals may be acquired and used to adjust the operating parameters of the RF system 120 or the gradient system 118, or to control the view order in which k- space is sampled. In still another example, the data acquisition server 112 may also process magnetic resonance signals used to detect the arrival of a contrast agent in a magnetic resonance angiography (“MRA”) scan. For example, the data acquisition server 112 may acquire magnetic resonance data and processes it in real-time to produce information that is used to control the scan.

[0028] The data processing server 114 receives magnetic resonance data from the data acquisition server 112 and processes the magnetic resonance data in accordance with instructions provided by the operator workstation 102. Such processing may include, for example, reconstructing two-dimensional or three-dimensional images by performing a Fourier transformation of raw k-space data, performing other image reconstruction algorithms (e.g., iterative or backproj ection reconstruction algorithms), applying filters to raw k-space data or to reconstructed images, generating functional magnetic resonance images, or calculating motion or flow images.

[0029] Images reconstructed by the data processing server 114 are conveyed back to the operator workstation 102 for storage. Real-time images may be stored in a data base memory cache, from which they may be output to operator display 102 or a display 136. Batch mode images or selected real time images may be stored in a host database on disc storage 138. When such images have been reconstructed and transferred to storage, the data processing server 114 may notify the data store server 116 on the operator workstation 102. The operator workstation 102 may be used by an operator to archive the images, produce films, or send the images via a network to other facilities.

[0030] The MRI system 100 may also include one or more networked workstations 142. For example, a networked workstation 142 may include a display 144, one or more input devices 146 (e.g., a keyboard, a mouse), and a processor 148. The networked workstation 142 may be located within the same facility as the operator workstation 102, or in a different facility, such as a different healthcare institution or clinic.

[0031] The networked workstation 142 may gain remote access to the data processing server 114 or data store server 116 via the communication system 140. Accordingly, multiple networked workstations 142 may have access to the data processing server 114 and the data store server 116. In this manner, magnetic resonance data, reconstructed images, or other data may be exchanged between the data processing server 114 or the data store server 116 and thenetworked workstations 142. such that the data or images may be remotely processed by a networked workstation 142.

[0032] Referring now to FIG. 2A, a flowchart of non-limiting example steps is shown for a process 200 that can provide analysis and images of an extended view based on medical images of a subject acquired with a medical imaging system over at least two partial views. The process 200 may be fully automated to conveniently provide an extended and detailed view of the musculoskeletal system to users. The process 200 can be completed on a suitable computer system. In some implementations (e.g., processes 200 and 260), segmentation and analysis can be performed prior to registration to maintain accuracy of the 3D model. Registration can be performed subsequently to provide a full visualization of the 3D model, which may include segmentation, structure labels, and analyses.

[0033] First, partial view medical images can be accessed in block 202. The partial view images may include 3D image volumes of a subject or patient acquired with MRI. For example, the 3D image volumes may be acquired as a combination of 2D slices or as 3D MRI data. Such volume images may be referred to as a set of images. The partial view images include a common or overlapping anatomical region or volume. The common region may include a landmark common to the partial view images. As a non-limiting example, a first partial view image set may provide images of the right shoulder, arm, and thorax, while a second partial view image set may provide images of the left shoulder, arm, and thorax. In this case, the common imaging volume may include the spinal region, which may be included in both the first and second partial view image sets.

[0034] The partial view images may be acquired within the same scan session, in which the subject or patient does not move between acquisitions. Alternatively, the partial view images may be acquired with different subject positions. For example, the patient may be asked to shift between scans such that their right shoulder is nearly centered in the bore during the first acquisition and their left shoulder is nearly centered in the bore during the second acquisition. In this way, the multiple acquisitions can take advantage of the highly homogeneous region of the bore, which is typically at the center, and circumvent the limited imaging field of view.

[0035] The partial view images may include images with any desired contrast. As nonlimiting examples, the partial view images may include water-fat images (e.g., DIXON), water- only images, fat-only images, Ti-weighted images, T2-weighted images, Short Tau Inversion Recovery images (STIR), diffusion-weighted images, ultrashort echo time (UTE) images or images of other contrasts. Accessing the partial view images can include accessing storedimages with a computer system or using a computer system to acquire images using an MRI system.

[0036] The partial view images can be used in block 204a to perform segmentation and analysis. To perform segmentation, a computer system can access a machine learning model that is trained to delineate and label a variety of anatomical features or structures. For example, the segmentation may delineate adipose tissue (i.e., fat) or individual muscles, bones, ligaments, tendons, and the like. As a non-limiting example, the segmentation may be applied using the methods described in US 11,526,993 B2, US 10,314,511, and Appl. No. 17 / 664,893, the entire contents of which are incorporated herein by reference. Such segmentation can provide a 3D model. For example, block 204 a can produce a partial 3D model for each of the partial view images.

[0037] The 3D models can be expanded to include structure labels or other analyses. For example, the segmented anatomical structures may be labeled by a machine learning or artificial intelligence algorithm. In this way, the 3D model may include anatomical labels, such as “fatty tissue”, “right pectoralis major”, and so on. The images and segmentations or 3D models may be further processed to provide various desired analyses. As non-limiting examples, the analyses may include estimating fat infiltration in muscles, measuring muscle sizes or volumes, identifying muscles asymmetries, measuring cross-sectional areas of tendons, measuring an extent of edema or scar tissue, measuring bone density, and others. Examples of these optional analyses are described in U.S. Provisional Patent Application Serial No. 63 / 534,916, the entire contents of which is incorporated herein by reference. The 3D models may include these analyses results.

[0038] A common imaging volume can be identified in the partial view images in block 206. Such common imaging volume may be an anatomical region that is imaged in each of the partial view images, as described above. Identifying the common volume can be accomplished in several ways. For example, the labeled segmentation 3D model generated in block 204a may be used to identify a common anatomical landmark within each of the partial views. For example, the common landmark may be a specific organ, bone, or other anatomical structure.

[0039] The common landmark may be determined a priori or a posteriori. For example, the spine may be determined as the common landmark, with knowledge that the subject’s spine was included in each of the partial views. In this case, the spine can be identified in each of the partial view image sets based on the segmentation labels. To determine the common landmark a posteriori, each of the partial view segmentations may be queried by the automated system to determine what labeled structures are common to each image. If there are several commonfeatures, the system may choose the common landmark feature based on a desired criterion, such as the largest structure by volume, the structure located nearest to the center of the subject, etc. Alternatively, the common landmark may be defined to include more than one anatomical structure, such as the spine and the sternum.

[0040] Once a common landmark is identified for each of the partial view image sets, the common imaging volume can be defined as a volume that contains the landmark. For example, the volume may be defined as the minimum cuboid or rectangular prism that fully contains the landmark structure. The volume may also be defined by dilating segmented landmark to a desired extent, such as by 3 voxels in each of x, y, and z.

[0041] Identifying the common volume may also be achieved using a priori knowledge of the image acquisition. For example, the common volume may be defined as any volume acquired in each of the partial view image sets according to the scanner coordinates and prescribed FOVs. Such FOV coordinates may be accessed in the imaging metadata or DICOM header, for example. Using such a method may be preferable when the machine learning-based labeling does not perform well, which may be caused by weak image contrast or enhanced noise.

[0042] Identifying the common volume may also include, as one example, identifying an overlapping region using computer vision techniques. Any of several computer vision methods may be used to identify commonalities within the images. For example, various candidate common volumes may be defined in each of the partial view image sets. The candidate common volumes can be ranked to indicate a level of similarity between the volumes. Such ranking may be based on a parameter like mutual information, sum of squared differences, correlation, other metrics, or a combination thereof. The region pair or group with the highest rank may be defined as the common volume.

[0043] The common imaging volume can be used at process bock 210 to determine a transformation. Tire transformation determined in block 210 can be represented as a transformation matrix or a deformation field, which may be spatially varying. The transformation can be applied to one of the partial view image sets to achieve a common patient coordinate system. The transformation may also be applied in part to each of the partial view image sets. The common patient coordinate system may be defined based on one of the partial views or may be an alternate coordinate system. For example, the common patient coordinate system may be defined between the coordinate system of a first partial view and a second partial view, which may advantageously center or straighten the common patient coordinatesystem. Such a newly defined coordinate system is illustrated in FIG. 3C.

[0044] Referring again to FIG. 2A, the transformation can be applied to one or more of the partial view image sets in block 212. The transformation may also be applied to the partial 3D models (e.g., produced in block 204a) to produce extended view 3D models. For example, the transformation can be applied to the partial view 3D segmentation models to produce extended view segmentation models, which may be labeled with the corresponding anatomical structure or otherwise analyzed. The transformation may also be applied to partial view quantitative maps (e.g., fat infiltration) or other analyses produced in block 204a.

[0045] Block 212 may also include applying the spatial transformation to other imaging data. Several imaging data sets may be accessed with correspondent FOV s. For example, these data sets may represent similar anatomies with varying contrasts. As a non-limiting example, the image registration may be determined in blocks 204a, 206, and 210 based on water-only images and applied in block 212 to the water images as well as the fat-only images. As another example, image registration may be performed in T2-contrast images and applied to both T2- contrast and Ti-contrast images, proton density images, diffusion-weighted images, or others.

[0046] The transformation between a first and second set of partial view images or 3D models may be applied fully to one set, or partially to each set. For example, the transformation may be applied to the first partial view set to register the first partial view images and model to the patient coordinate system of the second partial view set. Alternatively, the transformation may be applied by ¥2 to the first set and model and its inverse applied by ¥1 to the second set and model to transform each of the image sets and models to a common patient coordinate system. The transformation can be applied by other fractions to define a desired common patient coordinate system. For example, the desired common patient coordinate system may be defined to straighten the coordinate system such that the final images show the patient aligned with the z axis.

[0047] The transformation can also be determined and applied in multiple stages by repeating blocks 210 and 212 N times in block 230. The multi-stage registration can provide an iterative approach. For example, the iterative approach can improve the registration with each stage by initially applying broad transformations and subsequently applying more precise transformations.

[0048] As a non-limiting example, the first stage may be limited to linear transformations, and subsequent iterations can include non-rigid transformations. For example, the initial transformation can be determined using linear transformations in block 210. The linear transformation can be applied in block 212 to the cropped images. This initialtransformation can also be applied to the partial view images and 3D segmentation model. In this way the cropped images, partial view images, and 3D segmentation model can be registered to an intermediate patient coordinate system, providing an intermediate or “close” registration result. The registration can be further improved by repeating the steps in block 230, for N = 1 times, for example. In the second iteration, the transformations may be more precise by allowing non-rigid transformations, for example. The secondary transformation can be applied to the intermediate images and 3D model in block 212. In other implementations, blocks 210 and 212 may be repeated N > 0 times with various allowed transformations as desired for the specific application.

[0049] After applying the transformation, the corresponding partial view images, the transformed images, 3D models, segmentations, quantitative maps, and other analyses can be stitched together. This produces comprehensive images, 3D models, and analyses that have an extended view that spans all of the partial views. A report may be generated and viewed or stored in block 220. The report can include the extended images, which provide a comprehensive 3D view of the musculoskeletal system in an extended region. Such images may be in the form of DICOM images. The report may also include an extended view 3D model, which advantageously provides a visualization of an extended region of the musculoskeletal system. The 3D model may include segmentation and analysis of the extended view. The segmentation may be labeled with the corresponding anatomical structures. The report may be interactive, providing a 3D model of the extended anatomy that can be navigated by a user on a computer system (e.g., scroll through slices, rotate images and segmentations, click on anatomical structures to query the analyses). Such a report can advantageously provide a detailed look at the musculoskeletal system of a comprehensive anatomical region (e.g., fullbody, upper body). The model can be queried to provide metrics or other analysis of each structure (e.g., muscle volume).

[0050] FIG. 2B shows another non-limiting example of a flowchart that illustrates a process 250 that can provide analysis and images of an extended view. Process 250 includes similar steps to process 200; the descriptions of each block of process 200 applies similarly in process 250. In this example, however, once a common imaging volume is identified, the partial view images can be cropped at process block 208 to selectively include the common imaging volume. In this way, the cropped partial view images will contain common anatomical structures and information, hi this non- limiting example, a co-registration transformation can be determined in block 210 by performing image registration on the cropped images. Several different image registration techniques may be employed, including any suitable imageregistration method known in the art. For example, a measure of similarity can be reduced or minimized by applying a linear transformation to the cropped images. The similarity measure may include cross-correlation, mutual information, sum of squared intensity differences, or ratio image uniformity. The registration may also be performed in the frequency domain, for example, using a phase correlation method. Such registration can provide a linear transformation that may be applied in tire image or frequency domain. The transformation may include linear transformations, such as rotation, scaling, translation, or other affine transformations. The transformation may also include non-rigid, deforming, or elastic transformations. For example, a non-rigid deformation can be generated by reducing a cost function, such as mutual information of the cropped images.

[0051] Additionally, the registration and transformation of partial view images can be completed prior to the segmentation and analysis. The segmentation and analysis can be performed in block 204b by applying the segmentation and analysis (as described in block 204a) to the extended view images. Thus, block 204b can generate an extended view 3D model in block 204b. This contrasts process 200, in which the segmentation and analysis are applied in block 204a on the partial view images and the 3D model is produced in block 212. Process 200 may advantageously reduce propagation of error, while process 250 may increase simplicity and reduce computational needs. In either case, a comprehensive view of the musculoskeletal system can be provided including extended view images, a 3D model, segmentation, and analyses.

[0052] FIG. 2C illustrates another non-limiting example of a process 260. Many of the blocks of process 260 are similar to those of processes 200 and 250. However, in process 260, the registration may be determined using the 3D model rather than the medical images. As in process 200, partial view images can be accessed in block 202 and segmented in block 204a, providing partial view 3D models and corresponding analyses. In block 266, a common volume can be identified in the partial view models. The common volume may be identified according to the labeled 3D model. For example, the spine volume may be identified in each of the partial view 3D models. Other methods may be used, as described with respect to block 206.

[0053] The 3D models can be cropped to the common imaging volume in block 268. For example, the models can be cropped to a 3D cuboid surrounding the spine region. Using the cropped, partial view 3D models, a transformation can be determined in block 270. The transformation can be determined using the same methods described with respect to block 210. For example, image registration can be applied on the cropped 3D model, where the model includes a binary mask delineating an anatomical structure or a multi-layer mask delineatingseveral anatomical structures. The transformation can then be applied in block 212 to the partial view segmentation models and analyses and may also be applied to the partial view images.

[0054] FIGS. 3A and 3B provide non-limiting examples of applications for extended view images and analysis. Referring to FIG. 3A, an example is provided in which extended images of the hill upper body are achieved. In this case, the partial view images include a first view of the right arm, shoulder, and thorax and a second view of the left arm, shoulder, and thorax. The overlapping region includes the spine, which can be used as the common landmark. Using the spine as a landmark, the arms, shoulders, and outer edges of the thorax may be cropped out of the partial view images to leave a reduced view focusing on the spinal region. In this way, the cropped image may selectively include an overlapping anatomical region that was acquired in each set of partial view images. This overlapping region can be used to determine a registration, which can be applied to the partial views images or 3D models to produce an extended view of the whole upper body. Similarly, FIG. 3B shows an example application that provides an extended view of the full body. The partial views include the right upper body, left upper body, and lower extremities. The lumbar spine or sacrum may be used as the common anatomical landmark within the overlapping region.

[0055] To produce an extended view from three or more partial views, the segmentation and registration process (e.g., 200, 250, 260) can be applied simultaneously to all three (or more) partial views. Alternatively, the process (e.g., 200, 250, 260) can be repeated several times in which an intermediate partial view is produced from the previous iteration. For example, the left upper body and right upper body can be registered to produce a new partial view that includes the full upper body. Then, the process can be repeated to align the upper body and lower body to create a full-body extended view. In a non-limiting example, the left upper body and the right upper body can be registered, first, using two iterations (N = 1) in which the first iteration includes linear transformations, and the second iteration includes non- rigid transformations. Subsequently, the upper body and lower body can be registered with a single iteration (N = 0) using linear transformations.

[0056] FIG. 3C provides an example registr ation of three partial view 3D segmentation models. MRI images were acquired in three stages. The first partial view includes images of the subject’s right upper extremity and thorax, the second partial view includes images of the subject’s left upper extremity and thorax, and the third partial view includes the subject’s lower extremities and trunk. As the MRI images demonstrate, the subject shifted in the MRI scanner between the first and second partial view, leaning to the left and right to provide better coverage of the full shoulder and arm regions. In this example, the subject also moved on the scannertable between the second and third partial view.

[0057] The partial view MRI image sets were segmented to provide a 3D model of the bones and muscles of the subject’s full body. These segmentation models were combined using a naive registration, which was informed by the MRI imaging scanner coordinates. Because the patient moved between scans, such a naive combination provided an incoherent full-body model, as shown.

[0058] For comparison, the segmentation model was aligned manually using linear transformations, as shown in FIG. 3C. The manual registration did not fully register the partial views, as evident from the unaligned spines in the resulting extended 3D model. Thus, linear transformations were insufficient in this example. Moreover, manual segmentation is timeconsuming and prone to human error and variability. A high-quality registration may also require complex 3D rotations in addition to translation. Such 3D rotation may be difficult to visualize, making it challenging to determine a registration that is simultaneously successful in all three planes. In contrast, the disclosed automated registration produced a robust full-body 3D model in which the spine is fully aligned, as shown in FIG. 3C.

[0059] FIG. 4A shows parts of an example report that provide characterization of fat infiltration in the muscles of both the upper and lower body. The extended view report allows for a holistic view of the musculoskeletal system. For example, the report provides a color- coded view of the muscle-wise fat infiltration across the whole body. The report further includes a detailed quantification of the fat infiltration in each muscle. In this way, the fat infiltration can be visually and quantitatively compared across the whole body, particularly between the right and left sides for evaluation of asymmetries. The report also lists notable findings and the muscles with the highest and lowest levels of fat infiltration, which can help to highlight asymmetries and abnormalities for the user.

[0060] FIGS. 4B-4C show parts of an example report. The report advantageously includes a pixel-wise analysis of both the right and left pectoralis muscles (and other muscle groups not shown), allowing for comparison between sides, evaluation of asymmetries, and a comprehensive view of the upper body. The report provides an example of the 3D model, which includes a segmentation model of the subject’s muscles and a 3D pixel-by-pixel analysis of fat infiltration in the segmented muscles. The left and right pectoralis muscles are shown, for example, in FIG. 4B. As shown in FIG. 4C, the report also provides the muscle and fat content and the fat infiltration percentage for cross sectional areas along the muscle from the inferior to superior position.

[0061] FIG. 4D provides a part of an example report that corresponds to the vastuslateralis. The report includes the following analyses: fat infiltration with the cross-sectional area along the muscle, from distal to proximal; muscle and fat content; and a 3D pixel by pixel map of fat infiltration. These analyses are shown for an initial time point and at a second timepoint 12 months later, allowing for comparison between timepoints.

[0062] The above-described systems and methods may be implemented using a variety of systems and networks. FIG. 5 shows an example 500 of a system for automatically carrying out the processes described above. As shown in FIG. 5, a computing device 510 can receive multiple types of image data from an image source 502. In some configurations, computing device 510 can execute at least a portion of an automatic segmentation system 504 to automatically determine whether bone, fat, or tissue are present in images of a subject and to provide for a quantitative assessment, as described above.

[0063] Additionally or alternatively, in some embodiments, computing device 510 can communicate information about image data received from image source 502 to a server 520 over a communication network 508, which can execute at least a portion of automatic segmentation system 504 to automatically determine whether bone, fat, or tissue are present in images of a subject and to provide for a quantitative assessment. In such embodiments, server 520 can return information to computing device 510 (and / or any other suitable computing device) indicative of an output of automatic segmentation system 504 to determine whether bone, fat, or tissue are present in images of a subject and to provide for a quantitative assessment.

[0064] In some embodiments, computing device 510 and / or server 520 can be any suitable computing device or combination of devices, such as a desktop computer, a laptop computer, a smartphone, a tablet computer, a wearable computer, a server computer, a virtual machine being executed by a physical computing device, etc. In some configurations, automatic segmentation system 504 can extract features from labeled (e.g., labeled as including a condition or disease, or normal) image data using a convolutional neural network (CNN), UNet, and the like, trained as a general image classifier, and can perform a correlation analysis to calculate correlations between the features corresponding to the image data and a database. In some embodiments, the labeled data can be used to train a classification model, such as a support vector machine (SVM), to classify features as indicative of bone, tissue, or fat. In some embodiments, automatic segmentation system 504 can provide features for unlabeled image data to the trained classification model and can present a quantitative assessment, such as a fat infiltration score or ratio based on the output of the classification model (e.g., based on which class the SVM identifies the features with).

[0065] In some embodiments, image source 502 can be any suitable source of image data, such as an MRI, CT, or another computing device (e.g., a server storing image data), etc. In some embodiments, image source 502 can be local to computing device 510. For example, image source 502 can be incorporated with computing device 510 (e.g., computing device 510 can be configured as part of a device for capturing and / or storing images). As another example, image source 502 can be connected to computing device 510 by a cable, a direct wireless link, etc. Additionally or alternatively, in some embodiments, image source 502 can be located locally and / or remotely from computing device 510, and can communicate image data to computing device 510 (and / or server 520) via a communication network (e.g., communication network 508).

[0066] In some embodiments, communication network 508 can be any suitable communication network or combination of communication networks. For example, communication network 508 can include a Wi-Fi network (which can include one or more wireless routers, one or more switches, etc.), a peer-to-peer network (e.g., a Bluetooth network), a cellular network (e.g., a 3G network, a 4G network, etc., complying with any suitable standard, such as CDMA, GSM, LTE, LTE Advanced, WiMAX, etc.), a wired network, etc. In some embodiments, communication network 508 can be a local area network, a wide area network, a public network (e.g., the Internet), a private or semi-private network (e.g., a corporate or university intranet), any other suitable type of network, or any suitable combination of networks. Communications links shown in FIG. 5 can each be any suitable communications link or combination of communications links, such as wired links, fiber optic links, Wi-Fi links, Bluetooth links, cellular links, etc.

[0067] FIG. 6 shows an example 600 of hardware that can be used to implement image source 502, computing device 510, and / or server 520 in accordance with some embodiments of the disclosed subject matter. As shown in FIG. 6, in some embodiments, computing device 510 can include a processor 602, a display 604, one or more inputs 606, one or more communication systems 608, and / or memory 610. In some embodiments, processor 602 can be any suitable hardware processor or combination of processors, such as a central processing unit (CPU), a graphics processing unit (GPU), etc. In some embodiments, display 604 can include any suitable display devices, such as a computer monitor, a touchscreen, a television, etc. In some embodiments, inputs 606 can include any suitable input devices and / or sensors that can be used to receive user input, such as a keyboard, a mouse, a touchscreen, a microphone, etc.

[0068] In some embodiments, communications systems 608 can include any suitable hardware, firmware, and / or software for communicating information over communicationnetwork 508 and / or any other suitable communication networks. For example, communications systems 608 can include one or more transceivers, one or more communication chips and / or chip sets, etc. In a more particular example, communications systems 608 can include hardware, firmware and / or software that can be used to establish a Wi-Fi connection, a Bluetooth connection, a cellular connection, an Ethernet connection, etc.

[0069] In some embodiments, memory' 610 can include any suitable storage device or devices that can be used to store instructions, values, etc., that can be used, for example, by processor 602 to present content using display 604, to communicate with server 520 via communications system(s) 608, etc. Memory 610 can include any suitable volatile memory, non-volatile memory, storage, or any suitable combination thereof. For example, memory' 610 can include RAM, ROM, EEPROM, one or more flash drives, one or more hard disks, one or more solid state drives, one or more optical drives, etc. In some embodiments, memory 610 can have encoded thereon a computer program for controlling operation of computing device 510. In such embodiments, processor 602 can execute at least a portion of the computer program to present content (e.g., MRI images, user interfaces, graphics, tables, etc.) , receive content from server 520, transmit information to server 520, etc.

[0070] In some embodiments, server 520 can include a processor 612, a display 614, one or more inputs 616, one or more communications systems 618, and / or memory 620. In some embodiments, processor 612 can be any suitable hardware processor or combination of processors, such as a CPU, a GPU, etc. In some embodiments, display 614 can include any suitable display devices, such as a computer monitor, a touchscreen, a television, etc. In some embodiments, inputs 616 can include any suitable input devices and / or sensors that can be used to receive user input, such as a keyboard, a mouse, a touchscreen, a microphone, etc.

[0071] In some embodiments, communications systems 618 can include any suitable hardware, firmware, and / or software for communicating information over communication network 508 and / or any other suitable communication networks. For example, communications systems 618 can include one or more transceivers, one or more communication chips and / or chip sets, etc. In a more particular example, communications systems 618 can include hardware, firmware and / or software that can be used to establish a Wi-Fi connection, a Bluetooth connection, a cellular connection, an Ethernet connection, etc.

[0072] In some embodiments, memory 620 can include any suitable storage device or devices that can be used to store instructions, values, etc., that can be used, for example, by processor 612 to present content using display 614, to communicate with one or more computing devices 510, etc. Memory 620 can include any suitable volatile memory, non-volatile memory, storage, or any suitable combination thereof. For example, memory 620 can include RAM, ROM, EEPROM, one or more flash drives, one or more hard disks, one or more solid state drives, one or more optical drives, etc. In some embodiments, memory 620 can have encoded thereon a server program for controlling operation of server 520. In such embodiments, processor 612 can execute at least a portion of the server program to transmit information and / or content (e.g., MRI data, results of automatic diagnosis, a user interface, etc.) to one or more computing devices 510, receive information and / or content from one or more computing devices 510, and / or receive instructions from one or more devices (e.g., a personal computer, a laptop computer, a tablet computer, a smartphone, etc.).

[0073] In some embodiments, image source 502 can include a processor 622, imaging components 624, one or more communications systems 626, and / or memory 628. In some embodiments, processor 622 can be any suitable hardware processor or combination of processors, such as a CPU, a GPU, etc. In some embodiments, imaging components 624 can be any suitable components to generate image data corresponding to one or more imaging modes (e.g., T1 imaging, T2 imaging, fMRI, etc.). An example of an imaging machine that can be used to implement image source 502 can include a conventional MRI scanner (e.g., a 1.5 T scanner, a 3 T scanner), a high field MRI scanner (e.g., a 7 T scanner), an open bore MRI scanner, a CT system, an ultrasound scanner and the like.

[0074] Note that, although not shown, image source 502 can include any suitable inputs and / or outputs. For example, image source 502 can include input devices and / or sensors that can be used to receive user input, such as a keyboard, a mouse, a touchscreen, a microphone, a trackpad, a trackball, hardware buttons, software buttons, etc. As another example, image source 502 can include any suitable display devices, such as a computer monitor, a touchscreen, a television, one or more speakers, etc.

[0075] In some embodiments, communications systems 626 can include any suitable hardware, firmware, and / or software for communicating information to computing device 510 (and, in some embodiments, over communication network 508 and / or any other suitable communication networks). For example, communications systems 626 can include one or more transceivers, one or more communication chips and / or chip sets, etc. In a more particular example, communications systems 626 can include hardware, firmware and / or software that can be used to establish a wired connection using any suitable port and / or communication standard (e.g., VGA, DVI video, USB, RS-232, etc.), Wi-Fi connection, a Bluetooth connection, a cellular connection, an Ethernet connection, etc.

[0076] In some embodiments, memory 628 can include any suitable storage device ordevices that can be used to store instructions, values, image data, etc., that can be used, for example, by processor 622 to: control imaging components 624, and / or receive image data from imaging components 624; generate images; present content (e.g., MRI images, a user interface, etc.) using a display; communicate with one or more computing devices 510; etc. Memory 628 can include any suitable volatile memory, non-volatile memory, storage, or any suitable combination thereof. For example, memory 628 can include RAM, ROM, EEPROM, one or more flash drives, one or more hard disks, one or more solid state drives, one or more optical drives, etc. In some embodiments, memory 628 can have encoded thereon a program for controlling operation of image source 502. In such embodiments, processor 622 can execute at least a portion of the program to generate images, transmit information and / or content (e.g., MRI image data) to one or more computing devices 510, receive information and / or content from one or more computing devices 510, receive instructions from one or more devices (e.g., a personal computer, a laptop computer, a tablet computer, a smartphone, etc.), and the like.

[0077] The present disclosure has described one or more preferred embodiments, and it should be appreciated that many equivalents, alternatives, variations, and modifications, aside from those expressly stated, are possible and within the scope of the invention.

Claims

CLAIMS1. A system for generating a report providing an extended view 3D model based on medical images of a subject acquired with a medical imaging system over at least two partial views, the system comprising: a processor configured to: access at least a first set of partial view medical images and a second set of partial view medical images of the subject; segment the first set of partial view medical images to generate a first partial 3D model, wherein the first partial 3D model comprises delineations of a first plurality of anatomical structures; segment the second set of partial view medical images to generate a second partial 3D model, wherein the second partial 3D model comprises delineations of a second plurality of anatomical structures; identify an overlapping region in the first and second sets of partial view images that is common to the first and second sets of partial view images using at least one of the first plurality of anatomical structures or the second plurality of anatomical structures; determine a spatial transformation using the overlapping region; apply the spatial transformation to produce an extended view 3D model that aligns the first partial 3D model with the second partial 3D model; and generate a report comprising the extended view 3D model.

2. The system of claim 1 , wherein the processor is further configured to apply the spatial transformation to produce extended view images that align the first set of partial view images with the second set of partial view images, and wherein the report further comprises the extended view images.

3. The system of claim 1 , wherein the processor is further configured to: generate analysis of each of the first plurality of anatomical structures to produce a first partial view analysis, wherein the first partial 3D model further comprises the first partial view analysis; andgenerate analysis of each of the second plurality of anatomical structures to produce a second partial view analysis, wherein the second partial 3D model further comprises the second partial view analysis.

4. The system of claim 3 wherein: the first plurality of anatomical structures comprises a first muscle of the subject; the second plurality of anatomical structures comprises a second muscle of the subject; the first partial view analysis comprises a characterization of fat infiltration in the first muscle; the second partial view analysis comprises a characterization of fat infiltration in the second muscle; and the extended view 3D model further comprises the characterization of fat infiltration in the first muscle and the characterization of fat infiltration in the second muscle.

5. The system of claim 1 , wherein: the first set of partial view images includes magnetic resonance imaging (MRI) images of a left shoulder, a left arm, a left thorax, and at least part of a spine of the subject; the second set of partial view images includes MRI images of a right shoulder, a right arm, a right thorax, and the at least part of the spine of the subject; and the overlapping region includes the at least part of the spine of the subject.

6. The system of claim 1, wherein: the first set of partial view images includes MRI images of an upper body and at least part of a spine of the subject; the second set of partial view images includes MRI images of a lower body and the at least part of the spine of the subject; and the overlapping region includes the at least part of the spine of the subject.

7. The system of claim 1 , wherein applying the spatial transformation comprises applying a first part of the spatial transformation to the first partial 3D model and applying a second part of the spatial transformation to the second partial 3D model to align the first and second partial 3D models to a common patient coordinate system.

8. The system of claim 1, wherein the spatial transformation is a first spatial transformation and comprises linear transformations.

9. The system of claim 1, wherein the processor is configured to determine the spatial transform by: cropping the first set of partial view images to the overlapping region to generate a first set of cropped partial view images; cropping the second set of partial view images to the overlapping region to generate a second set of cropped partial view images; and registering the first set of cropped partial view images to the second set of cropped partial view images to determine the spatial transform.

10. The system of claim 9, wherein the processor is further configured to: apply the first spatial transformation to at least one of the first set of cropped partial view images or the second set of cropped partial view images to produce a first intermediate set of cropped partial view images and a second intermediate set of cropped partial view images; register the first intermediate set of cropped partial view images to the second intermediate set of cropped partial view images to determine a second spatial transformation, the second spatial transformation being a non-rigid transformation; and apply the second spatial transformation to produce an extended view 3D model that aligns the first partial 3D model with the second partial 3D model.

11. A method for generating a report providing an extended view 3D model based on medical images of a subject acquired with a medical imaging system over at least two partial views, the comprising: accessing at least a first set of partial view medical images and a second set of partial view medical images of the subject; segmenting the first set of partial view medical images to generate a first partial 3D model, wherein the first partial 3D model comprises delineations of a first plurality of anatomical structures;segmenting the second set of partial view medical images to generate a second partial 3D model, wherein the second partial 3D model comprises delineations of a second plurality of anatomical structures; identifying an overlapping region in the first and second sets of partial view images that is common to the first and second sets of partial view images using at least one of the first plurality of anatomical structures or the second plurality of anatomical structures; determining a spatial transformation using the overlapping region; applying the spatial transformation to produce an extended view 3D model that aligns the first partial 3D model with the second partial 3D model; and generating a report comprising the extended view 3D model.

12. The method of claim 11 , further comprising applying the spatial transformation to produce extended view images that align the first set of partial view images with the second set of partial view images, and wherein the report further comprises the extended view images.

13. The method of claim 11 , further comprising: analyzing of each of the first plurality of anatomical structures to produce a first partial view analysis, wherein the first partial 3D model further comprises the first partial view analysis; and analyzing each of the second plurality of anatomical structures to produce a second partial view analysis, wherein the second partial 3D model further comprises the second partial view analysis.

14. The method of claim 13 wherein: the first plurality of anatomical structures comprises a first muscle of the subject; the second plurality of anatomical structures comprises a second muscle of the subject; the first partial view analysis comprises a characterization of fat infiltration in the first muscle; the second partial view analysis comprises a characterization of fat infiltration in the second muscle; and the extended view 3D model further comprises the characterization of fat infiltration in the first muscle and the characterization of fat infiltration in the second muscle.

15. The method of claim 11 , wherein: the first set of partial view images includes magnetic resonance imaging (MRI) images of a left shoulder, a left arm, a left thorax, and at least part of a spine of the subject; the second set of partial view images includes MRI images of a right shoulder, a right arm, a right thorax, and the at least part of tire spine of the subject; and the overlapping region includes the at least part of the spine of the subject.

16. The method of claim 11, wherein: the first set of partial view images includes MRI images of an upper body and at least part of a spine of the subject; the second set of partial view images includes MRI images of a lower body and the at least part of the spine of the subject; and the overlapping region includes the at least part of the spine of the subject.

17. The method of claim 11, wherein applying the spatial trans formation comprises applying a first part of the spatial transformation to the first partial 3D model and applying a second part of the spatial transformation to the second partial 3D model to align the first and second partial 3D models to a common patient coordinate system.

18. The method of claim 11, wherein the spatial transformation is a first spatial transformation and comprises linear transformations.

19. The method of claim 11 , wherein determining the spatial transform includes: cropping the first set of partial view images to the overlapping region to generate a first set of cropped partial view images; cropping the second set of partial view images to the overlapping region to generate a second set of cropped partial view images; and registering the first set of cropped partial view images to the second set of cropped partial view images to determine the spatial transform.

20. The method of claim 19, further comprising: applying the first spatial transformation to at least one of the first set of cropped partial view images or the second set of cropped partial view images toproduce a first intermediate set of cropped partial view images and a second intermediate set of cropped partial view images; registering the first intermediate set of cropped partial view images to the second intermediate set of cropped partial view images to determine a second spatial transformation, the second spatial transformation being a non-rigid transformation; and applying the second spatial transformation to produce an extended view 3D model that aligns the first partial 3D model with the second partial 3D model.

Images