Modular algorithm for automatic localization of features in images

Abstract

Techniques are disclosed for processing image data to determine spinal characteristics. A system, device, and method support receiving an image, executing a first neural network to perform a first landmark detection using the image and executing a second neural network to perform a second landmark detection using the image. The first neural network may generate a vertebra patch, a sacrum region of interest (ROI), and / or a femoral ROI. The second neural network may generate vertebra center data, vertebra corner data, vertebra labeling data, sacrum endplate data, and / or femoral head data.

Description

MODULAR ALGORITHM FOR AUTOMATIC LOCALIZATION OF FEATURES INIMAGESFIELD OF INVENTION

[0001] This disclosure generally relates to image analysis and, more particularly, to automatically localizing anatomical features in images.BACKGROUND

[0002] In various medical software there is a requirement to perform localization of features in two-dimensional images, such as using X-ray, fluoroscopy, and EOS imaging modalities. Conventional systems for automatically localizing features suffer from inaccuracy, lack of robustness to data variability, complex development requirements, slow runtimes, incompatible software integrations, inability to expand with additional functions for other applications, and a requirement for significant amounts of training data.SUMMARY

[0003] In general, the disclosure describes techniques for personalizing artificial intelligence (Al) models that analyze images of patients. As discussed in this disclosure, a computing system may comprise: processing circuitry; and a storage medium, the processing circuitry configured to: receive an image; execute a first artificial intelligence system to perform a first landmark detection using the image to generate one or more of a vertebra patch, a sacrum region of interest (ROI), and a femoral ROI; execute at least a second artificial intelligence system to perform a second landmark detection using the image to generate one or more of vertebra center data, vertebra corner data, vertebra labeling data, sacrum endplate data, and femoral head data, based on the one or more of the vertebra patch, the sacrum ROI, and the femoral ROI; and output the one or more of vertebra center data, vertebra corner data, vertebra labeling data, sacrum endplate data, and femoral head data.

[0004] This summary is intended to provide an overview of the subject matter described in this disclosure. It is not intended to provide an exclusive or exhaustive explanation of the apparatus and methods described in detail within the accompanying drawings and description below. Further details of one or more examples are set forth in the accompanying drawings and the description below.

[0005] The details of one or more aspects of the disclosure are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the techniques described in this disclosure will be apparent from the description and drawings, and from the claims.

[0006] The preceding is a simplified summary of the disclosure to provide an understanding of some aspects of the disclosure. This summary is neither an extensive nor exhaustive overview of the disclosure and its various aspects, implementations, and configurations. It is intended neither to identify key or critical elements of the disclosure nor to delineate the scope of the disclosure but to present selected concepts of the disclosure in a simplified form as an introduction to the more detailed description presented below. As will be appreciated, other aspects, implementations, and configurations of the disclosure are possible utilizing, alone or in combination, one or more of the features set forth above or described in detail below.

[0007] Numerous additional features and advantages of the present disclosure will become apparent to those skilled in the art upon consideration of the implementation descriptions provided hereinbelow.BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

[0008] The accompanying drawings are incorporated into and form a part of the specification to illustrate several examples of the present disclosure. These drawings, together with the description, explain the principles of the disclosure. The drawings simply illustrate preferred and alternative examples of how the disclosure can be made and used and are not to be construed as limiting the disclosure to only the illustrated and described examples. Further features and advantages will become apparent from the following more detailed description of the various aspects, implementations, and configurations of the disclosure, as illustrated by the drawings referenced below.

[0009] FIG. 1 is a block diagram illustrating an environment for capturing and analyzing an image of a patient in accordance with the techniques of the disclosure.

[0010] FIG. 2 is a block diagram illustrating a system for analyzing an image of a patient in accordance with the techniques of the disclosure.

[0011] FIG. 3 is a block diagram of an example input image and an output image generated by an analysis engine based on the input image according to the techniques of the disclosure.

[0012] FIGS . 4A-4C are images of X-rays of a patient in accordance with one or more techniques of the present disclosure.

[0013] FIGS. 5 A and 5B illustrate a two-phase analysis system in accordance with the techniques of the disclosure.

[0014] FIGS. 6 and 7 illustrate annotated images of a patient in accordance with the techniques of this disclosure.

[0015] FIG. 8 is a flowchart illustrating an example method in which a computing system generates a result based on an image in accordance with the techniques of this disclosure.

[0016] Like reference characters refer to like elements throughout the figures and description.DETAILED DESCRIPTION

[0017] It should be understood that various aspects disclosed herein may be combined in different combinations than the combinations specifically presented in the description and accompanying drawings. It should also be understood that, depending on the example or implementation, certain acts or events of any of the processes or methods described herein may be performed in a different sequence, and / or may be added, merged, or left out altogether (e.g., all described acts or events may not be necessary to carry out the disclosed techniques according to different implementations of the present disclosure). In addition, while certain aspects of this disclosure are described as being performed by a single module or unit for purposes of clarity, it should be understood that the techniques of this disclosure may be performed by a combination of units or modules associated with, for example, a computing device and / or a medical device.

[0018] Medical imaging software often necessitates the ability to automatically and precisely localize features within two-dimensional images. Such images may be sourced from various modalities, such as X-ray, fluoroscopy, and EOS. Within the scope of the present disclosure, the utilization of an Al component is proposed which can serve as a fundamental module, designed specifically to execute individual tasks that together contribute to a more intricate, multi-stage algorithmic procedure.

[0019] An advantage of the systems and methods presented herein is an improvement over conventional feature detection techniques. The systems and methods described herein provide enhanced accuracy in localization of vertebra, sacrum, and femoral heads in spinal column images. The systems and methods described herein may be utilized to simplify the developmental process of Al systems, enhance execution speed, and facilitate a more streamlined integration into softwareplatforms. The design of the systems and methods described herein grants flexibility, making the expansion to encompass more functions or adapt to varied applications.

[0020] An Al system or module as described herein, which may be referred to as a landmark detection engine, in various embodiments is realized as a fully convolutional neural network (CNN) specifically tailored for detecting landmarks. The spectrum of tasks that the CNN is equipped to handle includes but is not limited to pinpointing the spinal region, detecting centers of vertebrae, accurately locating endplates, identifying the sacrum, pinpointing centers of femoral heads and gauging their radii, automating the labeling of vertebrae, and / or detecting neurostimulation electrodes.

[0021] In one or more examples, the described methods, processes, and techniques may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored as one or more instructions or code on a computer-readable medium and executed by a hardware-based processing unit. Alternatively, or additionally, functions may be implemented using machine learning models, neural networks, artificial neural networks, or combinations thereof (alone or in combination with instructions). Alternatively, or additionally, functions may be implemented using machine learning models, neural networks, artificial neural networks, or combinations thereof (alone or in combination with instructions). Computer-readable media may include non-transitory computer-readable media, which corresponds to a tangible medium such as data storage media (e.g., random access memory (RAM), read only memory (ROM), programmable read only memory (PROM), erasable programmable read only memory (EPROM), electronically erasable programmable read only memory (EEPROM), flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer).

[0022] Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors (e.g., Intel Core i3, i5, i7, or i9 processors; Intel Celeron processors; Intel Xeon processors; Intel Pentium processors; AMD Ryzen processors; AMD Athlon processors; AMD Phenom processors; Apple A10 or 10X Fusion processors; Apple Al l, A12, A12X, A12Z, or A13 Bionic processors; or any other general purpose microprocessors), graphics processing units (e.g., Nvidia GeForce RTX 2000-series processors, Nvidia GeForce RTX 3000-series processors, AMD Radeon RX 5000-series processors, AMD Radeon RX 6000-series processors, or any other graphics processing units), application specificintegrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Accordingly, the term “processor” as used herein may refer to any of the foregoing structure or any other physical structure suitable for implementation of the described techniques. Also, the techniques could be fully implemented in one or more circuits or logic elements.

[0023] Before any implementations of the disclosure are explained in detail, it is to be understood that the disclosure is not limited in its application to the details of construction and the arrangement of components set forth in the following description or illustrated in the drawings. The disclosure is capable of other implementations and of being practiced or of being carried out in various ways. Also, it is to be understood that the phraseology and terminology used herein is for the purpose of description and should not be regarded as limiting. The use of “including”, “comprising”, or “having” and variations thereof herein is meant to encompass the items listed thereafter and equivalents thereof as well as additional items. Further, the present disclosure may use examples to illustrate one or more aspects thereof. Unless explicitly stated otherwise, the use or listing of one or more examples (which may be denoted by “for example”, “by way of example”, “e.g.,” “such as”, or similar language) is not intended to and does not limit the scope of the present disclosure.

[0024] Fig. 1 illustrates an example of a computing system 100 that supports aspects of the present disclosure. The computing system 100 may be in communication with an imaging device 102, a database 104, a user device 106, and / or other components. Systems according to other implementations of the present disclosure may include more or fewer components than illustrated in Fig. 1. For example, systems and methods described herein may be utilized while omitting and / or including additional instances of one or more of the computing system 100, database 104, imaging device 102, and / or user device 106. In an example, the computing system 100 may omit any instance of the imaging device 102, database 104, and / or user device 106. The computing system 100 may support the implementation of one or more other aspects of one or more of the methods disclosed herein.

[0025] In some examples, computing system 100 may take the form of a handheld computing device, computer workstation, server or other networked computing device, smartphone, tablet, or external programmer that includes a user interface for presenting information to and receiving input from a user. In some examples, computing system 100 may include one or more devices thatimplement a machine learning system, such as a neural network, a deep learning system, or another type of machine learning system.

[0026] A user, such as a physician, technician, surgeon, electro-physiologist, or other clinician, may interact with computing system 100 to process images of patients, such as images from an imaging device 102. A user may also interact with computing system 100 to capture images of a patient 108 by controlling the imaging device 102. Computing system 100, as described in greater detail below, may include a processor configured to evaluate images from an imaging device 102 using a plurality of neural networks.

[0027] The imaging device 102 may in some implementations comprise a machine capable of capturing X-ray, fluoroscopy, EOS, and / or other images of a patient 108. Such images may be transmitted directly to the computing system 100 or may be stored in a database 104 accessible to the computing system 100, such as via a network connection. Images from the imaging device 102 may be still or moving images and may be two- or three-dimensional images.

[0028] Images from the imaging device 102 may in some implementations comprise or be associated with metadata. Metadata associated with or comprised by an image may include information such as patient name or ID, diagnosis information, or other information.

[0029] In one example, computing system 100 receives patient data of a patient 108 collected by an imaging device 102. In some examples, the patient data includes one or more images of the patient 108, physiological data for patient 108, or any other types of patient related data.

[0030] A database 104 may store image data for a patient 108. The database 104 may include processing circuitry and one or more storage mediums (e.g., RAM, ROM, PROM, EPROM, EEPROM, and / or flash memory. In some examples, the database 104 is a cloud computing system. In some examples, functions of the database 104 are distributed across a number of computing systems.

[0031] In some examples, a user device 106 takes the form of an external computer system or mobile device, such as a mobile phone, a smartphone, a laptop, a tablet computer, a personal digital assistant (PDA), etc. In some examples, the user device 106 is a CareLink™ monitor available from Medtronic, Inc. A user, such as a physician, technician, surgeon, electro-physiologist, or other clinician, may interact with the user device 106 to retrieve image or diagnostic information from the computing system 100. A user, such as patient 108 or a clinician may also interact with the user device 106 to program the computing system 100, e.g., select or adjust values for operationalparameters of applications executed by the computing system 100 and / or the imaging device 102 as described herein. The user device 27 may include processing circuitry, a memory, a user interface, and communication circuitry capable of transmitting and receiving information to and from each of the imaging device 102 and the computing system 100.

[0032] In some implementations, the computing system 100 may communicate with the imaging device 102, database 104, and / or user device 106 via a network. A network as described herein may include one or more computing devices (not shown), such as one or more non-edge switches, routers, hubs, gateways, security devices such as firewalls, intrusion detection, and / or intrusion prevention devices, servers, computer terminals, laptops, printers, databases, wireless mobile devices such as cellular phones or personal digital assistants, wireless access points, bridges, cable modems, application accelerators, or other network devices. A network as described herein may include one or more networks administered by service providers and may thus form part of a large- scale public network infrastructure, e.g., the Internet. A network as described herein may provide computing devices, such as computing system 100 and imaging device 102, access to the Internet, and may provide a communication framework that allows computing devices to communicate with one another. In some examples, a network as described herein may be a private network that provides a communication framework that allows computing system 100, imaging device 102, and the database 104 to communicate with one another. In some examples, communications between computing system 100, imaging device 102, user device 106, and database 104 are encrypted.

[0033] The user device 106 and computing system 100 may communicate via wireless or nonwireless communication over a network using any techniques known in the art. In some examples, the computing system 100 is a remote device that communicates with the user device 106 via an intermediary device located in a network, such as a local access point, wireless router, or gateway. Examples of communication techniques may include, for example, communication according to the Bluetooth® or BLE protocols. Other communication techniques are also contemplated. The computing system 100 may also communicate with one or more other external devices using a number of known communication techniques, both wired and wireless.

[0034] Fig. 2 is a block diagram of an example computing system 100 according to techniques of the disclosure. In the illustrated example, computing system 100 includes processing circuitry 200, memory 202, a communication interface 222, and a user interface 220. Computing systemsaccording to implementations of the present disclosure may include more or fewer components than the computing system 100 illustrated in Fig. 2.

[0035] Processing circuitry 200 may include any one or more of a microprocessor, a controller, a central processing unit (CPU), graphics processing unit (GPU), DSP, an ASIC, an FPGA, or equivalent discrete or analog logic circuitry. In some examples, processing circuitry 200 may include multiple components, such as any combination of one or more microprocessors, one or more controllers, one or more DSPs, one or more ASICs, or one or more FPGAs, as well as other discrete or integrated logic circuitry. The functions attributed to processing circuitry 200 herein may be embodied as software, firmware, hardware, or any combination thereof.

[0036] The processing circuitry 200 of the computing system 100 may be any processor described herein or any similar processor. The processing circuitry 200 may be configured to execute instructions stored in the memory 202, which instructions may cause the processing circuitry 200 to carry out one or more computing steps utilizing or based on data received from a database 104, an imaging device 102, a network, a user device 106, and / or other data sources.

[0037] Processing circuitry 200 may include hardware capable of implementing or executing a neural network, which may be embodied as hardware, firmware, software, or any combination thereof. The neural network module may comprise a dedicated hardware circuit, such as an ASIC, separate from other processing circuitry 200 components, such as a microprocessor, or a software module executed by a component of processing circuitry 200, which may be a microprocessor or ASIC. The neural network module may implement programmable neural networks such as landmark detection systems.

[0038] In some examples, processing circuitry 200 of computing system 100 implements one or more artificial intelligence (Al) and / or machine learning (ML) systems, such as convolutional neural networks (CNNs). For instance, processing circuitry 200 may apply a generalized CNN and / or a specialized CNN to images of a patient 108, as described elsewhere in this disclosure. Processing circuitry 200 may implement an Al system using special-purpose circuitry or by executing software instructions stored on a computer-readable medium, such as memory 202. The communication interface 222 may be configured to receive image data from an imaging device 102, database 104, a user device 106, and / or other data sources and to transmit results of image processing as described herein to external devices such as a user device 106, a database 104, and / or other devices capable of receiving data.

[0039] Processing circuitry 200, in one example, is configured to implement functionality and / or process instructions for execution within computing system 100. For example, processing circuitry 200 may be capable of processing instructions stored in memory 202 to, for example, implement or execute a neural network. Examples of processing circuitry 200 may include, any one or more of a microprocessor, a controller, a DSP, an ASIC, an FPGA, or equivalent discrete and / or integrated logic circuitry.

[0040] Memory 202 includes computer-readable instructions that, when executed by processing circuitry 200, cause the computing system 100 and processing circuitry 200 to perform various functions and execute various applications as described herein. Memory 202 may include any volatile, non-volatile, magnetic, optical, or electrical media, such as a RAM, ROM, non-volatile RAM (NVRAM), EEPROM, flash memory, or any other digital or analog media.

[0041] Memory 202 may be configured to store a variety of operational parameters, weights, training data 214, results data 218, and analysis application 204. In the example of Fig. 2, memory 202 may store landmark detection engine 206 data, detection data 208, pre-processing engine 210 data, and post-processing engine 212 data. In other examples, memory 202 may act as a temporary buffer for storing data until it can be uploaded to a user device 106, a database 104, and / or another data repository.

[0042] Memory 202 as described herein may refer to one or more storage devices which may be configured to store information within the computing system 100 during operation. Such storage devices may in some examples be described as a computer-readable storage medium. In some examples, a storage device may comprise temporary or volatile memory. Examples of volatile memories include RAM, DRAM, SRAM, and other forms of volatile memories. In some examples, a storage device may be used to store program instructions for execution by processing circuitry 200. For example, a storage device may be used by software or applications running on the computing system 100, such as an analysis application 204, to temporarily store information during program execution.

[0043] The memory 202 may store information or data associated with completing, for example, any step of the methods described herein, or of any other methods. The memory 202 may store, for example, instructions and / or machine learning models that support one or more functions of the computing system 100. For instance, the memory 202 may store content (e.g., instructions and / or machine learning models) that, when executed by the processing circuitry 200, enable one or morelandmark detection engines 206, pre-processing engines 210, post-processing engines 212, and other processes. Such content, if provided as in instruction, may, in some implementations, be organized into one or more applications, modules, packages, layers, or engines.

[0044] Alternatively, or additionally, the memory 202 may store other types of content or data (e.g., machine learning models, artificial neural networks, deep neural networks, etc.) that can be processed by the processing circuitry 200 to carry out various methods and features described herein. Thus, although various contents of memory 202 may be described as instructions, it should be appreciated that functionality described herein can be achieved through use of instructions, algorithms, and / or machine learning models. The data, algorithms, and / or instructions may cause the processing circuitry 200 to manipulate data stored in the memory 202 and / or received from or via the computing system 100, the database 104, the imaging device 102, and / or the user device 106.

[0045] The processing circuitry 200 may utilize data stored in memory 202 as one or more neural networks. In some aspects, the neural network may be or include one or more landmark detection engines 206. In some other aspects, the neural network may be or include any machine learning network such as, for example, a deep learning network, a CNN, a reconstructive neural network, a generative adversarial neural network, or any other neural network capable of accomplishing functions of the computing system 100 as described herein. Some elements stored in memory 202 may be described as or referred to as instructions or instruction sets, and some functions of the computing system 100 may be implemented using machine learning techniques.

[0046] An Al system, such as a neural network, may support various inputs supportive of implementing aspects of the present disclosure. For example, a neural network may support generating outputs based on model inputs including, but not limited to, image data (e.g., provided by an imaging device 102 or an image sensor).

[0047] A neural network may include various appropriate model types supportive of implementing aspects of the present disclosure. For example, a neural network may include deep learning models (e.g., a CNN, recurrent neural network, deep reinforcement network, deep belief network, transformer network, etc.). In some examples, machine learning model(s) may include vector machines (SVMs), CNN models, transformer models, or other machine learning models appropriate with implementing aspects of the present disclosure as described herein.

[0048] A neural network may support unsupervised machine learning algorithms (e.g., principal component analysis (PCA) algorithms), semi-supervised machine learning algorithms, and supervised machine learning algorithms. A neural network may support locked execution modes and continuous learning execution modes. A neural network may support providing outputs including content, classifications, predictions, recommendations, and decisions.

[0049] The processing circuitry 200 may support landmark detection engines 206 which may be machine learning model(s), and which may be trained and / or updated based on data (e.g., training data 214) provided or accessed by any of the computing system 100, the database 104, the imaging device 102, and / or the user device 106. The landmark detection engines 206 may be built and updated by the computing system 100 based on the training data 214.

[0050] For example, the landmark detection engines 206 may be trained with one or more training sets included in the training data 214. In some aspects, the training data 214 may include multiple training sets.

[0051] In an example, the training data 214 may include a first training set that includes X-ray or other types of images with annotations indicating information such as a spine ROI, vertebra patches, a sacrum ROI, and femoral head ROI(s). A first landmark detection engine 206 may be trained based on the first training set to output information such as spine ROI, vertebra patches, sacrum ROI, and / or femoral head ROIs.

[0052] The training data 214 may include a second training set that includes X-ray or other types of images with annotations indicating information such as vertebra corners and / or centers, gaps between vertebrae, vertebrae endplates, and / or other information. A second landmark detection engine 206 may be trained based on the second training set to output information such as vertebra corners, vertebra centers, gaps between vertebra, vertebrae endplates, and / or other information based on an input image.

[0053] The training data 214 may include a third training set that includes X-ray or other types of images with annotations indicating information such as sacrum location information. A third landmark detection engine 206 may be trained based on the third training set to output information such as sacrum detection and / or location information based on an input image.

[0054] The training data 214 may include a fourth training set that includes X-ray or other types of images with annotations indicating information such as femoral head location information. Afourth landmark detection engine 206 may be trained based on the fourth training set to output information such as femoral head detection and / or location information based on an input image.

[0055] In some examples, based on the data included in the training data 214, the landmark detection engines 206 may be used to perform one or more algorithms (e.g., processing algorithms) by the processing circuitry 200 supportive of the features described herein.

[0056] In some implementations, prior to being fed into a landmark detection engine 206, such as a CNN, images may undergo one or more pre-processing steps performed using a pre-processing engine 210. Such pre-processing steps may include normalization, augmentation, and resolution adjustment. For example, images may be resized to a consistent dimension, ensuring that the landmark detection engine 206 receives inputs of uniform size. Additionally, or alternatively, pixel values of input images may be normalized, such as ranging between 0 and 1. Pre-processing may include one or more data augmentation techniques. Techniques such as rotation, zooming, cropping, and horizontal flipping may be applied to original images.

[0057] Outputs of each landmark detection engine 206, such as the first, second, third, and fourth landmark detection engines 206 described above, may be processed and / or interpreted by one or more decoders. In some implementations, a decoder can be designed to handle and interpret outputs produced by a landmark detection engine 206, such as a CNN, to transform representations generated by the landmark detection engine 206 into a comprehensible or usable format. In some implementations, a decoder may upscale and / or reconstruct feature maps into segmentation masks, assigning each pixel a class label based on features recognized by the landmark detection engine 206.

[0058] In some implementations, a decoder may be equipped with specialized layers or mechanisms to refine and enhance a landmark detection engine's 206 output further. Such mechanisms may include, but should not be considered as limited to, transposed convolutional layers, skip connections from earlier CNN layers, or attention mechanisms that weigh the importance of different features.

[0059] The landmark detection engine 206 may be capable of deriving numerical coordinates of a subject. For example, the landmark detection engine 206 may include a mechanism capable of converting an output feature map of a CNN into numerical coordinates of a detected landmark. Such a mechanism may include, for example, a differentiable spatial to numerical transform(DSNT) layer or another algorithm which may transform peaks of a feature map into numerical coordinates.

[0060] In some implementations, after an image is processed by one or more landmark detection engines 206, outputs of the landmark detection engines 206 may be subsequently post-processed using a post-processing engine 212. Post-processing procedures may include performing morphological operations to smooth segmented regions, fill small holes, or remove small noise artifacts. Post-processing may be applied to map the landmark detection engines' 206 output to clinically relevant metrics or visual representations, facilitating easier interpretation by medical professionals.

[0061] The analysis application 204 may also include program instructions and / or data that are executable by computing system 100. Example application(s) executable by computing system 100 may include landmark detection engines 206, pre-processing engines 210, and / or post-processing engine 212. The analysis application 204 may be executed by the processing circuitry 200 to implement features relating to each of the landmark detection engines 206, pre-processing engines 210, post-processing engines 212, and / or detection data 208 as described above. Other additional applications not shown may alternatively or additionally be included to provide other functionality described herein and are not depicted for the sake of simplicity.

[0062] The computing system 100 may include an operating system 216. An operating system 216, in some examples, controls the operation of components of computing system 100. For example, the operating system 216 may facilitate communication of one or more applications within memory 202 with processing circuitry 200, the communication interface 222, the user interface 220, and / or other components.

[0063] Memory 202 of the computing system 100 may also store results data 218. Results data 218 may include outputs of various landmark detection engines 206, decoders, pre-processing engines 210, post-processing engines 212, and / or other data as described herein.

[0064] The communication interface 222 may include any suitable circuitry, firmware, software, or any combination thereof for communicating with another device, such as the imaging device 102 of FIG 1. For example, communication interface 222 may include one or more processors, memory, wireless radios, antennae, transmitters, receivers, modulation and demodulation circuitry, filters, amplifiers, or the like for radio frequency communication with other devices, such as user device 106. Processing circuitry 200 may provide data to be uplinked to other computing systemsand control signals for a telemetry circuit within the communication interface 222, e.g., via an address / data bus. In some examples, communication interface 222 may provide received data to processing circuitry 200 via a multiplexer. The communication interface 222 may include a network interface card, such as an Ethernet card, an optical transceiver, a radio frequency transceiver, or any other type of device that can send and receive information. Other examples of such network interfaces may include Bluetooth™, NFC, 3G, 4G, 5G, and WI-FI™ radios.

[0065] The computing system 100, in one example, also includes a user interface 220 which may comprise one or more user interface devices. User interface devices, in some examples, are configured to receive input from a user through tactile, audio, or video feedback. Examples of user interface devices(s) include a presence- sensitive display, a mouse, a keyboard, a voice responsive system, video camera, microphone, or any other type of device for detecting a command from a user. In some examples, a presence-sensitive display includes a touch-sensitive screen.

[0066] The user interface 220 may also or alternatively comprise one or more output devices included in the computing system 100. An output device, in some examples, is configured to provide output to a user using tactile, audio, or video stimuli. An output device, in one example, includes a presence- sensitive display, a sound card, a video graphics adapter card, or any other type of device for converting a signal into an appropriate form understandable to humans or machines. In some examples, output devices include a display device. Additional examples of output devices include a speaker, a cathode ray tube (CRT) monitor, a liquid crystal display (ECD), or any other type of device that can generate intelligible output to a user.

[0067] In some implementations, the computing system 100 may utilize a user interface 220 housed separately from one or more other components of the computing system 100. In some implementations, the user interface 220 may be located proximate one or more other components of the computing system 100, while in other implementations, the user interface 220 may be located remotely from one or more other components of the computer system 100.

[0068] Although shown in Fig. 2 as a stand-alone computing system 100 for purposes of example, computing system 100 may be any component or system that includes processing circuitry or other suitable computing environment for executing software instructions and, for example, need not necessarily include one or more elements shown in Fig. 2 (e.g., communication interface 222 or user interface 220; and in some examples components such as memory 202 maynot be co-located or in the same chassis as other components). In some examples, computing system 100 may be a cloud computing system distributed across a plurality of devices.

[0069] As illustrated in Fig. 3, an analysis application 204, such as may be executed by a computing system 100 may receive an input image 302 and output content 304. In some implementations, the input image 302 may be an X-ray image. X-ray imaging may be used to capture the internal structures of a body of a patient by passing a controlled amount of X-ray radiation through it and recording the resulting shadow on a detector. The resultant grayscale images present a visualization where denser materials, such as bones, appear lighter while softer tissues appear darker.

[0070] As described herein, a series of landmark detection engines may be employed to automatically and accurately detect, classify, or segment various anatomical structures or potential pathologies present in the input image 302. Layers of each landmark detection engine may be trained to recognize patterns associated with various features such as vertebra of a spine, sacrum, and femoral heads.

[0071] In some implementations, when an analysis application 204 processes an input image 302, the output content 304 can encompass a range of data points, representations, or analyses depending on the specific tasks for which the landmark detection engines of the analysis application 204 are trained. For example, the output of the analysis application 204 may include annotations or highlighted areas of the input image 302 where vertebra locations are indicated, sacrum areas are defined, and femoral heads are located. In some implementations, Each identified region may be accompanied by a confidence score, indicating the certainty of the analysis application 204 regarding the output localization information. The analysis application 204 may be designed to segment the X-ray image and differentiate and label each vertebra, disc space, sacrum, femoral heads, or other anatomical structures, thereby producing a segmented map of the spine.

[0072] Output content 304 generated by an analysis application 204 based on an input image 302 may comprise one or more annotated images and / or metadata associated with an annotated image. As illustrated in Figs. 4A, 4B, and 4C, output content 304 may comprise a cropped version of the input image 302. The output content 304 may comprise annotations such as bounding boxes representing ROIs, such as a spinal column ROI, a sacrum ROI, a femoral head(s) ROI, and / or other ROIs. The output content 304 may also or alternatively comprise visual indicators representing centers of each vertebra, gaps between each of the vertebrae, vertebrae endplates,visual indicators representing centers of femoral heads, circles highlighting femoral heads, and / or other information.

[0073] An analysis application 204 may implement a two-phase system comprising a plurality of CNNs which perform as landmark detection engines. A first phase 500, illustrated in Fig. 5A, may use a phase one landmark detection engine 504 to output vertebra patches 510, a sacrum ROI 512, and one or more femoral ROIs 514 based on an input image 502. A second phase 550, illustrated in Fig. 5B, may use one or more phase two landmark detection engines, such as a vertebrae landmark detection engine 516, a sacrum landmark detection engine 518, and a femoral landmark detection engine 520, to output vertebra content 522, sacrum content 524, and femoral head content 526 based on the input image 502 and the vertebra patches 510, a sacrum ROI 512, and one or more femoral ROIs 514 output by the phase one landmark detection engine 504.

[0074] Each of the landmark detection engines 504, 516, 518, 520 may be based on a single CNN. The CNN may be duplicated to create each of the landmark detection engines 504, 516, 518, 520. Each of the landmark detection engines may be separately and specifically trained to produce content.

[0075] For example, the phase one landmark detection engine 504 may be trained to output vertebra patches 510, a sacrum ROI 512, and one or more femoral ROIs 514 based on an input image 502. The vertebrae landmark detection engine 516 may be trained to output vertebra content 522 based on an input image 502 and vertebra patches 510 output by the phase one landmark detection engine 504. The sacrum landmark detection engine 518 may be trained to output sacrum content 524 based on the input image 502 and the sacrum ROI 512 output by the phase one landmark detection engine 504. The femoral landmark detection engine 520 may be trained to output femoral head content 526 based on the input image 502 and the one or more femoral ROIs 514 output by the phase one landmark detection engine 504.

[0076] Each landmark detection engine may be a deep learning model, such as a CNN. Each landmark detection engine may comprise a plurality of convolutional layers. For instance, in one example, a convolutional layer may follow an input layer of the type described above. A first convolutional layer neuron may receive input from a first set of input layer neurons consisting of a given number of consecutive input layer neurons; a second convolutional layer neuron may receive input from a second set of input layer neurons consisting of the same given number of consecutive input layer neurons, but offset from the first input layer neuron of the first set of inputlayer neurons by a stride length; a third convolutional layer neuron may receive input from a third set of input layer neurons consisting of the same given number of consecutive input layer neurons, but offset from the first input layer neuron of the second set of input layer neurons by the stride length; and so on. The given number of consecutive input neurons and the stride length are different hyperparameters of the CNN. One or more fully connected hidden layers may follow the convolutional layer.

[0077] In some implementations, each landmark detection engine may comprise an encoderdecoder CNN. An encoder-decoder CNN may be designed based on a U-Net architecture or other specialized CNN architecture. However, it should be appreciated that landmark detection engines and other Al-related processes and models may involve SVMs and / or transformer models in addition to or instead of a CNN.

[0078] Each landmark detection engine may comprise an encoding path and a corresponding decoding path. The encoding path may comprise a series of convolutional layers interspersed with max-pooling operations. As an input image progresses through the encoding path, its spatial dimensions may gradually decrease while the feature representation becomes increasingly abstracted, allowing the landmark detection engine to capture characteristics of the input image.

[0079] Opposite to the encoding path, the decoding path may gradually up-sample abstracted features to restore the original spatial dimensions of the input image through a series of transposed convolutions or up-convolutions. In some implementations, skip connections between mirrored layers in the encoding and decoding paths may be used. Such connections may enable the decoder to leverage both abstracted features from the encoder and more detailed spatial information from earlier layers. As a result, an output with spatial accuracy may be produced.

[0080] In some implementations, an encoder-decoder architecture may comprise a total of eight layers. Spatial hierarchies in the input image may be captured through the encoder and spatial dimensions may be up-sampled and reconstructed in the decoder to generate a desired output.

[0081] In some implementations, the encoder portion of a landmark detection engine may comprise the five layers, which may be based on ResNet34, a deep learning model known for its residual connections that aid in preventing the vanishing gradient problem. The five layers of the encoder portion may capture progressively abstract features from an input image while reducing its spatial dimensions. Following the encoder portion, the decoder may comprise the subsequent three layers which may be designed to recover the spatial details and produce the output. To retainspatial information lost during the encoding phase, skip connections may be introduced. For example, the second layer of the encoder may be directly connected to the eighth layer of the decoder, the third layer to the seventh, and the fourth layer to the sixth. Such skip connections may facilitate the transfer of spatial and contextual information from the encoder to the decoder.

[0082] Each of the landmark detection engines 504, 516, 518, 520 may be separately trained to output specific content. Once trained, each landmark detection engine 504, 516, 518, 520 may be locked into an execution mode.

[0083] In some implementations, supervised learning methodologies may be employed to train each landmark detection engine 504, 516, 518, 520 for outputting specific content as described below. Supervised learning as described herein may provide each landmark detection engine 504, 516, 518, 520 with input samples paired with corresponding desired output samples (which may be referred to as labels or ground truth). Each landmark detection engine 504, 516, 518, 520 may process each input sample and attempt to match its output to the provided label. During the training phase, discrepancies between the output of each landmark detection engine 504, 516, 518, 520 and the given label may be measured using a loss function. The aim is to minimize this loss, which indicates the error or difference between predicted outputs and true labels. As the training progresses, internal parameters, including weights and biases, of each landmark detection engine 504, 516, 518, 520 may be iteratively adjusted using optimization algorithms like stochastic gradient descent to better align predictions of each landmark detection engine 504, 516, 518, 520 with the true labels.

[0084] Once training is complete and each landmark detection engine 504, 516, 518, 520 consistently produces outputs that closely match or align with the desired content, each landmark detection engine 504, 516, 518, 520 can then be transitioned from a training mode to an execution mode. In the execution mode, parameters of each landmark detection engine 504, 516, 518, 520 may be locked or frozen, such that the engines are no longer updated or modified.

[0085] The processes of the analysis application illustrated in Figs. 5A and 5B may be executed by a computing system 100 such as illustrated in Fig. 2. As illustrated in Fig. 5A, an input image 502 may be provided to a phase one landmark detection engine 504. The input image 502 may be provided to the computing system 100 via an imaging device 102, a database 104, and / or a user device 106. The input image 502 may be as described above in relation to the input image 302 illustrated in Fig. 3.

[0086] The input image 502 may, prior to processing by the phase one landmark detection engine 504, be processed using a pre-processing engine 210 as described above.

[0087] The phase one landmark detection engine 504 may be trained to output content such as a heatmap 508. The content may be output from a decoder 506 or from a decoder within the phase one landmark detection engine 504. For example, a decoder 506 may comprise one or more convolutional layers which generate, based on an output of the phase one landmark detection engine 504, more refined or specific outputs. In some implementations, a decoder 506 can comprise a convolutional layer with, for example, a 3x3 kernel which may capture spatial features from a received input from the phase one landmark detection engine 504.

[0088] Following the 3x3 kernel, features may be further processed by another convolutional layer with, for example, a 1x1 kernel which may serve as a feature pooling layer. The decoder 506 may be configured to output a heatmap 508. The heatmap 508 may comprise a representation of specific spatial distributions or likelihoods of features include, for example, vertebra centers, vertebra corners, sacrum locations, femoral head locations, or other information.

[0089] The heatmap 508 may in some implementations comprise a two-dimensional representation in which the intensity or color of each pixel indicates a probability or confidence level of the pixel being a feature, such as a center or a corner of a vertebra. In some implementations, the heatmap 508 may be processed by the phase one landmark detection engine 504 one or more times to refine the output heatmap 508. The phase one landmark detection engine 504 may further be configured to identify one or more ROIs within the input image 502. For example, the phase one landmark detection engine 504 may, based on a processing of the input image 502, locate vertebra, sacrum, and femoral features within the input image 502. Based on the location of the vertebra, sacrum, and femoral features, the phase one landmark detection engine 504 may determine a bounding box perimeter of areas containing one or more of the vertebra, sacrum, and femoral features. In this way, the phase one landmark detection engine 504 may output one or more of a spinal ROI, one or more vertebra patches 510 or vertebra ROIs, one or more sacrum ROIs 512, and / or one or more femoral ROIs 514 or femoral head ROIs.

[0090] As an example, an ROI 600 may be as illustrated in Fig. 6 and may be a cropped version of an input image. Bounding boxes may illustrate different ROIs and may be used to distinguish one vertebra ROI from another, for example.

[0091] Using the output ROIs and / or patches from the phase one landmark detection engine 504, a second phase 550 of the analysis application may be implemented to process the input image 502 further using one or more second phase landmark detection engines, such as a vertebra landmark detection engine 516, a sacrum landmark detection engine 518, and / or a femoral landmark detection engine 520.

[0092] A vertebra landmark detection engine 516 may be provided an input of the input image 502, which was processed by the phase one landmark detection engine 504, and one or more vertebra patches 510 or vertebra ROIs.

[0093] A vertebra patch 510 may comprise a cropped section of the input image 502 or coordinates of pixels of the input image 502 which relate to vertebra. In some implementations, the vertebra landmark detection engine 516 may process a separate vertebra patch 510 for each vertebra identified by the phase one landmark detection engine 504 in the input image 502.

[0094] The vertebra landmark detection engine 516 may be configured to process input data of an input image 502 and vertebra patches 510 and to output content such as localization information indicating centers of any vertebra contained within the input image 502 and / or within a segment of the input image 502, such as within a cropped version of the input image 502, or within a vertebra patch 510. In some implementations, the vertebra landmark detection engine 516 may be configured to detect corners of vertebra and / or to label detected vertebrae.

[0095] For example, an output 522 of the vertebra landmark detection engine 516 may be annotation and / or image data indicating centers of vertebra contained within an input image 502. In some implementations, the vertebra landmark detection engine 516 may generate a plurality of outputs, and each output may be associated with a different vertebra. Each output may indicate a center of the vertebra, such as represented by a dot, and may include information such as edges of the vertebra, such as represented by lines.

[0096] A sacrum landmark detection engine 518 may be provided an input of the input image 502, which was processed by the phase one landmark detection engine 504, and one or more sacrum ROIs 512. Based on the input image 502 and / or sacrum ROIs 512, the sacrum landmark detection engine 518 may be configured to generate an output 524 comprising an indication of a location of sacrum portions of a spine and / or location information of endplates in the input image 502.

[0097] The sacrum landmark detection engine 518 may be configured to identify the location and orientation of the sacrum and / or endplates in an image, such as an X-ray, of the human spine.For example, an output 524 of a sacrum landmark detection engine 518 may comprise a line indicating an endplate and a dot representing a center of the sacrum.

[0098] A femoral landmark detection engine 520 may be provided an input of the input image 502, which was processed by the phase one landmark detection engine 504, and one or more femoral ROIs 514. Based on the input image 502 and / or femoral ROIs 514, the femoral landmark detection engine 520 may be configured to generate an output 526 comprising an indication of a location of femoral heads and / or measurements of a radius of one or more femoral heads in the input image 502.

[0099] The femoral landmark detection engine 520 may be configured to identify the location of femoral heads and / or to measure radii of femoral heads in an image, such as an X-ray, of the human spine. For example, an output 526 of a femoral landmark detection engine 520 may comprise circles outlining each femoral head and / or dots representing centers of each femoral head.

[0100] Additionally, or in alternative to one or more of the landmark detection engines illustrated in Fig. 5B, phase two of the analysis application may include one or more other landmark detection engines. For example, a landmark detection engine may be configured to detect neurostimulation electrodes, or other characteristics which may be represented in an input image 502.

[0101] In some implementations, each of the outputs 522, 524, 526 of the various landmark detection engines 516, 518, 520 of the second phase of the analysis application may be combined into a single output 528. For example, a wrapper algorithm may be executed to add content output by each of the vertebra landmark detection engine 516, the sacrum landmark detection engine 518, and the femoral landmark detection engine 520 to the input image 502 to create an ultimate output of the analysis application.

[0102] In some implementations, a wrapper algorithm may integrate the outputs of various landmark detection engines to produce a comprehensive result. Once each of the landmark detection engines process the input image 502, the wrapper algorithm may retrieve the outputs and combine the results. The combining process can be achieved through various methods, depending on the problem domain and the desired output.

[0103] Fig. 7 illustrates various outputs of the landmark detection engines illustrated in Figs. 5A and 5B. A heatmap 702 with a bounding box illustrating a spinal column ROI may be output by a phase one landmark detection engine 504. A phase one landmark detection engine 504 may also or alternatively output an indication of vertebra centers 704. A femoral landmark detection engine520 may output an indication of femoral heads 706. A vertebra landmark detection engine 516 may output an indication of vertebra corners 708 and / or vertebra labeling information 712. A sacrum landmark detection engine may output an indication of a location of a sacrum endplate 710. All of the information output by the various landmark detection engines of the analysis application may be combined, such as via a wrapper algorithm, into a single output such as the output 528 illustrated in Fig. 5B.

[0104] Fig. 8 is a flowchart illustrating an example method 800 in accordance with the techniques of the present disclosure. For convenience, Fig. 8 is described with respect to the analysis application illustrated in Figs. 5A and 5B as well as the computing system 100 illustrated in Figs. 1 and 2. The flowchart of this disclosure is presented as an example. Other examples in accordance with techniques of this disclosure may include more, fewer, or different actions, or actions may be performed in different orders or in parallel. The method 800 of FIG. 8 may be performed by an Al system implemented on a computing system 100 and / or other devices.

[0105] At 802, processing circuitry 200 of the computing system 100 may receive an input image. An input image may be received from memory 202 within the computing system 100 or, via a communication interface 222, from an external source, such as an imaging device 102, a database 104, and / or a user device 106.

[0106] At 804, the processing circuitry 200 may perform a landmark detection based on the input image by processing the input image with a phase one landmark detection engine. As described above, an output of the phase one landmark detection engine may be generated using a decoder, in some implementations, one or more refinement feedback processes may be performed to finetune the output of the phase one landmark detection engine.

[0107] The landmark detection performed at 804 may be a multi-stage process in which the precision of the landmark detection is enhanced by focusing on specific regions of interest in successive iterations. For example, an initial landmark detection stage may involve the landmark detection engine processing the entire input image to detect potential landmarks. The output from the initial landmark detection stage may include a set of preliminary landmark locations. Next, a refinement stage, in one or more iterations, may involve the processing circuitry 200 identifying regions of interest (ROIs) around each of the preliminary landmark locations and processing each ROI with the landmark detection engine. After processing the ROIs in a first refinement iteration, the landmark detection engine may produce updated landmark locations. In some implementations,this refinement process may repeat with greater specificity, each time extracting new ROIs centered around updated locations for even more focused analysis. After successive refinements, the output of the landmark detection engine may converge to one or more landmark locations.

[0108] At 806, the processing circuitry 200 may generate one or more vertebra patches and ROIs. As described above, the phase one landmark detection engine may be configured to output vertebra patches, a sacrum ROI, and / or a femoral ROI. In some implementations, the phase one landmark detection engine may be configured to identify a plurality of vertebrae in an input image and to output a separate vertebra patch for each identified vertebra. The vertebra patches may comprise cropped versions of the original input image or may be used by a vertebra landmark detection engine to locate a particular vertebra.

[0109] The output vertebra patches, sacrum ROI, and / or femoral ROI may each be received by a separate phase two landmark detection engine. At 808, each of the phase two landmark detection engines may process one or more of the vertebra patches, the sacrum ROI, and / or the femoral ROI. For example, a vertebra landmark detection engine may process one or more vertebra patches, a sacrum landmark detection engine may process the sacrum ROI, and a femoral detection engine may process the femoral ROI.

[0110] At 810, each of the phase two landmark detection engines may be used to generate vertebra detection, vertebra corner refinement, sacrum detection, femoral head detection, and / or other content. For example, a vertebra landmark detection engine may output location information of vertebra corners and / or centers. The vertebra corner information output by a vertebra landmark detection engine may be a corner refinement which may refine locations of corners of vertebra included in a heatmap output by the phase one landmark detection engine. As another example, a sacrum landmark detection engine may output sacrum location and / or detection information and a femoral landmark detection engine may output femoral head location and / or detection information.

[0111] At 812, the processing circuitry 200 may generate a combined output which may include information from each of the landmark detection engines. In some implementations, generating the combined output may comprise performing one or more post-processing functions before compiling the information from each of the landmark detection engines into the final output.

[0112] As noted above, the present disclosure encompasses methods with fewer than all of the steps identified in Fig. 8 (and the corresponding description of the process flows), as well as methods that include additional steps beyond those identified in Fig. 8 (and the correspondingdescription of the process flows). The present disclosure also encompasses methods that include one or more steps from one method described herein, and one or more steps from another method described herein.

[0113] The foregoing is not intended to limit the disclosure to the form or forms disclosed herein. In the foregoing Detailed Description, for example, various features of the disclosure are grouped together in one or more aspects, implementations, and / or configurations for the purpose of streamlining the disclosure. The features of the aspects, implementations, and / or configurations of the disclosure may be combined in alternate aspects, implementations, and / or configurations other than those discussed above. This method of disclosure is not to be interpreted as reflecting an intention that the claims require more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed aspect, implementation, and / or configuration. Thus, the following claims are hereby incorporated into this Detailed Description, with each claim standing on its own as a separate preferred implementation of the disclosure.

[0114] Moreover, though the foregoing has included description of one or more aspects, implementations, and / or configurations and certain variations and modifications, other variations, combinations, and modifications are within the scope of the disclosure, e.g., as may be within the skill and knowledge of those in the art, after understanding the present disclosure. It is intended to obtain rights which include alternative aspects, implementations, and / or configurations to the extent permitted, including alternate, interchangeable and / or equivalent structures, functions, ranges or steps to those claimed, whether or not such alternate, interchangeable and / or equivalent structures, functions, ranges or steps are disclosed herein, and without intending to publicly dedicate any patentable subject matter.

[0115] Example aspects of the present disclosure include any of the following:

[0116] Example 1. A computing system, comprising processing circuitry configured to receive an image; execute a first artificial intelligence system to perform a first landmark detection using the image to generate one or more of a vertebra patch, a sacrum region of interest (ROI), and a femoral ROI; and execute at least a second artificial intelligence system to perform a second landmark detection using the image to generate one or more of vertebra center data, vertebra corner data, vertebra labeling data, sacrum endplate data, and femoral head data, based on the one or more of the vertebra patch, the sacrum ROI, and the femoral ROI.

[0117] Example 2. The computing system of example 1, wherein the image comprises an x-ray of a side profile of a human.

[0118] Example 3. The computing system of example 1 or 2, The computing system of example 1, wherein each of the first artificial intelligence system and second artificial intelligence system comprises a convolutional neural network.

[0119] Example 4. The computing system of any one of examples 1-3, wherein the processing circuitry creates the first artificial intelligence system by training a model to generate the one or more of the vertebra patch, sacrum ROI, and femoral ROI, and to create the at least the second artificial intelligence system by training the model to generate the one or more of the vertebra center data, vertebra corner data, vertebra labeling data, sacrum endplate data, and femoral head data.

[0120] Example 5. The computing system of any one of examples 1-4, wherein the processing circuitry performs a decoder process to generate the one or more of the vertebra patch, sacrum ROI, and femoral ROI.

[0121] Example 6. The computing system of any one of examples 1-5, wherein the processing circuitry performs a decoder process to generate the one or more of the vertebra center data, vertebra corner data, vertebra labeling data, sacrum endplate data, and femoral head data.

[0122] Example 7. The computing system of any one of examples 1-6, wherein executing the at least the second artificial intelligence system comprises executing the second artificial intelligence system to generate the vertebra center data, vertebra corner data, and vertebra labeling data, executing a third artificial intelligence system to generate the sacrum endplate data, and executing a fourth artificial intelligence system to generate the femoral head data.

[0123] Example 8. The computing system of any one of examples 1-7, wherein the processing circuitry is further configured to: create the first artificial intelligence system by training a model to generate the one or more of the vertebra patch, sacrum ROI, and femoral ROI; create the second artificial intelligence system by training the model to generate the one or more of the vertebra center data, vertebra corner data, vertebra labeling data; create the third artificial intelligence system by training the model to generate the sacrum endplate data; and create the third artificial intelligence system by training the model to generate the femoral head data.

[0124] Example 9. A computer-readable data storage medium having instructions stored thereon that, when executed by processing circuitry, cause the processing circuitry to: receive animage; execute a first artificial intelligence system to perform a first landmark detection using the image to generate one or more of a vertebra patch, a sacrum region of interest (ROI), and a femoral ROI; and execute at least a second artificial intelligence system to perform a second landmark detection using the image to generate one or more of vertebra center data, vertebra corner data, vertebra labeling data, sacrum endplate data, and femoral head data, based on the one or more of the vertebra patch, the sacrum ROI, and the femoral ROI.

[0125] Example 10. The computer-readable data storage medium of example 9, wherein the image comprises an x-ray of a side profile of a human.

[0126] Example 11. The computer-readable data storage medium of example 9 or 10, wherein each of the first and second artificial intelligence system comprises a convolutional neural network.

[0127] Example 12. The computer-readable data storage medium of any one of examples 9-11, wherein the instructions further cause the process circuitry to create the first artificial intelligence system by training a model to generate the one or more of the vertebra patch, sacrum ROI, and femoral ROI, and to create the at least the second artificial intelligence system by training the model to generate the one or more of the vertebra center data, vertebra corner data, vertebra labeling data, sacrum endplate data, and femoral head data.

[0128] Example 13. The computer-readable data storage medium of any one of examples 9-12, wherein the instructions further cause the process circuitry to perform a decoder process to generate the one or more of the vertebra patch, sacrum ROI, and femoral ROI.

[0129] Example 14. The computer-readable data storage medium of any one of examples 9-13, wherein the instructions further cause the process circuitry to perform a decoder process to generate the one or more of the vertebra center data, vertebra corner data, vertebra labeling data, sacrum endplate data, and femoral head data.

[0130] Example 15. A method, comprising receiving an image; executing a first artificial intelligence system to perform a first landmark detection using the image to generate one or more of a vertebra patch, a sacrum region of interest (ROI), and a femoral ROI; and executing at least a second artificial intelligence system to perform a second landmark detection using the image to generate one or more of vertebra center data, vertebra corner data, vertebra labeling data, sacrum endplate data, and femoral head data, based on the one or more of the vertebra patch, the sacrum ROI, and the femoral ROI.

[0131] Example 16. The method of example 15, wherein the image comprises an x-ray of a side profile of a human.

[0132] Example 17. The method of example 15 or 16, wherein each of the first artificial intelligence system and the second artificial intelligence system comprises a convolutional neural network.

[0133] Example 18. The method of any one of examples 15-17, further comprising: creating the first artificial intelligence system by training a model to generate the one or more of the vertebra patch, sacrum ROI, and femoral ROI, and creating the at least the second artificial intelligence system by training the model to generate the one or more of the vertebra center data, vertebra corner data, vertebra labeling data, sacrum endplate data, and femoral head data.

[0134] Example 19. The method of any one of examples 15-18, further comprising performing a decoder process to generate the one or more of the vertebra patch, sacrum ROI, and femoral ROI.

[0135] Example 20. The method of any one of examples 15-19, further comprising performing a decoder process to generate the one or more of the vertebra center data, vertebra corner data, vertebra labeling data, sacrum endplate data, and femoral head data.

[0136] Various examples include any aspect in combination with any one or more other aspects, any one or more of the features disclosed herein, any one or more of the features as substantially disclosed herein, any one or more of the features as substantially disclosed herein in combination with any one or more other features as substantially disclosed herein, any one of the aspects / features / implementations in combination with any one or more other aspects / features / implementations and use of any one or more of the aspects or features as disclosed herein.

[0137] It is to be appreciated that any feature described herein can be claimed in combination with any other feature(s) as described herein, regardless of whether the features come from the same described implementation.

[0138] It is to be appreciated that any feature described herein can be claimed in combination with any other feature(s) as described herein, regardless of whether the features come from the same described implementation.

[0139] The phrases “at least one”, “one or more”, “or”, and “and / or” are open-ended expressions that are both conjunctive and disjunctive in operation. For example, each of the expressions “at least one of A, B and C”, “at least one of A, B, or C”, “one or more of A, B, and C”, “one or more 1of A, B, or C”, “A, B, and / or C”, and “A, B, or C” means A alone, B alone, C alone, A and B together, A and C together, B and C together, or A, B and C together.

[0140] The term “a” or “an” entity refers to one or more of that entity. As such, the terms “a” (or “an”), “one or more”, and “at least one” can be used interchangeably herein. It is also to be noted that the terms “comprising”, “including”, and “having” can be used interchangeably.

[0141] The term “automatic” and variations thereof, as used herein, refers to any process or operation, which is typically continuous or semi-continuous, done without material human input when the process or operation is performed. However, a process or operation can be automatic, even though performance of the process or operation uses material or immaterial human input, if the input is received before performance of the process or operation. Human input is deemed to be material if such input influences how the process or operation will be performed. Human input that consents to the performance of the process or operation is not deemed to be “material”.

[0142] Aspects of the present disclosure may take the form of an implementation that is entirely hardware, an implementation that is entirely software (including firmware, resident software, micro-code, etc.) or an implementation combining software and hardware aspects that may all generally be referred to herein as a “circuit”, “module”, or “system”. Any combination of one or more computer-readable medium(s) may be utilized. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium.

[0143] A computer-readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhau stive list) of the computer-readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, RAM, ROM, EPROM, or Flash memory, an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer-readable storage medium may be any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device.

[0144] A computer-readable signal medium may include a propagated data signal with computer-readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including, but not limited to,electro-magnetic, optical, or any suitable combination thereof. A computer-readable signal medium may be any computer-readable medium that is not a computer-readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any appropriate medium, including, but not limited to, wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0145] The terms “determine”, “calculate”, “compute”, and variations thereof, as used herein, are used interchangeably, and include any type of methodology, process, mathematical operation, or technique.

Claims

CLAIMSWhat is claimed is:

1. A computing system, comprising processing circuitry to: receive an image (502); execute a first artificial intelligence system (500) to perform a first landmark detection (504) using the image to generate one or more of a vertebra patch (510), a sacrum region of interest (ROI) (512), and a femoral ROI (514); and execute at least a second artificial intelligence system (550) to perform a second landmark detection (516, 518, 520) using the image to generate one or more of vertebra center data (522), vertebra corner data (522), vertebra labeling data (522), sacrum endplate data (524), and femoral head data (526), based on the one or more of the vertebra patch, the sacrum ROI, and the femoral ROI.

2. The computing system of claim 1, wherein the image comprises an x-ray (502) of a side profile of a human.

3. The computing system of claims 1 or 2, wherein each of the first artificial intelligence system and second artificial intelligence system comprises a convolutional neural network (500, 550).

4. The computing system of any one of claims 1 to 3, wherein the processing circuitry creates the first artificial intelligence system by training a model (516, 518, 520) to generate the one or more of the vertebra patch, sacrum ROI, and femoral ROI, and to create the at least the second artificial intelligence system by training the model to generate the one or more of the vertebra center data, vertebra corner data, vertebra labeling data, sacrum endplate data, and femoral head data.

5. The computing system of any one of claims 1 to 4, wherein the processing circuitry performs a decoder process (506) to generate the one or more of the vertebra patch, sacrum ROI, and femoral ROI.

6. The computing system of any one of claims 1 to 5, wherein the processing circuitry performs a decoder process (506) to generate the one or more of the vertebra center data, vertebra corner data, vertebra labeling data, sacrum endplate data, and femoral head data.

7. The computing system of any one of claims 1 to 6, wherein executing the at least the second artificial intelligence system comprises executing the second artificial intelligencesystem to generate the vertebra center data, vertebra corner data, and vertebra labeling data, executing a third artificial intelligence system (518) to generate the sacrum endplate data, and executing a fourth artificial intelligence system (520) to generate the femoral head data.

8. The computing system of any one of claims 1 to 7, wherein the processing circuitry is further to: create the first artificial intelligence system (504) by training a model to generate the one or more of the vertebra patch, sacrum ROI, and femoral ROI; create the second artificial intelligence system (516) by training the model to generate the one or more of the vertebra center data, vertebra corner data, vertebra labeling data; create the third artificial intelligence system (518) by training the model to generate the sacrum endplate data; and create the third artificial intelligence system (520) by training the model to generate the femoral head data.

9. A computer-readable data storage medium (502) having instructions stored thereon that, when executed by processing circuitry (200), cause the processing circuitry to: receive an image (502); execute a first artificial intelligence system (500) to perform a first landmark detection (504) using the image to generate one or more of a vertebra patch (510), a sacrum region of interest (ROI) (512), and a femoral ROI (514); and execute at least a second artificial intelligence system (550) to perform a second landmark detection (516, 518, 520) using the image to generate one or more of vertebra center data (522), vertebra corner data (522), vertebra labeling data (522), sacrum endplate data (524), and femoral head data (526), based on the one or more of the vertebra patch, the sacrum ROI, and the femoral ROI.

10. The computer-readable data storage medium of claim 9, wherein the image comprises an x-ray (502) of a side profile of a human.

11. The computer-readable data storage medium of claims 9 or 10, wherein each of the first and second artificial intelligence system comprises a convolutional neural network (500, 550).

12. The computer-readable data storage medium of any one of claims 9 to 11, wherein the instructions further cause the process circuitry to create the first artificial intelligence system by training a model (516, 518, 520) to generate the one or more of the vertebra patch, sacrum ROI,and femoral ROI, and to create the at least the second artificial intelligence system by training the model to generate the one or more of the vertebra center data, vertebra corner data, vertebra labeling data, sacrum endplate data, and femoral head data.

13. The computer-readable data storage medium of any one of claims 9 to 12, wherein the instructions further cause the process circuitry to perform a decoder process (506) to generate the one or more of the vertebra patch, sacrum ROI, and femoral ROI.

14. The computer-readable data storage medium of any one of claims 9 to 13, wherein the instructions further cause the process circuitry to perform a decoder process (506) to generate the one or more of the vertebra center data, vertebra corner data, vertebra labeling data, sacrum endplate data, and femoral head data.

15. A method (800), comprising: receiving an image (502); executing a first artificial intelligence system (500) to perform a first landmark detection (504) using the image to generate one or more of a vertebra patch (510), a sacrum region of interest (ROI) (512), and a femoral ROI (514); and executing at least a second artificial intelligence system (550) to perform a second landmark detection (516, 518, 520) using the image to generate one or more of vertebra center data (522), vertebra corner data (522), vertebra labeling data (522), sacrum endplate data (524), and femoral head data (526), based on the one or more of the vertebra patch, the sacrum ROI, and the femoral ROI.

Images