System and method for analyzing medical images based on spatio-temporal data

Abstract

A system, method, and program product for analyzing spatiotemporal medical images using an artificial neural network are provided, the method including: capturing a series of medical images of a patient, the series of medical images including visual movement of at least one entity; tracking time-varying spatial data associated with the at least one entity based on the visual movement; generating spatiotemporal data by correlating the time-varying spatial data with the series of medical images; and analyzing the series of medical images based on an artificial neural network including multiple layers, one or more layers of the multiple layers each combining features from at least three different scales, and at least one layer of the multiple layers of the artificial neural network configured to learn spatiotemporal relationships based on the spatiotemporal data.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Patent Application No. 62 / 904,728, filed Sep. 24, 2019, the disclosure of which is incorporated by reference in its entirety.

[0002] 1. Field The present disclosure relates generally to artificial neural networks and, in non-limiting embodiments, to systems, methods, and computer program products for analyzing medical images based on spatio-temporal data using artificial neural networks. [Background technology]

[0003] 2.Technical issues Medical images acquired using optical coherence tomography (OCT), ultrasound, MRI, or other sequential acquisition methods may include a sequence of cross-sectional slices (or volumes, e.g., full frame OCT) taken through a portion of a patient's body. These images undergo changes from slice (or volume) to slice (or volume) based on a variety of different types of motion and / or orientation of the patient (internal or external) and / or the equipment used (e.g., ultrasound probe), etc. Additionally, existing neural networks used to analyze such images consider each image in the sequence independently of all other images, and therefore, these neural networks cannot model motion, take into account previous images in the sequence, or take into account changing motion and / or orientation of the patient and / or equipment.

[0004] Intima-Media Thickness (IMT) is a parameter that quantifies risk, such as the formation of atherosclerotic plaques, in clinical applications. However, specifically, IMT can be used to track the functional progress of hand transplant recipients (or recipients of other composite tissue allografts), where the gold standard for monitoring change is currently histopathology. Recently, Ultra-High Frequency Ultrasound (UHFUS) has been shown to quantitatively measure IMT with a resolution of vasculature at 0.03 mm within a shallow tissue depth of 1 cm. However, this improved resolution also results in increased speckle noise that compromises vessel boundaries, in contrast to conventional ultrasound and high frequency ultrasound (HFUS) imaging devices. Furthermore, vessels at shallow depths are highly distorted (due to transducer pressure and motion) in contrast to vessels deeper in the body, such as the carotid artery. It is therefore desirable to have a system that accounts for sub-millimeter localization of rapidly moving and pulsating vascular contours and other entities in UHFUS and HFUS sequences to compare changes in IMT over time.

[0005] Prior art vessel-based segmentation methods for ultrasound sequences fall into two categories: the first category, such as state-of-the-art level-set methods for HFUS and UHFUS, are quick to implement but lack the robustness required for clinical applications due to parameter fine-tuning; the second category, including learning-based methods, are robust to changes in scan settings and image quality variations but are task-specific without properly exploiting frame-to-frame vessel dynamics and therefore cannot be applied to a variety of different biomedical imaging techniques. Summary of the Invention [Means for solving the problem]

[0006] According to a non-limiting embodiment or aspect, provided is a method of analyzing spatio-temporal medical images using an artificial neural network, the method comprising: capturing a series of medical images of a patient using an imaging device, the series of medical images including visual movement of at least one entity including at least a portion of at least one of the patient and an object, tracking, using a computing device, time-varying spatial data associated with the at least one entity based on the visual movement, generating spatio-temporal data by correlating, using a computing device, the time-varying spatial data with the series of medical images, and analyzing, using a computing device, the series of medical images based on an artificial neural network comprising a plurality of layers, wherein each of one or more layers of the plurality of layers combines features from at least three different scales, and wherein at least one layer of the plurality of layers of the artificial neural network is configured to learn spatio-temporal relationships based on the spatio-temporal data.

[0007] According to non-limiting embodiments or aspects, the one or more layers combining features from the at least three different scales include dilated convolutions of different scales. According to non-limiting embodiments or aspects, the one or more layers combining features from the at least three different scales include tight and / or residual connections between at least a subset of layers of the plurality of layers, the at least a subset of layers including features from at least three different scales. According to non-limiting embodiments or aspects, the one or more layers combining features from the at least three different scales include convolutions of at least two different scales and connections for a subset of layers of the plurality of layers including features from at least two different scales, resulting in features of at least three different scales. According to non-limiting embodiments or aspects, the at least one entity includes at least one of the following: equipment, the imaging device, a physical artifact, a manifest artifact, or any combination thereof.

[0008] According to non-limiting embodiments or aspects, tracking the time-varying spatial data includes tracking at least one of the following: translational / rotational position of the at least one entity, velocity of the at least one entity, acceleration of the at least one entity, inertial measurements of the at least one entity, or any combination thereof. According to non-limiting embodiments or aspects, tracking the time-varying spatial data is based on at least one of the following: inertial measurement unit, tracking system, position sensor, robot kinematics, inverse kinematics, or any combination thereof. According to non-limiting embodiments or aspects, the spatio-temporal data includes at least one of the following: data representative of internal motion within the patient's body, data representative of external motion of the patient's body, data representative of a motion of an instrument, data representative of an angle of the instrument, data representative of a motion resulting in a deformation of the patient's body, or any combination thereof. According to non-limiting embodiments or aspects, the artificial neural network includes an encoder and a decoder, and at least one of the decoder and the encoder is configured to utilize the spatio-temporal data as an input. According to a non-limiting embodiment or aspect, the artificial neural network includes at least one of the following: a long short-term memory (LSTM) unit, a gated recurrent unit (GRU), a temporal convolutional network, or any combination thereof.

[0009] According to non-limiting embodiments or aspects, the spatial data includes a position and / or orientation of the patient and / or device. According to non-limiting embodiments or aspects, analyzing the series of medical images includes identifying at least one anatomical structure in the series of images, the at least one anatomical structure being one of the following: vessels, arteries, veins, ligaments, nerves. , muscle fibers, fascial fibers or fascial networks, fat nodules, grafted fat nodules, lymphatic tissue, skin patches, tendons, bones, cartilage fragments, pulmonary pleural lines, heart valves, atrial chambers, cardiac surfaces, trachea, brain regions, tubes, trabecular meshwork, corneal layers, retinal layers, eye lenses, ocular surfaces, soft tissues, Vogt's palisades of the corneal limbus, organs, extracellular structures, intercellular structures, cells, or any combination thereof. According to non-limiting embodiments or aspects, the series of medical images includes at least one of the following: ultrasound images, optical coherence tomography (OCT) images, CT images, MRI images, PET images, SPECT images, fluoroscopic images, X-ray images, mammography images, tomosynthetic images, photoacoustic images, audio-optical images, endoscopic images, microscopic images, fundus images, scanning laser ophthalmoscope (SLO) images, smartphone images, 3D (depth) images, focal stack images, bright field images, visible light images, infrared images, ultraviolet images, thermal images, multispectral images, tomographic images, projection images, integrated images, reconstructed images, or any combination thereof. According to non-limiting embodiments or aspects, analyzing the series of medical images includes segmenting one or more vessels shown in the series of medical images.

[0010] According to non-limiting embodiments or aspects, at least a portion of the artificial neural network includes dilated convolutions. According to non-limiting embodiments or aspects, at least a portion of the artificial neural network includes residual connections and / or skip connections. According to non-limiting embodiments or aspects, at least a portion of the artificial neural network includes dilated convolutions. According to non-limiting embodiments or aspects, at least a portion of the artificial neural network includes residual connections and / or skip connections.

[0011] According to a non-limiting embodiment or aspect, provided is a system for analyzing spatio-temporal medical images using an artificial neural network, the system comprising: a computing device programmed or configured to: capture, with an imaging device, a series of medical images of a patient, the series of medical images including visual movement of at least one entity including at least a portion of at least one of the patient and an object; track, with a computing device, time-varying spatial data associated with the at least one entity based on the visual movement; generate spatio-temporal data by correlating, with a computing device, the time-varying spatial data with the series of medical images; and analyze, with a computing device, the series of medical images based on an artificial neural network comprising a plurality of layers, wherein each of one or more layers of the plurality of layers combines features from at least three different scales, and wherein at least one layer of the plurality of layers of the artificial neural network is configured to learn spatio-temporal relationships based on the spatio-temporal data.

[0012] According to non-limiting embodiments or aspects, the one or more layers combining features from the at least three different scales include dilated convolutions of different scales. According to non-limiting embodiments or aspects, the one or more layers combining features from the at least three different scales include tight and / or residual connections between at least a subset of layers of the plurality of layers, the at least a subset of layers including features from at least three different scales. According to non-limiting embodiments or aspects, the one or more layers combining features from the at least three different scales include convolutions of at least two different scales and connections for a subset of layers of the plurality of layers including features from at least two different scales, resulting in features of at least three different scales. According to non-limiting embodiments or aspects, the at least one entity includes at least one of the following: equipment, the imaging device, a physical artifact, a manifest artifact, or any combination thereof.

[0013] According to a non-limiting embodiment or aspect, tracking the time-varying spatial data includes tracking the translation / rotation position of the at least one entity, the velocity of the at least one entity, , an acceleration of the at least one entity, an inertial measurement of the at least one entity, or any combination thereof. According to non-limiting embodiments or aspects, tracking the time-varying spatial data is based on at least one of the following: an inertial measurement unit, a tracking system, a position sensor, robot kinematics, inverse kinematics, or any combination thereof. According to non-limiting embodiments or aspects, the spatio-temporal data includes at least one of the following: data representative of internal motion within the patient's body, data representative of external motion of the patient's body, data representative of a motion of an instrument, data representative of an angle of the instrument, data representative of a motion resulting in a deformation of the patient's body, or any combination thereof. According to non-limiting embodiments or aspects, the artificial neural network includes an encoder and a decoder, and at least one of the decoder and the encoder is configured to utilize the spatio-temporal data as an input. According to non-limiting embodiments or aspects, the artificial neural network includes at least one of the following: a long short-term memory (LSTM) unit, a gated recurrent unit (GRU), a temporal convolutional network, or any combination thereof.

[0014] According to non-limiting embodiments or aspects, the spatial data includes a position and / or orientation of the patient and / or device. According to non-limiting embodiments or aspects, analyzing the series of medical images includes identifying at least one anatomical structure in the series of images, the at least one anatomical structure including at least one of the following: vessels, arteries, veins, ligaments, nerves, muscle fibers, fascial fibers or meshwork, fat nodules, grafted fat nodules, lymphatic tissue, skin patches, tendons, bones, cartilage fragments, pulmonary pleural lines, heart valves, atrial chambers, cardiac surfaces, trachea, brain regions, tubes, trabecular meshwork, corneal layers, retinal layers, eye lenses, ocular surfaces, soft tissues, Vogt's palisades of the corneal limbus, organs, extracellular structures, intercellular structures, cells, or any combination thereof. According to non-limiting embodiments or aspects, the series of medical images includes at least one of the following: ultrasound images, optical coherence tomography (OCT) images, CT images, MRI images, PET images, SPECT images, fluoroscopic images, X-ray images, mammography images, tomosynthetic images, photoacoustic images, audio-optical images, endoscopic images, microscopic images, fundus images, scanning laser ophthalmoscope (SLO) images, smartphone images, 3D (depth) images, focal stack images, bright field images, visible light images, infrared images, ultraviolet images, thermal images, multispectral images, tomographic images, projection images, integrated images, reconstructed images, or any combination thereof. According to non-limiting embodiments or aspects, analyzing the series of medical images includes segmenting one or more vessels shown in the series of medical images.

[0015] According to a non-limiting embodiment or aspect, provided is a computer program product for analyzing medical images using a neural network, the computer program product including at least one non-transitory computer-readable medium including instructions that, when executed by a computing device, cause the computing device to: capture a series of medical images of a patient using an imaging device, the series of medical images including visual movement of at least one entity including at least a portion of at least one of the patient and an object, track, using a computing device, time-varying spatial data associated with the at least one entity based on the visual movement, generate spatio-temporal data by correlating, using a computing device, the time-varying spatial data with the series of medical images, and analyze, using a computing device, the series of medical images based on an artificial neural network including a plurality of layers, wherein each of one or more layers of the plurality of layers combines features from at least three different scales, and wherein at least one layer of the plurality of layers of the artificial neural network is configured to learn spatio-temporal relationships based on the spatio-temporal data.

[0016] According to a non-limiting embodiment or aspect, there is provided an artificial neural network. 16. A method for analyzing spatio-temporal medical images using an artificial neural network, the method comprising: capturing a series of medical images of a patient using an imaging device, the series of medical images including visual movement of at least one entity including at least a portion of at least one of the patient and an object, tracking using a computing device time-varying spatial data associated with the at least one entity based on the visual movement, generating spatio-temporal data by correlating the time-varying spatial data with the series of medical images using a computing device, and analyzing the series of medical images based on an artificial neural network comprising a plurality of layers, the artificial neural network including dilated convolutions and / or tight coupling between a plurality of layers of different scales and resolutions that combine features from at least three different scales, and at least one layer of the plurality of layers configured to learn spatio-temporal relationships based on the spatio-temporal data. According to non-limiting embodiments or aspects, provided is a system for analyzing spatio-temporal medical images using an artificial neural network, the system comprising: a computing device programmed or configured to: capture, with an imaging device, a series of medical images of a patient, the series of medical images including visual movement of at least one entity including at least a portion of at least one of the patient and an object; track, with a computing device, time-varying spatial data associated with the at least one entity based on the visual movement; generate spatio-temporal data by correlating, with a computing device, the time-varying spatial data with the series of medical images; and analyze, with a computing device, the series of medical images based on an artificial neural network comprising a plurality of layers, the artificial neural network including dilated convolutions and / or tight coupling between a plurality of layers of different scales and resolutions that combine features from at least three different scales, and at least one of the plurality of layers configured to learn spatio-temporal relationships based on the spatio-temporal data.According to a non-limiting embodiment or aspect, provided is a computer program product for analyzing medical images using a neural network, the computer program product including at least one non-transitory computer-readable medium including instructions that, when executed by a computing device, cause the computing device to: capture a series of medical images of a patient using an imaging device, the series of medical images including visual movement of at least one entity including at least a portion of at least one of the patient and an object, track time-varying spatial data associated with the at least one entity based on the visual movement, generate spatio-temporal data by correlating the time-varying spatial data with the series of medical images, and analyze the series of medical images based on an artificial neural network including a plurality of layers, the artificial neural network including dilated convolutions and / or tight coupling between a plurality of layers of different scales and resolutions that combine features from at least three different scales, and at least one of the plurality of layers configured to learn spatio-temporal relationships based on the spatio-temporal data.

[0017] Other non-limiting embodiments or aspects are described in the following numbered clauses:

[0018] Clause 1: A method of analyzing spatio-temporal medical images using an artificial neural network, comprising: capturing a series of medical images of a patient using an imaging device, the series of medical images including visual movement of at least one entity including at least a portion of at least one of the patient and an object, tracking, using a computing device, time-varying spatial data associated with the at least one entity based on the visual movement; generating spatio-temporal data by correlating, using a computing device, the time-varying spatial data with the series of medical images; and analyzing, using a computing device, the series of medical images based on an artificial neural network comprising a plurality of layers, wherein one or more layers of the plurality of layers each include at least and combining features from at least three different scales, and at least one layer of the plurality of layers of the artificial neural network is configured to learn spatio-temporal relationships based on the spatio-temporal data.

[0019] Clause 2: The method of clause 1, wherein the one or more layers that combine features from the at least three different scales include dilated convolutions at different scales.

[0020] Clause 3: The method of clause 1 or 2, wherein the one or more layers combining features from the at least three different scales comprises dense and / or residual coupling between at least a subset of layers of the plurality of layers, the at least a subset of layers comprising features from at least three different scales.

[0021] Clause 4: The method of any of clauses 1 to 3, wherein the one or more layers combining features from the at least three different scales comprises a convolution of at least two different scales and a combination for a subset of layers of the plurality of layers comprising features from at least two different scales, resulting in features of at least three different scales.

[0022] Clause 5: The method of any of clauses 1 to 4, wherein the at least one entity comprises at least one of the following: equipment, the imaging device, a physical artifact, a manifest artifact, or any combination thereof.

[0023] Clause 6: The method of any of clauses 1 to 5, wherein tracking the time-varying spatial data includes tracking at least one of the following: translational / rotational position of the at least one entity, velocity of the at least one entity, acceleration of the at least one entity, inertial measurements of the at least one entity, or any combination thereof.

[0024] Clause 7: A method according to any of clauses 1 to 6, wherein tracking the time-varying spatial data is based on at least one of the following: an inertial measurement unit, a tracking system, a position sensor, robot kinematics, inverse kinematics, or any combination thereof.

[0025] Clause 8: The method of any one of clauses 1 to 7, wherein the spatiotemporal data includes at least one of the following: data representing internal movements within the patient's body, data representing external movements of the patient's body, data representing movements of an equipment, data representing an angle of the equipment, data representing movements resulting in deformation of the patient's body, or any combination thereof.

[0026] Clause 9: The method of any of clauses 1 to 8, wherein the artificial neural network comprises an encoder and a decoder, and at least one of the decoder and the encoder is configured to utilise the spatio-temporal data as input.

[0027] Clause 10: The method of any of clauses 1 to 9, wherein the artificial neural network comprises at least one of the following: a long short-term memory (LSTM) unit, a gated recurrent unit (GRU), a temporal convolutional network, or any combination thereof.

[0028] Clause 11: The method of any of clauses 1 to 10, wherein the spatial data includes a position and / or orientation of the patient and / or equipment.

[0029] Clause 12: Analyzing the series of medical images includes identifying at least one anatomical structure in the series of images, and the at least one anatomical structure is selected from the group consisting of vessels, arteries, veins, ligaments, nerves, muscle fibers, fascial fibers or fascial meshwork, fat nodules, grafted fat nodules, lymphatic tissue, skin patches, tendons, bones, cartilage fragments, pulmonary pleural lines, heart valves, atriums, cardiac surfaces, and the like. 12. The method of any of clauses 1-11, comprising at least one of: a surface, a trachea, a brain region, a tube, a trabecular meshwork, a corneal layer, a retinal layer, an eye lens, an ocular surface, a soft tissue, a Vogt's palisade of the limbus, an organ, an extracellular structure, an intercellular structure, a cell, or any combination thereof.

[0030] Clause 13: The method of any of clauses 1 to 12, wherein the series of medical images includes at least one of the following: ultrasound images, optical coherence tomography (OCT) images, CT images, MRI images, PET images, SPECT images, fluoroscopic images, X-ray images, mammography images, tomographic composite images, photoacoustic images, audio-optical images, endoscopic images, microscopic images, fundus images, scanning laser ophthalmoscope (SLO) images, smartphone images, 3D (depth) images, focal stack images, bright field images, visible light images, infrared images, ultraviolet images, thermal images, multispectral images, tomographic images, projection images, integrated images, reconstructed images, or any combination thereof.

[0031] Clause 14: A method according to any of clauses 1 to 13, wherein analysing the series of medical images includes segmenting one or more vessels shown in the series of medical images.

[0032] Clause 15: A system for analyzing spatiotemporal medical images using an artificial neural network, comprising: capturing a series of medical images of a patient using an imaging device, the series of medical images including visual movement of at least one entity including at least a portion of at least one of the patient and an object; and tracking, using a computing device, time-varying spatial data associated with the at least one entity based on the visual movement; 1. A system comprising: a computing device programmed or configured to: generate, with a computing device, spatio-temporal data by correlating the time-varying spatial data with the series of medical images; and analyze, with a computing device, the series of medical images based on an artificial neural network comprising a plurality of layers, wherein one or more layers of the plurality of layers each combine features from at least three different scales, and wherein at least one layer of the plurality of layers of the artificial neural network is configured to learn spatio-temporal relationships based on the spatio-temporal data.

[0033] Clause 16: The system described in clause 15, wherein the one or more layers that combine features from the at least three different scales include dilated convolutions at different scales.

[0034] Clause 17: The system described in Clause 15 or 16, wherein the one or more layers combining features from the at least three different scales comprises dense and / or residual connections between at least a subset of layers of the plurality of layers, the at least a subset of layers comprising features from at least three different scales.

[0035] Clause 18: A system described in any of clauses 15 to 17, wherein the one or more layers combining features from the at least three different scales includes a convolution of at least two different scales and a combination on a subset of layers of the plurality of layers comprising features from at least two different scales, resulting in features of at least three different scales.

[0036] Clause 19: A system described in any of clauses 15 to 18, wherein the at least one entity includes at least one of the following: equipment, the imaging device, a physical artifact, a manifest artifact, or any combination thereof.

[0037] Clause 20. The system of any of clauses 15 to 19, wherein tracking the time-varying spatial data comprises tracking at least one of the following: translational / rotational position of the at least one entity, velocity of the at least one entity, acceleration of the at least one entity, inertial measurements of the at least one entity, or any combination thereof. .

[0038] Clause 21: A system described in any of clauses 15 to 20, wherein tracking the time-varying spatial data is based on at least one of the following: an inertial measurement unit, a tracking system, a position sensor, robot kinematics, inverse kinematics, or any combination thereof.

[0039] Clause 22: A system described in any of clauses 15 to 21, wherein the spatiotemporal data includes at least one of the following: data representing internal movement within the patient's body, data representing external movement of the patient's body, data representing movement of an equipment, data representing the angle of the equipment, data representing movement causing a deformation of the patient's body, or any combination thereof.

[0040] Clause 23: A system described in any of clauses 15 to 22, wherein the artificial neural network includes an encoder and a decoder, and at least one of the decoder and the encoder is configured to utilise the spatio-temporal data as input.

[0041] Clause 24: The system described in any of clauses 15 to 23, wherein the artificial neural network includes at least one of the following: a long short-term memory (LSTM) unit, a gated recurrent unit (GRU), a temporal convolutional network, or any combination thereof.

[0042] Clause 25: A system described in any of clauses 15 to 24, wherein the spatial data includes a position and / or orientation of the patient and / or equipment.

[0043] Clause 26: The system of any of clauses 15 to 25, wherein analyzing the series of medical images includes identifying at least one anatomical structure in the series of images, the at least one anatomical structure including at least one of the following: vessels, arteries, veins, ligaments, nerves, muscle fibers, fascial fibers or networks, fat nodules, grafted fat nodules, lymphatic tissue, skin patches, tendons, bones, cartilage fragments, pulmonary pleural lines, heart valves, atrial chambers, cardiac surfaces, trachea, brain regions, tubes, trabecular meshwork, corneal layers, retinal layers, ocular lens, ocular surface, soft tissue, Vogt's palisade of the corneal limbus, organs, extracellular structures, intercellular structures, cells, or any combination thereof.

[0044] Clause 27: The system of any of clauses 15 to 26, wherein the series of medical images includes at least one of the following: ultrasound images, optical coherence tomography (OCT) images, CT images, MRI images, PET images, SPECT images, fluoroscopic images, X-ray images, mammography images, tomographic synthetic images, photoacoustic images, audio-optical images, endoscopic images, microscopic images, fundus images, scanning laser ophthalmoscope (SLO) images, smartphone images, 3D (depth) images, focal stack images, bright field images, visible light images, infrared images, ultraviolet images, thermal images, multispectral images, tomographic images, projection images, integrated images, reconstructed images, or any combination thereof.

[0045] Clause 28: The system described in clauses 15 to 27, wherein analyzing the series of medical images includes segmenting one or more blood vessels shown in the series of medical images.

[0046] Clause 29: A computer program product for analyzing medical images using a neural network, the computer program product, when executed by a computing device, comprising: capturing a series of medical images of a patient using an imaging device, the series of medical images including visual movement of at least one entity including at least a portion of at least one of the patient and an object, tracking, using a computing device, time-varying spatial data associated with the at least one entity based on the visual movement; generating spatiotemporal data by correlating, using a computing device, the time-varying spatial data with the series of medical images; and generating spatiotemporal data by correlating, using a computing device, the time-varying spatial data with the series of medical images. and analyzing, using a computing device, the set of medical images based on an artificial neural network comprising a plurality of layers, wherein each of one or more layers of the plurality of layers combines features from at least three different scales, and wherein at least one layer of the plurality of layers of the artificial neural network is configured to learn spatio-temporal relationships based on the spatio-temporal data.

[0047] Clause 30. The method of any of clauses 1 to 14, wherein at least a part of the artificial neural network comprises dilated convolutions.

[0048] Clause 31: The method of any of clauses 1 to 14 and 30, wherein at least a portion of the artificial neural network comprises residual connections and / or skip connections.

[0049] Clause 32: A system according to any of clauses 15 to 28, wherein at least a portion of the artificial neural network includes dilated convolutions.

[0050] Clause 33: A system according to any of clauses 15 to 28 and 32, wherein at least a portion of the artificial neural network comprises residual connections and / or skip connections.

[0051] Clause 34: A method of analysing spatio-temporal medical images using an artificial neural network, comprising: capturing a series of medical images of a patient using an imaging device, the series of medical images including visual movement of at least one entity including at least a portion of at least one of the patient and an object, tracking using a computing device time-varying spatial data associated with the at least one entity based on the visual movement, generating spatio-temporal data by correlating the time-varying spatial data with the series of medical images using a computing device, and analysing the series of medical images based on an artificial neural network comprising a plurality of layers, the artificial neural network including dilated convolutions and / or tight coupling between a plurality of layers of different scales and resolutions that combine features from at least three different scales, and at least one layer of the plurality of layers configured to learn spatio-temporal relationships based on the spatio-temporal data.

[0052] Clause 35: A system for analyzing spatiotemporal medical images using an artificial neural network, comprising: capturing a series of medical images of a patient using an imaging device, the series of medical images including visual movement of at least one entity including at least a portion of at least one of the patient and an object; and tracking, using a computing device, time-varying spatial data associated with the at least one entity based on the visual movement; 1. A system comprising: a computing device programmed or configured to: generate spatio-temporal data by correlating, with a computing device, the time-varying spatial data with the series of medical images; and analyze, with a computing device, the series of medical images based on an artificial neural network comprising a plurality of layers, the artificial neural network including dilated convolutions and / or tight connections between a plurality of layers of different scales and resolutions that combine features from at least three different scales, and at least one layer of the plurality of layers configured to learn spatio-temporal relationships based on the spatio-temporal data.

[0053] Clause 36: A computer program product for analyzing medical images using a neural network, the computer program product, when executed by a computing device, comprising: capturing a series of medical images of a patient using an imaging device; and capturing a series of medical images of the patient, the series of medical images including at least a portion of at least one of the patient and the object. 16. A computer program product comprising: at least one non-transitory computer-readable medium comprising instructions to: track time-varying spatial data associated with the at least one entity based on the visual movement; generate spatio-temporal data by correlating the time-varying spatial data with the series of medical images; and analyze the series of medical images based on an artificial neural network comprising a plurality of layers, the artificial neural network comprising dilated convolutions and / or tight coupling between a plurality of layers of different scales and resolutions that combine features from at least three different scales, and at least one layer of the plurality of layers configured to learn spatio-temporal relationships based on the spatio-temporal data.

[0054] The above and other features and characteristics of the present disclosure, as well as the method of operation and function of the associated elements of construction, combination of parts and economies of manufacture, will become more apparent from a consideration of the following detailed description and the appended claims, all of which form a part hereof, with reference to the accompanying drawings in which like reference characters represent corresponding parts in the various views, and in which: It is to be expressly understood, however, that the drawings are for the purpose of illustration and description only and are not intended as a definition of the limits of the invention.

[0055] Further advantages and details are explained in more detail below with reference to non-limiting exemplary embodiments shown in the attached drawings. [Brief description of the drawings]

[0056] [Figure 1] 1 shows a schematic diagram of a system for analyzing spatiotemporal medical images using an artificial neural network, according to a non-limiting embodiment. [Diagram 2] 1 illustrates an artificial neural network model for use in a system for analyzing spatiotemporal medical images using artificial neural networks, according to a non-limiting embodiment. [Diagram 3] 1 illustrates an input structure and an output structure for use in a system for analyzing spatiotemporal medical images using artificial neural networks, according to a non-limiting embodiment. [Figure 4] 1 illustrates a coding block structure for use in a system for analyzing spatiotemporal medical images using artificial neural networks, according to a non-limiting embodiment. [Diagram 5] 1 illustrates a decoding block structure for use in a system for analyzing spatiotemporal medical images using artificial neural networks, according to a non-limiting embodiment. [Figure 6] FIG. 1 shows a flow diagram for analyzing spatiotemporal medical images using an artificial neural network, according to a non-limiting embodiment. [Figure 7] 1 illustrates example components of a computing device that may be used in connection with non-limiting embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0057] It is to be understood that the embodiments may take various alternative variations and step sequences unless expressly specified otherwise. It is also to be understood that the specific devices and processes described in the following specification are merely exemplary embodiments or aspects of the present disclosure. Thus, specific dimensions and other physical characteristics relating to the embodiments or aspects disclosed herein are not to be considered limiting. No aspect, component, element, structure, act, step, function, and / or instruction, etc., used herein is to be considered critical or essential unless expressly described as such. Also, in this specification, the articles "a" and "an" are intended to include one or more items and may be used interchangeably with "one or more" and "at least one". Also, in this specification, terms such as "has," "have," or "having" are intended to be open-ended terms. Furthermore, the phrase "based on" is intended to mean "based at least in part on," unless expressly described otherwise. "ed at least partially on."

[0058] As used herein, the term "computing device" may refer to one or more electronic devices configured to process data. A computing device may include components necessary to receive, process, and output data, such as a processor, a display, a memory, an input device, and / or a network interface, in some examples. A computing device may be a mobile device. A computing device may also be a desktop computer or other form of non-mobile computer. In non-limiting embodiments, a computing device may include a GPU. In non-limiting embodiments, a computing device may be composed of multiple circuits.

[0059] Non-limiting embodiments provide systems, methods, and computer program products that use artificial neural networks (e.g., convolutional neural networks (CNNs) and spatio-temporal data to analyze a sequence of medical images (e.g., anatomical, physiological, functional, and / or other biomedical images). In some non-limiting embodiments, CNN-based computer vision techniques are utilized to automatically identify and label anatomical structures visible in cross-sectional tomographic image sequences, such as, but not limited to, ultrasound or optical coherence tomography (OCT). Non-limiting embodiments enable the simultaneous tracking of spatial information, such as motion and orientation data, along with tracking of changes to entities, such as anatomical structures. This allows parameters of anatomical structures, such as shape, to be tracked across space and time. Such variations to shape may include, for example, vessel compressions or branching points, by way of example.

[0060] 1, a system 1000 for analyzing a series of medical images 104 is shown according to a non-limiting embodiment. The system 1000 includes an imaging device 102, which may include an ultrasound scanner, an OCT scanner, or the like, that captures a series of medical images 104 of a patient P over a period of time. The medical images may include ultrasound images, OCT images, CT images, MRI images, PET images, SPECT images, fluoroscopic images, X-ray images, mammography images, tomosynthetic images, photoacoustic images, audio-optical images, endoscopic images, microscopic images, fundus images, scanning laser ophthalmoscope (SLO) images, smartphone images, 3D (depth) images, focal stack images, bright field images, visible light images, infrared images, ultraviolet images, thermal images, multispectral images, tomographic images, projection images, integrated images, and / or reconstructed images, or the like. The imaging device 102 may be in communication with an instrument 106 for operating the imaging system 102, such as an ultrasound probe, although a variety of instruments may be utilized. The imaging device 102 is also in communication with the computing device 100 , which is in communication with a data storage device 108 .

[0061] 1, the sequence of medical images 104 may include images of a patient's body that may change from one slice (or volume) to the next based on visual movement of the patient and / or entities such as objects. For example, such movement may include internal movement (e.g., beating vessels, a moving fetus, etc.), external movement (e.g., movement of the patient's body, movement of the instrument 106 or other tools, changing the angle of the OCT scanning beam, etc.), and / or interactive movement (e.g., pressing an ultrasound transducer into the patient's body, thereby deforming internal anatomical structures). The moving object may be a physical artifact such as one or more anatomical structures (e.g., vessels, arteries, veins, ligaments, nerves, muscle fibers, fascial fibers or networks, fat nodules, grafted fat nodules, lymphatic tissue, skin patches, tendons, bones, cartilage fragments, pulmonary pleural lines, pulmonary consolidation, heart valves, atrial chambers, cardiac surfaces, trachea, brain regions, tubes, trabecular meshwork, corneal layers, retinal layers, ocular lens, ocular surface, soft tissues, Vogt's palisade of the corneal limbus, organs, extracellular structures, intercellular structures, and / or cells) and / or a biological artifact that is not physically present but is one or more biological structures. The artifacts may include visible artifacts, such as visual effects caused by the imaging process and / or the tools used therein, that are indicative of the physical properties of the ultrasound imaging device. Such visual effects may include, for example, needle-related ultrasound artifacts (e.g., reverberation, side lobes, and / or bayonet artifacts, etc.), and lung-related artifacts and structures (e.g., A-lines, B-lines, Z-lines, and / or commit tails, etc.). Various other artifacts may also be tracked.

[0062] 1, the computing device 100 is configured to track time-varying spatial data of an entity based on the entity's visual movement in one or more images of the sequence of images 104. Given various possible changes that may occur between successively captured images, the non-limiting embodiment tracks the entity's location as these values ​​vary across multiple images in the sequence of images 104 (e.g., at least a subset of the sequence of images 104). Based on the tracked time-varying spatial data spanning multiple images in the sequence of images 104, the computing device 100 may generate spatio-temporal data by correlating the time-varying spatial data with the multiple images in the sequence of images 104. For example, values ​​and / or changes in values ​​in the spatial data may be associated with one or more specific images by being linked to those specific images. The spatio-temporal data may indicate changes in shape, position, and / or orientation over time. The linked data may be represented in a memory in the data storage device 108.

[0063] In a non-limiting embodiment, with further reference to FIG. 1, the generated spatiotemporal data may be stored and analyzed in the data storage device 108. For example, the spatiotemporal data may be input to an artificial neural network executed by the computing device 100, such as, but not limited to, a convolutional neural network (CNN). For example, a CNN enhanced with spatiotemporal data may be used to analyze structural tissue changes over time in ultrasound video sequences of a vessel (e.g., in a hand), such as to measure intima-media thickness. As another example, an enhanced CNN may be used to analyze structural changes in the anterior segment, such as, for example, reconstructing individual volumes on a daily basis and then quantifying changes in the Voigt-Pallisade stem cell niche at the limbus over multiple days. It will be appreciated that a variety of other uses and applications are possible.

[0064] In non-limiting embodiments, the artificial neural network may be configured with a U-Net architecture that includes tight and / or residual connections between successive downsampling and upsampling layers, such that such layers process inputs generated at different scales. In such embodiments, or in other non-limiting U-Net embodiments (which may not include tight or residual connections, for example), the U-Net may include blocks or layers that use dilated (and normal) convolutions to compute features across different scales. In contrast to prior art U-Net architectures, such individual layers or blocks may be configured to compute features across at least three scales by a combination of convolutions at one or more scales and connections to other layers that include one or more scales. One or more layers of the downsampling and / or upsampling layers may be configured to learn spatiotemporal relationships. Spatiotemporal data may be incorporated into the artificial neural network in various ways. For example, in some non-limiting embodiments, a Long-Short Term Memory (LSTM) is incorporated into the decoder portion of the CNN architecture. Through the use of LSTM-based multi-scale networks, multi-scale features are intelligently combined to retain relevant features across multiple video time steps and update features only when necessary. In some non-limiting embodiments, the artificial neural network architecture uses LSTMs and / or other forms of memory (e.g., gated recurrent units (GRUs)) in the encoder and / or decoder portions of the network. The present invention may be modified to further incorporate other architectural elements, such as a time-domain convolutional network (e.g., a time-domain ecurrent unit) or a "temporal" convolutional network.

[0065] In other non-limiting embodiments, other network architectures, such as Residual Neural Network (ResNet) or Coarse-to-Fine Context Memory (CFCM) networks, may be augmented to compute multi-scale and spatio-temporal features and / or relationships. In other non-limiting embodiments, multi-scale networks, such as High Resolution Network (HRNet), may be configured to learn spatio-temporal features and / or relationships.

[0066] In a non-limiting embodiment, incorporating spatiotemporal data into an artificial neural network results in a reinforced neural network that can be used for numerous purposes. For example, the reinforced neural network can be used to analyze structural tissue changes over time in ultrasound image sequences of a vessel (e.g., in the hand), such as to measure intima-media thickness. In another example, the reinforced neural network can be used to analyze structural changes in the anterior segment, such as reconstructing individual volumes for each day and then quantifying changes in the Voigt-Pallisade stem cell niche at the limbus over multiple days. It will be appreciated that a variety of other uses and applications are possible.

[0067] In a non-limiting embodiment, a series of medical images 104 are acquired in a spatio-temporal sequence such that as the instrument 106 (e.g., an ultrasound transducer, etc.) moves across the body of the patient P, the representation of the internal anatomical structures moves and changes in the ultrasound footage. The LSTM of the network infers how the instrument 106, the patient P, or any tools used in the process were moving, so that the user (e.g., a technician, physician, or other operator or analyst) does not need to know how the instrument 106 was actually moved. In some examples, further information (e.g., motion information) regarding how the instrument 106, the patient P, and / or the tools are moving may be available, such as by tracking translational / rotational position, velocity, acceleration, and / or other output from, for example, inertial measurement units, tracking systems (e.g., spatial tracking systems for any number of dimensions), position sensors, robot kinematics, and / or inverse kinematics. For example, one or more sensors located on the equipment 106, the patient P, and / or the tool may provide motion information that is incorporated into the LSTM so that the computing device 100 can better determine how an entity (such as a moving equipment 106, the patient P, and / or the tool) was moving relative to other entities.

[0068] Referring now to Figure 2, an artificial neural network 200 according to a non-limiting embodiment is shown. The network 200 includes a downsampling encoder (e.g., the portion of the network 200 including the encoding block 206) and an LSTM-based decoder (e.g., the portion of the network 200 including the decoding block 208). The encoding block 206 computes features from the image at a sequence of scales, and the feature maps decrease in resolution from block 203 down the encoder portion of the network 200 as their individual kernels compute features from a larger proportion of their input feature maps (and therefore have a larger receptive field in the original input image). Similarly, the decoding block 208 computes features at a sequence of scales, and the feature maps increase in resolution from block 212 up the decoder portion of the network 200 as their individual kernels compute features from a smaller proportion of their input feature maps. Block-by-block iterations 214 may be included in the network 200 (e.g., 2, 3, 4, 6, 3 iterations down a series of blocks). For example, the decoder may be or incorporate a convolutional LSTM network (ConvLSTM). The model of the network 200 is a U-Net that treats each frame (e.g., image) in a sequence independently. Different from the segmentation model, the LSTM-based model and architecture shown in Fig. 2 implements a memory mechanism that considers correlations between images (e.g., video frames) (e.g., using LSTM cells in the decoding block 208) to preserve the appearance of entities (e.g., vessels) across multiple scales for dense pixel-by-pixel prediction. By combining the spatial context collected in the encoder portion of the network 200 (e.g., the encoding block 206) with the LSTM cells from the decoder portion of the network 200 (e.g., the decoding block 208) via the transfer of such information using the communication path 210, spatiotemporal entity-related features are estimated for improved segmentation.

[0069] 2-5, the symbols and letters represent the following: C (convolution function), D (dilated convolution function), BN (batch normalization function), ReLU (rectified linear activation unit), T (output class: binary(2), multi(2, …)), N (number of feature maps, e.g., {32, 64, 128, 56, 512}), Ht (hidden state at time i), Ct (cell state at time t), · (element-wise multiplication function), a (sigmoid activation), × (convolution), and + (element-wise addition function).

[0070] 2 and 3, an artificial neural network 200 receives a sequence of images as input 202 and begins encoding the images using block 203. The network 200 decodes from block 212 and result block 205 and outputs a sequence of images with one or more segmented entities as output 204. The ReLU follows the BN and holds the rectifier (e.g., activation function).

[0071] 2 and 4, the encoder portion of the network 200 includes an encoding block 206 that uses dilated convolutions and residual connections to extract meaningful representations of the appearance of entities across multiple scales. The feature maps characterized in the first few layers of the coder section depict distinct properties (such as edges, corners, and / or curves) that are considered as low-level attributes limited by their smaller receptive field. In deeper layers of the network, coarse but complex attributes are seen with less distinct properties (e.g., entity contours). At this level, more parts of the image are seen on a global scale due to the larger receptive field of the individual kernels that compute the feature maps. Residual connections and dilated convolutions gather more spatial information, especially about faintly discernible boundaries, and teach (e.g., pass) this information from one block to the next to prevent gaps in the final segmentation. Dilated convolutions gather contextual information about the broader surrounding image content to accurately segment entity boundaries (e.g., object or tissue boundaries). As an example, dilated convolutions can "fill in" gaps to perform better than prior art methods in areas with poor boundary contrast. Such a hierarchical representation can be ,While it may not independently model the dynamics of entity movement (e.g., vascular movement) in a sequence of images, it can be used to improve the segmentation of entities. For example, by transmitting feature maps extracted at different scales from the encoder part to the LSTM cells of the decoder part, the LSTM cells can retain relevant features of interest in memory and thus incorporate them into the network model to produce segmentations of better quality and accuracy.

[0072] 2 and 5, the decoder portion of the network 200 includes a decoding block 208. Every encoding block 206 communicates its output feature map to an LSTM memory unit of the decoder portion of the network 200 (e.g., to a corresponding decoding block 608 via communication path 210). M-cells may be incorporated into the network 200 and may be configured to implement a memory mechanism to consider the output of each encoding block 206 as a single time step and to integrate feature maps extracted at multiple scales in a coarse-to-fine manner. In a non-limiting embodiment, such integration may be performed using a gated logic structure in the decoding block 208 that regulates the removal or addition of new information to the cell state. In this way, global context information from the deepest encoder layer (e.g., the bottom encoding block 206 and all its iterations) is observed by the LSTM unit first, and as the receptive field decreases, finer details about the entities (e.g., more information about the vessel boundary) are added.

[0073] 2 and 5, each decoding block 208 incorporates an LSTM unit that utilizes three feature sets (input state, hidden state, and cell state) as input and outputs information using three logic gates (forget gate, input gate, and output gate). The forget gate is configured to remove information from the cell state feature set. The input gate is configured to determine new information to be incorporated into the cell state feature set. The output gate is configured to adjust the output of the respective LSTM unit. The LSTM unit of each decoding block 208 utilizes convolution and ReLU to improve segmentation accuracy, although various structures are possible for the LSTM units. The initial hidden states and initial cell states of the initial decoding block at the deepest level of the network 200 (e.g., block 212) may be initialized to zero so that the hidden states and cell states of each of the other LSTM units (e.g., the portion of the decoding block 208 other than 212) are upsampled from the LSTM unit below it. The use of a structured LSTM-based decoding block 208, such as a ConvLSTM block, facilitates the network 200 to preserve shape attributes of entities and segment the entities in each of one or more images.

[0074] Referring now to FIG. 6, shown is a flow diagram of a method for analyzing a series of medical images according to a non-limiting embodiment. It will be appreciated that the order of steps shown in FIG. 6 is for illustrative purposes only and that non-limiting embodiments may involve more steps, fewer steps, different steps, and / or a different order of steps. In step 600, an artificial neural network is created. In a non-limiting embodiment, the artificial neural network is created with dense and / or residual connections between layers. In such and other non-limiting embodiments, the artificial network may include multiple layers, with one or more layers of the multiple layers each combining features from at least three different scales / resolutions. In some examples, the layer combining features from at least three different scales may include, in part, dilated convolutions of different scales, dense connections between at least a subset of layers including features from the three different scales, and / or residual connections between at least a subset of layers including features from the three different scales.

[0075] The network may be trained in various ways, such as by supervised and / or unsupervised methodologies. In a non-limiting example, still images may be used to train the non-temporal portion of the network. Once the non-temporal portion of the network is trained, video may be used to train the entire network with spatio-temporal data. In step 602, a series of medical images is captured using an imaging device, such as an ultrasound scanner and / or an OCT scanner. The series of medical images may include frames from the video that show the movement of an entity, such as a patient, an object, and / or a portion thereof. In some examples, one or more entities may move within the captured frames (e.g., images), and in some examples, one or more entities outside the frame (e.g., an ultrasound transducer capturing the images) may move relative to the entities within the frame.

[0076] With further reference to FIG. 6, in step 604, spatial data associated with the movement of at least one entity within or outside the frame is tracked. The spatial data may be tracked, for example, as absolute or relative spatial coordinates in two-dimensional or three-dimensional space. The spatial data may include translational / rotational position, velocity, acceleration, and / or other output from, for example, an inertial measurement unit, a tracking system (e.g., a spatial tracking system for any number of dimensions), a position sensor, robot kinematics, and / or inverse kinematics. In step 606, spatio-temporal data is generated by correlating the spatial data tracked in step 604 with the set of medical images captured in step 602. The spatio-temporal data may include associations (e.g., links) to one or more databases. In step 608, the set of medical images is analyzed using the artificial neural network created in step 600. The artificial neural network may be trained to identify spatio-temporal relationships of entity movements based on incorporating LSTM cells as described herein. The result of step 608 may be a series of medical images in which one or more entities are segmented so that the movement of the one or more entities throughout the series of images (e.g., in a video) may be observed and recorded.

[0077] The system was tested using imaging sequences from two scanners: a Visualsonics Vevo 2100 UHFUS machine (Fujifilm, Canada) and a Diasus HFUS scanner (Dynamic Imaging, UK). The UHFUS scanner provided a 50 MHz transducer with a physical resolution of 30 μm and a pixel spacing of 11.6 μm. Fifty-eight UHFUS sequences were used, each containing 100 2D B-scans with dimensions of 832 × 512 pixels. The HFUS scanner had a 10-22 MHz transducer with a pixel spacing of 92.5 μm. Twenty-six HFUS sequences were used, each containing a variable number of 2D B-scans (50-250) with dimensions of 280 × 534 pixels. All of the sequences included the arteries of the hand (e.g., superficial palmar arch) with a wide range of adjustable gain settings (40-70 dB). A wide range of probe motions, such as longitudinal scanning, pulsating vessels, and / or out-of-plane vessel deformations, were also captured. An experienced human grader annotated all 84 UHFUS and HFUS sequences. To demonstrate general applicability, the system was also tested on an x-ray dataset containing 138 annotated images with 58 abnormal and 80 normal cases.

[0078] Of the 58 UHFUS sequences used for testing, 20 were selected for training and the remaining 38 were used for testing. Similarly, from the 26 HFUS sequences, 20 were selected for training and the remaining 6 were used for testing. Tree-fold cross-validation for the vessel segmentation task was performed. To simulate clinical applications, the ensemble of the two best models with the smallest validation loss (by single fold) was used for testing. Tree-fold cross-validation for the lung segmentation task was also performed on the x-ray dataset. For the vessel segmentation task, the error was compared with that of a level set-based method and two LSTM-based segmentation techniques. For the lung segmentation task, the results were compared with the state-of-the-art models. The sequences contained variable image sizes, and training the ConvLSTM with full-sized images was limited by the RAM of the GPU. Therefore, the artificial neural network was trained by scaling each B-scan to 256 × 256 pixels. Data augmentation (e.g., elastic deformation and / or blurring) was performed to increase the training set to 120,000 images. To compare with other methods, each baseline result was compared to expert annotations. To quantify the error, we calculated the following metrics: 1) Dice Similarity Coefficient (DSC) [6], 2) Hausdorff Distance (HD) [6] in millimeters, 3) Mean Absolute Deviation (MAD) in millimeters, 4) distinct false positive and false negative distances, and 5) Precision. (Pree.) and 6) reproducibility (Rec.).

[0079] Table 1 shows the segmentation error comparison for UHFUS (USVS-Net values ​​above) and HFUS (USVS-Net values ​​below) image sequences compared to other methods.

[0080] [Table 1]

[0081] Based on these tests, existing level set methods were successful in segmenting vessels in only 33 of 38 sequences, while the LSTM-based method was successful in segmenting vessels in all sequences. The system and network architecture described herein produced output that matched the expert annotations with the highest accuracy and the lowest error. The system processed and output sub-millimeter vessel localization in UHFUS sequences with increased speckle and large vessel motion.

[0082] 7, shown is a diagram of example components of a computing device 900 for implementing and performing the systems and methods described herein, according to a non-limiting embodiment. In some non-limiting embodiments, the device 900 may include additional components, fewer components, different components, or components arranged differently than those shown in FIG. 7. The device 900 may include a bus 902, a processor 904, a memory 906, a storage component 908, an input component 910, an output component 912, and a communication interface 914. The bus 902 may include components that enable communication between the components of the device 900. In some non-limiting embodiments, the processor 904 may be implemented in hardware, firmware, or a combination of hardware and software. For example, processor 904 may include a processor (e.g., a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), etc.), a microprocessor, a digital signal processor (DSP), and / or any processing component that can be programmed to perform a function (e.g., a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), etc.). Memory 906 may include random access memory (RAM), read only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic memory, optical memory, etc.) that stores information and / or instructions used by processor 904.

[0083] 7, the storage component 908 may store information and / or software related to the operation and use of the device 900. For example, the storage component 908 may include a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optical disk, a solid-state disk, etc.) and / or another type of computer-readable medium. The input component 910 may receive user input (e.g., using a touch screen display, a keyboard, a keypad, a mouse, a button, a switch, a microphone, etc.). The input components 910 may include components that enable the device 900 to receive information, such as by inputting a signal from a network, a display, a computer, a mobile phone, a mobile device, a mobile telephone, a mobile phone, a mobile device ... For example, the communication interface 914 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi interface, and / or a cellular network interface.

[0084] The device 900 can perform one or more processes described herein. The device 900 can perform these processes based on the processor 904 executing software instructions stored by a computer-readable medium, such as the memory 906 and / or the storage component 908. The computer-readable medium can include any non-transitory memory device. The memory device includes memory space within a single physical storage device or memory space spread across multiple physical storage devices. The software instructions may be loaded into the memory 906 and / or the storage component 908 from another computer-readable medium or from another device via the communication interface 914. When executed, the software instructions stored in the memory 906 and / or the storage component 908 can cause the processor 904 to perform one or more processes described herein. Additionally or alternatively, hardwired circuitry may be used in place of or in combination with software instructions to perform one or more processes described herein. Thus, the embodiments described herein are not limited to any specific combination of hardware circuitry and software. As used herein, the term "programmed or configured" refers to the arrangement of software, hardware circuitry, or any combination thereof on one or more devices.

[0085] While the embodiments have been described in detail for purposes of illustration, it is to be understood that the above details are for such purposes only, and that the disclosure is not intended to be limited to the disclosed embodiments, but rather to cover modifications and equivalent arrangements within the spirit and scope of the appended claims. For example, it is to be understood that the present disclosure contemplates that, to the extent possible, one or more features of any embodiment can be combined with one or more features of any other embodiment.

Claims

1. 1. A method for analyzing spatiotemporal medical images using an artificial neural network, comprising: capturing a series of medical images of a patient using an imaging device, the series of medical images including visual movement of at least one entity including at least a portion of at least one of the patient and an object; tracking, with a computing device, time-varying spatial data associated with the at least one entity based on the visual movement of the at least one entity within one or more images of the sequence of medical images; generating spatio-temporal data by correlating the time-varying spatial data with the series of medical images using a computing device; and analyzing, with a computing device, the series of medical images based on an artificial neural network input with the spatio-temporal data; the artificial neural network includes a plurality of layers; 13. A method according to claim 12, wherein each of the one or more layers of the plurality of layers combines features from at least three different scales, and at least one layer of the plurality of layers of the artificial neural network is configured to learn spatio-temporal relationships based on the spatio-temporal data.

2. The method of claim 1 , wherein the one or more layers that combine features from the at least three different scales include different-scale dilated convolutions.

3. 2. The method of claim 1 , wherein the one or more layers combining features from the at least three different scales comprises dense and / or residual connections between at least a subset of layers of the plurality of layers, the at least a subset of layers comprising features from at least three different scales.

4. 2. The method of claim 1 , wherein the one or more layers that combine features from the at least three different scales includes a convolution of at least two different scales and a union for a subset of layers of the plurality of layers that includes features from at least two different scales, resulting in features of at least three different scales.

5. 2. The method of claim 1, wherein the at least one entity comprises at least one of the following: equipment, the imaging device, a physical artifact, a manifest artifact, or any combination thereof.

6. 2. The method of claim 1 , wherein tracking the time-varying spatial data comprises tracking at least one of the following: translational / rotational position of the at least one entity, velocity of the at least one entity, acceleration of the at least one entity, inertial measurements of the at least one entity, or any combination thereof.

7. The method of claim 1 , wherein tracking the time-varying spatial data is based on at least one of the following: an inertial measurement unit, a tracking system, a position sensor, robot kinematics, inverse kinematics, or any combination thereof.

8. 2. The method of claim 1, wherein the spatiotemporal data comprises at least one of the following: data representing internal motion within the patient's body, data representing external motion of the patient's body, data representing motion of an instrument, data representing an angle of the instrument, data representing motion causing a deformation of the patient's body, or any combination thereof.

9. The method of claim 1 , wherein the artificial neural network includes an encoder and a decoder, at least one of the decoder and the encoder configured to utilize the spatio-temporal data as an input.

10. 2. The method of claim 1, wherein the artificial neural network comprises at least one of the following: long short-term memory (LSTM) units, gated recurrent units (GRUs), temporal convolutional networks, or any combination thereof.

11. The method of claim 1 , wherein the spatiotemporal data comprises a position and / or orientation of the patient and / or equipment.

12. 2. The method of claim 1, wherein analyzing the series of medical images comprises identifying at least one anatomical structure in the series of images, the at least one anatomical structure comprising at least one of the following: vessels, arteries, veins, ligaments, nerves, muscle fibers, fascial fibers or meshwork, fat nodules, grafted fat nodules, lymphatic tissue, skin patches, tendons, bones, cartilage fragments, pulmonary pleural lines, heart valves, atrial chambers, cardiac surfaces, trachea, brain regions, tubes, trabecular meshwork, corneal layers, retinal layers, eye lenses, ocular surfaces, soft tissues, Vogt's palisades of the corneal limbus, organs, extracellular structures, intercellular structures, cells, or any combination thereof.

13. 2. The method of claim 1, wherein the sequence of medical images comprises at least one of the following: ultrasound images, optical coherence tomography (OCT) images, CT images, MRI images, PET images, SPECT images, fluoroscopic images, X-ray images, mammography images, tomosynthetic images, photoacoustic images, audio-optical images, endoscopic images, microscope images, fundus images, scanning laser ophthalmoscope (SLO) images, smartphone images, 3D (depth) images, focal stack images, bright field images, visible light images, infrared images, ultraviolet images, thermal images, multispectral images, tomographic images, projection images, integrated images, reconstructed images, or any combination thereof.

14. The method of claim 1 , wherein analyzing the series of medical images includes segmenting one or more vessels shown in the series of medical images.

15. 1. A system for analyzing spatiotemporal medical images using an artificial neural network, comprising: capturing a series of medical images of a patient using an imaging device, the series of medical images including visual movement of at least one entity including at least a portion of at least one of the patient and an object; using a computing device to track time-varying spatial data associated with the at least one entity based on the visual movement of the at least one entity within one or more images of the sequence of medical images; using a computing device to generate spatio-temporal data by correlating the time-varying spatial data with the series of medical images; a computing device programmed or configured to analyze the series of medical images based on an artificial neural network input with the spatio-temporal data, the artificial neural network includes a plurality of layers; wherein each of the one or more layers of the plurality of layers combines features from at least three different scales, and at least one layer of the plurality of layers of the artificial neural network is configured to learn spatio-temporal relationships based on the spatio-temporal data.

16. 16. The system of claim 15, wherein the one or more layers that combine features from the at least three different scales include dilated convolutions at different scales.

17. 16. The system of claim 15, wherein the one or more layers that combine features from the at least three different scales include dense and / or residual connections between at least a subset of layers of the plurality of layers, the at least a subset of layers including features from at least three different scales.

18. 16. The system of claim 15, wherein the one or more layers that combine features from the at least three different scales includes a convolution of at least two different scales and a union for a subset of layers of the plurality of layers that includes features from at least two different scales, resulting in features of at least three different scales.

19. 16. The system of claim 15, wherein the at least one entity comprises at least one of the following: equipment, the imaging device, a physical artifact, a manifest artifact, or any combination thereof.

20. 16. The system of claim 15, wherein tracking the time-varying spatial data comprises tracking at least one of the following: translational / rotational position of the at least one entity, velocity of the at least one entity, acceleration of the at least one entity, inertial measurements of the at least one entity, or any combination thereof.

21. 16. The system of claim 15, wherein tracking the time-varying spatial data is based on at least one of the following: an inertial measurement unit, a tracking system, a position sensor, robot kinematics, inverse kinematics, or any combination thereof.

22. 16. The system of claim 15, wherein the spatiotemporal data includes at least one of the following: data representing internal motion within the patient's body, data representing external motion of the patient's body, data representing motion of an instrument, data representing an angle of the instrument, data representing motion causing deformation of the patient's body, or any combination thereof.

23. 16. The system of claim 15, wherein the artificial neural network includes an encoder and a decoder, at least one of the decoder and the encoder configured to utilize the spatio-temporal data as an input.

24. 16. The system of claim 15, wherein the artificial neural network comprises at least one of the following: long short-term memory (LSTM) units, gated recurrent units (GRUs), temporal convolutional networks, or any combination thereof.

25. The system of claim 15 , wherein the spatiotemporal data comprises a position and / or orientation of the patient and / or equipment.

26. 16. The system of claim 15, wherein analyzing the series of medical images comprises identifying at least one anatomical structure in the series of images, the at least one anatomical structure comprising at least one of the following: vessels, arteries, veins, ligaments, nerves, muscle fibers, fascial fibers or meshwork, fat nodules, grafted fat nodules, lymphatic tissue, skin patches, tendons, bones, cartilage fragments, pulmonary pleural lines, heart valves, atrial chambers, cardiac surfaces, trachea, brain regions, tubes, trabecular meshwork, corneal layers, retinal layers, eye lenses, ocular surfaces, soft tissues, Vogt's palisades of the corneal limbus, organs, extracellular structures, intercellular structures, cells, or any combination thereof.

27. 16. The system of claim 15, wherein the series of medical images includes at least one of the following: ultrasound images, optical coherence tomography (OCT) images, CT images, MRI images, PET images, SPECT images, fluoroscopic images, X-ray images, mammography images, tomosynthesis images, photoacoustic images, audio-optical images, endoscopic images, microscope images, fundus images, scanning laser ophthalmoscope (SLO) images, smartphone images, 3D (depth) images, focal stack images, bright field images, visible light images, infrared images, ultraviolet images, thermal images, multispectral images, tomographic images, projection images, integrated images, reconstructed images, or any combination thereof.

28. The system of claim 15 , wherein analyzing the series of medical images includes segmenting one or more vessels shown in the series of medical images.

29. 1. A computer program product for analyzing medical images using a neural network, the computer program product, when executed by a computing device, causing the computing device to: capturing a series of medical images of a patient using an imaging device, the series of medical images including visual movement of at least one entity including at least a portion of at least one of the patient and an object; tracking, with a computing device, time-varying spatial data associated with the at least one entity based on the visual movement of the at least one entity within one or more images of the sequence of medical images; generating spatio-temporal data by correlating the time-varying spatial data with the series of medical images using a computing device; and analyzing, with a computing device, the series of medical images based on an artificial neural network input with the spatio-temporal data; the artificial neural network includes a plurality of layers; wherein each of the one or more layers of the plurality of layers combines features from at least three different scales, and at least one layer of the plurality of layers of the artificial neural network is configured to learn spatio-temporal relationships based on the spatio-temporal data. Computer program products.

30. The method of claim 1 , wherein at least a portion of the artificial neural network comprises dilated convolutions.

31. The method of claim 1 , wherein at least a portion of the artificial neural network includes residual connections and / or skip connections.

32. 16. The system of claim 15, wherein at least a portion of the artificial neural network comprises a dilated convolution.

33. 16. The system of claim 15, wherein at least a portion of the artificial neural network includes residual connections and / or skip connections.

34. 1. A method for analyzing spatiotemporal medical images using an artificial neural network, comprising: capturing a series of medical images of a patient using an imaging device, the series of medical images including visual movement of at least one entity including at least a portion of at least one of the patient and an object; tracking, with a computing device, time-varying spatial data associated with the at least one entity based on the visual movement of the at least one entity within one or more images of the sequence of medical images; generating spatio-temporal data by correlating the time-varying spatial data with the series of medical images using a computing device; and analyzing, with a computing device, the series of medical images based on an artificial neural network input with the spatio-temporal data; the artificial neural network includes a plurality of layers; 11. The method of claim 1, wherein the artificial neural network includes dilated convolutions and / or tight connections between multiple layers of different scales and resolutions that combine features from at least three different scales, and at least one of the multiple layers is configured to learn spatio-temporal relationships based on the spatio-temporal data.

35. 1. A system for analyzing spatiotemporal medical images using an artificial neural network, comprising: capturing a series of medical images of a patient using an imaging device, the series of medical images including visual movement of at least one entity including at least a portion of at least one of the patient and the object; using a computing device to track time-varying spatial data associated with the at least one entity based on the visual movement of the at least one entity within one or more images of the sequence of medical images; generating spatio-temporal data by correlating the time-varying spatial data with the series of medical images using a computing device; a computing device programmed or configured to analyze the series of medical images based on an artificial neural network input with the spatio-temporal data, the artificial neural network includes a plurality of layers; 1. The system of claim 1, wherein the artificial neural network includes dilated convolutions and / or tight connections between multiple layers of different scales and resolutions that combine features from at least three different scales, and at least one of the multiple layers is configured to learn spatio-temporal relationships based on the spatio-temporal data.

36. 1. A computer program product for analyzing medical images using a neural network, the computer program product, when executed by a computing device, causing the computing device to: capturing a series of medical images of a patient using an imaging device, the series of medical images including visual movement of at least one entity including at least a portion of at least one of the patient and an object; tracking time-varying spatial data associated with the at least one entity based on the visual movement of the at least one entity within one or more images of the sequence of medical images; generating spatio-temporal data by correlating the time-varying spatial data with the series of medical images; and analyzing the series of medical images based on an artificial neural network input with the spatio-temporal data; and the artificial neural network includes a plurality of layers; 11. The computer program product of claim 1, wherein the artificial neural network includes dilated convolutions and / or tight connections between multiple layers of different scales and resolutions that combine features from at least three different scales, and at least one of the multiple layers is configured to learn spatio-temporal relationships based on the spatio-temporal data.

Images