Systems and methods for system-independent automated detection of cardiovascular anomalies and / or other features

Abstract

A system and method are provided for processing image data generated by medical imaging devices such as ultrasound or echocardiography systems, using artificial intelligence and machine learning, and for determining the presence of one or more congenital cardiac defects (CHDs) and / or other cardiovascular anomalies in the image data in a manner independent of the type of imaging system, software, and / or hardware. Image data from various types of imaging systems, software, and / or hardware, which generate various styles of imaging data, can be processed to determine the image style. Input image data for analysis can then be processed, along with representative style image data, to generate style-determined input images for each style.

Description

Technical Field

[0001] (Cross - Reference to Related Applications) This application claims priority to U.S. Patent Application No. 18 / 330,819, filed on June 7, 2023 (currently U.S. Patent No. 11,861,838) and European Patent Application No. 23305903.9, filed on June 7, 2023, the entire contents of each of which are incorporated herein by reference.

[0002] This technology generally relates to image processing systems, for example, image processing systems with artificial intelligence and machine learning capabilities for detecting cardiovascular anomalies.

Background Art

[0003] Today's imaging technologies allow healthcare providers to examine the interior of a patient's body and detect abnormalities and conditions without the need for surgical procedures. For example, imaging technologies such as ultrasound imaging enable medical technicians to obtain two - and three - dimensional views of a patient's anatomical structures such as the patient's heart chambers. For example, an echocardiogram uses high - frequency sound waves to generate pictures of a patient's heart. Various views can be obtained by manipulating the orientation of the ultrasound sensor relative to the patient.

[0004] Medical imaging can be used by healthcare providers to perform medical examinations of a patient's anatomical structures without the need for surgical procedures. For example, healthcare providers can examine images generated with respect to visible deviations from normal anatomical structures. In addition, healthcare providers can make measurements using medical images, compare the measurements to known normal ranges, and identify anomalies.

[0005] In one embodiment, a healthcare provider may use echocardiography to identify a cardiac defect, such as a ventricular septal defect, which is an abnormal connection between the lower ventricles (i.e., ventricles) of the heart. The healthcare provider may visually identify the connection in the medical image and make a diagnosis based on the image. This diagnosis may then lead to surgical intervention or other treatment.

[0006] While healthcare providers frequently detect anomalies such as cardiac defects through medical imaging, defects and various other abnormalities often go undetected due to human error, insufficient training, minute visual cues, and various other reasons. This is especially true with respect to complex anatomical structures and prenatal imaging. For example, congenital cardiac defects (CHD) in the fetus are particularly difficult to detect. It is estimated that CHD occurs in about 1 percent of pregnancies. However, 50–70 percent of CHD cases are not properly detected by the practitioner. Detection of CHD during pregnancy would enable healthcare providers to make a diagnosis and / or provide immediate intervention, which could lead to improved fetal and infant health and fewer infant mortality.

[0007] Artificial intelligence and imaging processing systems are being developed to analyze images to help medical practitioners detect anomalies and other visual cues within images, such as still frames. However, such systems are susceptible to bias, which can lead to inaccurate and / or misleading results. For example, a training set for such a system may contain a large number of anomalies with respect to one type of imaging system (e.g., from one brand, model, specific sensor, or transducer) and, conversely, a large number of normal, unremarkable images from another type of imaging system.

[0008] Due to unintended biases in the training set, models trained using this data may associate anomalies with images from one type of imaging system and / or normal images with another type of imaging system. For example, an image may contain other information consistently generated by a particular imaging system, such as visual arrangement, logos and / or borders, text placement, text size, font, general image style, certain colors, or equivalents. As a result, a model may be trained to associate such stylistic information with anomalies, or conversely, with normal images. For example, a model may incorrectly determine whether an image is normal or abnormal based on the image arrangement, rather than necessarily the image content generated by the image sensor.

[0009] Therefore, there is a need for improved methods and systems for analyzing and / or processing medical imaging, including ultrasound imaging for detecting cardiovascular anomalies such as CHD. [Overview of the project] [Means for solving the problem]

[0010] Provided herein are systems and methods for analyzing a set of styled medical images to include style data from multiple types of imaging systems and / or sensors in order to determine cardiovascular anomalies such as congenital heart disease (CHD), and to overcome any bias in a trained image analysis system for optionally detecting standard views, anatomical structural keypoints, determining measurements, and / or segmentation. The systems and methods may include a step of determining style data from the image data, which can be used to process image data generated by multiple imaging systems of different imaging system types and to determine representative images for each type of imaging system.

[0011] When a new set of images is received from the imaging system, the new set of images and representative images may be processed by a style transfer generator (e.g., a trained network and / or model) to determine several versions of the input set of images, each modified to incorporate a style corresponding to the representative style image. The set of styled input images may then be processed by an image analysis system (e.g., an anomaly detection model or network), which may be a spatiotemporal neural network, to identify cardiovascular anomalies in the styled set of input images, for example. Alternatively, the image analyzer may be trained using a set of styled images that have a standard style. The standard style may be achieved using a style transfer generator.

[0012] A method for determining the likelihood of the presence of one or more cardiovascular anomalies in a patient is provided herein. The method may include the steps of: determining a plurality of sets of image data corresponding to a plurality of style groups, each set of image data representing a portion of the cardiovascular system of a sample patient and comprising a series of image frames; determining a plurality of representative sets of image data based on the plurality of sets of image data, each comprising at least one representative image frame corresponding to one of the plurality of style groups; determining a first set of image data representing a first portion of the cardiovascular system of the patient and comprising a first series of image frames; processing the first set of image data and each representative set of image data from the plurality of representative sets of image data using a style transfer generator to generate a set of styled image data for each representative set of image data; and processing each set of styled image data from each representative set of image data using an image analyzer to determine the likelihood of the presence of one or more cardiovascular anomalies for each set of styled image data.

[0013] The method may further include a step of processing multiple sets of image data using a classification model and generating style data for each set of image data in the multiple sets of image data. The style data may correspond to feature maps associated with each set of image data in the multiple sets of image data. The method may further include a step of processing the style data using a clustering model and determining multiple sets of styles corresponding to the multiple sets of image data. The first set of image data may be generated by a first imaging system corresponding to a first imaging system type, and the multiple sets of image data may include a second set of image data generated by a second imaging system corresponding to a second imaging system type, where the first imaging system type may differ from the second imaging system type. The first imaging system may include a first imaging sensor corresponding to a first imaging sensor type, and the second imaging system may include a second imaging sensor corresponding to a second imaging sensor type, where the first imaging sensor type may differ from the second imaging sensor type. The step of determining multiple representative sets of image data may include a step of using one or more of the Akaike Information Criterion (AIC) and the Bayesian Information Criterion (BIC).

[0014] The first set of image data may include one or more video clips having sequential image frames. The first set of image data may include a first image frame and a second image frame arranged immediately after the first image frame, and the step of processing each representative set of image data from the first set of image data and multiple representative sets of image data using a style transfer generator further includes the step of determining optical flow data based on the first image frame and the second image frame. The method may also include the step of determining at least one constraint region in the second image frame based on the optical flow data. The style transfer generator may be a convolutional neural network. The style transfer generator may be a recurrent neural network having an encoder / decoder portion and a multi-instance normalization portion, and the recurrent neural network may be a spatiotemporal neural network.

[0015] A system for determining the likelihood of the presence of one or more cardiovascular anomalies in a patient is provided herein. The system may include a memory designed to store computer executable instructions, and at least one computer processor designed to access the memory and determine multiple sets of image data corresponding to multiple style groups, each set of image data representing a portion of the cardiovascular system of a sample patient and including a series of image frames, determine multiple representative sets of image data based on the multiple sets of image data, each including at least one representative image frame corresponding to one of the multiple style groups, representing a first portion of the patient's cardiovascular system and including a first series of image frames, process the first set of image data and each representative set of image data from the multiple representative sets of image data using a style transfer generator to generate a set of styled image data corresponding to each representative set of image data, process the styled sets of image data using an image analyzer and execute computer executable instructions to determine the likelihood of the presence of one or more cardiovascular anomalies in the set of styled image data.

[0016] A computer processor may use a classification model to process multiple sets of image data, generate style data for each set of image data, the style data may correspond to a feature map associated with each set of image data, and may use a clustering model to process the style data and execute computer executable instructions to determine multiple sets of styles corresponding to multiple sets of image data. The first set of image data may be generated by a first imaging system which may correspond to a first imaging system type, and the multiple sets of image data may include a second set of image data generated by a second imaging system which may correspond to a second imaging system type, and the first imaging system type may differ from the second imaging system type.

[0017] The first imaging system may include a first imaging sensor corresponding to a first imaging sensor type, and the second imaging system may include a second imaging sensor corresponding to a second imaging sensor type, and the first imaging sensor type may differ from the second imaging sensor type. The step of determining a plurality of representative sets of image data may include the step of using one or more of the Akaike Information Criterion (AIC) and the Bayesian Information Criterion (BIC). The first set of image data may include one or more video clips having sequential image frames. The first set of image data may include a first image frame and a second image frame arranged immediately after the first image frame, and the step of processing each representative set of image data of the first set of image data and the plurality of representative sets of image data using a style transfer generator may include the step of determining optical flow data based on the first image frame and the second image frame. The computer processor may execute computer executable instructions to determine at least one constraint region in the second image frame based on the optical flow data. The style transfer generator may be a convolutional neural network. The style transfer generator may be a recurrent neural network having an encoder / decoder part and a multi-instance normalization part, and the recurrent neural network may be a spatiotemporal neural network.

[0018] Another method is provided for determining the likelihood of the presence of one or more cardiovascular anomalies in a patient. This method may include the steps of: determining multiple sets of image data corresponding to multiple style groups; determining representative image data corresponding to a representative style group; processing the multiple sets of image data and the representative image data using a style transfer generator to generate a set of styled image data corresponding to a representative style group; training an image analyzer to determine the likelihood of the presence of one or more cardiovascular anomalies using the set of styled image data; determining a set of patient image data representing a portion of the patient's cardiovascular system; and processing the set of patient image data and the representative image data using a style transfer generator to generate a set of styled patient image data corresponding to a representative style group. This method may also include the step of processing the set of styled patient image data using an image analyzer to determine the likelihood of the presence of one or more cardiovascular anomalies present in the set of styled patient image data.

[0019] A set of patient image data may be generated by a first imaging system corresponding to a first imaging system type, and a plurality of sets of image data may include a second set of image data generated by a second imaging system corresponding to a second imaging system type, the first imaging system type may differ from the second imaging system type. The first imaging system may include a first imaging sensor corresponding to a first imaging sensor type, and the second imaging system may include a second imaging sensor corresponding to a second imaging sensor type, the first imaging sensor type may differ from the second imaging sensor type. A set of patient image data may include one or more video clips having sequential image frames. A set of patient image data may include a first image frame and a second image frame arranged immediately after the first image frame, and the step of processing the set of patient image data using a style transfer generator further includes the step of determining optical flow data based on the first and second image frames. The method may include the step of determining at least one constraint region in the second image frame based on the optical flow data. The style transfer generator may be a convolutional neural network. The style transfer generator may be a recurrent neural network that includes an encoder / decoder part and a multi-instance normalization part, and the recurrent neural network is a spatiotemporal neural network. The representative image data may include representative image frames.

[0020] The above summary is illustrative and not intended to be limiting in any way. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features will become apparent by referring to the following drawings and detailed description. [Brief explanation of the drawing]

[0021] [Figure 1]Figure 1 illustrates an image processing system for determining a styled input image and the presence of one or more cardiovascular anomalies, according to some aspects of the present invention.

[0022] [Figure 2] Figure 2 illustrates a schematic diagram of the data flow between the imaging system of the image processing system, the analyst device, and the backend.

[0023] [Figure 3] Figure 3 illustrates a schematic diagram of a classification model for determining content data and style data.

[0024] [Figure 4] Figure 4 illustrates a schematic diagram of a clustering model for determining style groups.

[0025] [Figure 5] Figure 5 illustrates a process flow for determining a representative style image, determining a styled input image based on the representative style image, and determining the likelihood of the presence of a cardiovascular anomaly.

[0026] [Figure 6] Figure 6 is a schematic block diagram of a computing device.

[0027] The foregoing and other features of the present invention will become apparent from the following description and the appended claims, considered in conjunction with the accompanying drawings. It is to be understood that the drawings depict only some embodiments in accordance with the present disclosure and are not to be considered limiting of its scope, and that the present disclosure will be described with additional specificity and detail through the use of the accompanying drawings.

Mode for Carrying Out the Invention

[0028] Detailed Description of the Present Invention The present invention relates to an image processing system that uses artificial intelligence and machine learning to determine the likelihood of the presence of cardiovascular anomalies, such as congenital heart disease (CHD) and / or other cardiovascular-related anomalies, in patients, such as fetuses during pregnancy. The image processing system may also optionally detect standard views, determine anatomical structural keypoints, determine measurements, and / or perform segmentation. For example, medical imaging (e.g., still frames and / or video clips) may be generated using an imaging system such as an ultrasound system (e.g., an echocardiography system) and may be processed by a neural network and / or model to determine the likelihood of the presence of one or more cardiovascular anomalies. Medical imaging may include video clips and / or a series of consecutive still frame images.

[0029] The imaging processing system can overcome any bias in the system trained to detect the presence of one or more cardiovascular anomalies, for example, by generating a set of styled image data for each set of input image data. In addition, or alternatively, the images used for training may be styled to eliminate or reduce any bias. The set of styled input data may incorporate style data from representative style images from multiple different imaging systems. For example, a single input image frame may be styled using image data from several (e.g., 4, 8, 20, etc.) imaging systems, resulting in several input images, each corresponding to a representative style of a particular imaging system. The styled input images may be processed by an anomaly model (e.g., a classification neural network) to detect cardiovascular anomalies within the set of styled input data.

[0030] Representative style images corresponding to each imaging system can be determined using classification and / or clustering models, derived from the image file itself (e.g., from metadata), or manually selected or determined. Various different approaches may be applied to perform style transfer on an input video clip and / or a series of consecutive images, in order to maintain consistency between consecutive image frames. For example, optical flow between frames may be determined and used to inform style transfer so that certain regions of consecutive frames can be constrained to maintain consistency between frames.

[0031] In another embodiment, the style transfer model may be a neural network comprising an encoder / decoder portion and a multi-instance normalization portion. The neural network may be a spatiotemporal neural network. However, it should be understood that any other suitable approach for performing style transfer with respect to input image frames and / or video clips may be used.

[0032] Referring here to Figure 1, an image processing system 100 is illustrated. The image processing system 100 may be designed to receive medical images, process the medical images using artificial intelligence and machine learning, and determine the likelihood of the presence of one or more cardiovascular anomalies (e.g., CHD and / or other cardiovascular anomalies). For example, the image processing system 100 may receive image data showing the anatomical structure of a fetus, process the image data, and automatically determine the likelihood of the presence of one or more cardiovascular anomalies in the fetus. In addition, the image processing system 100 may optionally perform standard views, anatomical keypoint detection, measurement determination, and / or segmentation. Image processing system 100 and / or any other image processing systems referenced herein may be identical to, include in whole or in part, and / or used together with, any image processing systems described and / or illustrated in U.S. Patent Application No. 18 / 183,937 (now U.S. Patent No. 11,869,188), filed March 14, 2023; U.S. Patent Application No. 18 / 183,942 (now U.S. Patent No. 11,875,507), filed March 14, 2023; U.S. Patent Application No. 18 / 406,446, filed January 8, 2024; and / or U.S. Patent Application No. 18 / 412,325, filed June 12, 2024 (each of which is incorporated herein by reference in its entirety).

[0033] The image processing system 100 may include one or more imaging systems 102 that can communicate with the server 104. For example, the imaging system 102 may be any well-known medical imaging system that generates medical image data (e.g., still frames and / or video clips containing RGB pixel information), such as an ultrasound system, an echocardiography system, an X-ray system, a computed tomography (CT) system, a magnetic resonance imaging (MRI) system, a positron emission tomography (PET) system, and equivalents.

[0034] As shown in Figure 1, the imaging system 102 may be an ultrasound imaging system including a sensor 108 and an imaging device 106. The sensor 108 may include a piezoelectric sensor device and / or a transducer, and / or any well-known medical imaging device. The imaging device 106 may be any well-known computing device including a processor and a display, and may have a wired or wireless connection to the sensor 108.

[0035] Sensor 108 may be used by a healthcare provider to acquire image data of the anatomical structure of a patient (e.g., patient 110). Sensor 108 may generate two- or three-dimensional images corresponding to the orientation of sensor 108 relative to patient 110. Image data generated by sensor 108 may be communicated to imaging device 106. Imaging device 106 may transmit image data to remote server 104 via any well-known wired or wireless system (e.g., Wi-Fi, cellular network, Bluetooth®, Bluetooth Low Energy (BLE), Short Field Communication Protocol, etc.).

[0036] In addition, or alternatively, image data may be received and / or read from one or more photoarchive and communication systems (PACS). For example, the PACS system may use the Medical Digital Imaging and Communication (DICOM) format. Any results from this system may be shared with the PACS.

[0037] Image data may be a set of image data, a video clip, an image, a still frame, a series of consecutive image frames, or equivalent. For example, image data may include image 105, which may include a single frame of a two-dimensional representation of a cross-section of a patient's cardiovascular anatomical structure (e.g., a ventricle of the heart). Image 105 may include information such as patient information, information about the number of scans, video, sensor devices in use, time, data, models of image device 106 and / or sensor 108, company and / or manufacturer logos, technician's name, physician's name, name of medical facility, and / or any other information commonly found on medical images.

[0038] The remote server 104 may be any computing device having one or more processors capable of performing the operations described herein. In the embodiment illustrated in Figure 1, the remote server 104 may be one or more servers, desktop or laptop computers, or equivalent, and / or may be located in a different location from the imaging system 102. The remote server 104 may launch one or more local applications to facilitate communication between the imaging system 106, the data store 112, and / or the analyst device 116.

[0039] The data store 112 may be one or more drives having memory dedicated to storing digital information such as information unique to a particular patient, professional, facility, and / or device. For example, the data store 112 may include, but is not limited to, volatile (e.g., random access memory (RAM)), non-volatile (e.g., read-only memory (ROM)), flash memory, or any combination thereof. The data store 112 may be integrated into the server 104, or it may be separate from the server 104 and distinctly different. In one embodiment, the data store 112 may be a photo archive and communications system (PACS).

[0040] The remote server 104 may communicate with the data store 112 and / or the analyst device 116 via any well-known wired or wireless system (e.g., Wi-Fi, cellular network, Bluetooth®, Bluetooth Low Energy (BLE), Short Field Communication Protocol, etc.). The data store 112 may receive and store image data (e.g., image data 118) received from the remote server 104. For example, the imaging system 102 may generate image data (e.g., ultrasound image data) and transmit such image data to the remote server 104, which may then transmit the image data to the data store 112 for storage. It should be understood that the data store 112 may be optional, and / or more than one imaging system 102, remote server 104, data store 112, and / or analyst device 116 may be used.

[0041] The analyst device 116 may be any computing device having a processor and a display, capable of communicating with at least the remote server 104 and performing the operations described herein. The analyst device 116 may be any well-known computing device such as a desktop, laptop, smartphone, tablet, wearable, or equivalent. The analyst device 116 may launch one or more local applications to facilitate communication between the analyst device 116 and the remote server 104 and / or any other computing devices or servers described herein.

[0042] The remote server 104 may determine and / or receive representative images corresponding to a unique style for various types of imaging systems. Figure 1 illustrates the server 104 communicating with the imaging system 106, but the server 104 may communicate with multiple different imaging systems, which may include different types of imaging systems (e.g., imaging systems with different sensors, different hardware, different software, imaging systems made by different companies and / or manufacturers, different model numbers, etc.). Alternatively, the server 104 may receive images from the data store 112.

[0043] The remote server 104 may receive a set of input image data (e.g., video clips and / or image frames) from the imaging system 106 and / or the data store 112, expand the input image data into multiple types of input image data, and incorporate the styles of various different types of imaging systems into the input image data (e.g., styled image 107). Style may refer to the appearance or impression of the image produced by the probe, which may be a result of the patient's anatomical structure, or alternatively. For example, style may include the patient's BMI and / or age, which may result in a certain appearance in medical imaging. The remote server 104 may process the styled input image data using one or more trained models, such as a neural network (e.g., a convolutional neural network (CNN)), which is trained to detect one or more cardiovascular anomalies. For example, the likelihood of the presence of one or more cardiovascular anomalies may be determined, and may be processed automatically by the remote server 104 to determine the presence of one or more cardiovascular anomalies. In one embodiment, the remote server 104 and / or data store 112 may facilitate storage, processing, and / or analysis in the cloud.

[0044] The remote server 104 may share information regarding the likelihood of the presence of one or more cardiovascular anomalies with one or more computing devices (e.g., a user device, an analyst device, a medical device, a practitioner device, etc.). The remote server 104 may cause the analyst device 116 to display information regarding the likelihood of the presence of one or more cardiovascular anomalies. For example, the analyst device may display the patient ID number and the percentage of likelihood for one or more CHDs and / or other cardiovascular anomalies.

[0045] Referring here to Figure 2, a schematic diagram of the data flow between the imaging system of the image processing system, the analyst device, and the backend is depicted. As shown in Figure 2, the imaging system 202, which may be identical to or similar to the imaging system 102 in Figure 1, may include an image generator 204 that can generate image data 206. The image data 206 may include video clips and / or still frames, which may include RGB and / or grayscale pixel information. Alternatively, or in addition, the image data 206 may include Doppler image data. The imaging system 202 may be designed to generate grayscale image data and / or Doppler image data. For example, the image data 206 may include a two-dimensional representation of an ultrasound scan of the anatomical structure of a patient (e.g., a fetus). In addition, or alternatively, the image data 206 may include Doppler image information (e.g., color Doppler, power Doppler, spectral Doppler, duplex Doppler, and equivalents). It should be understood that various types of image data 206 can be processed simultaneously by the imaging system 202. In one embodiment, Doppler image data may be generated simultaneously with ultrasound image data.

[0046] The imaging system 202 may transmit image data 206, which may be input image data (e.g., a set of input image data), to a backend 208, which may be identical to or similar to the server 104 in Figure 1. The image data 206 may optionally be processed by a preprocessor 210. The preprocessor 210 may remove noise from unnecessary areas of the image data 206, reduce its file size, focus on it, crop it, resize it, and / or otherwise remove it to produce preprocessed image data 212. In addition, or alternatively, the preprocessor may generate a series of consecutive still frame images from the video clip, or otherwise segment the video clip.

[0047] Image data 206 may be applied to a style transfer generator 213, which can be trained using a representative image 211. Applying image data to a style transfer generator is understood throughout to mean either applying image data to a style transfer generator, or applying a style transfer generator to image data, so that the image data is processed by the style transfer generator, input to it, and / or analyzed by it. The representative image 211 may be manually selected from image data with a known style. For example, the style may be known in such a way that it may correspond to a recording acoustic inspection instrument and / or probe (e.g., of a certain model or manufacturer), and / or the style may be determined from metadata or other information in the image file. Alternatively, an image classifier 205 and a clusterer 209 may be used to determine a representative image 212 from image data 203.

[0048] Image data 203, which may be multiple sets of image data (e.g., video clips and / or image frames) from various different types of imaging devices, systems, models, units, etc., may be received by the backend 208. For example, image data 203 may include video clips representing medical images of cardiovascular regions of various patients. It should be understood that image data 203 may be a large amount of image data (e.g., hundreds, thousands, or more video clips) and may be received by the backend 208 at different points in time. In one embodiment, image data 203 may optionally be preprocessed by a processor similar to the preprocessor 210.

[0049] If the style is not known and / or a representative image 211 has not been manually selected, the image data 203 may be processed by an image classifier 205. The image classifier 205 may be a neural network, such as a classification neural network, which can be trained for image processing, detection, and / or recognition using a large set of images. For example, images from everyday life (e.g., cars, bicycles, apples, etc.) may generally be used to train the classifier for image recognition. In addition, or alternatively, the classifier may ultimately be trained or fine-tuned using a specific dataset corresponding to cardiovascular anatomical structures with or without CHD and / or other cardiovascular anomalies in order to recognize cardiovascular anomalies.

[0050] The image classifier 205 may be used to determine style data 207, which can be obtained based on the low-level feature maps of the classifier 205 (e.g., in the initial layers of a neural network). For each image (e.g., a still frame) of the image data 203 processed by the classifier 205, a Gram matrix relating to the feature map corresponding to that image may be calculated. For example, each Gram matrix may be a representation of the image's feature map in a certain layer (e.g., using correlation operations). In one embodiment, the Gram matrix may include the point-raised product of the feature maps.

[0051] The values ​​of the Gram matrix may be concatenated into a vector (e.g., a single style vector). The style data may include low-level feature maps or their representations, corresponding Gram matrices, and / or one or more vectors corresponding to the Gram matrices. The clusterer 209 may process the vectors representing the Gram matrices and determine their locations in multidimensional space (e.g., data points) based on the vectors. The clusterer may be any suitable cluster model trained to perform clustering (e.g., clustering of Gram matrices). In one embodiment, clustering may be performed using a Gaussian mixture model (GMM) method.

[0052] Once clustering is performed on vectors representing Gram matrices, groups of imaging system styles can be identified in multidimensional space based on proximity representations of such input vectors. For example, data points that are relatively close to each other may be determined to be in the same cluster, which will be referred to herein as a group of styles. Each image or set of images corresponding to data points in the same cluster will therefore be determined to have a similar style. A style may be an arrangement or presentation of data in an image and / or within an image (e.g., border style, text placement, size, or font, general image style, a certain color, or equivalent).

[0053] For each style group, a representative set of images or image frames may be determined. For example, an image corresponding to a data point at the central position of the style group in a multidimensional space may be determined. In one embodiment, the representative data point for a given style group in a multidimensional space may be determined using the Akaike Information Criterion (AIC) and / or the Bayesian Information Criterion (BIC). However, it should be understood that any other suitable approach may be used to determine the data point in a multidimensional space that best represents a given style group.

[0054] Image data used to determine the Gram matrix and vector inputs to the clusterar, ultimately yielding representative style data points, is determined and may be referred to as the representative image 211. The representative image 211 may therefore contain style data 207 that best represent a set of styles that can correspond to a given type of imaging system. Representative images for each set of styles in multidimensional space may be determined in the same manner. The representative image 211 may be a set of image frames, a single image frame, and / or a video clip.

[0055] Whether the representative image 211 is manually selected or determined using the image classifier 205 and clusterer 209, both the input image data 206 and the representative image 211 may be input to and processed by the style transfer generator 213 to expand the input image data 206 into multiple image frames, each incorporating a representative style for each style group identified by the clusterer 209. For example, for each representative image 211, a style-determined input may be generated by the style transfer generator 213. Alternatively, instead of using the image classifier 205 and clusterer 209 to determine the representative image 211, the representative image 211 may be manually determined and provided to the backend 208.

[0056] The style transfer generator 213 may be a model (e.g., one or more neural networks) trained to combine the content of image data 206 with the style of one of the representative images 212, resulting in an image frame having the content of image data 206 and the style of image data 203. The style transfer generator 213 may be trained to transfer a style for a given image frame and / or a style for a given video clip. For example, the style transfer generator 213 may be trained with images for which the style corresponding to such images is known. Optionally, other information corresponding to the training set of images may also be known, such as the manufacturer of the imaging system, model number, probe type or style, patient conditions and / or other biometric information, and / or equivalents that may be known. Based on this information, the model may be trained to map a given input image to any known style. This type of training may be supervised training.

[0057] In embodiments in which the style transfer generator 213 transfers a style relating to a given image frame, the style transfer generator 213 may use the technique described in "A Neural Algorithm of Artistic Style" by L. Gatys, et al., arXiv:1508.06576v2, September 2, 2015 (which is incorporated herein in whole by reference). Specifically, feature maps relating to representative images may be determined using a convolutional neural network. For example, a lower-level convolutional neural network (CNN) may be used to determine, approximate, represent, and / or extract style information. In addition, the input image (e.g., image data 206) may be processed by the CNN to determine content information. For example, higher-level content information may be determined, approximated, represented, and / or extracted in higher layers of the CNN.

[0058] Style information from a representative image (e.g., representative image 211) and content information from an input image (e.g., image data 206) may be combined by finding an image that simultaneously matches the content information of the input image and the style information of the representative image.

[0059] Rather than performing a style transfer on a single image frame in isolation, it may be desirable to perform a style transfer on a video clip or multiple image frames in sequence. For example, the style transfer may be performed using the style transfer generator 213 and the approach outlined in "Artistic style transfer for videos and spherical images" by Ruder, et al., arXiv:1708.04538v3, August 5, 2018 (which is incorporated herein in whole by reference). Specifically, the style transfer may be performed on a video clip (e.g., a series of consecutive image frames) in a manner in which each image frame is styled in part based on preceding image frames. For example, the deviation between two consecutive image frames may be determined by determining the optical flow on the image frames. With a known deviation, a multi-pass algorithm may be used to process the video clip in alternating temporal directions using both forward and reverse flows. To maintain consistency in a video clip, the constraint region may be determined based on the optical flow for each image frame, and each subsequent image frame may be styled, at least partially, based on a previously styled image frame in a sequence that takes a previously styled image frame as input and is warped according to the optical flow.

[0060] In another embodiment, style transfer for a video clip may include steps using a style transfer generator 213 and the approach outlined in "Real-Time Neural Style Transfer for Videos" by H. Huang, et al., Conference: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 1, 2017 (incorporated herein in whole by reference). For example, the style transfer model may be a feedforward convolutional neural network including a style determination network and a loss network. The stylization network may process input image frames and output stylized image frames. The loss network may include a classification network for determining, approximating, representing, and / or extracting features of the stylized image frames and outputting loss data indicating the spatial loss in each of the stylized stylized image frames. In addition to spatial loss leading to frame-by-frame style transfer, the style transfer model may further incorporate temporal loss to enforce temporal consistency between adjacent frames. To determine the temporal loss, two consecutive frames are fed into the network simultaneously. Time loss is defined as the mean squared error between the styled output at time t and the warped version of the styled output at time t-1.

[0061] In another embodiment, style transfer on a video clip using the style transfer generator 213 is based on the approach outlined by W. Gao in “Fast Video Multi-Style Transfer” Conference: 2020 IEEE Conference Winter Conference on Applications of Computer Vision (WAVC), March 1-5, 2020 (which is incorporated herein in whole by reference). For example, the style transfer generator 213 may include multiple modules such as an encoder / decoder, a multi-instance normalization block, and a convolutional long-term memory (ConvLSTM). To avoid retraining the network for each different style, the network may learn multiple styles using an instance normalization layer with multiple sets of parameters. For example, each style may be associated with a pair of parameters. Each parameter pair can be considered an embedding of a specific style in the instance normalization layer. The ConvLSTM may be two ConvLSTM modules that can be inserted into the encoder / decoder network. For example, using a recurrent network may combine all previous and current frame information to infer the output. Specifically, ConvLSTM can compress the entire previous input sequence into a hidden state tensor, and based on the hidden state, it can predict the current state.

[0062] Style-determined image data 214 may be generated using a style transfer generator 213, which may use any of the style transfer approaches described herein and / or any other preferred style transfer approaches. The style image data may include multiple image frames, each corresponding to image data 206 but with a different style. For example, an image frame for each style identified (e.g., by a clusterer 209 or manually) may be generated for each image frame of image data 206. For example, if eight styles are determined, eight distinctly different image frames may be generated by the style transfer generator 213, each with a different style but with the content of image data 206.

[0063] In another embodiment, the image analyzer 216 may be trained using image data that has been styled according to a certain style (e.g., a standard style) (e.g., using a style transfer generator 213). The image data 206 may then be analyzed by the image analyzer 216. Alternatively, the image data 206 may also be applied to the style transfer generator 213 so that it is styled using the same style (e.g., standard) as the training data was styled. In this embodiment, the style transfer generator 213 may output only the styled images for each input image (e.g., each image of image data 206) based on the same style type (e.g., a standard style).

[0064] Each image frame within the styled image data 214 may be processed by an image analyzer 216. The image analyzer 216 may be one or more neural networks trained to process image data (e.g., image frames and / or video clips), such as medical image data, to detect cardiovascular anomalies (e.g., CHD), and optionally determine or detect standard views, anatomical keypoints (e.g., identification of valve ends), measurements (e.g., measurement of heart size and / or heart area or volume), and / or perform segmentation (e.g., identification of heart contours). For example, the image analyzer 216 may process the styled image data 214 and detect one or more cardiovascular anomalies within each image frame.

[0065] In one embodiment, the image analyzer 216 may be a spatiotemporal CNN trained to determine cardiovascular anomalies. For example, the image analyzer 216 may include a spatial stream and a temporal stream that can be fused together. The styled image data may be applied to a spatial model, which may be a spatial CNN, such as a spatial CNN, trained for image processing, to produce a spatial output. In addition, optical flow data may be generated based on the styled image data 214, which may allow the network to better account for the movement of image data over time.

[0066] Optical flow data may be applied to a time model, which may be a time CNN, such as a time CNN, trained for image processing and / or trained to process optical flow data, in order to generate a time output. Both the spatial and time outputs may be input to a fuser to generate a spatiotemporal output. For example, the fuser may combine the architectures of the spatial and time models at several levels.

[0067] In another embodiment, as described above, the image analyzer 216 may be trained using image data applied to the style transfer generator 213, resulting in all training datasets having the same style (e.g., a standard style). As a result, the image analyzer 216 can avoid or mitigate any bias regarding a particular style, since all input images used for training purposes may have the same style. In this embodiment, the image data 206 may optionally be processed by the style transfer 213 to transfer the same style used for the training data to the image data 206 prior to being input to the image analyzer 216.

[0068] The image analyzer 216 may output analyzed data 218, which may indicate the likelihood of the presence of one or more anomalies in the stylized image data, and optionally, the presence of standard views, anatomical keypoints, measurement information, and / or segmentation information. For example, a value between 0 and 1 may be generated for each type of potential anomaly, indicating the presence of that particular anomaly. Alternatively, any other suitable image processing model (e.g., a classification model) may be used to process the stylized image data 214 and detect the likelihood of the presence of one or more cardiovascular anomalies in the stylized image data 214.

[0069] The analyzed data 218 may be processed by an output analyzer 234, which may generate an analyzed output 236 that may indicate the presence of one or more cardiovascular anomalies in the style-determined image data 214. For example, the output analyzer 234 may calculate a weighted average based on the analyzed data 218 and / or filter out a portion of the analyzed data 218. In one embodiment, the analyzed output 236 may indicate a risk of the presence of one or more morphological abnormalities or defects and / or the presence of one or more pathological findings. For example, the analyzed output 236 may indicate the presence of atrial septal defect, atrioventricular septal defect, aortic coarctation, double outlet right ventricle, dextrorotatory transposition of the aorta, Epstein anomaly, hypoplastic left heart syndrome, aortic arch interruption, ventricular disproportion (e.g., left or right ventricle larger than the other), abnormal cardiac size, ventricular septal defect, abnormal atrioventricular junction, increased or abnormal posterior area of ​​the left atrium, abnormal left ventricle and / or aortic junction, abnormal right ventricle and / or pulmonary junction, aortic size mismatch (e.g., aorta larger or smaller than the pulmonary artery), right aortic arch anomaly, abnormal size of the pulmonary artery, transverse aortic arch, and / or superior vena cava, visible additional vessels, and / or any other morphological abnormalities, defects, and / or pathological findings. In one embodiment, it should be understood that the analyzed output 236 may indicate one or more CHDs, or features associated with one or more CHDs.

[0070] The backend 208 may communicate information based on the analyzed output 236 and / or the analyzed data 218 to an analyst device 240, which may be identical to or similar to the analyst device 116. The analyst device 240 may be different from or identical to the devices in the imaging system 202. The display module 238 may generate a user interface on the analyst device 240 to generate and display a representation of the analyzed output 236 and / or the analyzed data 218. For example, the display may show a representation of image data (e.g., an ultrasound image) with an overlay indicating the location of the detected risk or possibility of CHD and / or other cardiovascular anomalies. In one embodiment, the overlay may be a box or any other visual indicator (e.g., an arrow).

[0071] The user input module 242 may receive user input 244 and communicate user input 244 to the backend 208. User input 244 may be instructions from the user to generate a report or other information such as instructions that the results generated by one or more of the image classifier 205, clusterer 209, style transfer generator 213, image analyzer 216, and / or output analyzer 234 are inaccurate. For example, if user input 244 indicates inaccuracy, user input 244 may be used to further train one or more of the aforementioned models and / or networks.

[0072] If user input 244 indicates a request for a report, user input 244 may communicate with a report generator 246 that can generate a report. For example, the report may include the analyzed output 236, the analyzed data 218, and / or some or all of the analyses, graphs, plots, and tables relating thereto. The report 248 may then be communicated to an analyst device 240 for display (e.g., by a display module 238), which may also be printed out by the analyst device 240.

[0073] Referring here to Figure 3, a classification model for determining content and style data is depicted. For example, model 302 may be identical to or similar to image classifier 205 in Figure 2. As shown in Figure 3, image frame 304 may be input to model 302. Image frame 304 may be similar to image 105 in Figure 1 and may contain a single frame of a two-dimensional representation of a cross-section of a patient's cardiovascular anatomical structure (e.g., a ventricle of the heart).

[0074] Image frame 304 may include content data 306 that may represent a representation of the patient's anatomical structure. In addition, image frame 304 may include style data 308, which includes style information such as patient information, information about the number of scans, video, sensor device in use, time, data, model of image device 106 and / or sensor 108, company and / or manufacturer logos, name of technician, name of physician, name of medical facility, and / or any other information commonly found on medical images. Style data 308 may further include the appearance and / or impression of the image, such as color, spatial arrangement, text style, text font, border, icons for navigation, and equivalents. Alternatively, or in addition, information about the imaging device (e.g., model and / or manufacturer) and / or other style information may be determined, for example, from image file information such as metadata and / or other information in the Medical Digital Imaging and Communication (DICOM) file.

[0075] Feature maps from the low level 310 of Model 302 may be used to determine, approximate, represent, and / or extract style data 312, which can capture the texture and other style information of the image. Style data 312 may represent style data 308 but without content data 306. For example, style data 312 may include subtle differences such as contrast, hue, color, transparency, edges, and equivalents. In addition, content data 316 may be determined at a higher layer 314 of Model 302. Content data 316 may represent content data 306 and may include representations of the patient's anatomical structure. Thus, one or more neural networks may map one image to similar images (e.g., with identical content but with a certain style).

[0076] Referring to Figure 4, a clustering model for forming style groups and style data corresponding to each style group is illustrated. As shown in Figure 4, a set of input data 402, which may be identical to or similar to the image data 206 (e.g., input data and / or a set of input data), may be processed by an image classifier to determine the style data. The style data may be represented by vectors, which may then be processed by a clusterer, which may output spatial information representing the style data in a multidimensional space 404. A two-dimensional plot is illustrated in Figure 4, but it should be understood that dimensions greater than two can be generated.

[0077] The multidimensional space 404 may be used to represent multiple distinct groups of style groups, which may be groups or clusters of data points representing images having style data that are close together within the multidimensional space 404. For example, style group 406, style group 408, and style group 410 may be determined from the multidimensional space 404. Each data point in a given style group may represent style data having similar styles. For example, style group 406 may correspond to image data generated using the same ultrasound software, resulting in similarly arranged images.

[0078] As shown in Figure 4, style data 412, which may correspond to style group 406, may represent an image with a thin border, text and navigation icons at the top, and may be arranged vertically. Style data 414, which may correspond to style group 408, may present an image with a thick border, text in the upper right, a logo in the lower right, and navigation icons in the lower left. Style data 416, which may correspond to style group 410, may represent an image without a border, with text data and a logo or other image on the right, and several navigation icons on the left. Each data point in each style group may not correspond to strictly identical type of style data, but proximity within the multidirectional space 404 indicates at least some degree of similarity in style arrangement or selection.

[0079] Referring here to Figure 5, a process flow for demonstrating the possibility of CHD and / or other cardiovascular anomalies, independent of the type of imaging system (e.g., ultrasound), transducer and / or sensor, and / or other style input, is depicted. Some or all of the blocks of the process flow of this disclosure may be carried out in a distributed manner across any number of devices (e.g., servers such as server 104 in Figure 1, computing devices, imaging or sensor devices, or equivalent). Some or all of the operations of the process flow may be arbitrary and may be carried out in different orders.

[0080] In block 502, computer-executable instructions stored in the memory of a device such as a server may be executed to determine a set of image data (e.g., still frames and / or video data) from one or more imaging systems. For example, the set of image data may be produced by different companies, manufacturers, and / or generated by different imaging systems having different sensors and / or hardware. In an optional block 504, computer-executable instructions stored in the memory of a device such as a server may be executed to apply the set of image data to a trained classification model (e.g., the image classifier 205 in Figure 2).

[0081] In an optional block 506, a computer-executable instruction stored in the memory of a device such as a server may be executed to determine style data from a set of image data. For example, low-level feature maps from a trained classification model may be used to determine style data for a set of image data. In an optional block 508, a computer-executable instruction stored in the memory of a device such as a server may be executed to apply the style data and / or a representation of the style data to a clustering model (e.g., clusterer 209 in Figure 2) to determine a set of styles. For example, a Gram matrix corresponding to the lower layers of a classification model may be determined, and a vector representing such a Gram matrix may be input to the clustering model.

[0082] In block 510, computer-executable instructions stored in the memory of a device such as a server may be executed to determine a representative set of image data and / or a representative image frame for each style group. For example, representative data points for a given style group in multidimensional space may be determined using any preferred approach to determine data points in multidimensional space that best represent the given style group. Alternatively, representative image frames for each style may be selected manually.

[0083] In block 512, a computer-executable instruction stored in the memory of a device such as a server may be executed to determine input image data (e.g., a set of input image data, image frames, and / or video clips). The input image data may be identical to or similar to image data 206 in Figure 2. In block 514, a computer-executable instruction stored in the memory of a device such as a server may be executed to process the input image data using a style transfer generator (e.g., style transfer generator 213). A representative image frame from block 510 may also be input to the style transfer generator. The style transfer generator may output a set of style-determined imaging data (e.g., style-determined image data 214 in Figure 2) for each determined style (e.g., for each group of styles). For example, if five styles are determined, the style transfer generator may generate five distinctly different style-determined images for each input image data, and each style-determined image may correspond to one of the five styles.

[0084] In block 516, computer executable instructions stored in the memory of a device such as a server may be executed to apply a set of style-determined image data to an image analyzer (e.g., image analyzer 216 in Figure 2) and generate analyzed data (e.g., analyzed data 218 in Figure 2). It should be understood that the image analyzer may optionally determine or detect standard views, anatomical keypoints, measurements, and / or perform segmentation. In embodiments where multiple different style-determined images are output by a style transfer generator with respect to a given input image, the analyzed data may be aggregated to result in aggregated analyzed data. For example, if the analyzed data is a vector or matrix, each vector or matrix may be added or otherwise combined to result in a single vector or matrix.

[0085] Alternatively, the image analyzer may be trained using styled training data. In this embodiment, block 513 may be executed to apply a set of image data (e.g., training data) and optionally input image data to a style transfer generator to generate one or more styled sets of image data and / or styled input image data based on multiple different style types, or alternatively, based on a single standard style. In block 515, a computer-executable instruction stored in the memory of a device such as a server may be executed to train the image analyzer using a set or multiple sets of styled image data. In one embodiment, the training data may be styled using multiple different types of styles such that, for each image frame of the training data, multiple styled image frames may be generated, depending on the number of styles. Alternatively, the training data may be styled using only one style, which may be a standard style. In block 517, the input image data or styled input image data may be applied to and / or processed by an image analyzer to generate analyzed data.

[0086] Block 518 may begin after either block 516 or 517, where a computer executable instruction stored in the memory of a device such as a server may be executed to process the analyzed data and / or aggregated analyzed data and determine the likelihood of a cardiovascular anomaly. For example, the analyzed data and / or aggregated analyzed data may be a number between 0 and 1, and the analyzed data and / or aggregated analyzed data may be processed to determine whether the anomaly satisfies a certain threshold (e.g., 0.7), in which case it may be determined that there is a high probability that a cardiovascular anomaly exists. In block 520, a computer executable instruction stored in the memory of a device such as a server may be executed to cause a computing device (e.g., an analyst device or any other device) to present the data and / or the likelihood of a cardiovascular anomaly that has been analyzed on the computing device (e.g., an analyst device).

[0087] Referring now to Figure 6, a schematic block diagram of server 600 is illustrated. Server 600 may be identical to or similar to server 104 in Figure 1 or one or more of the servers in Figure 1-5B. It should be understood that the imaging system, analyst device, and / or datastore may, in addition or alternatively, include one or more of the components illustrated in Figure 6, and that server 600 may perform one or more of the operations of server 600 described herein, either alone or in combination with any of the foregoing.

[0088] Server 600 may be designed to communicate with one or more servers, imaging systems, analyst devices, data stores, other systems, or equivalents. Server 600 may also be designed to communicate over one or more networks. Such networks may include, but are not limited to, one or more different types of communication networks, such as cable networks, public networks (e.g., the Internet), private networks (e.g., Frame Relay networks), wireless networks, cellular networks, telephone networks (e.g., public switched telephone networks), or any other suitable private or public packet-switched or circuit-switched networks.

[0089] In an illustrative configuration, the server 600 may include one or more processors 602, one or more memory devices 604 (also referred to herein as memory 604), one or more input / output (I / O) interfaces 606, one or more network interfaces 608, one or more transceivers 610, one or more antennas 634, and a data storage device 620. The server 600 may further include one or more buses 618 that functionally connect the various components of the server 600.

[0090] Bus 618 may include at least one of the system bus, memory bus, address bus, or message bus, and may enable the exchange of information (e.g., data (including computer executable code), signaling, etc.) between various components of the server 600. Bus 618 may also include, but is not limited to, a memory bus or memory controller, peripheral buses, accelerated graphics ports, etc. Bus 618 may be associated with any preferred bus architecture.

[0091] Memory 604 may include volatile memory (memory that maintains its state when power is supplied) such as random access memory (RAM) and / or non-volatile memory (memory that maintains its state even when power is not supplied) such as read-only memory (ROM), flash memory, and ferroelectric RAM (FRAM®). Persistent data storage devices, as the term is used herein, may include non-volatile memory. In various implementations, memory 604 may include several different types of memory, such as various types of static random access memory (SRAM), various types of dynamic random access memory (DRAM), various types of immutable ROM, and / or electrically erasable programmable read-only memory (EEPROM), and writable variations of ROM such as flash memory.

[0092] The data storage device 620 may include, but is not limited to, removable and / or non-removable storage devices, including magnetic storage devices, optical disk storage devices, and / or tape storage devices. The data storage device 620 may provide non-volatile storage for computer-executable instructions and other data. The removable and / or non-removable memory 604 and data storage device 620 are embodiments of a computer-readable storage medium (CRSM) as the term is used herein. The data storage device 620 may store computer-executable code, instructions, or equivalents that are loadable into memory 604 and can be executed by the processor 602 to cause the processor 602 to perform or initiate various operations. In addition, the data storage device 620 may store data that can be copied to memory 604 for use by the processor 602 during the execution of computer-executable instructions. Output data generated as a result of the execution of computer-executable instructions by the processor 602 may first be stored in memory 604 and finally copied to the data storage device 620 for non-volatile storage.

[0093] The data storage device 620 may store one or more operating systems (O / S) 622, one or more optional database management systems (DBMS) 624, and one or more program modules, applications, engines, computer executable code, scripts, or equivalents, such as one or more implementation modules 626, style module 627, communication module 628, content module 629, style transfer module 630, and / or image analyzer module 631. Some or all of these modules may be submodules. Any of the components described as to be stored in the data storage device 620 may include any combination of software, firmware, and / or hardware. The software and / or firmware may include computer executable code, instructions, or equivalents that can be loaded into memory 604 for execution by one or more of the processors 602. Any of the components described as to be stored in the data storage device 620 may support the functionality described with reference to the corresponding named component mentioned above in this disclosure.

[0094] Referring here to other illustrative components described as being stored in data storage device 620, O / S622 may be loaded from data storage device 620 into memory 604 and may provide an interface between other application software running on server 600 and the hardware resources of server 600. More specifically, O / S622 may include a set of computer executable instructions for managing the hardware resources of server 600 and providing common services to other application programs (e.g., managing memory allocation between various application programs). In one exemplary embodiment, O / S622 may control the execution of other program modules for content rendering. O / S622 may include, but is not limited to, any server operating system, any mainframe operating system, or any other proprietary or non-proprietary operating system, including any operating system that is currently known or may be developed in the future.

[0095] An optional DBMS624 may be loaded into memory 604 and may support functionality for accessing, reading, storing, and / or manipulating data stored in memory 604 and / or data stored in data storage device 620. DBMS624 may use any of various database models (e.g., relational model, object model, etc.) and may support any of various query languages. DBMS624 may access data represented in one or more data schemas and stored in any preferred data repository, including, but not limited to, databases (e.g., relational, object-oriented, etc.), file systems, flat files, distributed data stores where data is stored on more than one node of a computer network, peer-to-peer network data stores, or equivalents.

[0096] The optional input / output (I / O) interface 606 may facilitate the reception of input information by the server 600 from one or more I / O devices, and the output of information from the server 600 to one or more I / O devices. The I / O devices may include any of the following components: a display or display screen having a touch surface or touchscreen, an audio output device for producing sound such as a speaker, an audio capture device such as a microphone, an image and / or video capture device such as a camera, and so on. Any of these components may be integrated into the server 600 or may be separate.

[0097] The server 600 may further include one or more network interfaces 608 through which the server 600 can communicate with any of various other systems, platforms, networks, devices, etc. The network interfaces 608 may, for example, enable communication with one or more wireless routers, one or more host servers, one or more web servers, and equivalents via one or more networks.

[0098] Antenna 634 may include any suitable type of antenna, for example, depending on the communication protocol used to transmit or receive signals through Antenna 634. Non-limiting embodiments of suitable antennas may include directional antennas, omnidirectional antennas, dipole antennas, folded dipole antennas, patch antennas, multiple-input multiple-output (MIMO) antennas, or equivalents. Antenna 634 may be communicatively coupled to one or more transceivers 610 or wireless components from which signals can be transmitted or received. Antenna 634 may include, but is not limited to, a cellular antenna for transmitting or receiving signals to and from a cellular network infrastructure, an antenna for transmitting or receiving Wi-Fi signals to and from an access point (AP), a Global Navigation Satellite System (GNSS) antenna for receiving GNSS signals from GNSS satellites, a Bluetooth® antenna for transmitting or receiving Bluetooth® signals, including BLE signals, a Near Field Communication (NFC) antenna for transmitting or receiving NFC signals, a 900 MHz antenna, and the like.

[0099] The transceiver 610 may include any suitable radio components for transmitting or receiving radio frequency (RF) signals within a bandwidth and / or channel corresponding to a communication protocol used by the server 600 to communicate with other devices, in cooperation with the antenna 634. The transceiver 610 may potentially include hardware, software, and / or firmware for modulating, transmitting, or receiving communication signals in cooperation with any of the antennas 634, but not limited to, one or more Wi-Fi and / or Wi-Fi Direct protocols, one or more non-Wi-Fi protocols, or one or more cellular communication protocols or standards discussed above. The transceiver 610 may further include hardware, firmware, or software for receiving GNSS signals. The transceiver 610 may include any known receiver and baseband suitable for communication via the communication protocol used by the server 600. The transceiver 610 may further include a low-noise amplifier (LNA), an additional signal amplifier, an analog-to-digital (A / D) converter, one or more buffers, a digital baseband, or equivalent.

[0100] Referring here to the functionality supported by the various program modules depicted in Figure 6, the implementation module 626 may include computer executable instructions, code, or equivalents that can perform functions in response to execution by one or more of the processors 602, including, but not limited to, supervision of coordination and interaction between one or more modules in the data storage device 620 and computer executable instructions, determination of user-selected actions and tasks, determination of actions associated with user interaction, determination of actions associated with user input, initiation of commands locally or on remote devices, and equivalents.

[0101] The style module 627 may include computer-executable instructions, code, or equivalents that, in response to execution by one or more of the processors 602, perform functions including, but are not limited to, analyzing and processing image data (e.g., still frames and / or video clips), and determining one or more sets of styles and / or representative image frames corresponding to a particular style from the image data.

[0102] The communication module 628 may include computer executable instructions, code, or equivalents that can perform functions in response to execution by one or more of the processors 602, including, but not limited to, communication with one or more devices via wired or wireless communication, communication with servers (e.g., remote servers), communication with data stores and / or databases, communication with imaging systems and / or analyst devices, sending or receiving notifications or commands / instructions, communication with cache memory data, communication with computing devices, and equivalents.

[0103] The content module 629 may include computer-executable instructions, code, or equivalents that, in response to execution by one or more of the processors 602, can perform functions including, but not limited to, determining an input image, processing an input image, segmenting an input image, and / or determining the content within an input image.

[0104] The style transfer module 630 may include computer-executable instructions, code, or equivalents that, in response to execution by one or more of the processors 602, perform functions including, but not limited to, processing an input image and / or a representative style image to generate a style-determined input image that incorporates a style from a representative style image.

[0105] The image analyzer module 631 may include computer-executable instructions, code, or equivalents that, in response to execution by one or more of the processors 602, perform functions including, but not limited to, processing styled input images and detecting cardiovascular anomalies based on styled input images, and / or optionally, performing standard views, anatomical keypoints, measurement determination or detection, and / or segmentation based on styled input images.

[0106] While specific embodiments of the Disclosure are described, those skilled in the art will recognize that numerous other modifications and alternative embodiments are also within the scope of the Disclosure. For example, any of the functionalities and / or processing capabilities described with respect to a particular device or component may be implemented by any other device or component. Furthermore, while various illustrative implementations and architectures are described according to embodiments of the Disclosure, those skilled in the art will understand that numerous other modifications of the illustrative implementations and architectures described herein are also within the scope of the Disclosure.

[0107] Certain aspects of this disclosure are described above with reference to block and flow diagrams of systems, methods, apparatus, and / or computer program products according to exemplary embodiments. It should be understood that one or more blocks in the block diagrams and flow diagrams, and combinations of blocks in the block diagrams and flow diagrams, can be implemented by the execution of computer executable program instructions. Similarly, some blocks in the block diagrams and flow diagrams may not necessarily be implemented in the order presented, or may not necessarily be implemented in all cases, according to some embodiments. Furthermore, additional components and / or operations other than those depicted within the blocks of the blocks and / or flow diagrams may exist in some embodiments.

[0108] Therefore, the blocks in block diagrams and flowcharts support combinations of means for performing a defined function, combinations of elements or steps for performing a defined function, and program instructions for performing a defined function. Furthermore, it should be understood that each block in block diagrams and flowcharts, and combinations of blocks in block diagrams and flowcharts, can be implemented by a dedicated hardware-based computer system, or a combination of dedicated hardware and computer instructions, for performing the defined function, element, or step.

[0109] The program modules, applications, or equivalents disclosed herein may include one or more software components, for example, software objects, methods, data structures, or equivalents. Each such software component may include computer executable instructions that, in response to execution, cause to perform at least one of the functionalities described herein (for example, one or more operations of the illustrative methods described herein).

[0110] Software components may be coded in any of various programming languages. The illustrative programming language may be a lower-level programming language, such as assembly language, associated with a specific hardware architecture and / or operating system platform. Software components containing assembly language instructions may require conversion to executable machine code by an assembler prior to execution by the hardware architecture and / or platform.

[0111] Another exemplary programming language may be a higher-level programming language that may be portable across multiple architectures. Software components containing higher-level programming language instructions may require conversion to an intermediate representation by an interpreter or compiler prior to execution.

[0112] Other embodiments of the programming language include, but are not limited to, macro languages, shell or command languages, job control languages, scripting languages, database query or search languages, or reporting languages. In one or more exemplary embodiments, a software component containing instructions in one of the aforementioned embodiments of the programming language may be executed directly by an operating system or other software component without the need to first be converted to another form.

[0113] Software components may be stored as files or other data storage structures. Similar types or functionally related software components may be stored together, for example, in a specific directory, folder, or library. Software components may be static (e.g., pre-established or fixed) or dynamic (e.g., created or modified at runtime).

[0114] A software component may call, or be called by, other software components through one of a wide variety of mechanisms. The software components that are called or call may include other custom-developed application software, operating system functionality (e.g., device drivers, data storage (e.g., file management) routines, other common routines, and services, etc.), or third-party software components (e.g., middleware, encryption or other security software, database management software, file transfer or other network communication software, mathematical or statistical software, image processing software, and format conversion software).

[0115] Software components associated with a particular solution or system may reside and run on a single platform, or they may be distributed across multiple platforms. These multiple platforms may be associated with one or more hardware vendors, underlying chip technologies, or operating systems. Furthermore, software components associated with a particular solution or system may initially be written in one or more programming languages, but they may also call software components written in other programming languages.

[0116] Computer program instructions may be loaded onto a dedicated computer or other specific machine, processor, or other programmable data processing device to produce a particular machine, such that the execution of the instructions on the computer, processor, or other programmable data processing device causes one or more functions or operations defined in a flow diagram. These computer program instructions may also be stored in a computer-readable storage medium (CRSM) in which the instructions stored in the computer-readable storage medium can instruct a computer or other programmable data processing device to function in a particular manner, depending on the execution, to produce a product that includes instruction means for implementing one or more functions or operations defined in a flow diagram. Computer program instructions may also be loaded onto a computer or other programmable data processing device to produce a computer implementation process by causing a series of operating elements or steps to be executed on the computer or other programmable device.

[0117] Additional types of CRSM that may be present in any of the devices described herein may include, but are not limited to, programmable random access memory (PRAM), SRAM, DRAM, RAM, ROM, electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage devices, magnetic cassettes, magnetic tapes, magnetic disk storage devices or other magnetic storage devices, or any other medium that may be used to store and access information. Any combination of the above also falls within the scope of CRSM. Alternatively, computer-readable communication medium (CRCM) may include computer-readable instructions, program modules, or other data transmitted in data signals such as carrier waves or other transmissions. However, as used herein, CRSM does not include CRCM.

[0118] While embodiments are described in language specific to structural features and / or methodological actions, it should be understood that this disclosure is not necessarily limited to the specific features or actions described. Rather, specific features and actions are disclosed as illustrative forms that implement the embodiments. In particular, conditional language such as “can,” “could,” “might,” or “may,” unless specifically otherwise described or understood differently in the context in which they are used, is generally intended to convey that one embodiment may include certain features, elements, and / or steps, while other embodiments may not. Accordingly, such conditional language is generally not intended to imply that features, elements, and / or steps are required for any one or more embodiments, or that one or more embodiments necessarily include logic for determining whether these features, elements, and / or steps should be included or implemented in any particular embodiment, with or without user input or prompting.

[0119] It should be understood that any of the computer operations described herein may be implemented, at least in part, as computer-readable instructions stored in computer-readable memory. Naturally, it should be understood that the embodiments described herein are illustrative, and that components may be arranged, substituted, combined, and designed in a wide variety of different configurations, all of which are assumed and fall within the scope of this disclosure.

[0120] The foregoing description of illustrative embodiments is presented for illustrative and explanatory purposes only. It is not intended to be exhaustive or to limit to any precise form disclosed, and modifications and variations may be conceivable in light of the foregoing teachings or obtained from the practice of the disclosed embodiments. The scope of the invention is intended to be defined by the claims appended herein and their equivalents.

Claims

1. A method for determining the likelihood of the presence of one or more cardiovascular anomalies in a patient, wherein the method is: The process involves determining multiple sets of image data corresponding to multiple style groups, wherein each set of image data represents a portion of the cardiovascular system of a sample patient and comprises a series of image frames. Determining a plurality of representative sets of image data based on a plurality of sets of image data, wherein each of the plurality of representative sets of image data comprises at least one representative image frame corresponding to one of the plurality of style groups, To determine a first set of image data comprising a first series of image frames, which shows a first portion of the cardiovascular system of the patient, Using a style transfer generator, the first set of image data and each representative set of image data from a plurality of representative sets of image data are processed to generate a style-determined set of image data for each representative set of image data. Using an image analyzer, process each set of styled image data for each representative set of image data, and determine the likelihood of the presence of one or more cardiovascular anomalies for each set of styled image data. Methods that include...

2. The process involves using a classification model to process multiple sets of the image data and generating style data for each set of image data among the multiple sets of image data, wherein the style data corresponds to a feature map associated with each set of image data among the multiple sets of image data. Using a clustering model, process the style data and determine multiple style groups corresponding to multiple sets of image data. The method according to claim 1, further comprising:

3. The first set of image data is generated by a first imaging system corresponding to a first imaging system type. The plurality of sets of image data comprises a second set of image data generated by a second imaging system corresponding to a second imaging system type, The first imaging system type is different from the second imaging system type. The method according to claim 1.

4. The method according to claim 3, wherein the first imaging system comprises a first imaging sensor corresponding to a first imaging sensor type, and the second imaging system comprises a second imaging sensor corresponding to a second imaging sensor type, wherein the first imaging sensor type is different from the second imaging sensor type.

5. The method according to claim 1, wherein determining a number of representative sets of image data includes using one or more of the Akaike Information Criterion (AIC) and the Bayesian Information Criterion (BIC).

6. The method according to claim 1, wherein the first set of image data comprises one or more video clips having sequential image frames.

7. The method according to claim 6, wherein the first set of image data comprises a first image frame and a second image frame arranged immediately after the first image frame, and processing the first set of image data and each representative set of image data from a plurality of representative sets of image data using the style transfer generator further comprises determining optical flow data based on the first image frame and the second image frame.

8. The method according to claim 7, further comprising determining at least one constraint region within the second image frame based on the optical flow data.

9. The method according to claim 1, wherein the style transfer generator is a convolutional neural network.

10. The method according to claim 1, wherein the style transfer generator is a recurrent neural network comprising an encoder / decoder portion and a multi-instance normalization portion, and the recurrent neural network is a spatiotemporal neural network.

11. A system for determining the likelihood of the presence of one or more cardiovascular anomalies in a patient, wherein the system is Memory configured to store computer executable instructions, At least one computer processor, the at least one computer processor accesses memory, The process involves determining multiple sets of image data corresponding to multiple style groups, wherein each set of image data represents a portion of the cardiovascular system of a sample patient and comprises a series of image frames. Determining a plurality of representative sets of image data based on a plurality of sets of image data, wherein each of the plurality of representative sets of image data comprises at least one representative image frame corresponding to one of the plurality of style groups, To determine a first set of image data comprising a first series of image frames, which shows a first portion of the cardiovascular system of the patient, Using a style transfer generator, process the first set of image data and each representative set of image data from among multiple representative sets of image data to generate a set of style-determined image data corresponding to each representative set of image data. Using an image analyzer, process the styled set of image data and determine the likelihood of the presence of one or more cardiovascular anomalies in the styled set of image data. A computer processor configured to execute the computer executable instructions and A system equipped with these features.

12. The aforementioned computer processor further, The process involves using a classification model to process multiple sets of the image data and generating style data for each set of image data among the multiple sets of image data, wherein the style data corresponds to a feature map associated with each set of image data among the multiple sets of image data. Using a clustering model, process the style data and determine multiple style groups corresponding to multiple sets of image data. The system according to claim 11, configured to execute the computer executable instructions to perform the following:

13. The first set of image data is generated by a first imaging system corresponding to a first imaging system type. The plurality of sets of image data comprises a second set of image data generated by a second imaging system corresponding to a second imaging system type, The first imaging system type is different from the second imaging system type. The system according to claim 11.

14. The system according to claim 13, wherein the first imaging system comprises a first imaging sensor corresponding to a first imaging sensor type, and the second imaging system comprises a second imaging sensor corresponding to a second imaging sensor type, wherein the first imaging sensor type is different from the second imaging sensor type.

15. The system according to claim 11, wherein determining a representative set of multiple image data includes using one or more of the Akaike Information Criterion (AIC) and the Bayesian Information Criterion (BIC).

16. The system according to claim 11, wherein the first set of image data comprises one or more video clips having sequential image frames.

17. The system according to claim 16, wherein the first set of image data comprises a first image frame and a second image frame arranged immediately after the first image frame, and processing the first set of image data and each representative set of image data from a plurality of representative sets of image data using the style transfer generator further includes determining optical flow data based on the first image frame and the second image frame.

18. The system according to claim 17, wherein the computer processor is further configured to execute the computer executable instructions to determine at least one constraint region in the second image frame based on the optical flow data.

19. The system according to claim 1, wherein the style transfer generator is a convolutional neural network.

20. The system according to claim 1, wherein the style transfer generator is a recurrent neural network comprising an encoder / decoder portion and a multi-instance normalization portion, and the recurrent neural network is a spatiotemporal neural network.

21. A method for determining the likelihood of the presence of one or more cardiovascular anomalies in a patient, wherein the method is: Determining multiple sets of image data corresponding to multiple style groups, To determine representative image data corresponding to representative style sets, Using a style transfer generator, process multiple sets of the image data and the representative image data to generate a set of style-determined image data corresponding to the representative style group, The image analyzer is trained to determine the likelihood of the presence of one or more cardiovascular anomalies using the aforementioned style-determined set of image data. To determine a set of patient image data showing a part of the cardiovascular system of the aforementioned patient, and Using a style transfer generator, process the set of patient image data and the representative image data to generate a set of style-determined patient image data corresponding to the representative style group. Methods that include...

22. The method according to claim 21, further comprising using the image analyzer to process the set of styled patient image data and determining the likelihood of the presence of one or more cardiovascular anomalies present in the set of styled patient image data.

23. The set of patient image data is generated by a first imaging system corresponding to a first imaging system type. The plurality of sets of image data comprises a second set of image data generated by a second imaging system corresponding to a second imaging system type, The first imaging system type is different from the second imaging system type. The method according to claim 21.

24. The method according to claim 23, wherein the first imaging system comprises a first imaging sensor corresponding to a first imaging sensor type, and the second imaging system comprises a second imaging sensor corresponding to a second imaging sensor type, wherein the first imaging sensor type is different from the second imaging sensor type.

25. The method according to claim 21, wherein the set of patient image data comprises one or more video clips having sequential image frames.

26. The method according to claim 21, wherein the set of patient image data comprises a first image frame and a second image frame arranged immediately after the first image frame, and processing the set of patient image data using the style transfer generator further comprises determining optical flow data based on the first image frame and the second image frame.

27. The method according to claim 26, further comprising determining at least one constraint region in the second image frame based on the optical flow data.

28. The method according to claim 21, wherein the style transfer generator is a convolutional neural network.

29. The method according to claim 21, wherein the style transfer generator is a recurrent neural network comprising an encoder / decoder portion and a multi-instance normalization portion, and the recurrent neural network is a spatiotemporal neural network.

30. The method according to claim 21, wherein the representative image data comprises a representative image frame.

Images