Intelligent annotation of recorded waveforms representative of physiological properties

Abstract

In some embodiments, a method includes training an autoencoder with a set of unlabeled input samples through unsupervised learning, and training a deep neural network through supervised learning using the trained autoencoder. The unlabeled input sample may be a recorded waveform representing a physiological characteristic of the human body. Training a deep neural network through supervised learning using a trained autoencoder may include: training a deep neural network with a first subset of manually labeled samples selected from the set of unlabeled samples; and iteratively training the deep neural network with a plurality of consecutive subsets of manually tagged samples extracted from the untagged samples until convergence or until untagged sample inputs are used up. Each consecutive subset includes a plurality of distanced unlabeled samples having the lowest confidence selected from among remaining unlabeled samples to which tags are propagated, the distance determination including using an automatic encoder for feature extraction.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] Priority and earlier effective filing date of U.S. Application Serial No. 63 / 354,541, filed on June 22, 2022, is hereby claimed for all purposes including rights of priority. This related application is also incorporated herein by reference for all purposes as if expressly set forth verbatim herein. Technical Field

[0003] The present disclosure relates to a deep neural network for classifying recorded waveforms representing physiological characteristics of a human body, and more particularly, to a technique for training such a deep neural network. Background Art

[0004] This section of this document presents information about and / or from the art that may provide background to or be relevant to the subject matter described herein and / or claimed below. It provides background information to facilitate a better understanding of the various aspects claimed below. This is a discussion of "related" art. Just because such art is related does not mean that it is also "prior" art. Related art may or may not be prior art. The discussion in this section of this document should be read in that light, and not as an admission that it is prior art.

[0005] In a clinical setting, a patient's physiological state is often represented by monitoring one or more physiological characteristics of the patient over time. Monitoring often includes capturing data using sensors, the captured data representing the physiological characteristics. Because the data represents the physiological characteristics, the data may be referred to as "physiological data."

[0006] In one common example, known as electrocardiography, multiple electrodes are placed at predetermined points on a patient's body. Each sensor generates a voltage signal that is captured and plotted. The plot, or electrocardiogram ("ECG" or "EKG"), provides a visual representation of the magnitude of the voltage signal over time. A trained clinician or technician can then determine how "normal" or "abnormal" the patient's heart beats. Thus, in this case, the captured physiological data represents the patient's heartbeat.

[0007] The data captured by such monitoring is usually sorted at least by the acquisition time and the attributes of the physiological characteristics represented in the data. In the above ECG example, the data is sorted not only by the acquisition time and the voltage amplitude, but also by the sensor through which the data is collected. The voltage collected by a single sensor can be presented as a graph or curve of voltage amplitude over time for human perception, which is called a "waveform". It is also common to refer to physiological data collectively as "waveforms". The result of the ECG is then a set of waveforms reflecting the patient's heartbeat.

[0008] As mentioned above with respect to the ECG, a trained clinician or technician can analyze the "waveform" to determine certain aspects of the patient's condition captured in the physiological data. For a variety of reasons, the art has turned to "smart machines," or computing machines that employ various artificial intelligences, to perform such analysis or assessments. However, like the clinician or technician, the smart machine is trained to perform such analysis. Summary of the invention

[0009] In some embodiments, a method includes training an autoencoder with a set of unlabeled input samples by unsupervised learning, and training a deep neural network by supervised learning using the trained autoencoder. The unlabeled input samples can be recorded waveforms representing physiological characteristics of a human body. Training the deep neural network by supervised learning using the trained autoencoder includes: training the deep neural network with a first subset of manually labeled samples selected from the set of unlabeled samples; and iteratively training the deep neural network with multiple continuous subsets of manually labeled samples extracted from the unlabeled samples until convergence or until the unlabeled sample input is exhausted. Each continuous subset includes multiple unlabeled samples with distances selected from the remaining unlabeled samples to which the labels are propagated, and the distance determination includes using an autoencoder for feature extraction.

[0010] In other embodiments, a method includes: training an autoencoder with a plurality of unlabeled input samples by unsupervised learning, and training a deep neural network by supervised learning using the trained autoencoder. The unlabeled input samples are recorded waveforms representing physiological characteristics of a human body. Training a deep neural network by supervised learning using the trained autoencoder includes: manually labeling a first predetermined number of unlabeled samples randomly selected from a plurality of unlabeled input samples to generate a first subset of labeled samples; training the deep neural network with the first subset of labeled samples, manually labeling a second predetermined number of selected, unlabeled samples with distances to generate a second subset of labeled samples, propagating labels to a second predetermined number of unlabeled samples with distances selected from the remaining unlabeled samples that are closest to the labeled samples; and iterating until convergence or the remaining unlabeled samples are exhausted. Manually labeling the selected second predetermined number of distant, unlabeled samples to generate a second subset of labeled samples may include: selecting multiple unlabeled samples with lowest confidence from the remaining unlabeled samples; and filtering the selected multiple unlabeled samples using a trained autoencoder for feature extraction of the compared selected unlabeled samples to discard the selected unlabeled samples that are too close to another selected unlabeled sample.

[0011] In other embodiments, there is a method for annotating a plurality of recorded waveforms representing physiological characteristics of a human body. The method includes: providing a set of unlabeled samples of the recorded waveforms; training a deep neural network with the unlabeled samples to develop an autoencoder; receiving a plurality of manual labels for a first predetermined number of randomly selected unlabeled samples; augmenting the manually labeled randomly selected samples; training a deep neural network with the augmented, manually labeled, randomly selected samples; applying the trained deep neural network to the remaining unlabeled samples; receiving a second predetermined number of selections of the remaining unlabeled samples; propagating the labels to a third predetermined number of remaining unlabeled samples that are closest to the labeled samples; and iterating until convergence or the remaining unlabeled samples are exhausted. In these embodiments, a second predetermined number of selections of the remaining unlabeled samples is received, the selections comprising: identifying a second predetermined number of candidate unlabeled samples with the lowest confidence; and filtering the identified unlabeled candidates. Then, filtering includes using the trained autoencoder for feature extraction to determine whether each identified candidate is too close to an immediately preceding identified candidate; discarding the identified candidate if the identified candidate is too close to the immediately preceding identified candidate; identifying a replacement candidate for the discarded candidate; and iterating the identifying and filtering until a second predetermined number of unlabeled samples have been identified and filtered;

[0012] In yet other embodiments, a computing device includes: a processor-based resource; and a memory in electronic communication with the processor-based resource. The memory may be encoded with instructions that, when executed by the processor-based resource, perform the methods set forth herein.

[0013] Still other embodiments include a non-transitory computer-readable memory encoded with instructions that, when executed by a processor-based resource, perform the methods set forth herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Illustrative embodiments of the subject matter claimed below will now be disclosed. For the sake of clarity, not all features of an actual implementation are described in this specification. It should be understood that in the development of any such actual embodiment, many implementation-specific decisions must be made to achieve the developer's specific goals, such as compliance with system-related and business-related constraints, which will vary from implementation to implementation. Furthermore, it should be understood that such development efforts, even if complex and time-consuming, will be a routine task for those of ordinary skill in the art who have the benefit of this disclosure.

[0015] Figure 1A-1B Depicted are normal beats from a set of acquired ECG waveforms that may be used as sample input in some embodiments of the techniques disclosed herein.

[0016] Figure 2A-2CDepicted are abnormal beats from a set of acquired ECG waveforms that may be used as sample input in some embodiments of the techniques disclosed herein.

[0017] Figure 3 The ECG process is illustrated by which, in some embodiments, sample inputs such as Figure 1A-1B and 2A-2C pulsations.

[0018] Figure 4 depicts an ECG waveform, such as may be seen in Figure 3 The ECG process is collected and processed to receive sample inputs such as Figure 1A-1B and 2A-2C pulsations.

[0019] Figure 5 One particular embodiment of a computing device is conceptually illustrated with which various aspects of the disclosed technology may be practiced, in accordance with one or more embodiments.

[0020] Figure 6A-6B One particular embodiment of an autoencoder that can be used in some embodiments of the presently claimed subject matter is depicted.

[0021] Figure 6C Deep neural networks, and more particularly, convolutional neural networks, are described, which may be used in some embodiments of the presently claimed subject matter.

[0022] Figure 7 Methods that may be practiced in accordance with one or more embodiments are illustrated.

[0023] Figure 8A-8C A partial computer-implemented method is illustrated in accordance with one or more specific embodiments.

[0024] Figure 9A-9C is a flowchart of a specific computer-implemented implementation of one or more embodiments.

[0025] Fig.10 The effect of the technology disclosed in this article is illustrated.

[0026] Figures 11A-11D The diagrams illustrate the presently disclosed technology in one or more embodiments.

[0027] Although the present invention is susceptible to various modifications and alternative forms, the accompanying drawings illustrate specific examples which are described in detail herein by way of example. However, it should be understood that the description of specific examples herein is not intended to limit the invention to the particular forms disclosed, but on the contrary, the invention is intended to cover all modifications, equivalents and substitutes falling within the spirit and scope of the invention as defined by the appended claims. DETAILED DESCRIPTION

[0028] Machine learning and deep learning are very powerful mathematical tools for solving classification and regression problems. Supervised learning and unsupervised learning are two main approaches.

[0029] Supervised learning makes predictions based on a set of labeled training examples provided by the user. This technique is useful when you know what the output should look like. Supervised learning is often used to predict future outputs (regression problems) or to classify input data (classification problems). In supervised learning, a person generates annotations / labels for training samples, then trains a model with these training samples, then tests your model and deploys it to your system.

[0030] In unsupervised learning, data points are not labeled - the algorithm labels them on its own by organizing the data or describing its structure. This technique is useful when it is not known what the output should look like, and one is trying to discover hidden structure within the dataset. For example, one might be provided with customer data and want to create segments of customers who like similar products. The data provided is not labeled, and the labels in the output are generated based on the similarities found between the input data.

[0031] One drawback of supervised learning is that labeling or annotating input samples can be laborious and time consuming. Labeling a set of input samples typically involves a knowledgeable and trained human examining each input sample, determining the "correct" output for each input sample, and then annotating each input sample with the corresponding correct output. The number of input samples can also be large. These kinds of factors, when present, cumulatively ensure that annotating input samples is a long and difficult task.

[0032] The present disclosure proposes a technology, including a method and apparatus for supervised training of neural networks, which greatly reduces the time, cost and effort for annotating training sets and training deep neural networks. This technology can be referred to as a "smart annotation" technology because it results in resource savings. As used herein, smart annotation technology is a technology that uses an autoencoder trained, for example, via unsupervised learning, and applies both supervised learning and unsupervised learning to train a neural network. Supervised learning includes a minimum confidence assessment combined with label propagation and an autoencoder for training a neural network. The technology includes not only this method, but also a computing device programmed to perform this method by executing instructions stored on a computer-readable non-transitory storage medium using processor-based resources.

[0033] The technology disclosed here assumes that a set of input samples have been previously collected. The input samples in the disclosed embodiments are ECG waveforms. However, it should be understood that this is for illustration only, and the embodiments may also operate on input samples of waveforms collected using other processes. For example, in patient care, it is common to monitor many physiological characteristics of patients, which can be collected using other processes and expressed as waveforms. The technology described here can be used in combination with such other waveforms collected by other processes and representing other physiological characteristics. Examples of such other physiological characteristics include, but are not limited to, measurements of invasive pressure (i.e., invasive blood pressure), respiratory gas output (i.e., oxygen, carbon dioxide), blood oxygen, internal pressure (i.e., intracranial pressure), anesthetic (i.e., nitrous oxide) concentrations, etc.

[0034] Furthermore, the input samples disclosed herein are "organic" in the sense that they are collected by actually performing an ECG procedure on a (multiple) patient. In other embodiments, the input samples may be synthetic in the sense that they are collected by artificially generating them. Still other embodiments may use a combination of organic and synthetic waveforms.

[0035] ECG waveforms are collected over time and represent the number of heartbeats that occur within a predetermined time window. These waveforms can be sampled as a portion of a larger ECG waveform, each portion representing a single heartbeat. The ECG waveforms in the input samples of the illustrated embodiment can therefore be referred to as "beats" because they are a portion of a larger waveform representing a single heartbeat. Therefore, the input samples of the disclosed technology can be obtained by sampling portions of a larger waveform.

[0036] This particular embodiment classifies input samples (or "beats") as "normal" or "abnormal" - that is, whether the heartbeats they represent are normal or abnormal. For example, in the illustrated embodiment where the input samples are beats, a beat showing atrial fibrillation (or "afib") would be classified as abnormal. Using the above terminology, each output of the evaluation process would be either "normal" or "abnormal." The terms "normal" and "abnormal" are clinically defined, and these definitions are well known to those skilled in the art of evaluating ECG waveforms.

[0037] For example, for ECG beat classification, each training sample includes one beat, such as Figure 1A – Figure 2CAs shown (about 75 sample points). Each beat needs to be marked / annotated as "marked" or "annotated" if it is a normal or abnormal beat. Samples are usually randomly selected for annotation. In order to achieve 95.1% classification accuracy in ECG beat classification, about 3080 training samples are required using traditional methods. This would be a huge effort to complete as a person would need to go through these samples one by one to annotate them. The currently disclosed method reduces the number of annotated samples to achieve the same accuracy. Using the currently disclosed technology, only about 952 training samples need to be annotated to achieve the same accuracy. Therefore, the annotation work is reduced to about one third.

[0038] In the illustrated embodiment, as described above, ECG beats are divided into two categories: one is normal beats, and the other is abnormal beats, including left bundle branch block, right bundle branch block, bundle branch block, atrial premature beats, ventricular premature beats, supraventricular premature beats or ectopic beats (atrial or nodal), and ventricular escape beats. Figure 2A – Figure 2C Some examples are shown: Atrial fibrillation is an irregular and often rapid heart rate that can increase your risk of stroke, heart failure, and other heart-related complications. We applied a 1D convolutional neural network and achieved very good performance to classify Afib beats from normal beats. Atrial fibrillation is an irregular and often rapid heart rate that can increase your risk of stroke, heart failure, and other heart-related complications. The illustrated embodiment applied a 1D convolutional neural network and achieved very good performance to classify Afib beats from normal beats.

[0039] The disclosed technique uses unsupervised learning to build an autoencoder by training it with unlabeled data. An autoencoder is a neural network that can be used to learn a compressed representation of raw data. An autoencoder consists of an encoder and a decoder sub-model. The encoder compresses the input, and the decoder attempts to reconstruct the input from the compressed version provided by the encoder. After training, the encoder model is saved, and the decoder is discarded. A suitable autoencoder and its use for feature extraction are described below with respect to Figure 6A-6B Further description.

[0040] Then n samples are randomly picked and manually labeled. For example, in one embodiment, N=56. The labeled samples are then trained with a deep neural network with data augmentation (noise with small variance). Data augmentation reduces overfitting during training. The present technology then runs the trained deep neural network on the remaining unlabeled samples, using the lowest confidence to pick N samples, which are then manually labeled. This is a general method used in active learning, but the currently disclosed technology modifies the traditional method. In particular, if two new samples are close enough (features are extracted using a trained autoencoder, and the feature distance between samples is used to represent the sample distance), the present technology will ignore the second sample and continue to look for the next lowest confidence sample.

[0041] Next, label propagation is applied to the N samples. Within the unlabeled samples, the N samples closest to the labeled samples are identified and added for training. But only one sample is propagated for each labeled sample. Therefore, there are 2N new training samples for each iteration. N samples come from user annotations and N samples come from automatic label propagation. The method then returns to training the labeled samples with the deep neural network and iterates until the accuracy changes by less than 0.25% in five iterations.

[0042] The currently disclosed technology can also be Fig.11A Fig.11D Graphically illustrated as shown. Fig.11A This conceptually illustrates the feature space, where there are samples represented by circles. Fig. 11B As shown, N samples are selected and manually labeled. In this example, N=3 for ease of illustration. The deep neural network is then trained and the labels are propagated as described above. Fig. 11C As shown in the identification and Fig.11D As shown, three previously labeled samples, three samples with propagated labels, and three previously unlabeled samples with the lowest confidence are selected. Then use Fig.11D Retrain the deep neural network with the new set of samples. Iterate these last two steps: retraining, then selecting the label sample with the lowest confidence and propagating, until convergence.

[0043] More specifically, Figure 1A-1B depicts a normal pulsation, and Figure 2A-2C Abnormal beats are depicted, which are presented for human perception. One criterion for selecting an input sample set is that it contains a statistically significant number of each potential outcome. Thus, in the illustrated embodiment, there should be a statistically significant number of both "normal" and "abnormal" beats. Note that, as discussed above, a beat is a waveform that is actually an ordered data set.

[0044] However, note that other embodiments may be used to analyze and / or classify input samples representing some other physiological characteristics. Examples of such other physiological characteristics include, but are not limited to, measurements of invasive pressure (e.g., invasive blood pressure), respiratory gas output (e.g., oxygen, carbon dioxide), blood oxygen, internal pressure (e.g., intracranial pressure), anesthetic (e.g., nitrous oxide) concentrations, etc. Therefore, in some embodiments, the classification of input samples may vary to accommodate aspects of the physiological characteristics of interest to clinicians. The number of classifications may also vary from two to three or more. For example, in some embodiments, instead of classifying input samples as "normal" or "abnormal," samples may be classified as "high," "normal," or "low." Yet other physiological characteristics and classifications will be apparent to those skilled in the art who benefit from the present disclosure.

[0045] Although the disclosed techniques assume that input samples have been previously collected, for the sake of completeness, the collection of input samples in the ECG environment will now be discussed. Figure 3 An ECG process 300 according to the present disclosure is illustrated. Figure 3 In the embodiment of the present invention, a patient 303 is undergoing an ECG procedure 300 performed using an ECG system 306. The ECG system 306 includes an ECG monitor 309, a plurality of electrical leads 312, and a plurality of ECG electrodes 315 (only one is shown). Figure 3 A common configuration, and a configuration that can be practiced by the currently disclosed technology, is a 10-electrode, 12-lead configuration for measuring 10 voltages on the human body.

[0046] The ECG system 306 acquires a plurality of ECG waveforms 320, such as Figure 4 4. The ECG waveform 320 is then processed to generate individual input samples or beats 322. This processing may be performed by the ECG monitor 309. However, it is more likely that the ECG waveform 320 will be output to another computing device or computing system (not shown) where processing is performed.

[0047] This processing can be performed manually by a user through a user interface, for example, on a workstation. Thus, for example, in the further discussion below Figure 5 In some embodiments, user 500 may manually process one or more ECG waveforms through user interface 512 of computing device 503. In other embodiments, the processing may be automatically performed by a neural network (not shown) trained for beat detection in ECG waveforms. Figure 3 , an ECG waveform 320 and input samples 322 are shown as being presented for human perception.

[0048] Figure 5A user 500 at a computing device 503 is illustrated, with which certain computer-implemented aspects of the disclosed technology can be performed. Depending on the task being performed and the skills involved, the user 500 can be a person who is trained and skilled in classifying input samples. For example, in embodiments such as disclosed herein where the input sample is a beat, the user 500 can be a clinician or technician who can classify and label the input sample as normal or abnormal. Alternatively, the user 500 can be a person such as a software engineer who is more skilled in the computation of the disclosed technology. Alternatively, the user 500 can be a person with appropriate skills to perform labeling and classification and computation-oriented tasks.

[0049] The computing device 503 may be, for example, a workstation. In this embodiment, the computing device 503 includes a processor-based resource 506, a memory 509, and a user interface 512. The user interface 512 includes a display 515, one or more peripheral devices, such as a keyboard 518 and / or a mouse 521, and a software component ("UI") 524 resident on the memory 509. The UI software component 524 is executed by the processor-based resource 506 to provide a presentation (not separately shown) on the display 515, with which the user 500 can interact using (multiple) peripheral components.

[0050] Those skilled in the art who have the benefit of this disclosure will appreciate that the term "processor" is understood in the art to have a clear structural connotation. A processor can be hardware, software, or a combination of both. Figure 5 In the illustrated embodiment, processor-based resource 506 is a programmed hardware processor, such as a controller, microcontroller, or central processing unit ("CPU"). However, in alternative embodiments, processor-based resource 506 may be a digital signal processor ("DSP"), a graphics processor, a processor chipset, an application-specific integrated circuit ("ASIC"), a suitably programmed electrically programmable read-only memory ("EPROM"), a suitably programmed electrically erasable programmable read-only memory ("EEPROM"), or any other equivalent integrated or discrete logic circuitry, as well as any combination of such components and one or more machine learning algorithms.

[0051] Processor-based resources 506 execute machine executable instructions 527 residing in memory 509 to implement the functions of the technology described herein. Instructions 527 can be embedded in memory 509 as firmware, or encoded as routines, subroutines, applications, etc. Memory 509 is a computer-readable non-temporary storage medium, and can be local or remote. Memory 509 can be distributed across computing clouds, for example. Memory 509 can include read-only memory ("ROM"), random access memory ("RAM"), or a combination of the two. Memory 509 is usually installed memory, but it can also be removable. Memory 509 can be a primary storage device, a secondary storage device, a tertiary storage device, or some combination thereof implemented using electromagnetic, optical or solid-state technology. Therefore, in various embodiments, memory 509 can be part of a mass storage device, a hard drive, a solid-state drive, an external drive (whether disk or solid state), an optical disk, a magnetic disk, a portable external drive, a jump drive, etc.

[0052] Thus, in the illustrated embodiment, computing device 503 performs the computer-implemented functions of the presently disclosed technology. More specifically, processor-based resources 506 execute instructions 527, both in Figure 5 , to implement the programming functions of the disclosed intelligent annotation technology disclosed herein. However, the subject matter currently claimed is not limited thereto.

[0053] For example, in some embodiments, computing device 503 may be some other type of personal computing device, and data may be accessed through a network such as a local area network ("LAN"). In other embodiments, computing device 503 may be a computing system. In such a computing system, processing and memory resources may be distributed on the cloud, for example, accessed from a workstation or other personal computing device through a public or private network such as the Internet. Still other variations of computing device 503 may be implemented by those skilled in the art having the benefit of this disclosure.

[0054] Still reference Figure 5 , a set of input samples 530 resides in memory 509. Input samples 530 may include labeled samples 533 and unlabeled samples 536. For present purposes, "manually labeled" means labeled by a person such as user 500 on a computing device such as computing device 503 through a user interface such as user interface 512. As will be discussed further below, at the beginning of the intelligent annotation technique disclosed herein, all input samples 530 are unlabeled samples 536. The technique includes labeling some of the unlabeled samples 536 so that the number of labeled samples 533 grows as the actions comprising the technique are performed.

[0055] The autoencoder 540 and the deep neural network 545 also reside in the memory 509. The autoencoder 540 will be trained using unsupervised learning with the unlabeled samples 536 in a manner that will be described more fully below. The deep neural network 545 will be trained using intelligent annotation techniques in a manner that will be described more fully below. Note that it is not required that the input samples 530, the autoencoder 540, and the deep neural network 545 reside on the same memory device or the same computing resource as the instructions 527. As discussed above, the computing device 503 can be implemented as a distributed computing system. Therefore, in some embodiments, the input samples 530, the autoencoder 540, the deep neural network 545, and the instructions 527 can reside in different memory devices and / or different computing resources.

[0056] Now we will turn specifically to the autoencoder 540, which is a specific type of neural network that can be used to learn a compressed representation of raw data. The autoencoder includes an "encoder" 541 and a "decoder" 542. The encoder 541 compresses the input, and the decoder 542 attempts to reconstruct the input from the compressed version provided by the encoder. This is sometimes called "feature extraction". The compressed input produced by the encoder 541 is sometimes called the "code" of the autoencoder 540. After training, the encoder 541 is saved and the decoder 542 can be discarded.

[0057] Figure 6A-6B A particular autoencoder 600 suitable for practicing the techniques disclosed herein is depicted. Note that there are many types of autoencoders and many ways to implement each encoder known in the art. For the sake of clarity, and in order not to obscure the content of the following claim, a complete discussion of all these types and implementations is omitted. However, those skilled in the art who benefit from this disclosure will appreciate other types of autoencoders and other implementations that may also be effective in implementing the autoencoder 540.

[0058] The automatic encoder 600 includes an encoder 603 and a decoder 606. The encoder 603 includes a plurality of modules 609a-609k, and the decoder 606 includes a plurality of modules 612a-612k. In the context of the present disclosure, a "module" is an identifiable piece of software that performs a specific function when executed. Each of the modules 609a-609k and 612a-612k is identified using terms known in the art, and performs functions known in the art. Therefore, those skilled in the art will be able to easily implement the automatic encoder 600 according to the disclosure herein. As described above, both the encoder 603 and the decoder 606 are used to train the automatic encoder 600 by unsupervised training, after which the encoder 603 can be discarded.

[0059] The term "module" is used herein in its customary meaning to those skilled in the art. A module may be, for example, an identifiable executable code segment residing in a memory that, when executed, performs a specific function. Depending on the embodiment (or context in which the term is used), a module may also or alternatively be an identifiable hardware segment and / or an identifiable combination of software and hardware that performs a specific function. The software and / or hardware of a module may be dedicated to a single use, or may be used for multiple uses. Other definite meanings may be recognized by those skilled in the art having the benefit of this disclosure.

[0060] Now back to Figure 5 , in addition to the autoencoder 540, the computing device 503 also includes a deep neural network 545, as first mentioned above. The deep neural network 545 can be any kind of deep neural network known in the art that is trained by supervised learning. Therefore, the deep neural network 545 is a different kind of neural network than the autoencoder 540. In the illustrated embodiment, the deep neural network 545 is a convolutional neural network, although alternative embodiments may employ other kinds of deep neural networks. One example convolutional neural network that can be used is ResNet. For example, alternative embodiments may employ, without limitation, a recursive neural network or some other deep neural network trained by supervised training.

[0061] Figure 6C A particular deep neural network 650 is depicted that is suitable for practicing the techniques disclosed herein. Note that there are multiple categories of deep neural networks and multiple ways of implementing each category of deep neural networks known in the art. For the sake of clarity, and so as not to obscure what is claimed below, a full discussion of all of these categories and implementations is omitted. However, those skilled in the art who have the benefit of this disclosure may appreciate other categories of deep neural networks and other implementations that may also be effective in implementing deep neural networks.

[0062] In this particular embodiment, the deep neural network 650 is a convolutional neural network ("CNN"). Note that Resnet is a specific type of CNN. The deep neural network 650 includes a plurality of modules 660a-660m. Again, in the context of the present disclosure, a "module" is an identifiable piece of software that performs a specific function when executed. Each of the modules 660a-660m is identified using terms known in the art and performs functions known in the art. Thus, one skilled in the art will be able to readily implement the deep neural network 650 based on the disclosure herein.

[0063] Back again Figure 5, part of the supervised training of a deep neural network, such as deep neural network 545, is "feature extraction". More specifically, feature extraction typically involves processes such as convolution and pooling. According to the presently disclosed technology, when the deep neural network 545 is trained by supervised training, the deep neural network 545 uses the autoencoder 540 trained by unsupervised learning for feature extraction. Therefore, the embodiment illustrated here employs a convolutional neural network that uses the decoder 606 of the autoencoder 600 for feature extraction during unsupervised training.

[0064] Figure 7 A method 700 is illustrated that can be practiced according to one or more embodiments of the subject matter claimed below. Typically, an autoencoder is trained first. Then, the method starts deep neural network training. After each iteration, the autoencoder is used to determine whether the new sample is close to the previously selected sample. The goal is to select samples with different categories (normal, abnormal, and halfway between normal and abnormal).

[0065] Now common reference Figure 5 and Figure 7 , the method 700 begins by training (at 710) an autoencoder 540 by unsupervised learning with a set of unlabeled input samples 536, which are recorded waveforms representing physiological characteristics. The autoencoder 540 can be, for example, the autoencoder 600 shown in FIG6 , although other embodiments may use other suitable autoencoders. Note that, as discussed above with respect to FIG6 , both the encoder 603 and the decoder 606 are used to train the autoencoder 600.

[0066] In the illustrated embodiment, the unlabeled samples 536 used to train (at 710) the autoencoder 540 are individual “beats” such as Figure 3 322 as shown and discussed above. Thus, in this particular embodiment, the input samples 536 used to train (at 710) the autoencoder 600 represent individual heartbeats of individuals. The autoencoder 600 is trained to identify whether the individual input samples 322 are "normal" or "abnormal". However, in alternative embodiments where the input samples represent some other physiological characteristics, the autoencoder may be trained to classify the input samples differently. Once the autoencoder 600 is trained, the encoder 603 may be discarded.

[0067] The method 700 then continues by training (at 720) a deep neural network 545 by supervised learning using the trained (at 710) autoencoder 540. The deep neural network 545 may be, for example, a convolutional neural network. Typically, the deep neural network 545 is trained to iteratively perform a process on a set of inputs that includes feature extraction of the set of inputs and classifying the inputs based on the extracted features. In the method 700, the neural network 545 uses the trained autoencoder 540 for feature extraction.

[0068] More specifically, in accordance with the presently disclosed techniques, method 700 iteratively trains (at 740) a deep neural network with multiple successive subsets of manually labeled samples 533 extracted from unlabeled samples 536 until convergence or until the unlabeled sample input 536 is exhausted. Each successive subset includes a plurality of unlabeled samples with distances having the lowest confidence selected from the remaining unlabeled samples to which the labels are propagated. Again, distance determination includes using an autoencoder 540 for feature extraction.

[0069] Figure 8A-8C A partially computer-implemented method 800 is depicted in accordance with one or more specific embodiments. Fig. 8A , method 800 begins by training (at 810) an autoencoder using a plurality of unlabeled input samples by unsupervised learning. The unlabeled input samples are recorded waveforms representing physiological characteristics of the human body. For example, the samples may be "beats" or individual patient heartbeats, such as Figure 3 The input sample 322 shown, and for another example, can be used for training Figure 5 Once the autoencoder is trained (at 810), the method 800 proceeds to train (at 820) a deep neural network using the trained autoencoder through supervised learning. The deep neural network may be, for example, Figure 5 A deep neural network 545 is shown.

[0070] First in Fig. 8A The training of the deep neural network shown in (at 820) is Figure 8B . In this particular embodiment, training (at 820) may begin by manually labeling (at 830) a first predetermined number of unlabeled samples randomly selected from a plurality of unlabeled input samples to generate a first subset of labeled samples. Note that this action also creates a subset of unlabeled samples that is smaller than the initial set of unlabeled samples. Furthermore, as unlabeled samples are selected and labeled, they are then removed from the set of unlabeled samples such that the number of unlabeled samples is reduced.

[0071] Once the first subset of labeled samples is obtained (at 830), training (at 820, Fig. 8A) continues by training (at 840) a deep neural network with the first subset of labeled samples. The training (at 840) may be supervised training. Next, manually labeling (at 850) a second predetermined number of selected, distanced, unlabeled samples generates a second subset of labeled samples. Figure 8C As shown, manual labeling (at 850) includes selecting (at 853) a plurality of unlabeled samples with the lowest confidence from the remaining unlabeled samples. The selected plurality of unlabeled samples are then filtered (at 856) using the trained autoencoder for feature extraction of the selected unlabeled samples for comparison to discard the selected unlabeled samples that are too close to another selected unlabeled sample. Note that after discarding, the selected unlabeled samples can be returned to the subset of unlabeled samples.

[0072] The predetermined number of labeled samples of the second subset generated by manual labeling (at 850) should exceed a threshold number. This threshold number can be achieved in a variety of ways. For example, the number of unlabeled samples initially selected for manual labeling (at 850) can be a statistically determined number such that even after the unlabeled samples are discarded, the remaining labeled samples are sufficient in number to exceed the threshold. Alternatively, if the discarding causes the number of labeled samples in the second group to be below the threshold, additional unlabeled samples can be selected and processed. Or some combination of these methods can be used. Still other methods may become apparent to those skilled in the art having the benefit of this disclosure.

[0073] Back to Figure 8B Once the manual labeling of the second set of labeled samples has been performed (at 850), the labels are propagated (at 860) to a second predetermined number of unlabeled samples with distances that are closest to the labeled samples selected from the remaining unlabeled samples. Iteration (at 870) as Figure 8B The training shown (at 820) continues until convergence or the remaining unlabeled samples are exhausted. In this context, "convergence" means that the results do not change and can be objectively quantified. An implementation with objective quantification will now be discussed.

[0074] Figure 9A-9C A method 900 is illustrated for annotating a plurality of recorded waveforms representing a physiological characteristic of a human body. As discussed above with respect to earlier disclosed embodiments, the physiological characteristic of the illustrated embodiment may be a "beat," or a previously acquired heartbeat of an individual patient as discussed with respect to FIGS. 1 and 2 . However, in other embodiments, the physiological characteristic may be some other physiological characteristic, such as blood pressure, respiration, blood oxygen, etc. Furthermore, the present embodiment assumes that the input samples are previously acquired and have been stored awaiting their use in method 900. However, some embodiments may include data acquisition and processing to prepare and condition the input samples for annotation.

[0075] Method 900 is computer-implemented. For current purposes, the discussion will assume that the method is performed in Figure 5 503 of the computing device 504. However, as described above, the computer-implemented aspects of the subject matter claimed below are not limited to being implemented on a computing device such as a computing device. For example, some embodiments may be implemented in a computing environment distributed across various computing and storage resources of a cloud accessed from a workstation via the Internet.

[0076] In this particular embodiment, method 900 is performed by processor-based resources 506 by executing instructions 527. User 500 may invoke method 900 through user interface 512. More specifically, user 500 may invoke method 900 using a peripheral input device such as keyboard 518 or mouse 521 to interact with a user interface such as a graphical user interface (“GUI”) presented by UI 524.

[0077] Reference now Fig. 9A and Figure 5 , method 900 begins by accessing (at 905) a set of unlabeled samples 536 of recorded waveforms. The unlabeled input samples are recorded waveforms that represent physiological characteristics of the human body. For example, the samples may be "beats" or individual patient heartbeats, such as Figure 3 Input samples 322 are shown. In the illustrated embodiment, the number of unlabeled samples may be, for example, 500 samples.

[0078] The method 900 then trains (at 910) a deep neural network with unlabeled samples 536 to develop an autoencoder 540. Since the samples are unlabeled, the training (at 910) is unsupervised training. As discussed above, the autoencoder 540 includes an encoder 541 and a decoder 542. The encoder 541 may be used only in the training (at 910) of the autoencoder 540 and may be discarded afterwards.

[0079] The method 900 then receives (at 920) a plurality of manual labels for a first predetermined number of randomly selected unlabeled samples 536. More specifically, the user 500 randomly selects a number N of samples from the unlabeled samples 536 through the user interface 512 and manually labels them. In the illustrated embodiment, N=56, but may have other values. For example, in other embodiments, N may be 40 or 80. The processor-based resource 506 receives manual labels for the randomly selected unlabeled samples 536 from the user 500 to create labeled samples 533. Note that as samples are selected and labeled, they are removed from the pool of unlabeled samples 536 to join the pool of labeled samples 533.

[0080] The manually labeled, randomly selected samples are then "augmented" (at 930). Augmentation is a process known in the art by which distortions or "noise" with relatively small variations are intentionally introduced into the samples to avoid a phenomenon known as "overfitting." In this context, overfitting describes a situation in which training causes the deep neural network to adapt to the samples on which it was trained, which impairs the ability of the deep neural network to accurately classify other samples on which it was not trained. This is an optional step and may be omitted in some embodiments.

[0081] Those skilled in the art who benefit from this disclosure will appreciate that expanding samples is only one way to adjust samples. There are other techniques known in the art by which samples can be adjusted. Therefore, other embodiments may also use other techniques instead of or in addition to expansion to mitigate overfitting and solve other problems that may be encountered with unadjusted samples.

[0082] The method 900 then trains (at 940) a deep neural network 545 using the augmented, manually labeled, randomly selected samples 533. As mentioned above, the deep neural network 545 is a convolutional neural network, although alternative embodiments may employ other kinds of deep neural networks. The training is an example of supervised training because the samples used are labeled.

[0083] The method 900 then applies (at 950) the trained deep neural network 545 to the remaining unlabeled samples 536. This application results in additional unsupervised training of the deep neural network 545, since the unlabeled samples 536 are unlabeled. Thus, the presently disclosed techniques may be referred to as hybrid supervised-unsupervised learning techniques. Likewise, the deep neural network 545 at the end of the training may be referred to as hybrid supervised-unsupervised trained.

[0084] Next, the method 900 continues by selecting (at 960) a second predetermined number of the remaining unlabeled samples 536. In this embodiment, the second predetermined number is equal to the first predetermined number (at 920), but other embodiments may use a second predetermined number that is different from the first predetermined number. Although some aspects of the selection (at 960) can be performed manually in some embodiments, in this embodiment the selection (at 960) is performed in an automatic manner. The selection (at 960) is performed by the processor-based resource 506 executing the instructions 527.

[0085] like Fig. 9BAs shown, the selection (at 960) begins by identifying (at 961) a second predetermined number of candidate unlabeled samples 536 from the previous round of training with the lowest confidence. "Confidence" in this context refers to the confidence that the output of the deep neural network is correct. "Lowest confidence" then describes the unlabeled sample 536 where the previous application of the neural network (at 950) produced the "lowest confidence" that the correct output has been achieved. That is, the confidence that the classification of a given unlabeled sample 536 is correct is the lowest.

[0086] The "lowest confidence" unlabeled samples 536 that have been identified (at 961) are then filtered (at 962). Fig. 9C As shown, filtering may include using (at 963) a trained autoencoder for feature extraction to determine whether each identified candidate is too close to the immediately preceding identified candidate. In this particular embodiment, "too close" is measured by the trained autoencoder 540 for feature calculation and is set to 5% of the maximum magnitude and distance on the vector. However, other embodiments may use other measures of "too close", using other than Fig. 9C Some filtering technique other than that shown, or even omitting the filtering (at 963).

[0087] If (at 964) an identified candidate is too close to an immediately preceding identified candidate, the identified candidate may be discarded. A replacement candidate for the discarded candidate may then be identified (at 965). The process of identifying (at 963), discarding (at 964), and replacing (at 965) may be iterated (at 966) until a second predetermined number of unlabeled samples have been identified and filtered.

[0088] Note that an alternative embodiment may select (at 960) a second predetermined number of selections of the remaining unlabeled samples 536, which differs from the above with respect to Fig. 9B and Fig. 9C As described. The point of this step is to obtain a second predetermined number of unlabeled samples 536. Therefore, in all embodiments, the selection (at 960) process does not necessarily require filtering, discarding, and identifying replacements. For example, some embodiments may identify a third predetermined number of candidates greater than the second number, and then discard less desirable candidates until a second predetermined number of candidates is obtained. Those skilled in the art who benefit from this disclosure may appreciate that other suitable variations of the selection (at 960) may be performed.

[0089] Back to Fig. 9A, after selecting (at 960) a second predetermined number of remaining unlabeled samples 536, method 900 then propagates (at 970) the labels to a third predetermined number of remaining unlabeled samples that are closest to the labeled samples, and adds these labeled (at 970) samples to the training set of labeled samples 533.

[0090] The method 900 then iterates (at 980) until convergence or the remaining unlabeled samples are exhausted. More specifically, the method 900 first checks (at 982) whether convergence has been reached. In the illustrated embodiment, convergence is reached when the accuracy changes by less than 0.25% in five iterations. If convergence has been reached, the method 900 ends (at 984).

[0091] If convergence has not occurred (at 982), the method 900 checks (at 986) whether the unlabeled samples 536 have been exhausted. Recall that each iteration removes samples from the set of unlabeled samples 536 by labeling them, and therefore, in some embodiments, the set of unlabeled samples 536 may be exhausted by a sufficient number of iterations. If the unlabeled samples 536 are exhausted (at 986), the method 900 ends (at 984). If there are additional unlabeled samples 536 remaining (at 986), execution returns to the selection of unlabeled samples 536 (at 960).

[0092] Fig.10 The effect of the technology disclosed here is illustrated in FIG. Fig.10 The figure shows a graphical representation of classification accuracy versus number of samples as described above. The currently disclosed technique is represented by curve 1000, while the conventional technique using random selection of input samples is represented by curve 1002. Fig.10 As shown, the currently disclosed technology achieves a higher level of confidence with the same or fewer number of samples. In addition, as the number of samples increases, the benefits of the disclosed technology increase. Therefore, a trained deep neural network for classifying unlabeled input samples, which are recorded waveforms representing physiological characteristics of the human body, can be developed more quickly and accurately.

[0093] A deep neural network trained in the manner described here, such as Figure 5 The deep neural network 545 shown can then be used in a clinical environment, such as that shown in FIG. 1 , to treat and / or diagnose a patient's condition. As described above, the deep neural network can reside on a workstation or in the cloud and be accessed locally or remotely by a user. The collected samples representing physiological characteristics of the human body, such as Figure 3The pulse 322 shown is used to classify the collected sample. The process can be automated in the sense that the process is computer-implemented and is performed without human interaction or intervention. A deep neural network trained as described herein can perform automatic classification more consistently, more accurately, and more economically than current neural networks used for this purpose.

[0094] The features of several embodiments are summarized above so that those of ordinary skill in the art can better understand various aspects of the present disclosure. Those of ordinary skill in the art should understand that they can easily use the present disclosure as a basis for designing or modifying other processes and structures to achieve the same purposes and / or achieve the same advantages of the various embodiments described herein. Those of ordinary skill in the art should also recognize that such equivalent structures do not depart from the spirit and scope of the present disclosure, and that they can make various changes, substitutions and modifications herein without departing from the spirit and scope of the present disclosure.

[0095] Although the subject matter has been described in language specific to structural features or methodological acts, it should be understood that the subject matter of the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing at least some of the claims.

[0096] Various operations of the embodiments are provided herein. The order in which some or all operations are described should not be interpreted as implying that these operations are necessarily order-dependent. With the benefit of this description, alternative sequencing will be understood. In addition, it will be understood that not all operations must be present in each embodiment provided herein. In addition, it will be understood that in some embodiments, not all operations are necessary.

[0097] It should be understood that in some embodiments, for the purpose of simplicity and ease of understanding, the layers, features, elements, etc. described herein are illustrated with specific dimensions relative to each other, such as structural dimensions or orientation, and the same actual dimensions are significantly different from the dimensions illustrated herein. In addition, the "exemplary" used here refers to being used as an example, instance, illustration, etc., and is not necessarily advantageous. As used in this application, "or" is intended to represent an inclusive "or", rather than an exclusive "or". In addition, "one" and "an" used in this application and the appended claims are generally interpreted as representing "one or more", unless otherwise specified or clearly pointed to the singular form from the context. At least one of A and B, etc. represents A or B or both A and B. In addition, in terms of using "including", "having", "containing", "with" or its variants, such terms are intended to be included in a manner similar to the term "comprising". In addition, unless otherwise specified, "first", "second", etc. are not intended to imply time, space, order, etc. On the contrary, such terms are only used as identifiers, names, etc. for features, elements, items, etc. For example, the first element and the second element generally correspond to element A and element B, or two different or two identical elements, or the same element.

[0098] The detailed description ends here. The particular examples disclosed above are illustrative only, as the invention may be modified and practiced in different but equivalent manners apparent to those skilled in the art having the benefit of the teachings herein. Furthermore, no limitations are intended to the details of construction or design shown herein, except as described in the claims below. Therefore, it is evident that the particular examples disclosed above may be altered or modified, and all such variations are considered within the scope and spirit of the invention. Therefore, the protection sought herein is set forth in the claims below.

Claims

1. A method comprising: training an autoencoder using a set of unlabeled input samples by unsupervised learning, wherein the unlabeled input samples are recorded waveforms representing physiological characteristics of a human body; as well as Use trained autoencoders to train deep neural networks via supervised learning, including: training a deep neural network using a first subset of manually labeled samples selected from the set of unlabeled samples; as well as A deep neural network is iteratively trained with multiple consecutive subsets of manually labeled samples extracted from the unlabeled samples until convergence or until the unlabeled sample input is exhausted, each consecutive subset comprising a plurality of unlabeled samples with distances selected with lowest confidence from the remaining unlabeled samples to which the labels are propagated, the distance determination comprising using an autoencoder for feature extraction.

2. The method of claim 1 further comprises adjusting the first subset of labeled samples to avoid overfitting before training the deep neural network with the first subset of labeled samples. The method of claim 1 , wherein adjusting the first subset of labeled samples comprises expanding the first set of labeled samples.

4. The method of claim 1, wherein training a deep neural network with a first subset of manually labeled samples selected from the set of unlabeled samples comprises: receiving a plurality of manual labels for a first predetermined number of randomly selected unlabeled samples; as well as Train a deep neural network with manually labeled randomly selected samples.

5. The method of claim 4, wherein training the deep neural network with a first subset of manually labeled samples selected from the set of unlabeled samples further comprises augmenting the manually labeled randomly selected samples.

6. The method of claim 1, wherein iteratively training a deep neural network with multiple consecutive subsets of manually labeled samples comprises: receiving a selection of a second predetermined number of remaining unlabeled samples; as well as The labels are propagated to a third predetermined number of remaining unlabeled samples that are closest to the labeled samples. The method of claim 6 , wherein the first predetermined amount is equal to the second predetermined amount.

8. The method of claim 6, wherein receiving a second predetermined number of selections comprises: identifying a second predetermined number of candidate unlabeled samples having the lowest confidence; as well as The identified candidate unlabeled samples are filtered to enforce distances between the candidate unlabeled samples.

9. The method of claim 8, wherein filtering the identified candidate unlabeled samples comprises: using the trained autoencoder for feature extraction to determine whether each identified candidate is too close to an immediately previously identified candidate; If the identified candidate is too close to the immediately previously identified candidate, the identified candidate is discarded; identifying replacement candidates for the discarded candidates; as well as The identifying and filtering are repeated until a second predetermined number of unlabeled samples have been identified and filtered.

10. The method according to claim 8, wherein: The identified candidates are too close in the sense that the identified candidates are less than 5% of the maximum magnitude and distance on the vector.

11. The method according to claim 1, wherein: Convergence was reached when the accuracy changed by less than 0.25% over five iterations.

12. A computing device comprising: Processor-based resources; as well as A memory in electronic communication with a processor-based resource and encoded with instructions which, when executed by the processor-based resource, carry out the method of any one of claims 1 to 11.

13. A non-transitory computer readable memory encoded with instructions which, when executed by a processor-based resource, carry out the method of any one of claims 1 to 11.

14. A method comprising: training an autoencoder using a plurality of unlabeled input samples by unsupervised learning, wherein the unlabeled input samples are recorded waveforms representing physiological characteristics of a human body; as well as Use trained autoencoders to train deep neural networks via supervised learning, including: manually labeling a first predetermined number of randomly selected unlabeled samples from the plurality of unlabeled input samples to generate a first subset of labeled samples; Training a deep neural network using a first subset of labeled samples; Manually marking the selected second predetermined number of distanced, unlabeled samples to generate a second subset of labeled samples, comprising: Selecting a plurality of unlabeled samples with the lowest confidence from the remaining unlabeled samples; and filtering the selected plurality of unlabeled samples using the trained autoencoder for feature extraction of the compared selected unlabeled samples to discard a selected unlabeled sample that is too close to another selected unlabeled sample; Propagate the labels to a selected second predetermined number of distanced unlabeled samples that are closest to the labeled samples among the remaining unlabeled samples; and Iterate until convergence or the remaining unlabeled samples are exhausted.

15. The method of claim 14, further comprising augmenting the first subset of labeled samples before training the deep neural network with the first subset of labeled samples to avoid overfitting.

16. The method according to claim 14, wherein: The first predetermined number is equal to the second predetermined number.

17. The method according to claim 14, wherein: The identified candidates are too close in the sense that the identified candidates are less than 5% of the maximum magnitude and distance on the vector.

18. The method according to claim 14, wherein: Convergence was reached when the accuracy changed by less than 0.25% over five iterations.

19. A method for annotating a plurality of recorded waveforms representing physiological characteristics of a human body, the method comprising: providing a set of unlabeled samples of recorded waveforms; Train deep neural networks with unlabeled samples to develop autoencoders; receiving a plurality of manual labels for a first predetermined number of randomly selected unlabeled samples; Augmenting manually labeled randomly selected samples; Train a deep neural network with augmented, manually labeled, randomly selected samples; Apply the trained deep neural network to the remaining unlabeled samples; Receiving a selection of a second predetermined number of remaining unlabeled samples, the selection comprising: identifying a second predetermined number of candidate unlabeled samples having the lowest confidence; The identified candidate unlabeled samples are filtered by the following steps: using the trained autoencoder for feature extraction to determine whether each identified candidate is too close to an immediately previously identified candidate; If the identified candidate is too close to the immediately previously identified candidate, the identified candidate is discarded; identifying replacement candidates for the discarded candidates; and iterating said identifying and filtering until a second predetermined number of unlabeled samples have been identified and filtered; Propagating the labels to a third predetermined number of remaining unlabeled samples that are closest to the labeled samples; and Iterate until convergence or the remaining unlabeled samples are exhausted.

20. The method according to claim 19, wherein: The first predetermined number is equal to the second predetermined number.

21. The method according to claim 19, wherein: The identified candidates are too close in the sense that the identified candidates are less than 5% of the maximum magnitude and distance on the vector.

22. The method according to claim 19, wherein: Convergence was reached when the accuracy changed by less than 0.25% over five iterations.

23. A computing device comprising: Processor-based resources; as well as A memory in electronic communication with the processor-based resource and encoded with instructions which, when executed by the processor-based resource, carry out the method of any one of claims 19 to 22.

24. A non-transitory computer readable memory encoded with instructions which, when executed by a processor-based resource, carry out the method of any one of claims 19 to 22.

Images