Techniques for quality assessment of algorithm results

Abstract

A method includes generating a model representing first data used to train an algorithm and generating a decision metric for determining whether a similarity between second data input into the algorithm and the first data is sufficient for the algorithm to generate a valid output based on a pattern of the second data explained by an interpretation of the first data identified in the model.

Description

Cross-reference to related applications

[0001] This patent application claims priority to U.S. Provisional Patent Application 63 / 610,202, filed December 14, 2023, which is incorporated herein by reference in its entirety. Technical Field

[0002] This disclosure relates to systems, methods, and computer-readable storage media for generating quality assessments of results produced by algorithms and for utilizing such quality assessments. Background Technology

[0003] Over the past few decades, medical devices have provided a wide range of therapeutic benefits to recipients. Medical devices can include internal or implantable components / devices, external or wearable components / devices, or combinations thereof (e.g., devices having an external component that communicates with the implantable component). Medical devices, such as conventional hearing aids, partially or fully implantable hearing prostheses (e.g., bone conduction devices, mechanical stimulators, cochlear implants, etc.), pacemakers, defibrillators, functional electrical stimulation devices, and other medical devices, have been successful for many years in performing life-saving and / or lifestyle improvement functions and / or recipient monitoring.

[0004] Over the years, the types of medical devices and the range of functions they perform have increased. For example, many medical devices, sometimes referred to as “implantable medical devices,” now typically include one or more instruments, devices, sensors, processors, controllers, or other functional mechanical or electrical components that are permanently or temporarily implanted into a recipient’s body. These functional devices are typically used to diagnose, prevent, monitor, treat, or manage diseases / injuries or their symptoms, or to study, replace, or modify anatomical structures or physiological processes. Many of these functional devices utilize power and / or data received from an external device that is part of or operates in conjunction with the implantable component. Summary of the Invention

[0005] According to a first embodiment disclosed herein, a method includes: generating a model representing first data for training an algorithm; and generating a decision metric for determining whether the similarity between the second data and the first data is sufficient for the algorithm to generate a valid output based on patterns of interpretation of the first data identified in the model and input into the algorithm.

[0006] According to a second embodiment disclosed herein, a computing system includes one or more processing units that generate a model including information from first data used to develop an algorithm. The computing system generates a trust estimator that uses the model and the results of the algorithm generated based on second data to provide a representation of the relationship between the first data and the second data, for estimating whether the results of the algorithm are trustworthy.

[0007] According to a third embodiment disclosed herein, a non-transitory computer-readable storage medium includes computer-readable instructions stored thereon, the computer-readable instructions being configured to cause a computing system to: generate a quality assessment of the relationship between the first dataset and a second dataset using a model including features of a first dataset; and determine a similarity between the first dataset and the second dataset using a trust estimator that processes the quality assessment to evaluate the credibility of the results generated by an algorithm in response to the second dataset. The algorithm is developed using the first dataset.

[0008] According to a fourth embodiment disclosed herein, a computer-implemented method for estimating the confidence of the output of an algorithm that has been trained using training data includes: generating a representation of the relationship between the training data and input data using a model that includes a description of the training data, wherein the algorithm generates the output using the input data; and generating a confidence value for the output of the algorithm based on the output of the algorithm and based on the representation of the relationship between the training data and the input data using a decision metric. Attached Figure Description

[0009] Figure 1A Schematic diagrams are depicted of exemplary cochlear implants that can be configured to implement aspects of the techniques presented herein, according to some exemplary embodiments.

[0010] Figure 1B Depicting Figure 1A Functional block diagram of cochlear implants.

[0011] Figure 1C This is a diagram illustrating an example of a hearing prosthesis that may include one or more embodiments disclosed herein.

[0012] Figure 1D This is a functional block diagram of an exemplary fully implantable cochlear implant.

[0013] Figure 2A It is a diagram depicting an example of a training configuration for a system that can generate training embedding models and decision metrics during the training phase.

[0014] Figure 2BThis is a diagram depicting an example of a clinical configuration of a system that can be used to generate quality assessments of the results of an algorithm during the clinical application phase.

[0015] Figure 3A This is a diagram depicting an example of a training configuration for a system that uses the results of a pre-trained objective algorithm to generate a trained embedding model and decision metrics during the training phase.

[0016] Figure 3B This is a diagram depicting an example of a clinical configuration of a system that can be used to generate quality assessments of the algorithm's results during the clinical application phase.

[0017] Figure 4A This is a diagram illustrating an example training configuration of a system that can generate principal component analysis (PCA) transformations and residuals during the training phase to evaluate the reliability of the algorithm's results.

[0018] Figure 4B It is a graph depicting the result of an example of the target algorithm and an example of the trust value for that result.

[0019] Figure 4C A graph depicting a probability curve, which is formed by... Figure 4A An example of a trust value generated by a trust estimator.

[0020] Figure 4D This is a diagram depicting an example of a clinical configuration of a system that can be used to generate quality assessments of the results of the algorithm using PCA transformation during the clinical application phase.

[0021] Figure 5A This is a diagram illustrating an example training configuration of a system that can train the K-nearest neighbor algorithm during the training phase to evaluate the reliability of the results of the target algorithm.

[0022] Figure 5B This is a diagram depicting an example of a clinical configuration of a system that can be used to generate quality assessments of the results of a target algorithm using the K-nearest neighbor algorithm during the clinical application phase.

[0023] Figure 6A This is a diagram illustrating an example of a training configuration for a system that generates a model using the statistical properties of preprocessed training data during the training phase.

[0024] Figure 6B It is a graph that depicts a model that includes the statistical properties of preprocessing from the dataset and selected features.

[0025] Figure 6C This is a diagram illustrating a clinical configuration example of a system that can be used to generate a quality assessment of the results of a target algorithm using the statistical properties of preprocessed data during the clinical application phase.

[0026] Figure 7A This is a diagram illustrating an example training configuration of a system that can train a generative adversarial neural network during the training phase to evaluate the credibility of the results of a target algorithm.

[0027] Figure 7B This is a diagram depicting an example of a clinical configuration of a system that can be used to generate quality assessments of the results of a target algorithm using a discriminator network during the clinical application phase.

[0028] Figure 8 The diagram illustrates an example of a computing system within which one or more of the disclosed embodiments may be implemented. Detailed Implementation

[0029] For ease of description only, the techniques presented herein are described primarily with reference to illustrative medical devices (i.e., cochlear implant systems). However, it should be understood that the techniques presented herein can also be used with a variety of other medical devices that can benefit from the teachings used herein in other medical devices while providing a wide range of therapeutic benefits to recipients, patients, or other users. For example, any technique described herein for a type of hearing prosthesis (such as a cochlear implant system) corresponds to the disclosure of another embodiment that uses such teachings with another hearing prosthesis and also with other electro-analogous hearing prostheses (e.g., auditory brain stimulators), such as bone conduction devices (percutaneous, active transdermal, and / or passive transdermal), middle ear hearing prostheses, and direct acoustic stimulators. The technologies presented in this article can also be used with vestibular devices (e.g., vestibular implants), visual devices (i.e., bionic eyes), sensors, pacemakers, drug delivery systems, defibrillators, functional electrical stimulation devices, catheters, seizure devices (e.g., devices for monitoring and / or treating epileptic events), sleep apnea devices, electroporation devices, and the like.

[0030] The teachings detailed herein can be implemented in or with sensory prostheses such as hearing implants. Other types of sensory prostheses may include retinal implants. Therefore, unless otherwise stated, any teachings herein concerning sensory prostheses correspond to disclosures regarding their use in / with hearing implants and in / with retinal implants, provided that it is possible in the art to do so. Furthermore, any teachings herein, unless otherwise indicated, correspond to disclosures regarding their use in / with cochlear implants, bone conduction devices (active and passive transdermal bone conduction devices, and percutaneous bone conduction devices), and middle ear implants, provided that it is possible in the art to do so. It should be understood that any teachings herein concerning a particular sensory prosthesis correspond to disclosures regarding their use in / with any of the aforementioned hearing prostheses, and vice versa. It is therefore inferred that at least some of the teachings detailed herein can be implemented in somatosensory implants and / or chemosensory implants. Therefore, any teachings in this document concerning sensory prostheses correspond to the disclosures regarding the use of such teachings with or in connection with somatosensory implants and / or chemosensory implants.

[0031] While the teachings detailed herein are primarily described with respect to hearing prostheses, it should be noted, consistent with the foregoing, that any disclosure herein regarding hearing prostheses corresponds to a disclosure of another embodiment utilizing the associated teachings with respect to any other device or prosthesis mentioned herein (whether a hearing prosthesis or a sensory prosthesis, such as a retinal prosthesis). In this respect, unless explicitly indicated and / or unless it is not possible to achieve this in the art, any disclosure herein regarding the induction of auditory perception corresponds to a disclosure of inducing other types of neural perception (such as visual / sight perception, tactile perception, olfactory perception, or gustatory perception) in other embodiments. Any disclosure herein regarding devices, systems, and / or methods for or generating the final stimulation of the auditory nerve corresponds to a disclosure of similar stimulation of the optic nerve utilizing similar components, methods, and / or systems.

[0032] Figure 1A This is a schematic diagram of an exemplary conventional cochlear implant 100 configured to implement various aspects of the techniques presented herein. Figure 1B yes Figure 1A A block diagram of a standard cochlear implant 100. For ease of explanation, it will be described together. Figure 1A and 1B The cochlear implant 100 includes an external component 102 and an internal / implantable component 104. The external component 102 is attached directly or indirectly to the recipient's body and typically includes an external coil 106 and a magnet typically fixed relative to the external coil 106. Figure 1A-1B(Not shown in the image). External component 102 also includes one or more input elements / devices 113 for receiving input signals at sound processing unit 112. In this example, one or more input devices 113 include sound input devices 108 configured to capture / receive input signals (e.g., a microphone, pickup coil, etc., located by the receiver's auricle 110), one or more auxiliary input devices 109 (e.g., an audio port such as a direct audio input (DAI), a data port such as a universal serial bus (USB) port, a cable port, etc.), and a wireless transmitter / receiver (transceiver) 111, each located in, on, or near sound processing unit 112.

[0033] The sound processing unit 112 also includes, for example, at least one power supply 107, a radio frequency (RF) transceiver 121, and a processing module 125. The processing module 125 includes a plurality of elements, including an environment classifier 131, a sound processor 133, and an individualized self-speech detector 134. Each of the environment classifier 131, the sound processor 133, and the individualized self-speech detector 134 may be formed by one or more processors (e.g., one or more digital signal processors (DSPs), one or more processing cores, etc.) arranged to perform the operations described herein, firmware, software, etc. That is, the environment classifier 131, the sound processor 133, and the individualized self-speech detector 134 may each be implemented as a firmware element, partially or entirely using digital logic gates in one or more application-specific integrated circuits (ASICs), partially or entirely using software, etc.

[0034] exist Figure 1A and 1B In the example, the sound processing unit 112 is a behind-the-ear (BTE) sound processing unit, which is configured to be attached to and worn adjacent to the recipient's ear. However, it should be understood that the sound processing unit 112 may have other arrangements, such as an off-the-ear (OTE) processing unit (e.g., a component having a generally cylindrical shape and configured to be magnetically coupled to the recipient's head), a mini or micro BTE unit, an intracanal unit configured to be located in the recipient's ear canal, a body-worn sound processing unit, etc.

[0035] exist Figure 1A and 1BIn an exemplary embodiment, the implantable component 104 includes an implant body (main module) 114, a lead area 116, and an intracochlear stimulation assembly 118, all configured to be implanted beneath the recipient's skin / tissue (tissue) 105. The implant body 114 typically includes a hermetically sealed housing 115, in which an RF interface circuitry 124 and a stimulator unit 120 are disposed. The implant body 114 also includes an internal / implantable coil 122, which is typically outside the housing 115 but connected via a hermetically sealed feedthrough (…). Figure 1B (Not shown) is connected to the RF interface circuit system 124.

[0036] As described above, the stimulation component 118 is configured to be at least partially implanted in the cochlea 137 of a recipient. The stimulation component 118 includes a plurality of longitudinally spaced intracochlear electrical stimulation contacts (electrodes) 126, which together form a contact or electrode array 128 for delivering electrical stimulation (current) to the cochlea 137 of the recipient. The stimulation component 118 extends through an opening in the recipient's cochlea (e.g., cochlear fenestration, round window, etc.) and has a lead area 116 and an airtight feedthrough (…). Figure 1B (Not shown) is connected to the proximal end of the stimulator unit 120. The lead region 116 includes a plurality of conductors (wires) that electrically couple the electrode 126 to the stimulator unit 120.

[0037] As noted, the cochlear implant 100 includes an external coil 106 and an implantable coil 122. Coils 106 and 122 are typically wire antenna coils each comprising multiple turns of electrically insulated single-strand or multi-strand wire. Generally, a magnet is fixed in place relative to each of the external coil 106 and the implantable coil 122, but the magnet can be rotated or reoriented. In some embodiments, the external component 102 and / or the implantable component 104 may include magnet assemblies each having more than one magnet component. The magnets fixed relative to the external coil 106 and the implantable coil 122 facilitate operational alignment of the external coil and the implantable coil. This operational alignment of coils 106 and 122 enables the external component 102 to transmit data, and possibly power, to the implantable component 104 via a tightly coupled wireless link formed between the external coil 106 and the implantable coil 122. In some examples, the tightly coupled wireless link is a radio frequency (RF) link. However, various other types of energy transfer (such as infrared (IR), electromagnetic, capacitive, and inductive transfer) can be used to transfer power and / or data from external components to implantable components, and thus, Figure 1B Only one exemplary arrangement is shown.

[0038] As described above, the sound processing unit 112 includes a processing module 125. The processing module 125 is configured to convert an input audio signal into a stimulation control signal 136 for stimulating the receiver's first ear (i.e., the processing module 125 is configured to perform sound processing on the input audio signal received at the sound processing unit 112). In other words, the sound processor 133 (e.g., one or more processing elements implementing firmware, software, etc.) is configured to convert the captured input audio signal into a stimulation control signal 136 representing electrical stimulation to be delivered to the receiver. The input audio signal processed and converted into a stimulation control signal may be an audio signal received via the sound input device 108, a signal received via the auxiliary input device 109, and / or a signal received via the wireless transceiver 111.

[0039] exist Figure 1B In one embodiment, a stimulation control signal 136 is provided to an RF transceiver 121, which transceives the stimulation control signal 136 (e.g., in an encoded manner) transdermally to the implantable component 104 via an external coil 106 and an implantable coil 122. That is, the stimulation control signal 136 is received at the RF interface circuitry 124 via the implantable coil 122 and provided to the stimulator unit 120. The stimulator unit 120 is configured to use the stimulation control signal 136 to generate an electrical stimulation signal (e.g., a current signal) for delivery to the recipient's cochlea via one or more stimulation contacts 126. In this way, the cochlear implant 100 electrically stimulates the recipient's auditory nerve cells to induce the recipient to perceive one or more components of the input audio signal, bypassing the missing or defective hair cells that typically translate acoustic vibrations into neural activity.

[0040] Figure 1C This is a diagram illustrating an example of a hearing prosthesis 150 that may include one or more embodiments disclosed herein. Figure 1C The auditory prosthesis 150 is an example of a cochlear implant. As a more specific example, the auditory prosthesis 150 may be a partially implantable cochlear implant (MICI) or a fully implantable cochlear implant (TICI).

[0041] The hearing prosthesis 150 includes an internal / implantable component 154. In some embodiments, the hearing prosthesis 150 may also have an external component (not shown) that is positioned via the recipient's auricle 159 and configured to attach to and be worn adjacent to the recipient's ear. However, the external component may have other arrangements, such as an off-ear (OTE) processing unit (e.g., a component configured to be magnetically coupled to the recipient's head), an intracanal unit configured to be located in the recipient's ear canal 156, etc.

[0042] The implantable component 154 includes an implant body 160, a lead area 116, and a slender cochlear stimulation assembly 118, all configured to be implanted beneath the recipient's skin / tissue 155. The implant body 160 includes an hermetically sealed shell housing the various components. The shell of the implant body 160 serves as a protective barrier between the components within the shell of the implant body 160 and the recipient's tissues and bodily fluids.

[0043] The stimulation component 118 is configured to be at least partially implanted in the cochlea 162 of a recipient. The stimulation component 118 includes a plurality of longitudinally spaced intracochlear electrical stimulation contacts (electrodes) 126, which together form a contact or electrode array 128 for delivering electrical stimulation (current) to the cochlea 162 of the recipient. The stimulation component 118 extends through an opening in the recipient's cochlea (e.g., cochlear fenestration, round window, etc.) and has a lead area 116 and an airtight feedthrough (…). Figure 1C (Not shown) Connected to the proximal end of the stimulator unit in the implant body 160. Lead region 116 includes a plurality of conductors (wires) electrically coupling stimulation contacts 126 to the stimulator unit.

[0044] Figure 1D This is a functional block diagram of an exemplary fully implantable cochlear implant 170. Because the cochlear implant 170 is fully implantable, all components of the cochlear implant 170 are configured to be implanted under the recipient's skin / tissue 175. Since all components are implantable, the cochlear implant 170 operates in a "hidden hearing" mode without the need for an external device, at least for a limited period of time. An external device 172 may be used, for example, to charge the internal power source (battery) 177. The external device 172 may be a dedicated charger or a conventional cochlear implant sound processor.

[0045] As described above, the cochlear implant 170 includes an implant body (primary implantable component) 174, one or more input elements for capturing / receiving input audio signals (e.g., using one or more implantable microphones 178 and a wireless transceiver 181), an implantable coil 182, and an elongated intracochlear stimulation assembly. The microphones 178 and / or the implantable coil 182 may be located in or electrically connected to the implant body 174. The implant body 174 also includes a battery 177, an RF (radio frequency) interface circuitry system 184, a processing module 185, and a stimulator unit 180. The processing module 185 may be similar to the aforementioned processing module and includes an environmental classifier 191, a sound processor 193, and an individualized self-speech detector 195.

[0046] exist Figure 1DIn one embodiment, one or more implantable microphones 178 are configured to receive input audio signals. A processing module 185 is configured to convert the received input audio signals into stimulation control signals 196 for stimulating the recipient's first ear. In other words, a sound processor 193 is configured to convert the input audio signals into stimulation control signals 196 representing electrical stimulation to be delivered to the recipient.

[0047] exist Figure 1D In one embodiment, the processing module 185 is implanted in the recipient's body. Therefore, in Figure 1D In one embodiment, the stimulation control signal 196 does not traverse the RF link, but is instead provided directly to the stimulator unit 180. The stimulator unit 180 is configured to use the stimulation control signal 196 to generate an electrical stimulation signal, which is delivered to the recipient's cochlea via one or more stimulation channels including a lead region 116 and a stimulation assembly 118 having an electrode array 128.

[0048] In addition to the sound processing operations, as further described below, the environment classifier 191 is configured to determine the environment classification of the sound environment associated with the input audio signal, and the individualized self-speech detector 195 is configured to perform individualized self-speech detection (OVD).

[0049] The stimulatory component of a cochlear implant can be inserted into the recipient's cochlea during cochlear implant surgery. In rare cases, the electrode array of the cochlear implant may become bent, folded, or twisted during or after the cochlear implant surgery. The cochlear implant can be measured after surgery to generate a transimpedance matrix (TIM). A transimpedance matrix (TIM) measurement is a measurement of the propagation of the electric field within the recipient's cochlea. Features of the TIM can be used to estimate the placement of the cochlear implant's electrode array within the recipient or whether the electrode array has become bent, twisted, or folded. Transimpedance matrices (TIMs) generated from the inner ears of many cochlear implant recipients can be used to train machine learning (ML) algorithms. The trained ML algorithm can then be used to determine whether the electrode array of a cochlear implant already implanted in a recipient has become bent, folded, or twisted, using the TIM generated from the electrode array. In some cases, the amount of training data containing TIMs generated from electrode arrays implanted in recipients is insufficient to train an ML algorithm to correctly identify each instance of a bent, folded, or twisted electrode array.

[0050] According to some embodiments disclosed herein, a model is generated that describes the training data used in developing algorithms, such as machine learning (ML) algorithms, or another type of algorithm. This model can be used to identify the closeness of clinical input data to training data, and therefore can be used to define the confidence level of the algorithm's output. This model allows for the development of algorithms using a reduced training dataset when collecting a large amount of representative data for a specific output (such as detecting rare events in a dataset) is not feasible. As a specific example, not intended to be limiting, clinical input data may include TIM generated from measuring electrode arrays from many cochlear implant recipients, and the ML algorithm can be trained to detect electrode arrays in cochlear implants that have been bent, folded, or twisted. According to other examples, not intended to be limiting, clinical input data may include data generated from measuring a specific type of implantable medical device, and the training data can be used to train the ML algorithm to detect features of measurements obtained using the same type of implantable medical device. For example, measurements obtained using an implantable medical device include electrophysiological and / or tissue-related responses to stimulation, such as action potentials and / or impedance measurements. This model may optionally be combined with preprocessing, intermediate algorithmic products, or algorithmic outputs to enhance the confidence definition.

[0051] In some embodiments, a quality assessment is generated that provides a high-quality rating for the algorithm's output when it is correct and a low-quality rating when it is incorrect. A low-quality rating is provided when the probability of the algorithm outputting incorrect information is high. For example, the quality assessment could be a probability rating or a rating based on a log-likelihood function. The quality assessment can be used to determine whether the algorithm's output is trustworthy or untrustworthy. When the algorithm is processing edge cases in the input data that would cause it to generate an incorrect assessment, the quality assessment generates indications for further evaluation.

[0052] Quality assessment may include a training phase and a clinical application phase. During the training phase, a model is built for the training dataset, and decision metrics (e.g., including thresholds and functions) are defined to generate trustworthy outputs. According to some embodiments, a method may be performed during the training phase. This method includes generating a training embedding model representing the training data used to develop the algorithm. The method also includes generating a decision metric used to determine whether the similarity between the algorithm's input data and the training data is sufficient for the algorithm to generate valid outputs, based on patterns in the interpretive input data described by the training embedding model. According to other embodiments, the method may include generating a decision metric to indicate a representation of the relationship between the algorithm's input data and the training data used to develop the algorithm, for estimating the validity of the outputs generated by the algorithm based on patterns in the interpretive input data described by the model and the algorithm's outputs.

[0053] After generating the training embedding model and decision metric, during the clinical application phase, the training embedding model and decision metric are applied to the input data (e.g., data including clinical measurement vectors) to provide a confidence assessment of the quality of the algorithm's output for evaluating the credibility of the algorithm's output. During the training and clinical application phases, the algorithm can be treated as a black box. As an example, a method can be implemented during the clinical application phase that includes generating an assessment of the relationship between the algorithm's input data and the training data used to train the algorithm using the training embedding model. The training embedding model includes features of the training data. The method also includes determining the similarity between the input data and the training data for evaluating the credibility of the algorithm's output using the decision metric processed in the assessment.

[0054] For example, two configurations of quality assessment can be used during the training and clinical application phases. In the first configuration, direct training data assessment is performed. During direct training data assessment, the algorithm paired with the quality assessment is not used during the development of the quality assessment. This quality assessment can determine the similarity between the input data and the training data. In the second configuration, when the decision metric estimates the reliability of the algorithm output, the algorithm output is included as one of the inputs to the model embedded in the training data.

[0055] Figure 2A It is a diagram depicting an example of a training configuration for a system that can generate training embedding models and decision metrics during the training phase. Figure 2A The system includes an optional enhancement level 202, an optional preprocessing level 203, a training embedding model 204, and a decision metric 205. Figure 2A The training dataset 201, switch 207, and trust value 206 generated by decision metric 206 are also shown.

[0056] Figure 2A The system can be in Figure 2A The various training configurations are graphically illustrated using switch 207. Switch 207 can be adjusted to select one of the training configurations. Any of these training configurations can be used during the training phase.

[0057] According to some examples, augmentation stage 202 can optionally be used in various training configurations to expand training dataset 201 using one or more data augmentation transformations. In these examples, switch 207 is adjusted to prevent training dataset 201 from being directly fed to training embedding model 204. Alternatively, augmentation stage 202 performs data augmentation on training dataset 201 to generate augmented training dataset. In these examples, augmentation stage 202 is used to expand training dataset 201 used to develop training embedding model 204 beyond the training dataset 201 used in developing pre-trained algorithms. Switch 207 can optionally be adjusted according to various training configurations to feed augmented training dataset to preprocessing stage 203 or directly to training embedding model 204.

[0058] An example of a data augmentation transformation that can be performed at augmentation level 202 involves adding zero-mean Gaussian white noise to the measurements in training dataset 201. Another example of a data augmentation transformation that can be performed at augmentation level 202 involves requantizing the measurements in training dataset 201 with a variable bit depth. Yet another example of a data augmentation transformation that can be performed at augmentation level 202 involves selectively contaminating or removing a portion of the measurement data in training dataset 201. Still another example of a data augmentation transformation that can be performed at augmentation level 202 involves adding scaling to the measurements to account for gain inaccuracies in training dataset 201.

[0059] exist Figure 2A In this embodiment, the data augmentation at augmentation level 202 is an optional procedure because some forms of data augmentation (e.g., adding white noise to the training dataset 201 when using principal component analysis embedding techniques) may not affect the derived principal components of the training dataset 201. If the training dataset 201 is augmented at augmentation level 202, both the original training dataset 201 and the augmented training dataset are used to generate a training embedding model 204, thereby increasing the size of the training set and improving the generalization ability of the embeddings in the training embedding model 204.

[0060] According to other examples, preprocessing stage 203 can be optionally used in various training configurations to transform and augment the training dataset in order to reduce the dimensionality of the data, mitigate the effects of measurement noise in the data, select benchmark information in the data, or transform the data into a more meaningful representation (e.g., in the frequency domain). In these examples, switch 207 can be adjusted to prevent the augmented training dataset generated at augmentation stage 202 from being directly provided to the training embedding model 204. Alternatively, preprocessing stage 203 uses preprocessing techniques to transform the augmented training dataset to generate transformed training data provided to the training embedding model 204.

[0061] Examples of preprocessing techniques that can be performed by preprocessing stage 203 on the augmented training dataset to generate transformed training data include: smoothing or filtering the augmented training dataset, aggregating the currently augmented training data with previously augmented training data or output states, or normalizing the augmented training dataset to a fixed dynamic range. Other examples of preprocessing techniques that can be performed by preprocessing stage 203 include: performing Fourier transforms, performing phase angle estimation, performing eigenvalue decomposition, generating principal components or independent components, or performing interpolation of missing data using the augmented training dataset to generate transformed training data. For example, principal component analysis (PCA) can be used to generate weights that are applied to the augmented training dataset to generate transformed training data.

[0062] According to the control of switch 207 Figure 2A The training configuration provides transformed training data, augmented training datasets, and / or the original training dataset 201 to the training embedding model 204. It can also be set based on whether augmentation level 202 and preprocessing level 203 are enabled or disabled. Figure 2A The training configuration.

[0063] Then according to Figure 2A The training configuration is configured such that the training embedding model 204 is generated by embedding transformed training data, augmented training datasets, and / or the original training dataset 201 into the training embedding model 204. The training embedding model 204 represents the training dataset 201 used to develop the pre-trained algorithm. The training embedding model 204 is designed to provide a co-entropy estimate between the input dataset (e.g., clinical measurement data) and the training dataset 201 used to develop the pre-trained algorithm.

[0064] Embedding data into the training embedding model 204 is performed by a training embedding algorithm. Examples of training embedding algorithms include principal component analysis (PCA), Huffman coding of the input vectors, compression of the input data using a pre-trained dictionary, probability density function (PDF) of the baseline features, and the K-nearest neighbor algorithm. The K-nearest neighbor algorithm can be used to select the K most similar measurement vectors from the transformed training data, the augmented training dataset, and / or the original training dataset 201.

[0065] The output of the trained embedding model 204 is fed into the decision metric 205. Figure 2AIn the training configuration, a decision metric 205 is generated using the trained embedding model 204. Examples of decision metric 205 include expert knowledge of the effective input range or form, correlation coefficients, residual magnitudes, the compressed size of the input vector (i.e., when using a predefined compressed dictionary), the discriminator neural network, the logistic regression of the output of the pre-trained algorithm relative to the trained embedding or decision metric, and the product of PDF baseline features. The output of decision metric 205 can be a single or multiple values ​​that indicate the trust metric 206 (also referred to as the trust output) of the output of the pre-trained algorithm.

[0066] Figure 2B This is a diagram depicting an example of a clinical configuration of a system that can be used to generate quality assessments of the results of an algorithm during the clinical application phase. Figure 2B The system includes an optional preprocessing stage 203, a trained embedding model 204, a decision metric 205, and a pre-trained target algorithm 213. The pre-trained target algorithm 213 has been previously trained using the entire training dataset 201. The pre-trained target algorithm 213 can be a machine learning (ML) algorithm, an expert system, or any other type of trainable algorithm.

[0067] Figure 2B Clinical dataset 211, switch 212, and trust output 206 generated by decision metric 205 are also shown. Figure 2B The system can be in Figure 2B The different clinical configurations are illustrated graphically via switch 212. Switch 212 can be adjusted to select one of the clinical configurations. Any of these clinical configurations can be used during the clinical application phase.

[0068] exist Figure 2B In the example, clinical dataset 211 is provided to preprocessing stage 203. Preprocessing stage 203 uses preprocessing techniques to transform clinical dataset 211 to generate transformed data to be provided to training embedding model 204. During preprocessing stage 203, the techniques described above can be used... Figure 2A Discuss any preprocessing techniques used to generate transformed data. Figure 2B In the example, the pre-trained target algorithm 213 generates a result 214 (also referred to as the output) based on the state of the switch 212, the clinical dataset 211, and / or the transformed data generated by the preprocessing stage 203.

[0069] Figure 2B The training embedding model 204 in the above text can be used to train the embedding model. Figure 2AThe training embedding algorithm mentioned above generates the similarity estimate between the clinical dataset 211 and the training dataset 201. As described above, the training dataset 201 is used to develop the pre-trained target algorithm 213 and is embedded into the training embedding model 204 during the training phase. The training embedding model 204 then outputs the similarity estimate between the clinical dataset 211 and the training dataset 201 to the decision metric 205.

[0070] exist Figure 2B In the example, decision metric 205 is a representation of the relationship between clinical dataset 211 and training dataset 201, used to determine whether clinical dataset 211 and training dataset 201 are sufficiently similar for algorithm 213 to produce a valid result 214. Decision metric 205 uses an estimate of the similarity between clinical dataset 211 and training dataset 201 generated by training embedding model 204 to generate a trust output 206. The trust output 206 generated by decision metric 205 indicates whether clinical dataset 211 and training dataset 201 are sufficiently similar for the target algorithm 213 to produce a valid result 214. Decision metric 205 generates trust output 206 based on the estimate of the similarity between clinical dataset 211 and training dataset 201 generated by training embedding model 204. Trust output 206 is a quality assessment of the result 214 of algorithm 213, indicating whether result 214 is correct or incorrect. Decision metric 205 may include the above-mentioned... Figure 2A Any decision metric discussed.

[0071] Figure 3A This is a diagram depicting an example of a training configuration for a system that uses the results of a pre-trained objective algorithm to generate a trained embedding model and decision metrics during the training phase. Figure 3A The system includes an optional enhancement stage 202, an optional preprocessing stage 203, a trained embedding model 204, a decision metric 205, and a pre-trained target algorithm 213. Enhancement stage 202, preprocessing stage 203, and trained embedding model 204 are as described above regarding... Figure 2A It works as described. Figure 3A The training dataset 201, switches 207 and 310, the trust output 206 generated by the decision metric 205, and the result 214 of the pre-trained target algorithm 213 are also shown. The pre-trained target algorithm 213 was previously trained using the entire training dataset 201.

[0072] Figure 3A The system can be in Figure 3A The various training configurations used are graphically illustrated via switches 207 and 310. (The above text is about...) Figure 2ADifferent training configurations provided by adjusting switch 207 are discussed. Switch 310 can also be adjusted according to additional training configurations to provide the input of the pre-trained target algorithm 213 with either an augmented training dataset generated by augmentation stage 202 or transformed training data generated by preprocessing stage 203. Any one or more of these training configurations can be used during the training phase.

[0073] The pre-trained target algorithm 213 then processes the augmented training dataset generated by augmentation stage 202 or the transformed training data generated by preprocessing stage 203 to produce result 214. Result 214 is then provided to the input of decision metric 205 via path 311. Figure 3A In the training configuration, the decision metric 205 is generated using the trained embedding model 204 and the result 214. Therefore, in Figure 3A In each training configuration, the result 214 of algorithm 213 is used to train decision metric 205 to generate trust metric 206. In some embodiments, algorithm 213 receives continuous input training data before generating classification result 214. (The above refers to...) Figure 2A An example of decision metric 205 has been published.

[0074] Figure 3B This is a diagram depicting an example of a clinical configuration of a system that can be used to generate quality assessments of the algorithm's results during the clinical application phase. Figure 3B The system includes an optional preprocessing stage 203, a trained embedding model 204, a decision metric 205, and a pre-trained target algorithm 213. The preprocessing stage 203 and the trained embedding model 204 are as described above regarding... Figure 2B It works as described.

[0075] Figure 3B The clinical dataset 211, switch 212, trust output 206 generated by decision metric 205, and result 214 of algorithm 213 are also shown. Figure 2B As in the example, it can be adjusted Figure 3B The system uses switch 212 to select one of two clinical configurations. Either of these clinical configurations can be used during the clinical application phase. For example... Figure 2B As in the example, Figure 3B The pre-trained target algorithm 213 in the system generates result 214 based on the state of switch 212, clinical dataset 211 and / or transformed data generated by preprocessing stage 203.

[0076] The result 214 is provided to the input of decision metric 205 via path 311. As described above, decision metric 205 is a representation of the relationship between clinical dataset 211 and training dataset 201. Figure 3BIn the example, decision metric 205 compares clinical dataset 211 with training dataset 201 indicated by training embedding model 204 and evaluates the result 214 of the target algorithm 213 to provide an estimate of the validity of result 214. For example, decision metric 205 can be used to estimate whether result 214 is valid or trustworthy based on the algorithm's result 214 and the degree to which patterns in training dataset 201 identified in training embedding model 204 interpret clinical input data 211. Decision metric 205 uses the estimate of the validity of result 214 to generate a trust output 206. As a result, trust output 206 is a quality assessment of the result 214 of algorithm 213, indicating the credibility of result 214. Decision metric 205 may include the above-mentioned... Figure 2A Any decision metric discussed.

[0077] Figure 4A This is a diagram illustrating an example training configuration of a system that can generate principal component analysis transformations and residuals during the training phase to evaluate the reliability of the algorithm's results. Figure 4A The system includes a principal component extraction level 402, a comparison of explained variances 403, a principal component weighting level 404, a principal component analysis (PCA) transformation level 405, an inverse transformation level 407, a residual level 406, and a confidence estimator 445. Figure 4A Training dataset 401 and target algorithm 410 are also shown. Target algorithm 410 can be a machine learning (ML) algorithm that has been pre-trained using training dataset 401 or any other type of algorithm.

[0078] A training dataset 401 is provided to the target algorithm 410. The target algorithm 410 uses the training dataset 401 to generate results. Figure 4B In the figure 420 shown, examples of the results of algorithm 410 are illustrated as positive and negative. These examples are not intended to be limiting. As a more specific example, which is also not intended to be limiting, algorithm 410 may generate a positive in figure 420 based on the corresponding TIM in training dataset 401 to indicate a cochlear implant electrode array that has been bent or folded, and algorithm 410 may generate a negative in figure 420 based on the corresponding TIM in training dataset 401 to indicate a cochlear implant electrode array that has not been bent or folded.

[0079] Training dataset 401 is also provided to PCA transform stage 405, residual stage 406, and principal component extraction stage 402. Principal component extraction stage 402 extracts principal components from training dataset 401 using the principal component analysis (PCA) algorithm. The principal components extracted from training dataset 401 at stage 402 correspond to vectors in training dataset 401 that explain most of the variance in training dataset 401.

[0080] Then, at comparison 403, the principal components extracted from training dataset 401 are compared to determine whether the explained variance is greater than the percentage k%. If the explained variance is not greater than k% at comparison 403, additional principal components are extracted from training dataset 401 at level 402 to increase the number of extracted principal components, and the additional principal components extracted at level 402 are compared again at comparison 403.

[0081] If the explained variance at comparison 403 is greater than k%, then principal component weights are generated at principal component weight level 404 for the principal components extracted from training dataset 401 at level 402. The principal component weights generated at level 404 are vector descriptions of the variance explained in training dataset 401. These principal component weights generated at level 404 are then fed into PCA transformation level 405.

[0082] PCA transform stage 405 then multiplies each principal component weight received from stage 404 by the corresponding value in training dataset 401, and then sums the results of these multiplications together to generate a reconstruction. As an example, PCA transform stage 405 can multiply each principal component by the gain used for the corresponding TIM in training dataset 401 to generate a result, and then sum all the results together to generate a reconstruction.

[0083] The reconstruction generated by PCA transform stage 405 is provided to inverse transform stage 407. Inverse transform stage 407 then performs an inverse transform on the reconstruction generated by PCA transform stage 405 to generate an inverse transform reconstruction, which is provided to residual stage 406. Residual stage 406 subtracts the inverse transform reconstruction received from inverse transform stage 407 from the corresponding values ​​in training dataset 401 to generate residuals. In this embodiment, PCA transform stage 405, inverse transform stage 407, and / or residual stage 406 with principal component weights operate as training embedding model 204.

[0084] Trust estimator 445 then generates a trust value 446 for the results of the target algorithm 410 based on the values ​​of the residuals generated by residual stage 406 and the results of the target algorithm 410. Trust value 446 indicates the credibility of the corresponding results of the target algorithm 410. Therefore, trust estimator 445 uses the results of algorithm 410 and the residuals to generate a trust value 446 indicating the reliability of the results of algorithm 410. Trust value 446 can indicate which results of algorithm 410 deviate significantly from the expected results, and which results of algorithm 410 are relevant to the expected results. In this embodiment, trust estimator 445 functions as decision metric 205.

[0085] Figure 4BThe diagram 420 also shows an example of a trust value 446. The trust value 446 can be either true or false. True values ​​are shown in the left half of diagram 420, and false values ​​are shown in the right half. Figure 4B In the example, false values ​​are most likely to be generated in the middle range of the residuals, which is around 0.4.

[0086] As an example, the corresponding negative result of the true negative indicator algorithm 410 in Figure 420 may be correct. As another example, the corresponding positive result of the true positive indicator algorithm 410 in Figure 420 may be correct. As yet another example, the corresponding positive result of the false positive indicator algorithm 410 in Figure 420 may be incorrect.

[0087] Figure 4C Figures 421-422 show examples of probability curves, which are examples of trust values ​​446 generated by trust estimator 445. Figure 421 shows an example of the correct probability of a positive result (i.e., the trust level of a positive result) generated by algorithm 410 based on the residuals. Figure 422 shows an example of the correct probability of a negative result (i.e., the trust level of a negative result) generated by algorithm 410 based on the residuals.

[0088] Figure 4D This is a diagram depicting an example of a clinical configuration of a system that can be used to generate quality assessments of the results of the algorithm using PCA transformation during the clinical application phase. Figure 4D The system includes a principal component analysis (PCA) transformation stage 405, an inverse transformation stage 407, a residual stage 406, and a confidence estimator 445. Figure 4D The diagram also shows a clinical dataset 450, a target algorithm 410, a result 451 from the target algorithm 410, and a trust value 452. The target algorithm 410 generates result 451 by processing the clinical dataset 450. Result 451 indicates a positive or negative result.

[0089] exist Figure 4D In the clinical configuration, PCA transform stage 405 multiplies each principal component weight by the corresponding value in the clinical dataset 450, and then sums the results of these multiplications to generate a reconstruction. The reconstruction generated by PCA transform stage 405 is provided to inverse transform stage 407. Inverse transform stage 407 then performs an inverse transform on the reconstruction generated by PCA transform stage 405 to generate an inverse transformed reconstruction, which is provided to residual stage 406. Residual stage 406 subtracts the inverse transformed reconstruction from the corresponding value in the clinical dataset 450 to generate residuals. For example, the residuals may correspond to the difference between the observed value and the estimated value in the clinical dataset 450.

[0090] The result 451 of algorithm 410 and the residual generated at residual stage 406 are then passed to the trust estimator 445. The trust estimator 445 will then target... Figure 4A The probability curve of the trust value generated in the training configuration and in Figure 4D In the clinical configuration, residuals generated for each result 451 are compared at residual level 406 to generate a confidence value 452 for the residuals. Each confidence value 452 for the residuals indicates the confidence level of the corresponding result 451 of the target algorithm 410. The confidence value 452 can be, for example, a true value or a false value. The confidence value 452 indicates whether the clinical dataset 450 is sufficiently similar to the training dataset 401 for the target algorithm 410 to produce a valid result 451. Figure 4A Trust values ​​generated during the training phase and in Figure 4C The probability curves shown in Figures 421-422 illustrate examples that can be used by the trust estimator 445.

[0091] Figure 5A This is a diagram illustrating an example training configuration of a system that can train the K-nearest neighbor algorithm during the training phase to evaluate the reliability of the results of the target algorithm. Figure 5A The system includes the K nearest neighbor level 503 and the error and correlation levels 502. Figure 5A Also shown are augmented training dataset 501, training dataset 504 excluding input cardinality, and graphs 511-512. Augmented training dataset 501 can be generated by augmentation level 202 using data augmentation of the original training dataset, as described in this paper. Figure 2A What has been made public.

[0092] Augmented training datasets 501 and 504 are provided as input to the K-nearest neighbor stage 503. The K-nearest neighbor stage 503 compares each item in training datasets 501 and 504 with each other item in training datasets 501 and 504. For each item in training datasets 501 and 504, the K-nearest neighbor stage 503 performs the K-nearest neighbor algorithm relative to each other item in training datasets 501 and 504 to identify the error and correlation for that item. The K-nearest neighbor stage 503 performs the K-nearest neighbor algorithm relative to each other item in training datasets 501 and 504 to generate the error and correlation.

[0093] At error and correlation level 502, a probability distribution function (PDF) for the error and correlation of each item in the training dataset, generated in the K-nearest neighbor level 503, is created. As an example, the error can correspond to the residual, indicating the difference between the observed value and the estimated or expected value in the training dataset. The correlation indicates the degree of correlation of each item with the training dataset. As an example, the correlation could be a Pearson correlation. Figures 5A-5BIn the example, the results of the target algorithm were not used during the development error and correlation period.

[0094] In some exemplary embodiments, training datasets 501 and 504 may include training matrices of cochlear implant electrode arrays (e.g., TIMs for determining cochlear axis proximity) already implanted in a recipient. Based on an example where the training dataset includes training matrices, graphs 511 and 512 are plots generated by error and correlation levels 502. Graph 511 shows an example of the number of training matrices with various numbers of errors, and graph 512 shows an example of the number of training matrices with various correlation values ​​(shown as absolute values ​​of the correlations). In the example shown in graph 511, all errors are less than the noise floor. In the example shown in graph 512, all correlations are less than a threshold.

[0095] Figure 5B This is a diagram depicting an example of a clinical configuration of a system that can be used to generate quality assessments of the results of a target algorithm using the K-nearest neighbor algorithm during the clinical application phase. Figure 5B The system includes a K-nearest neighbor level 503, an error and correlation level 502, and a trust estimator 545. Figure 5B The clinical dataset 521, training dataset 522, target algorithm 531, results of target algorithm 531 532, trust value 523, and graphs 521-522 are also shown.

[0096] The target algorithm 531 can be a pre-trained machine learning (ML) algorithm or any other type of algorithm. A clinical dataset 521 is provided as input to the target algorithm 531. The target algorithm 531 generates a result 532 by processing the clinical dataset 521.

[0097] Clinical dataset 521 and training dataset 522 are provided to a K-nearest neighbor hierarchy 503. Training dataset 522 may be included in... Figure 5A The training configuration uses training datasets 501 and 504, or may only include the original training dataset. For each item in the clinical dataset 521, the K-nearest neighbor level 503 performs the K-nearest neighbor algorithm on each other item in the clinical dataset 521 to identify the error and correlation for that item in the clinical dataset 521. The error and correlation level 502 creates the probability distribution function (PDF) of the error and correlation for each item in the clinical dataset 521 generated in the K-nearest neighbor level 503. As an example, the correlation may be based on the degree to which the clinical dataset 521 can be compressed to match a compressed version of the training dataset 522. The K-nearest neighbor level 503 and / or the error and correlation level 502 function as training embedding model 204 in this embodiment.

[0098] Trust estimator 545 compares the error and correlation PDFs for each item in clinical dataset 521 with the error and correlations in training dataset 522 to generate trust values ​​523 that indicate the credibility of the results 532 of algorithm 531. Trust values ​​523 are generated based on the similarity between the error and correlations in clinical dataset 521 and the error and correlations in training dataset 522. Each trust value 523 indicates whether one or more items in clinical dataset 521 are sufficiently similar to those in training dataset 522 for the target algorithm 531 to produce valid results 532. Trust estimator 545 functions as decision metric 205 in this embodiment. Figure 5B Figures 511 and 512 are also shown as examples of errors and related factors.

[0099] Figure 6A This is a diagram illustrating an example of a training configuration for a system that generates a model using the statistical properties of preprocessed training data during the training phase. Figure 6A The system includes a preprocessing and feature selection stage 603, a model 604, and a trust estimator 616. Figure 6A The training dataset 601-602 is also shown. Figure 6B A graphical representation of model 604, including preprocessing and statistical properties of selected features, is shown.

[0100] exist Figure 6A During the training phase, training datasets 601 and 602 are provided to the preprocessing and feature selection stage 603. The preprocessing and feature selection stage 603 preprocesses the signals (e.g., complex signals) in the training datasets 601-602 and then selects benchmark features from the preprocessed signals as features for the preprocessed training data. Examples of preprocessing that can be performed by the preprocessing and feature selection stage 603 on the training datasets 601-602 to generate preprocessed signals include estimating the noise, phase angle, amplitude, frequency components, covariates, and / or autocorrelation of one or more signals in the training datasets.

[0101] The preprocessing and feature selection stage 603 can use expert knowledge of the feature space of the preprocessed training data features, combined with statistical analysis of the preprocessed training data features, to define a feasible expected range for the preprocessed training data features. The preprocessing and feature selection stage 603 then adds the preprocessed training data features (e.g., within the feasible expected range) to model 604, which in this embodiment operates as the training embedding model 204. Expert knowledge about the training dataset can also be added to model 604 as additional features. The preprocessing and feature selection stage 603 can also compare the preprocessed training data features and then add the interactions between the preprocessed training data features generated from the comparison to model 604. Figure 6BA graphical representation of an example of model 604 is shown, which includes the statistical properties of the features of the preprocessed training data generated by the preprocessing and feature selection stage 603.

[0102] The preprocessed training data features selected by the preprocessing and feature selection stage 603, along with any additional features (e.g., from expert knowledge) and the interactions between these features, are compared with the probability quality function of each of these features and combined in model 604. Trust estimator 616 then generates an estimate of the probability that this combined feature set appears in training dataset 601-602. The probability of occurrence of all these features in training dataset 601-602 can be combined in model 604 and estimated by trust estimator 616, for example, via a Bayesian estimator.

[0103] The trust estimator 616 can assign a trust-equal value to each feature in model 604. The trust-equal value is based on a probability quality function or distribution of the feature in the training datasets 601-602, indicating the probability of that feature occurring. Figures 6A-6C In the example, the results of the target algorithm were not used during the development of model 604.

[0104] The rarer features in training datasets 601-602 have a relatively small impact on the training of the algorithms used in preprocessing and feature selection stage 603. Therefore, any determinations made by the trust estimator 616 incorporating rare features from training datasets 601-602 have low trust in the algorithmic evaluation of these features. When generating model 604, these statistically derived features can be augmented by expert-defined interaction probabilities or value limits.

[0105] As a specific example not intended to be limiting, training datasets 601-602 may include biosignals (e.g., auditory signals) from recipients of medical devices such as cochlear implants. For example, biosignals in training datasets 601-602 may be generated in response to external stimuli. Figure 6B The graphical representation of model 604 shown is a histogram, which illustrates examples of four features from the training dataset along the diagonal of the graph, with the comparison results of each of these four features with each of the other four features arranged in the corresponding rows and columns of the histogram. Therefore, model 604 can include features from the training dataset, as well as how these features interact with each other.

[0106] Figure 6C This is a diagram illustrating a clinical configuration example of a system that can be used to generate a quality assessment of the results of a target algorithm using the statistical properties of preprocessed data during the clinical application phase. Figure 6C The system includes a preprocessing and feature selection stage 603, a model 604, an objective algorithm 614, and a trust estimator 616. Figure 6C The results 615 of the clinical datasets 611-612, the target algorithm 614, and one or more trust values ​​617 are also shown. The target algorithm 614 can be a pre-trained ML algorithm or any other type of algorithm.

[0107] exist Figure 6C During the clinical application phase, clinical datasets 611-612 are provided to the preprocessing and feature selection stage 603. The preprocessing and feature selection stage 603 preprocesses the signals in the clinical datasets 611-612 and then selects benchmark features from the preprocessed signals as preprocessed clinical data features. The preprocessing and feature selection stage 603 can perform preprocessing operations on the signals in the clinical datasets 611-612, such as estimating the noise, phase angle, amplitude, and frequency components of one or more signals.

[0108] The preprocessing and feature selection stage 603 can use expert knowledge of the feature space of the preprocessed clinical data features, combined with statistical analysis of the preprocessed clinical data features, to define a feasible expected range for the preprocessed clinical data features. The preprocessed clinical data features within the feasible expected range are then provided to the target algorithm 614 and the trust estimator 616. The target algorithm 614 then processes the preprocessed clinical data features within the feasible expected range to generate result 615.

[0109] Trust estimator 616 is based on Figure 6A Features generated in model 604 during the training phase are assigned trust values ​​617 to preprocessed clinical data features received from stage 603. Trust estimator 616 assigns trust values ​​617 to preprocessed clinical data features based on estimates of the probability of occurrence of the preprocessed clinical data feature set, inferred from the probability mass function, distribution, covariance, and / or joint probability mass function of the features in training datasets 601-602 in model 604. As a result, the trust value 617 for the preprocessed clinical data features indicates the similarity between these features and features from the training dataset represented in model 604. As a specific example, trust estimator 616 can compare the histograms of preprocessed clinical data features with those of features in model 604 to generate corresponding trust values ​​617 for the preprocessed clinical data features.

[0110] Figure 7A This is a diagram illustrating an example training configuration of a system that can train a generative adversarial network during the training phase to evaluate the credibility of the results of a target algorithm. Figure 7A The system includes an input selector 702, a discriminator network 703, and a generative network 706. Figure 7AThe augmented training dataset 701, discriminator loss 704, generator loss 705, and trust value 707 are also shown. The augmented training dataset 701 can be generated by augmentation level 202 using data augmentation of the original training dataset, as described in this paper. Figure 2A What has been made public.

[0111] Figure 7A The system has a generative adversarial network comprising two neural networks. The first of these two neural networks is a generative network 706. The generative network 706 generates data that matches the real training data as closely as possible without accessing the training data. The second of these two neural networks is a discriminator network 703. The discriminator network 703 distinguishes between the real training data and the data generated by the generative network 706. Figure 7A The goal of the generative adversarial network is to enable the generative network 706 to produce outputs that are indistinguishable from the training data, and to enable the discriminator network 703 to correctly identify the valid training data.

[0112] Generative network 706, in response to random noise and generator loss 705, generates data that most closely matches (i.e., indistinguishable) the training data in augmented training dataset 701 without accessing the training data. The data generated by generative network 706 is provided to input selector 702. Augmented training dataset 701 is also provided to input selector 702. Input selector 702 provides the augmented training dataset 701 and / or the data generated by generative network 706 to discriminator network 703.

[0113] Discriminator network 703, in response to discriminator loss 704, distinguishes between augmented training dataset 701 and data generated by generative network 706. The output of discriminator network 703 indicates the difference between augmented training dataset 701 and data generated by generative network 706. Discriminator network 703 generates discriminator loss 704 and generator loss 705. Discriminator network 703 also generates a confidence value 707, which indicates the probability that the difference identified by discriminator network 703 is real.

[0114] The generator loss 705 indicates the error in the output of the generative network 706. The generative network 706 uses the generator loss 705 during backpropagation to... Figure 7A In each iteration of the training phase, the weights in the nodes of the neural network are adjusted to reduce error. The discriminator loss 704 indicates the error in the output of the discriminator network 703. The discriminator network 703 uses the discriminator loss 704 in backpropagation to... Figure 7A In each iteration of the training phase, the weights in the nodes of the neural network are adjusted to reduce error. After a sufficient number of iterations in the training phase, the training phase is complete, and the discriminator network 703 can be used in clinical applications, as illustrated in this paper for example. Figure 7B What has been made public.

[0115] Figure 7B This is a diagram depicting an example of a clinical configuration of a system that can be used to generate quality assessments of the results of a target algorithm using a discriminator network during the clinical application phase. Figure 7B The system includes a discriminator network 703 and a target algorithm 724. Figure 7B The diagram also shows a clinical dataset 721, the result 725 of the target algorithm 724, and one or more trust values ​​723. The target algorithm 724 can be a pre-trained ML algorithm or any other type of algorithm. The target algorithm 724 processes the clinical dataset 721 to generate the result 725.

[0116] exist Figure 7B In the embodiment, the discriminator network 703 has been... Figure 7A After training in the training phase, the discriminator network 703 was used to train the embedding model 204 and the decision metric 205. Figure 7B In the clinical application phase, a clinical dataset 721 is provided to the discriminator network 703. If the discriminator network 703 has already passed... Figure 7A If the training phase is successful, the discriminator network 703 is used together with the target algorithm 724 to generate a trust value 723, which is an estimate of the probability that the algorithm 724 has previously encountered data similar to the clinical dataset 721. In an alternative embodiment, the output of the discriminator network 703 can be provided as the input to the target algorithm 724.

[0117] As a specific example, not intended to be limiting, the target algorithm 724 can be trained to generate a result 725 indicating whether the TIM generated from the electrode array in the cochlear implant indicates bending or folding. In this example, in Figure 7A During the training phase, the discriminator network 703 is trained using TIMs generated from the electrode array of the cochlear implant. During the clinical application phase, the discriminator network 703 generates a trust value 723 based on whether the TIMs in the clinical dataset 721 are sufficiently similar to the TIMs in the training dataset. This trust value indicates whether the results 725 regarding the TIMs are trustworthy.

[0118] Figure 8 This is a diagram illustrating an example of a computing system 800 within which one or more of the disclosed embodiments may be implemented. For example, the computing system 800 may be used to generate control signals and transmit them to the present document. Figure 1A-7B One or more of the disclosed medical devices or systems provide control signals.

[0119] Computing systems, environments, or configurations suitable for use with the examples described herein include, but are not limited to, personal computers, server computers, handheld devices, laptop devices, multiprocessor systems, microprocessor-based systems, programmable consumer electronics (e.g., smartphones), network computers, minicomputers, mainframe computers, tablet computers, distributed computing environments including any of the aforementioned systems or devices, and so on. Computing system 800 may be a single virtual or physical device operating in a networked environment via a communication link to one or more remote devices. The remote device may be an auditory prosthesis (e.g., an auditory phallus). Figure 1A-1D Any device or system in any of the figures), personal computer, server, router, network personal computer, peer device or other public network node.

[0120] The computing system 800 includes at least one processing unit 802 and a memory 804. The processing unit 802 includes one or more hardware or software processors (e.g., a central processing unit) capable of receiving and executing instructions. The processing unit 802 can communicate with and control the performance of other components of the computing system 800. The memory 804 is one or more software-based or hardware-based computer-readable storage media operable to store information accessible by the processing unit 802.

[0121] Memory 804 may store instructions executable by processing unit 802 to implement an application or enable the operations described herein, and may also store other data. Memory 804 may be volatile memory (e.g., random access memory or RAM), non-volatile memory (e.g., read-only memory or ROM), or a combination thereof. Memory 804 may include temporary or non-temporary memory. Memory 804 may also include one or more removable or non-removable storage devices. In the example, memory 804 may include non-temporary computer-readable storage media such as RAM, ROM, EEPROM (electrically erasable programmable read-only memory), flash memory, optical disc storage devices, magnetic storage devices, solid-state storage devices, or any other memory medium that can be used to store information for later access. In the example, memory 804 encompasses modulated data signals (e.g., signals whose one or more characteristics are set or changed in a manner such as encoding information in the signal), such as carrier waves or other transmission mechanisms, and includes any information delivery medium. By way of example and not limitation, memory 804 may include wired media (such as a wired network or direct wired connection), and wireless media (such as acoustic, radio frequency, infrared and other wireless media) or combinations thereof.

[0122] In the example shown, system 800 also includes a network adapter 806, one or more input devices 808, and one or more output devices 810. System 800 may include other components such as a system bus, component interfaces, a graphics system, a power supply (e.g., a battery), and other components.

[0123] Network adapter 806 is a component of computing system 800 that provides network access to network 812. Network adapter 806 can provide wired or wireless network access and can support one or more of various communication technologies and protocols, such as Ethernet, cellular, Bluetooth, near-field communication, and RF (radio frequency), among others. Network adapter 806 may include one or more antennas and associated components configured for wireless communication according to one or more wireless communication technologies and protocols.

[0124] One or more input devices 808 are means by which the computing system 800 receives input from a user. One or more input devices 808 may include physically actuable user interface elements (e.g., buttons, switches, or dial pads), touchscreens, keyboards, mice, pens, and voice input devices, as well as other input devices.

[0125] One or more output devices 810 are means through which the computing system 800 can provide output to a user. Output devices 810 may include displays, speakers, printers, and other output devices.

[0126] Unless otherwise expressly indicated, any embodiment or feature disclosed herein may be combined with any one or more other embodiments and / or other features disclosed herein. The use of any embodiment or feature disclosed herein in conjunction with any one or more other embodiments and / or other features disclosed herein is expressly excluded unless otherwise expressly indicated. It should be noted that any method detailed herein also corresponds to the disclosure of one or more or all of the method actions of an apparatus, computer-readable storage medium, and / or system as detailed herein. It should also be noted that any disclosure of an apparatus, computer-readable storage medium, and / or system detailed herein corresponds to methods of making and / or using the apparatus, computer-readable storage medium, and / or system, including methods of using the apparatus, system, or computer-readable storage medium according to the functions detailed herein.

[0127] The foregoing description of exemplary embodiments of the invention has been presented for illustrative purposes. The foregoing description is not intended to be exhaustive or to limit the invention to the examples disclosed herein. In some instances, features of the invention may be used without correspondingly using other features set forth. Many modifications, substitutions, and variations are possible in accordance with the foregoing teachings without departing from the scope of the invention.

Claims

1. A method comprising: Generate a model representing the first data, which is used to train the algorithm; as well as A decision metric is generated, which is used to determine whether the similarity between the second data and the first data is sufficient for the algorithm to generate a valid output based on the pattern of the second data into the second data identified in the model and the interpretation of the first data.

2. The method of claim 1, wherein the algorithm is trained with the first data to detect features of measurements obtained using an implantable medical device to generate the output.

3. The method of claim 2, wherein the measurement results obtained using the implantable medical device include electrophysiological or tissue-related responses to stimulation.

4. The method according to any one of claims 1-3, further comprising: Using data augmentation to enhance the first data to generate third data, wherein generating the model further includes making the model represent the third data.

5. The method according to any one of claims 1-4, further comprising: The first data is preprocessed to generate preprocessed data by at least one of the following: smoothing or filtering the first data, aggregating the first data, normalizing the first data to a fixed dynamic range, reducing the dimensionality of the first data, mitigating the influence of measurement noise in the first data, selecting reference information in the first data, or transforming the first data into another representation, wherein generating the model further includes making the model represent the preprocessed data.

6. The method according to any one of claims 1-5, wherein generating the model representing the first data comprises: The model is generated by executing the K-nearest neighbor algorithm, which compares each item in the first data with each other item in the first data to identify errors and correlations.

7. The method according to any one of claims 1-6, wherein generating the model representing the first data comprises: Select features from the first data; The features are compared to identify interactions between them; as well as Add the interaction between the features to the model.

8. The method according to any one of claims 1-7, wherein generating the decision metric comprises: The decision metric is generated to produce a trust value, which indicates the probability of the feature appearing in the second data based on a probability quality function or distribution of the feature in the first data.

9. The method according to any one of claims 1-8, wherein generating the model representing the first data comprises: A discriminator network is trained to distinguish the first data from third data generated by a generative network, which matches the third data with the first data, wherein the generative network and the discriminator network are adversarial networks.

10. The method according to any one of claims 1-9, wherein generating the decision metric comprises: The decision metric indicates a representation of the relationship between the second data input to the algorithm and the first data, for use in estimating whether the output generated by the algorithm is trustworthy using the output of the algorithm.

11. The method according to any one of claims 1-10, wherein generating the model representing the first data comprises: Principal component analysis is used to generate weights for the principal components extracted from the first data, the weights being descriptions of vectors that explain the variance in the first data.

12. The method according to any one of claims 1-11, wherein the method is implemented by a computing system comprising at least one processing unit and a memory.

13. A computing system, comprising: One or more processing units generate a model, the model including information from first data used to develop the algorithm. The computing system generates a trust estimator that uses the model and the results of the algorithm generated based on the second data to provide a representation of the relationship between the first data and the second data, for estimating whether the results of the algorithm are trustworthy.

14. The computing system of claim 13, wherein the computing system causes the trust estimator to provide the representation of the relationship between the first data and the second data based on the pattern in the first data that interprets the second data.

15. The computing system according to any one of claims 13-14, wherein the one or more processing units generate the model based on principal component weights, the principal component weights being derived from the first data using principal component analysis and being a description of a vector explaining most of the variance in the first data.

16. The computing system of claim 15, wherein the one or more processing units generate the model by multiplying the principal component weights by values ​​in the first data to generate a result and then summing the results to generate a reconstruction.

17. The computing system of claim 16, wherein the one or more processing units generate the model by performing an inverse transformation on the reconstruction to generate an inverse transformation reconstruction and subtracting the inverse transformation reconstruction from the value in the first data to generate a residual, and wherein the trust estimator generates a trust value for the result of the algorithm based on the residual.

18. The computing system according to any one of claims 13-17, wherein the algorithm is a machine learning algorithm.

19. The computing system according to any one of claims 13-18, wherein the one or more processing units generate the model using a first measurement result from a first implantable medical device for developing the algorithm, and wherein the result of the algorithm is generated based on a second measurement result from a second implantable medical device.

20. A non-transitory computer-readable storage medium, the non-transitory computer-readable storage medium comprising computer-readable instructions stored thereon, the computer-readable instructions being configured to cause a computing system to: A quality assessment of the relationship between the first and second datasets is generated using a model that includes features from the first dataset; and The similarity between the first dataset and the second dataset is determined using a trust estimator that processes the quality assessment to evaluate the credibility of the results generated by the algorithm in response to the second dataset, wherein the algorithm was developed using the first dataset.

21. The non-transitory computer-readable storage medium of claim 20, wherein the computer-readable instructions further cause the computing system to: Enhance or transform the third dataset to generate the first dataset.

22. The non-transitory computer-readable storage medium according to any one of claims 20-21, wherein the computer-readable instructions further cause the computing system to: The trust estimator is used to compare the features of the second dataset with the probability density or quality function for the first dataset to calculate the probability of the features appearing in the second dataset.

23. The non-transitory computer-readable storage medium according to any one of claims 20-22, wherein the computer-readable instructions further cause the computing system to: By performing the K-nearest neighbor algorithm on the second dataset to identify errors and correlations in the second dataset, the model is used to generate the quality assessment of the relationship between the first dataset and the second dataset; Create the error and the associated probability distribution function in the second dataset; as well as The confidence estimator is used to compare the error and the associated probability distribution function with the first dataset to evaluate the confidence of the algorithm's results.

24. The non-transitory computer-readable storage medium according to any one of claims 20-23, wherein the computer-readable instructions further cause the computing system to: A quality assessment of the relationship between the first dataset and the second dataset is generated using a discriminator network, wherein the discriminator network is trained on the first dataset using a generative network; and The discriminator network used to process the quality assessment determines the similarity between the first dataset and the second dataset to evaluate the credibility of the algorithm's results.

25. The non-transitory computer-readable storage medium according to any one of claims 20-24, wherein the computer-readable instructions further cause the computing system to: The trust estimator performs a quality assessment of the relationship between the first dataset and the second dataset based on the patterns in the first dataset that interpret the second dataset.

26. The non-transitory computer-readable storage medium according to any one of claims 20-25, wherein the second dataset comprises at least one of electrical measurement results of an electrode array of a cochlear implant in the recipient's body or medical images.

27. A computer-implemented method for estimating the confidence of the output of an algorithm trained with training data, the computer-implemented method comprising: A model including a description of the training data is used to generate a representation of the relationship between the training data and the input data, wherein the algorithm uses the input data to generate the output; as well as Using a decision metric, a trust value is generated for the output of the algorithm, based on the output of the algorithm and the representation of the relationship between the training data and the input data.

28. The computer-implemented method of claim 27, wherein generating the representation of the relationship between the training data and the input data using the model further comprises: The weights of the principal components derived from the training data are multiplied by the values ​​in the input data to generate the result; The results are then combined to generate a reconstruction. as well as The inverse transformation of the reconstruction is subtracted from the value in the input data to generate a residual.

29. The computer-implemented method of claim 28, wherein generating the trust value for the output of the algorithm using the decision metric further comprises: The trust probability curve generated in the training configuration of the decision metric is compared with the residual to generate the trust value for the output.

30. The computer-implemented method according to any one of claims 27-29, wherein generating the trust value for the output of the algorithm using the decision metric further comprises: The trust value for the output of the algorithm is generated based on the patterns in the training data that interpret the input data.

31. The computer-implemented method according to any one of claims 27-30, wherein the input data comprises a transimpedance matrix generated from an electrode array of a cochlear implant implanted in a recipient.

32. The computer-implemented method according to any one of claims 27-31, wherein generating the representation of the relationship between the training data and the input data using the model comprises generating the representation of the relationship using the model including the description of the training data from the first implantable medical device, and wherein the algorithm generates the output based on measurements from the second implantable medical device.

Images