Intelligent Computer Aided Decision Support System

Abstract

The present invention relates to a method for assisting an inquiring party to decide a response action in response to an inquiry between the inquiring party and the inquired party. The method comprises: providing a processing unit and inputting the voice of the inquired party into the processing unit as an electronic signal; in parallel with the occurrence of the inquiry, processing the electronic signal by means of the processing unit. The method also comprises an abnormal routine, the abnormal routine contains a statistical learning model, and determines by means of the statistical learning model that the corresponding sample number of the sample sequence is an abnormal state of the statistical learning model and returns to the abnormal routine to be processed by the abnormal routine on the subsequent sample number of the sample sequence.

Description

Technical Field

[0001] The present specification relates to a method for assisting an interviewing party by a computer so that the interviewing party can decide a response action faster or more reliably. A statistical learning model is envisioned to assist the interviewing party. Background Art

[0002] The setting envisioned by this specification may be an emergency situation, such as a traffic accident, cardiac arrest, or a problem encountered by an airplane.

[0003] Another scenario contemplated by this specification may be a doctor's office visit, i.e., a doctor's visit or hospital rounds (visits by medical staff to patients in a hospital to make decisions about the patient's care), where a patient and doctor meet to make decisions about the patient's care.

[0004] The inventors recognize that, in the above two scenarios, how to assist the inquiring party to decide on a response action more quickly or more reliably is a technical problem that needs to be solved urgently. Summary of the invention

[0005] If the aircraft has a problem such as an engine failure, the cockpit communicates the problem to the nearby tower, and the tower may advise or instruct the cockpit on what actions to take - for example, whether the aircraft should return to the departure airport or make an immediate emergency landing if possible.

[0006] Typically, the communication takes the form of an inquiry between the pilot in the cockpit (the interviewee party) and the air traffic controller in the tower (the inquirer). The tower may request the cockpit's readings on cockpit instruments such as oil pressure. These questions may be defined by a protocol and arranged in a sequence that the tower follows to narrow the scope of the questions. During the communication or inquiry, the tower may arrive at one or more instructions for the cockpit to follow.

[0007] It is envisaged that computer systems could assist the tower so that the correct instructions could be sent to the cockpit more quickly or with a higher chance of being correct, ie, instructions that are appropriate to the problem encountered by the aircraft.

[0008] Communications between the cockpit and the tower are preferably recorded in real time and routed to a processing unit of a computer system where they are processed using statistical models to diagnose the problem and determine the most likely cause of action to respond to the threat to the safety and operation of the aircraft. The results are presented to the tower, which may or may not take into account the computer's recommendations.

[0009] If an emergency such as cardiac arrest occurs, a key component of providing appropriate care is summoning emergency medical services (usually an ambulance) by calling for help using the appropriate local emergency number (emergency call). Early recognition that cardiac arrest has occurred is key to survival - for every minute a patient is in cardiac arrest, their chances of survival decrease.

[0010] Emergency calls are received by dispatchers at public safety answering points (PSAPs), emergency call centers. Similar to an airplane encounter, the dispatcher will question or interrogate the caller to diagnose the emergency and determine if and what emergency medical services should be dispatched.

[0011] It is contemplated that a computer system similar to that described for aircraft emergencies can assist dispatchers in making response actions more quickly, i.e., communications between callers and dispatchers are recorded and routed to a processing unit of the computer system where they are processed using statistical models to diagnose the emergency and determine expected response actions. The expected response actions are presented to the dispatcher, who can follow the recommendations made by the computer system.

[0012] The response action may include which treatment is appropriate or dispatching emergency medical services such as an ambulance and / or may provide the caller with a set of pre-arrival instructions for the caller to perform pre-arrival emergency medical services, such as first aid if the caller is a bystander to an injured person with a medical emergency. Alternatively, the injured person is instructed to seek medical assistance at an emergency room or a local doctor's office. The response action may also be an instruction to the cockpit.

[0013] The above objects and advantages and many other objects and advantages which are obvious from the description of the invention are achieved according to the first aspect of the invention by:

[0014] A method for assisting an inquiring party to decide a response action in response to an inquiry between the inquiring party and an inquired party, the method comprising the following steps:

[0015] - providing a processing unit and inputting the voice of the queried party into the processing unit as an electronic signal,

[0016] - in parallel with the occurrence of said inquiry, processing of said electronic signal by means of said processing unit, said processing lasting for the duration of said inquiry or until said inquiry party has decided on said response action,

[0017] The processing includes:

[0018] separating the electronic signal into a sequence of samples in the time domain or in a domain representing the frequency content of the electronic signal,

[0019] The processing also includes:

[0020] a) an anomaly routine, the anomaly routine comprising a statistical learning model, and

[0021] The following are determined with the aid of the statistical learning model:

[0022] - the corresponding number of samples in the sample sequence is an outlier of the statistical learning model; or

[0023] - the corresponding number of samples is the normality of the statistical learning model,

[0024] b) when the corresponding sample number is determined to be an outlier of the statistical learning model:

[0025] - returning to the abnormal routine to process a subsequent number of samples of the sample sequence,

[0026] c) when the corresponding sample size is determined to be a normal state of the statistical learning model:

[0027] - providing an audio recognition model, and detecting audio features of the corresponding number of samples by means of the audio recognition model,

[0028] - providing an action prediction model and analyzing the audio features with the aid of the action prediction model to determine an expected response action,

[0029] - providing a display or a speaker, and presenting the expected response action to the inquiring party by means of the display or the speaker to assist the inquiring party,

[0030] - returning to the anamorphic routine to process a subsequent number of samples of the sequence of samples by the anamorphic routine.

[0031] The action prediction model or analysis model can be part of the action prediction routine and constitute a diagnostic routine for determining the response action.

[0032] The response action may also be determined with the aid of a lookup table that relates diagnostic actions to response actions.

[0033] The outcome of an action prediction routine (diagnosis) can be a response action, or whether the right questions were asked, or suggestions for questions to ask, or which symptoms describe the general condition.

[0034] Anomalies can be explained by a data point that lies outside the probability distribution of the data on which the statistical model is learned, i.e., the data point is an unlikely event.

[0035] The statistical model can be constructed so that it encodes the input signal or converts the input signal into a potential probability distribution (e.g., with the aid of a neural network). The potential distribution can be constructed of a lower dimension than the dimension of the probability distribution of the input signal. Samples from the potential distribution can then be decoded back to an approximation of the distribution of the input signal by a second statistical model (i.e., a decoder, which can also be in the form of a neural network).

[0036] During the learning process, the latent distribution may be conditioned on a known prior distribution, so an approximation to the true latent distribution is known. The combination of the latent distribution, the prior distribution, and the approximation of the input distribution can be collected into an approximation of confidence. When this approximation is calculated based on the input data point, it explains how this data point behaves in the dataset that the statistical model is learning. If the difference exceeds a certain threshold, the data point is classified as an outlier.

[0037] The audio features may be speech, noise, or sounds such as breathing patterns.

[0038] The corresponding number of samples may be a first number of samples, which should not be understood as a sample representing the voice of the party being questioned in the first part of the inquiry, but may be one or more samples representing the voice of the party being questioned in a window somewhat entering the inquiry, for example, the corresponding number of samples may represent a 2ms window, which occurs 1 second into the inquiry.

[0039] Processing can also be terminated at a fixed time after the query is initiated, for example between 2 minutes and 10 minutes into the query, processing can be terminated. Alternatively, between 2 minutes and 8 minutes, or 2 minutes and 5 minutes, or 3 minutes and 8 minutes.

[0040] The sample numbers constitute information in the form of real or production data, ie data processed by the statistical learning model during execution of the method, the data being processed according to a set of instructions defined by the method.

[0041] The inquirer inquires the inquired party by asking questions to the inquired party, and the method continuously determines whether the voice of the inquired party contains reliable information that can be processed to return a reliable output. Reliable information is continuously processed and any noise is skipped. In this way, noise can be excluded from processing, which can increase the speed of processing and the reliability of output. In the current context, noise will be understood as audio that deviates from the audio used in the training of the statistical learning model so that the statistical learning model does not return a useful output.

[0042] According to the present invention, the inquiring party may be an air traffic controller in an air traffic control tower, or a dispatcher at a public safety answering point, or a medical professional making rounds at a hospital.

[0043] Accordingly, the party being questioned may be a pilot in the cockpit of an aircraft experiencing problems, or a person involved in an incident such as a car accident, or a patient in a hospital.

[0044] The purpose of an inquiry is to gather information from the party being inquired in order to diagnose the problem and decide on a response action based on the information gathered and the experience of the inquiring party.

[0045] The processing unit operates in parallel with the questioning and processes the audio signal representing the voice of the questioned party. The processing can determine the words spoken by the questioned party and can diagnose the problem based on the recognized words to determine a response action.

[0046] The processing unit independently or without interaction with the inquirer determines a response action and presents the response action to the inquirer. The response action determined by the processing unit is an expected response action because it is based on the learned data and the data collected from the inquiry.

[0047] The processing unit may be part of a system that may include a plurality of microphones for recording the voice of the interrogating party and / or the interrogated party. The system may also include a display and / or a speaker for presenting the expected response action to the interrogating party.

[0048] The speech may be sampled in conjunction with a recording of the speech. The sampling frequency may be 16kHz, which is useful for human speech.

[0049] The system may also include a telecommunications network in the case where the interrogator and the interrogated are located at a distance from each other, such as in an air traffic control tower and in the cockpit of an aircraft.

[0050] The telecommunication network may be a telephone or satellite network connecting the communication terminals of the inquiring party and the inquired party respectively.

[0051] The voice of the inquired party can be recorded by a microphone at the location of the inquired party and converted into an electronic signal, which can be transmitted to the communication terminal of the inquired party in a wired or wireless manner and routed to the processing unit. The microphone may be associated with the communication terminal of the inquired party (such as a phone or headset).

[0052] The voice of the interrogator may also be processed by the processing unit so that the processing unit determines the expected response action. The voice of the interrogator may be recorded by a microphone, converted into an electronic signal, and routed into the processing unit.

[0053] In situations where the questioner and the questioned are in the same room, such as during hospital rounds, it is conceivable that a microphone may be used to record the conversation / question. The microphone may be connected to a terminal of a portable computer, such as a laptop or tablet computer, which a medical professional may carry during hospital rounds or during a doctor's office visit. Furthermore, the system may assist the medical professional by suggesting an expected diagnosis, so that the medical professional may decide more quickly or more reliably on the type of treatment or care for the patient.

[0054] The term "assist" in the present context refers to the inquirer receiving information from the system to help decide on response actions that the inquirer may or may not use or utilize during the emergency call.

[0055] Therefore, the inquiring party decides on the response action, but obtains information from the system to help decide on the response action. Alternatively, the system can propose the response action for approval by the inquiring party.

[0056] The statistical learning model may be a hidden Markov model or a neural network (artificial neural network), and the statistical learning model may be trained with training data to find a prediction function relating inputs to outputs.

[0057] The training data are samples from a probability distribution according to which the inputs and outputs are distributed. The statistical learning model is trained so that the prediction function relates the inputs to the outputs with the smallest possible error (i.e., the loss function is defined and minimized).

[0058] The anomaly routine / detection is a test of whether the corresponding interval is an outlier / anomaly of the distribution on which the statistical learning model has been trained, i.e., a test of whether the samples used to train the statistical learning model are also representative of the samples (corresponding number of samples) provided during the field execution of the system.

[0059] In cases where the corresponding number of samples are outliers / anomalous, the statistically learned model will not produce useful output - it can be said that the statistical learning model was then trained with too few training samples, which do not represent the number of samples such that useful output can be expected.

[0060] A plurality of samples is to be understood as a group or set of samples where the samples follow each other in time in a chronological order. Thus, the number of samples represents a time series or signal which is an interval or window of the entire query - the window may be as small as a single sample.

[0061] Each sample can be represented as an integer or a real number.

[0062] The processing unit can operate on one sample at a time, that is, the sample size is a single sample, so that a single sample can constitute the input of the statistical learning model.

[0063] Alternatively, a plurality of samples may constitute the input of the statistical learning model. The plurality of samples may include one or more new (previously unprocessed) samples and one or more samples that have been previously processed.

[0064] A sequence of samples is to be understood as samples that follow each other in time - originating from the sampled speech of the interrogated party.

[0065] A subsequent number of samples is to be understood as a number of samples having a first sample starting at a later point in time than the first sample in the previous number of samples.

[0066] According to the second aspect of the present invention, the above objects and advantages are achieved by:

[0067] A system for assisting an inquiring party to decide a response action in response to an inquiry between the inquiring party and an inquired party, the system comprising:

[0068] - a receiver for receiving the voice of the queried party and presenting the voice of the queried party as an electronic signal,

[0069] a processing unit for processing the electronic signal unit in parallel with the occurrence of the inquiry, the processing lasting for the duration of the inquiry or until the inquiry party determines the response action,

[0070] The processing comprises separating the electronic signal into a sequence of samples in the time domain or in a domain representing the frequency content of the electronic signal,

[0071] The processing unit comprises:

[0072] a) A heterogeneous routine comprising a statistical learning model for determining:

[0073] - the corresponding number of samples in the sample sequence is an outlier of the statistical learning model; or

[0074] - the corresponding number of samples is a norm for the statistical learning model,

[0075] b) a loop for returning to the outlier routine to process a subsequent number of samples of the sample sequence when the corresponding number of samples is determined to be an outlier of the statistical learning model,

[0076] c) an audio recognition model and a diagnostic model for detecting speech of the corresponding sample quantity and diagnosing the audio features, respectively, to determine an expected response action to the inquiry when the corresponding sample quantity is detected as a normal state of the statistical learning model,

[0077] The system further comprises:

[0078] A feedback unit is used to present the expected response action to the inquiring party.

[0079] The receiver may be a microphone for converting the voice of the inquired party from sound into an electronic signal, or a telephone for receiving the voice of the inquired party as an electronic signal.

[0080] According to the third aspect of the present invention, the above objects and advantages are achieved by:

[0081] A method for assisting an inquirer to determine a response action in response to an inquiry between the inquirer and an inquired party:

[0082] - providing a processing unit and inputting the voice of the queried party into the processing unit as an electronic signal,

[0083] - in parallel with the occurrence of said inquiry, processing of said electronic signal by means of said processing unit, said processing lasting for the duration of said inquiry or until said inquiry party has decided on said response action,

[0084] The processing includes:

[0085] separating the electronic signal into a sequence of samples in the time domain or in a domain representing the frequency content of the electronic signal,

[0086] The processing also includes:

[0087] a) an audio recognition routine, the audio recognition routine comprising an audio recognition model for detecting audio features of a corresponding number of samples of the sample sequence by means of a first statistical learning model,

[0088] The audio recognition routine returns a first audio output representing the detected audio features of the corresponding number of samples,

[0089] b) an action prediction routine, the action prediction routine comprising an analysis model for analyzing the first audio output with the aid of a second statistical learning model, the action prediction routine returning feedback to the inquiring party,

[0090] -Return to the audio recognition routine and the action prediction routine so that the corresponding subsequent number of samples of the sample sequence can be processed by the audio recognition routine and the action prediction routine to determine an expected response action based on the corresponding interval and the corresponding subsequent interval, and present the expected response action to the inquiring party with the aid of a display or a speaker.

[0091] According to a fourth aspect of the present invention, the above objects and advantages are achieved by:

[0092] A method for generating information, such as an image or voice, comprising: providing a storage unit and a processing unit, wherein the storage unit stores an instruction set, and the processing unit is used to execute the instruction set,

[0093] The instruction set includes a generator routine having a decoder formed from a statistical learning model,

[0094] The decoder is defined by observable variables and a set of random variables of a decoder hierarchy consisting of layers having in each layer at least one random variable from the set of random variables,

[0095] The observable variables and the set of random variables are jointly distributed according to a prior probability distribution,

[0096] The prior probability distribution is factorized to have:

[0097] - a first factor defined as a first probability distribution of said observable variable conditioned on said set of random variables,

[0098] - a second factor defined as a second probability distribution of random variables of the top layer of the decoder,

[0099] a third factor defined as the product of sequence of probability distributions of random variables of said set of random variables,

[0100] The random variable of each corresponding element in the product of the sequences is conditioned on the random variable in the higher level,

[0101] The method also includes sampling values ​​of the random variables at the top level and processing the values ​​through the levels such that the information is generated.

[0102] According to a fifth aspect of the present invention, the above objects and advantages are achieved by:

[0103] A method for identifying information, wherein the information is, for example, speech or text, the method comprising:

[0104] Provide a processing unit,

[0105] inputting the information into the processing unit as an electronic signal and processing the electronic signal,

[0106] The processing includes a recognition routine having an encoder formed from a statistical learning model,

[0107] The encoder is defined by the observable variables and the set of random variables at the encoder level,

[0108] The encoder hierarchy consists of layers,

[0109] The random variables of the set of random variables are divided into a bottom-up path and a top-down path through the encoder hierarchy,

[0110] The set of random variables is conditionally distributed on the observable variables according to a probability distribution used to approximate the posterior probability distribution,

[0111] The probability distribution is factorized into:

[0112] a first factor defined as a first probability distribution of random variables of the top layer of the encoder, conditioned on the observable variables and the corresponding random variables of the bottom-up path below the top layer,

[0113] a second factor defined as a sequence product of products between a probability distribution of random variables for the bottom-up path and a probability distribution of random variables for the top-down path,

[0114] the corresponding random variable of the bottom-up path for a given index in a sequence product is conditioned on the observable variable and the corresponding random variable of the bottom-up path for an index lower than the given index,

[0115] The corresponding random variable of the top-down path for a given index in the sequence product is conditioned on:

[0116] the observable variables,

[0117] at least one corresponding random variable of the bottom-up path,

[0118] the corresponding random variables of the top-down paths for indices higher than the given index,

[0119] The process also includes passing the information through the encoder hierarchy and outputting the identified information.

[0120] According to a sixth aspect of the present invention, the above objects and advantages are achieved by:

[0121] A method for determining when a data point is outlier, comprising:

[0122] Provide a processing unit,

[0123] inputting the data points into the processing unit as electronic signals and processing the electronic signals,

[0124] The process includes a statistical learning model having an encoder and a decoder,

[0125] The encoder has an encoder level and the decoder has a decoder level, the encoder level and the decoder level are defined by a set of random variables, the random variables of the set of random variables are divided into a bottom-up path and a top-down path, the encoder level and the decoder level are composed of the following layers: the layers have at least one random variable from the set of random variables in each layer,

[0126] The encoder defines a first probability distribution for approximating a posterior probability distribution, and the decoder defines a priori probability distribution,

[0127] defining an evidence lower bound function, the evidence lower bound function being a function of the first probability distribution of random variables in the set of random variables, the set of random variables including: more random variables from the bottom layer than from the top layer,

[0128] The evidence lower bound function is also a function of the prior probability distribution of random variables in the set of random variables, the set of random variables including: more random variables from the bottom layer than from the top layer,

[0129] The method further comprises determining the following with the aid of the statistical learning model:

[0130] - when the evidence lower bound function is integrated to a value less than a threshold, the data point is an outlier of the statistical learning model,

[0131] -When the evidence lower bound function is integrated to a value less than a threshold, the data point is a normal state of the statistical learning model. BRIEF DESCRIPTION OF THE DRAWINGS

[0132] The invention will be explained in more detail below by means of examples of embodiments with reference to very schematic drawings, in which:

[0133] Figure 1 Medical emergency and public safety answering points are shown.

[0134] Figure 2 A flow chart for assisting a medical call taker is shown.

[0135] Figure 3 A schematic diagram of a system for assisting a medical call taker is shown.

[0136] Figure 4 A flow chart for assisting a medical call taker is shown.

[0137] Figure 5a A specific model for the decoder is shown.

[0138] Figure 5b A specific model for the encoder is shown.

[0139] Figure 5c The top-down path of the encoder is shown. DETAILED DESCRIPTION

[0140] However, the present invention can be embodied in different forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to make this disclosure thorough and complete and to fully convey the scope of the present invention to those skilled in the art. The same reference numerals always refer to the same elements. Therefore, the same elements will not be described in detail with respect to the description of each figure.

[0141] Figure 1 Medical emergency and public safety answering points are shown.

[0142] The injured person 10 is experiencing an acute injury or illness and is illustrated lying on the ground.

[0143] The caller 12 is a bystander and requires assistance to help an injured person 10 who is experiencing a medical emergency in the form of an acute injury or illness.

[0144] A caller (inquired party) 12 is illustrated using a mobile phone 18 to call an emergency phone number (emergency call).

[0145] The emergency call is transmitted through the public switched telephone network having cell towers 16 , and the emergency call is routed to a public safety answering point 26 .

[0146] The emergency call is received at the public safety answering point as an electronic signal (an electrical current representing information). The electronic signal can be analog or digital.

[0147] A medical call taker (inquirer), hereinafter referred to as dispatcher 14, answers the emergency call and a conversation between the caller and the dispatcher begins. A receiver 20 (such as a telephone, which may include a handset for the dispatcher) may be used to answer the emergency call and convert the electronic signal into sound with the aid of a speaker.

[0148] During emergency calls, dispatchers follow a protocol for systematic caller questioning.

[0149] The protocol for the systematized questioning may include questions in a sequential order to determine the type of assistance required. The protocol may be stored by the processing unit, and the sequence of questions in the protocol may be rearranged by the processing unit based on information identified in processing - for example, certain cues or background noises may have relevance or be associated with certain emergency situations, meaning that the processing unit may ask questions related to such emergency situations earlier than the questioning party may realize that he or she should ask questions related to such emergency situations.

[0150] At some point during the conversation, the dispatcher decides on a response action, which typically includes dispatching emergency medical services (such as an ambulance) and providing a set of pre-arrival instructions to the caller.

[0151] The electronic signal is also routed to a processing unit 22, ie a computer associated with the public safety answering point. The processing unit comprises a set of instructions (algorithms) for processing the electronic signal.

[0152] Figure 2 A flow chart of a method of assisting an inquiring party, such as an emergency dispatcher or air traffic controller or a medical professional (doctor / nurse), is shown.

[0153] During an inquiry, the electronic signal is continuously routed or passed to the processing unit as long as the call / inquiry occurs, and the processing of the electronic signal continues for the duration of the inquiry.

[0154] However, the process may be suspended when the dispatcher / interrogating party has decided on a response action (such as dispatching an ambulance to the scene of the accident) or when the calling / interrogated party hangs up the phone.

[0155] The first sample size is selected and passed to the anomaly routine, which combines Figure 3 Further description: The first sample quantity may be composed of a number of samples between 1 sample and 300 samples (or even up to 1000 samples).

[0156] Figure 3 A schematic diagram showing an exemplary portion of an out-of-state routine is shown.

[0157] The outlier routine determines whether an interval is outlier. If it is not outlier, it is called normal.

[0158] The heterogeneous routine includes a statistical learning model, which may be a hidden Markov model or a neural network.

[0159] A statistical learning model is trained using training data to find a predictive function that relates inputs to outputs.

[0160] Training data are samples from a probability distribution according to which the inputs and outputs are distributed.

[0161] The statistical learning model is trained so that the prediction function relates the input to the output with the smallest possible error (ie, the loss function is defined and minimized).

[0162] Figure 3 Two examples of neural networks are shown. The left hand neural network takes three inputs (x 1 、x 2 、x 3 ), that is, the input layer has three input nodes. The left neural network also uses three outputs (y 1 ,y 2 ,y 3 ), that is, the output layer has three output nodes.

[0163] A single layer (one hidden layer) is shown between the input layer and the output layer of the neural network on the left hand side. The input and output are connected to each other via this single layer. Each node in this single layer can have a function f, for example, a hyperbolic tangent (tanh) function with a certain amplitude / constant in front of the function.

[0164] Figure 3The right-hand neural network shown in has the same structure as the left-hand neural network. The input of the right-hand neural network is the output from the left-hand neural network.

[0165] Figure 3 The two neural networks shown in are examples of neural networks, but illustrate the setup of a statistical learning model used in an outlier routine to determine whether a corresponding sample population is outlier or normal.

[0166] Therefore, during the process of processing the first sample quantity by the abnormal routine, a first neural network (encoder / inference model / recognition model) and a second neural network (decoder / generator) can be used to determine whether the corresponding sample quantity is abnormal or normal.

[0167] The first neural network may constitute an audio recognition neural network, and the second neural network may constitute an inverse neural network, i.e. a neural network that can reverse the predictions of the first neural network - predict the input of the first neural network based on the output of the first neural network.

[0168] The audio recognition neural network may be composed of a recursive neural network that can use previously recognized letters in a word to estimate the next letters in a word.

[0169] The input of the first neural network consists of the input vector is defined by, that is, by the input of the statistical learning model.

[0170] If the corresponding first number of samples represents 20 ms of audio sampled at 16 kHz, then the corresponding number of samples is 320, and the first neural network may have 320 inputs (320 input nodes - one input node for each sample).

[0171] The number of hidden layers and the number of hidden nodes in total should be selected so that the neural network does not underfit or overfit. These numbers can be selected based on the number of samples used to train the neural network and the number of input nodes and output nodes.

[0172] The first neural network may be defined by a matrix having up to 50 million elements.

[0173] The output of the first neural network is represented by the output vector To be limited, the output can have 320 output nodes.

[0174] The output of the first neural network in the heterogeneous routine may constitute the input of the second neural network, i.e., the input vector of the second neural network is With such a configuration, and in the case where the output layer of the first neural network has 320 outputs as exemplified above, the input layer of the second neural network may have 320 inputs.

[0175] The second neural network may be defined by a matrix having up to 130 million elements.

[0176] The output of the second neural network is represented by the output vector To limit.

[0177] Abnormal routine:

[0178] The heterogeneous routine can convert the input vector With the output vector If the two vectors do not compare, then the corresponding sample number is determined to be abnormal - or the corresponding sample number represents a signal / information that is abnormal. If the two vectors can be compared, then the corresponding sample number is determined to be normal.

[0179] In other words, the following is tested: whether the training samples used to train the statistical learning model (training data) are also representative of the corresponding sample population (real / production data), that is, whether the corresponding sample population is an outlier of the distribution based on which the statistical learning model is trained - for example, if the signal-to-noise ratio in the real data is too low, or if the pronunciation deviates to the extent that it cannot be recognized based on the data used to train the statistical learning model.

[0180] If the corresponding number of samples is an outlier, the statistically learned model will not produce useful output - it can be said that the statistical learning model is then trained with too few training samples, which do not represent the corresponding number of samples such that useful output can be expected.

[0181] The comparison of vectors may include determining the input vector With the output vector The correlation between them.

[0182] Input and output of statistical learning models and output vectors The correlation or difference between them can be compared with a threshold, that is, if the correlation or difference is above the threshold or below the threshold, the corresponding sample quantity is normal or abnormal.

[0183] For example, if the correlation is between 0 and 0.5, the corresponding interval is determined to be abnormal. For example, if the correlation is greater than 0.5, the corresponding sample size is determined to be normal.

[0184] As an alternative or in addition to the above, the outlier test / routine may include comparing the probability distribution of the training data with the probability distribution of the real data (corresponding sample numbers).

[0185] Where the probability function of the training data is unknown or has high dimensionality, the training data may undergo a transformation to reduce the dimensionality so that the probability function of the training data may be represented, for example, by a one-dimensional probability distribution such as an exponential function in one dimension.

[0186] The transformation can be linear or non-linear. Neural networks can be used to reduce dimensionality, i.e., perform the transformation.

[0187] The probability distribution of the lower dimension can be a priori given distribution (such as, for example, in one-dimensional Gaussian function or exponential function with limited parameters), and the conversion function can be trained to convert the probability function of the training data into the limited low-dimensional probability distribution. The same conversion function can be used as the basis for transforming the probability distribution of the corresponding sample number. The lower dimensional probability distribution of the corresponding sample number can then be compared with the lower dimensional probability distribution of the training data.

[0188] The comparison of probability distributions (whether high-dimensional or low-dimensional) can be done by integrating the two probability distributions and comparing the integral to a threshold. If the integral is below the threshold, it can be determined that the corresponding number of samples represents an outlier.

[0189] Alternatively or additionally, the test of whether the corresponding sample quantity is an outlier / abnormal can include determining how far the corresponding sample quantity is from the mean of the probability distribution of the training data (i.e., the probability distribution that defines the statistical learning model). For example, if the distance is more than 0.5 standard deviations, then the corresponding sample quantity can be determined to be abnormal.

[0190] The corresponding intervals can be Fourier transformed into the frequency domain, and the frequencies of the signal represented by the corresponding number of samples can constitute the input of the statistical learning model used by the heteromorphic routine, that is, the signal represented by the corresponding number of samples can be divided into frequencies according to the energy at the frequencies of the corresponding intervals.

[0191] Return to Figure 2 An exemplary flow chart of the invention, in which the results of the corresponding number of samples processed by the abnormal routine are abnormal, and the processing returns feedback to the inquiring party.

[0192] This feedback may constitute negative feedback, in a sense, the inquiring party is informed that the process is ongoing. The feedback may be by sound or displayed on the display 24.

[0193] Alternatively, the method may not provide feedback to the inquirer until it returns a suggestion for a response action, or until processing reaches a number of samples representing a signal that is not anomalous.

[0194] Processing can then continue with a subsequent number of samples (window) representing the portion of the interrogated party's speech following the previous number of samples (which is anomalous). Thus, the method loops back to the anomalous routine - possibly while the condition is that an emergency call is still in progress.

[0195] In the event that the corresponding number of samples that have been processed by the outlier routine results in not being outliers, then the processing can continue with the corresponding number of samples, which can be passed to the audio recognition routine to recognize the audio in the corresponding interval.

[0196] The audio recognition routine may have been performed in conjunction with the anomaly routine, i.e., the first neural network of the anomaly routine is a neural network used for audio recognition - the audio recognition routine may include an audio recognition neural network corresponding to the first neural network also used by the anomaly routine, so that there is compliance between the tests performed by the anomaly routine and the audio recognition routine.

[0197] The audio recognition routine determines the letters spoken by the party being questioned and combines the letters into words.

[0198] A corresponding number of samples of the recognized audio and words are passed to a diagnostic routine for diagnosing the emergency situation.

[0199] The diagnostic routine may include diagnosing the neural network.

[0200] It may be necessary to use audio from more than one sample set to make a diagnosis and propose a responsive action for the method.

[0201] However, only a few sample sets, such as only one sample set that has been through an audio recognition and diagnostic routine, may result in positive feedback to the inquirer in the form of one or more prompts or suggestions - alternatively, systematized caller questioning in the form of a rearrangement of the protocol, i.e., rearranging the order of the list of questions in the protocol so that the party being inquired has a higher probability of asking the correct question more quickly in certain emergency situations - feedback to the inquirer may include a rearrangement of the questions in the protocol.

[0202] When enough audio has been recognized for the diagnostic routine to make a diagnosis based on the recognized words, a responsive action is suggested to the dispatcher.

[0203] Processing may include similar testing as for the anomaly routine, where a test is performed on whether there is enough information available to the diagnostic routine to make a diagnosis, i.e., the diagnostic routine can be expected to return a diagnosis and a corresponding response action (is correct / possible).

[0204] Background noise and cues.

[0205] The audio recognition and diagnosis routine can also analyze and / or respond to background noise, i.e., specific background noise can indicate certain emergency situations. The process can identify such specific background noise before the inquiring party, thereby reducing response time.

[0206] A neural network for audio event classification can be used to identify background noise (such as breathing patterns) and make a diagnosis based on the identified background noise. It can be a dedicated neural network because the training data can represent specific audio events, i.e., the training data can consist of recordings of the sound of breathing of many people (such as 1000 people) experiencing heart attacks.

[0207] The diagnostic routine may also analyze and / or respond to special cues, such as specific words, that may have been observed to be highly correlated with certain emergency or medical conditions. The special words may be help, heart attack, engine failure, etc. Special neural networks may be used to diagnose special cues, and the special neural networks may be trained with data representing the special cues.

[0208] In addition, the amplitude and / or frequency of the voice of the party being questioned may also be analyzed during the process. For example, a high amplitude may indicate that an ambulance needs to be dispatched quickly to the scene of the accident.

[0209] The processing may include a language routine for determining the language of the caller. The language routine may include a language neural network. When an emergency call is received at a public safety answering point, the language routine may be immediately initialized.

[0210] Figure 4 A flow chart for assisting a medical call taker according to a third aspect of the present invention is shown.

[0211] The outlier routine is omitted in the method, and the sample sets are processed successively by the audio recognition routine and the diagnosis routine respectively.

[0212] Figure 5a A specific model of the decoder is shown.

[0213] Figure 5b A specific model of the encoder is shown.

[0214] The decoder and encoder can be used to Figure 2 and Figure 4 The processing routine shown. The model defined by the decoder and encoder (such as Figure 5a to Figure 5cAs shown in Figure 2, the bidirectional inference variational autoencoder (BIVA) is constructed.

[0215] exist Figure 5a In , the decoder consists of a hierarchy with three layers. The top layer (third layer) has a random variable z distributed according to the probability distribution 3 Typically, a normal (Gaussian) distribution can be used. The parameters of the probability distribution (mean and variance in the case of a normal distribution) can be determined with the help of neural networks - one neural network per parameter. Other statistical learning models can also be used to qualify random variables.

[0216] More than three layers may be used, and in the following, the number of layers is arbitrary and specific layers may be referred to by index numbers.

[0217] For the layer below the top layer (i.e., the second layer) and the bottom layer (i.e., the first layer), it can be seen that the random variables of the corresponding layers are divided into two components The superscript BU refers to the bottom-up encoder path, while the superscript TD refers to the top-down encoder path (combined with Figure 5b to explain the encoder).

[0218] The decoder can have a deterministic top-down path d L-1 , ..., d 1 (This path can be parameterized by a neural network), and each layer i in the hierarchy receives a random variable z i+1 as input.

[0219] This can be done by constraining the model to be fully convolutional and shifting the feature dimension With d i+1 Make the connection to complete.

[0220] Therefore, d i can be considered as a deterministic variable that aggregates random variables z from higher levels in the hierarchy >i All relevant information.

[0221] exist Figure 5a to Figure 5c In , the dotted lines show deterministic parameters that are shared within the decoder or between the decoder and the encoder.

[0222] random variable and Conditioned on all information in higher layers, and given z >i The cases are conditionally independent.

[0223] The joint distribution (prior probability distribution) of the decoder is p θ (x, z) is given by:

[0224]

[0225] where θ refers to the parameters of the decoder, i.e., in the case where a neural network is used to define random variables (or probability distributions for random variables in a layer), the parameters may be the weights of the neural network.

[0226] p θ (x|z) is a first factor defined as a first probability distribution of the observable variable x conditioned on a set of random variables (ie, z).

[0227] p θ (z L ) is the second factor, which is defined as the top level z of the decoder L The second probability distribution of a random variable (where index i=L). Figure 5a , L=3.

[0228] is a third factor defined as the sequence product of the probability distribution of the random variables for the set of random variables for the decoder - The random variable of each corresponding element in the product of the sequences is conditioned on one or more higher-level random variables in the higher level. For example, for index i=2, the random variable z 2 The random variable z 3 or 4 or higher random variables as conditions. The condition may also be several higher-level random variables, i.e., for example, z 3 or 4 For index or element i = L-1, the random variable z L-1 Only the top random variable z L As a condition.

[0229] The elements in the product sequence can be factorized into That is, the first factor is defined as the conditional probability distribution for bottom-up random variables (where information / data goes from the bottom of the hierarchy toward the top), and the second factor is defined as the conditional probability distribution for top-down random variables (where information / data goes from the top of the hierarchy toward the bottom).

[0230] The probability distribution can have a diagonal covariance, with one neural network for the mean and another for the variance.

[0231] because and The variables are at the same level in the decoder and have the same dimensions, so all deterministic parameters going to the layers below are shared.

[0232] Specifically, the decoder has a top-down path that starts from z L Through the intermediate random variables to x. Between each layer there is a ResNet block with M layers. Weight normalization is applied to all neural network layers.

[0233] ResNet's neural network function for layer j associated with layer i (decoder parameters θ The function of i，j .

[0234] The feature map is written as d i，j The decoder routine can then be iterated as and i=L-1,L-2,...,1:

[0235] d i,0 =z i+1

[0236]

[0237] exist Figure 5b In , the encoder includes a hierarchy with three layers.

[0238] Due to the nonlinearity in the neural network that parameterizes the decoder, the exact posterior distribution p θ (z|x) is difficult to solve and needs to be approximated. This can be done by approximating the variational distribution (the probability distribution used to approximate the posterior probability distribution) q φ (z|x) is used to limit.

[0239] Bottom-up (BU) and top-down (TD) encoder paths are defined and computed sequentially when building a posterior approximation for each data point x.

[0240] The variational distribution above the BU random variable depends on the data x and all BU variables lower in the hierarchy, i.e., - Alternatively, the condition may be a smaller number of BU variables, such as those in the layers just below the layer with corresponding index i BU Variable. φ represents BU All parameters of the path. can directly depend only on the following BU variables, namely,

[0241] Similar to the decoder, a deterministic bottom-up path To achieve of dependence.

[0242] TD variables also depend on BU variables and data lower in the hierarchy through the BU encoder path, but also depend on Figure 5c All variables in the TD encoder path in the upper layer. Therefore, the variational approximation for the TD variables is

[0243] All parameters of the TD path can be shared with the decoder and are therefore denoted as θ, while the parameters of the encoder are denoted as φ. The encoder can be factorized as follows:

[0244]

[0245] That is, according to the probability distribution q used to approximate the posterior probability distribution φ (z|x), the random variable z is distributed conditionally on the observable variable x.

[0246] The variational distribution of the BU and TD random variables can be Gaussian distributions, whose mean and diagonal covariance can be parameterized by a neural network that takes as input the concatenation of the feature dimensions of the conditioning variables.

[0247] First Factor It can be defined as a first probability distribution of random variables at the top level of the encoder, conditioned on observable variables and corresponding random variables of the bottom-up path below the top level. It can further be conditioned on all corresponding random variables of the bottom-up path below the top level - or it can exclude some random variables closest to the bottom of the hierarchy.

[0248] Second Factor:

[0249]

[0250] The probability distribution of the random variable that can be defined for the bottom-up path Probability distribution of random variables for top-down paths The sequence product of the products between .

[0251] The corresponding random variables for the bottom-up path for a given index (or element) in the sequence product are conditioned on the observable variables and the corresponding random variables for the bottom-up paths for indices lower than the given index.

[0252] The corresponding random variable for the top-down path at a given index in the product of sequences is conditioned on:

[0253] - observable variables,

[0254] - at least one corresponding random variable of the bottom-up path, and

[0255] - The corresponding random variables for the top-down paths to indices higher than the given index.

[0256] The corresponding random variables for the top-down path for a given index in the sequence product can be conditioned on: the random variables for the bottom-up path for indexes higher than the given index, and the random variables for the bottom-up path for indexes lower than the given index; so that all random variables for the bottom-up path except one of the given indices.

[0257] For the variational autoencoder, the training of the encoder and decoder can be performed by maximizing the Evidence Lower Bound (ELBO) using stochastic backpropagation and the reparameterization trick:

[0258]

[0259] In the standard ELBO above, the main contribution to the expected log-likelihood term comes from averaging over the variational distributions of the lower-level random variables. This will therefore emphasize the low-level statistics.

[0260] When using Figure 5a to Figure 5c The specific encoder and decoder shown and mentioned above perform outlier detection in a way that emphasizes contributions from higher layers.

[0261] The lower bound of evidence for outlier detection is the posterior probability distribution (or an approximation of it) q φ (z ＞k |x) and the prior probability distribution p θ (z ＞k ) function:

[0262]

[0263] Regarding the prior probability distribution, it is the random variable for layers higher than k, i.e., the random variable excluding the lower variables. For example, the choice of k may be 3, so that for a hierarchy with 6 layers, the prior probability distribution is the random variable for the fourth, fifth, and sixth layers. Thus, k may be viewed as a layer number defining a particular layer between the top and the bottom of the hierarchy. The layer number may correspond to a middle layer or a layer closer to the middle layer than the top or the bottom.

[0264] It is possible to exclude only one variable in the kth layer or layers below k, i.e., for example, to exclude only the random variables of the first or second layer, and the distribution is for the other random variables of the layers. In general, the prior probability distribution is for the random variables of the layers excluding at least one random variable from one of the bottom two layers, the bottom three layers, or the bottom four layers.

[0265] Regarding the posterior probability distribution, it is the random variable for layers above k. The random variable is conditioned on the observable variable x. The choice of k and the function may be the same as for the prior probability distribution.

[0266] The random variable can be said to belong to the category that is restricted to z = z 1 ,z 2 ,z 3 ,…,z L A set of random variables, where

[0267] The calculation of is similar to Monte Carlo integration.

[0268] You can get samples from the encoder by To execute from p θ (z ≤k |z ＞k )q φ (z ＞k |x), which are then used to sample from the conditional prior Samples in

[0269] By sampling only the top L to k variables from the variational approximation, we rely only on the high-level semantics encoded in the top variables in the hierarchy when evaluating the metric, and not on the low-level statistics encoded in the lower variables.

[0270] The following is a list of reference numbers used in the detailed description of the invention and in the accompanying drawings mentioned in the detailed description of the invention.

[0271] 10 Injured Persons

[0272] 12 Caller

[0273] 14 Medical Call Taker

[0274] 16 Cell Phone Towers

[0275] 18 Mobile phone

[0276] 20 Receiver

[0277] 22 Processing Units

[0278] 24 Display

[0279] 26 Public Safety Answering Point.

Claims

1. A method for assisting an inquiring party to decide a response action in response to an inquiry between the inquiring party and an inquired party, the method comprising the following steps: - providing a processing unit and inputting the voice of the queried party into the processing unit as an electronic signal, - processing said electronic signal by means of said processing unit in parallel with the occurrence of said inquiry, said processing lasting for the duration of said inquiry or until said inquiry party has decided on said response action, The processing includes: separating the electronic signal into a sequence of samples in the time domain or in a domain representing the frequency content of the electronic signal, The processing also includes: a) an anomaly routine, which contains a statistical learning model and by means of which the following are determined: - the corresponding number of samples in the sample sequence is an outlier of the statistical learning model; or - the corresponding number of samples is a norm for the statistical learning model, b) when the corresponding sample number is determined to be an outlier of the statistical learning model: - returning to said anomaly routine to process a subsequent number of samples of said sequence of samples, c) when the corresponding sample size is determined to be a normal state of the statistical learning model: - providing an audio recognition model, and detecting audio features of the corresponding number of samples by means of the audio recognition model, - providing an action prediction model and analyzing the audio features with the aid of the action prediction model to determine an expected response action, - providing a display or a speaker, and presenting the expected response action to the inquiring party by means of the display or the speaker to assist the inquiring party, - returning to the anamorphic routine to process a subsequent number of samples of the sequence of samples by the anamorphic routine.

2. The method according to claim 1, comprising inputting the voice of the inquiring party into the processing unit.

3. The method of claim 1 or 2, providing feedback to the inquiring party during the processing by means of the display or the speaker.

4. According to the method of claim 3, when the corresponding number of samples is detected as an anomaly of the statistical learning model, the feedback constitutes negative feedback. The method according to claim 4 , wherein the negative feedback indicates that the processing is being performed.

Images