Method, apparatus and system for implementing fixed state connection order classification

Abstract

Some disclosed methods involve receiving a sequence of observations, the sequence of observations comprising a plurality of extracted features, each of the plurality of extracted features corresponding to a sequence signal in a sequence of sequence signals; determining a posterior likelihood grid, the grid including a likelihood that each observation value sequence corresponds to one of the plurality of tag categories; and applying a loss function to the posterior likelihood grid according to the true value, where applying the loss function involves applying both order information and clustering boundary information. Some methods involve updating the parameters used to determine the grid according to the loss determined by the loss function, and performing the foregoing until one or more convergence criteria are met.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 513,294 filed on July 12, 2023 and U.S. Provisional Patent Application No. 63 / 401,042 filed on August 25, 2022, which are hereby incorporated by reference herein. Technical Field

[0003] The present disclosure relates to machine learning. Background Art

[0004] Some methods, devices, and systems for machine learning are known. Although existing devices, systems, and methods for machine learning can provide benefits in some situations, improved devices, systems, and methods would still be desirable.

[0005] Symbols and terminology

[0006] Throughout this disclosure, including in the claims, the terms "speaker," "loudspeaker," and "audio reproduction transducer" are used synonymously to refer to any sound-emitting transducer (or set of transducers) driven by a single speaker feed. A typical set of headphones includes two speakers. Speakers may be implemented to include multiple transducers (e.g., a woofer and a tweeter), which may be driven by a single common speaker feed or multiple speaker feeds. In some examples, the speaker signal(s) may undergo different processing in different circuit branches coupled to different transducers.

[0007] Throughout this disclosure, including in the claims, expressions referring to “performing an operation on” a signal or data (e.g., filtering, scaling, transforming, or applying a gain to the signal or data) are used in a broad sense to mean performing the operation directly on the signal or data or on a processed version of the signal or data (e.g., a version of the signal that has undergone preliminary filtering or preprocessing before the operation is performed on it).

[0008] Throughout this disclosure, including in the claims, the expression "system" is used in a broad sense to refer to a device, system, or subsystem. For example, a subsystem that implements a decoder may be referred to as a decoder system, and a system that includes such a subsystem (e.g., a system that generates X output signals in response to multiple inputs, where the subsystem generates M inputs and the other XM inputs are received from external sources) may also be referred to as a decoder system.

[0009] Throughout this disclosure, including in the claims, the term "processor" is used in a broad sense to refer to a system or device that is programmable or otherwise configurable (e.g., with software or firmware) to perform operations on data (e.g., audio or video or other image data). Examples of processors include field programmable gate arrays (or other configurable integrated circuits or chipsets), digital signal processors that are programmed and / or otherwise configured to perform pipeline processing of audio or other sound data, programmable general purpose processors or computers, and programmable microprocessor chips or chipsets.

[0010] Throughout this disclosure, including in the claims, the terms "couple" or "coupled" are used to mean either a direct or indirect connection. Thus, if a first device couples to a second device, that connection may be through a direct connection or through an indirect connection via other devices and connections.

[0011] As used herein, a "smart device" is an electronic device that can operate interactively and / or autonomously to some extent, and is typically configured to communicate with one or more other devices (or networks) via various wireless protocols such as Bluetooth, Zigbee, near field communication, Wi-Fi, Light Fidelity (Li-Fi), 3G, 4G, 5G, etc. Some well-known types of smart devices are smart phones, smart cars, smart thermostats, smart doorbells, smart locks, smart refrigerators, tablet phones and tablet computers, smart watches, smart bracelets, smart key chains, and smart audio devices. The term "smart device" may also refer to devices that exhibit some properties of ubiquitous computing such as artificial intelligence.

[0012] In this document, the expression "smart audio device" is used to represent a smart device, which is a single-purpose audio device or a multi-purpose audio device (e.g., an audio device that implements at least some aspects of the virtual assistant functionality). A single-purpose audio device is a device (e.g., a television (TV)) that includes or is coupled to at least one microphone (and optionally also includes or is coupled to at least one speaker and / or at least one camera) and is largely or primarily designed to implement a single purpose. For example, although a TV can generally play (and is considered to be able to play) audio from program material, in most instances, modern TVs run some kind of operating system on which applications (including applications for watching TV) run locally. In this sense, a single-purpose audio device with (multiple) speakers and (multiple) microphones is typically configured to run local applications and / or services to directly use the (multiple) speakers and (multiple) microphones. Some single-purpose audio devices can be configured to be combined together to enable audio to be played in a certain area or user-configured area.

[0013] A common type of multi-purpose audio device is an intelligent audio device (such as a "smart speaker") that can be configured to implement at least some aspects of a virtual assistant function, although other aspects of the virtual assistant function can be implemented by one or more other devices such as one or more servers, and the multi-purpose audio device is configured to communicate with the one or more servers. Such a multi-purpose audio device may be referred to as a "virtual assistant" in this article. A virtual assistant is a device (e.g., a smart speaker or voice assistant integrated device) that includes or is coupled to at least one microphone (and optionally also includes or is coupled to at least one speaker and / or at least one camera). In some examples, a virtual assistant can provide the ability to use multiple devices (other than a virtual assistant) for applications that are cloud-enabled in some sense or that are otherwise not fully implemented in or on the virtual assistant itself. In other words, at least some aspects of the virtual assistant function (e.g., speech recognition functionality) can be (at least partially) implemented by one or more servers or other devices, and the virtual assistant can communicate with the one or more servers or other devices via a network (such as the Internet). Virtual assistants can sometimes work together, for example, in a discrete and conditionally defined manner. For example, two or more virtual assistants can work together in the sense that one of them (e.g., the virtual assistant that is most confident of having heard the wake word) responds to the wake word. In some implementations, connected virtual assistants can form a constellation that can be managed by a master application that can be (or implement) a virtual assistant.

[0014] In this document, "wake-up word" is used in a broad sense to refer to any sound (e.g., a word spoken by a human or other sound) that the smart audio device is configured to wake up in response to detecting ("hearing") the sound (using at least one microphone included in or coupled to the smart audio device, or at least one other microphone). In this context, "waking up" means that the device enters a state of waiting (in other words, listening) for a voice command. In some instances, the so-called "wake-up word" in this document can include more than one word, for example, a phrase.

[0015] In this document, the expression "wake-up word detector" refers to a device configured to continuously search for alignment between real-time sound (e.g., speech) features and a trained model (or refers to software including instructions for configuring the device to continuously search for alignment between real-time sound features and a trained model). Typically, a wake-up word event is triggered whenever the wake-up word detector determines that the probability of detecting a wake-up word exceeds a predefined threshold. For example, the threshold may be a predetermined threshold that is adjusted to give a reasonable compromise between a false acceptance rate and a false rejection rate. After a wake-up word event, the device may enter a state (which may be referred to as an "awake" state or an "attention" state) in which the device listens for commands and passes received commands to a larger, more computationally intensive recognizer.

[0016] As used herein, the terms "program stream" and "content stream" refer to a collection of one or more audio signals, and in some instances a collection of video signals, at least a portion of which is intended to be heard together. Examples include a selection of music, movie soundtracks, movies, television programs, audio portions of television programs, podcasts, live voice calls, synthesized voice responses from smart assistants, and the like. In some instances, a content stream may include multiple versions of at least a portion of an audio signal, e.g., the same conversation in more than one language. In such instances, only one version of the audio data or portion thereof (e.g., a version corresponding to a single language) is intended to be reproduced at a time. Summary of the invention

[0017] At least some aspects of the present disclosure may be implemented via one or more methods. In some instances, method(s) may be implemented at least in part by a control system and / or via instructions (e.g., software) stored on one or more non-transitory media. Some methods may involve receiving a sequence of observations by a control system, the sequence of observations comprising a plurality of extracted features. Each of the plurality of extracted features may correspond to a sequential signal in a sequence of sequential signals. Some methods may involve determining a posterior likelihood grid by the control system. The grid may include the likelihood that each sequence of observations corresponds to a label category in a plurality of label categories.

[0018] Some methods may involve applying, by the control system, a loss function to the posterior likelihood grid based on true values. In some examples, applying the loss function may involve applying both sequential information and cluster boundary information. According to some examples, applying the loss function may involve determining one or more valid paths between observations in the grid.

[0019] Some methods may involve updating, by the control system, parameters used to determine the a posteriori likelihood grid based on the loss determined by the loss function. Some methods may involve continuing to perform the receiving, determining, applying, and updating processes until the control system determines that one or more convergence criteria are met.

[0020] According to some examples, performing the foregoing process may provide a trained neural network implemented by the control system. Some methods may involve performing one or more downstream tasks via the trained neural network.

[0021] In some examples, the sequential signal sequence may be or may include an audio signal sequence.In some such examples, the one or more downstream tasks may include acoustic event detection, speaker logging, automatic speech recognition, or a combination thereof.

[0022] According to some examples, the sequence of sequential signals may be or may include a sequence of handwritten images.In some such examples, the one or more downstream tasks may include handwriting recognition.

[0023] In some examples, the cluster boundary information may be incomplete cluster boundary information. However, in some alternative examples, the cluster boundary information may be complete cluster boundary information.

[0024] Some methods may involve determining, by the control system, the plurality of extracted features. In some such examples, the methods may involve updating, by the control system, parameters used to determine the plurality of extracted features based on a loss determined by the loss function.

[0025] In some examples, the order information of the cost function, the cluster boundary information of the cost function, or both may be variable.According to some examples, the loss function may combine elements of connection time classification and cross entropy loss.

[0026] Some or all of the operations, functions, and / or methods described herein may be performed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include memory devices as described herein, including but not limited to random access memory (RAM) devices, read-only memory (ROM) devices, etc. Thus, some innovative aspects of the subject matter described in this disclosure may be implemented via one or more non-transitory media having software stored thereon.

[0027] At least some aspects of the present disclosure can be implemented via a device. For example, one or more devices (e.g., a system including one or more devices) may be able to perform at least partially the methods disclosed herein. In some embodiments, the device is or includes an audio processing system with an interface system and a control system. The control system may include one or more general-purpose single-chip or multi-chip processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic, discrete hardware components, or a combination thereof. The control system may be configured to implement some or all of the methods disclosed herein.

[0028] The details of one or more embodiments of the subject matter described in this specification are set forth in the following drawings and description. Other features, aspects, and advantages will become apparent from the description, drawings, and claims. Note that the relative sizes of the following drawings may not be drawn to scale. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Like reference numbers and designations throughout the various drawings indicate like elements.

[0030] Figure 1A is a block diagram illustrating an example of components of an apparatus capable of implementing various aspects of the present disclosure.

[0031] Figure 1B An example of an audio environment is shown.

[0032] Figure 2 and Figure 3 Examples of blocks that may be included in some disclosed fixed state connection order classification (PS-CSC) implementations are shown.

[0033] Figure 4 Shown are example blocks involved in training a neural network for a speaker log use case using PS-CSC.

[0034] Figure 5A , Figure 5B and Figure 5C Experimental results involving conventional methods in a speaker logging use case are shown.

[0035] Fig. 6A and Figure 6B Experimental results involving the PS-CSC approach in a speaker logging use case are shown.

[0036] Figure 7 Results from two additional experiments are shown.

[0037] Figure 8 is a flow chart outlining one example of the disclosed method. DETAILED DESCRIPTION

[0038] Machine learning methods based on cross entropy loss and its variants and machine learning methods based on connection temporal classification (CTC) use different loss targets and are generally used for different purposes in the machine learning community. Machine learning methods based on cross entropy loss are generally used to train neural network models to solve classification problems, with use cases such as image-based face classification or speaker recognition based on speech signals. Machine learning methods based on cross entropy loss may involve obtaining and / or learning cluster boundary information. In contrast, machine learning methods based on CTC are generally applied to situations where sequential information needs to be explored, with use cases such as automatic speech recognition and handwriting recognition.

[0039] The present disclosure provides examples of machine learning methods based on cluster boundary information and sequential information. Such methods may be referred to herein as fixed state connection sequential classification (PS-CSC) methods. According to some PS-CSC examples, the loss function for training a neural network may be based on cluster boundary information and sequential information. In some PS-CSC examples, the cluster boundary information may be or may include incomplete cluster boundary information. In some PS-CSC examples, the loss function for training a neural network may combine elements of CTC and cross entropy loss.

[0040] In some PS-CSC examples, the disclosed neural network training process may involve receiving a sequence of observations, the sequence of observations comprising a plurality of extracted features. According to some examples, the sequence of observations may be a time series, while in other examples, the sequence of observations may be a spatial series. Each extracted feature may correspond to a sequential signal in a sequence of sequential signals. In some instances, the training process may involve determining a posterior likelihood grid. The grid may include a likelihood that each sequence of observations corresponds to one of a plurality of label categories. According to some examples, the labels may be monotonically aligned with the observations.

[0041] According to some PS-CSC examples, applying the loss function may involve determining one or more valid paths between observations in the grid. In some such examples, the fixed state of the observations in the grid may reduce the number of valid paths and thus reduce the time and computational overhead required for convergence during the neural network training process.

[0042] A neural network trained according to some disclosed methods may be configured to perform one or more downstream tasks. The downstream tasks may vary depending on the type of signal in the sequential signal sequence. In some instances, the sequential signal sequence may be or may include a sequence of handwritten images, and the downstream task may be handwriting recognition. In other examples, the sequential signal sequence may be or may include a sequence of audio signal sequences, and the downstream task(s) may be acoustic event detection, speaker logging, and / or automatic speech recognition.

[0043] Figure 1A is a block diagram showing an example of components of an apparatus capable of implementing various aspects of the present disclosure. As with other figures provided herein, Figure 1A The types, quantities, and arrangements of the elements shown in are provided as examples only. Other embodiments may include more, fewer, and / or different types, quantities, and arrangements of elements. According to some examples, the device 100 may be configured to perform at least some of the methods disclosed herein. In some embodiments, the device 100 may be or may include one or more components of an office workstation, one or more components of a home entertainment system, etc. For example, the device 100 may be a laptop computer, a tablet device, a mobile device (such as a cellular phone), a smart home hub, a television, or another type of device.

[0044] According to some alternative embodiments, the apparatus 100 may be or may include a server. In some such examples, the apparatus 100 may be or may include an encoder. In some examples, the apparatus 100 may be or may include a decoder. Thus, in some instances, the apparatus 100 may be a device configured for use within an environment such as a home environment, while in other instances, the apparatus 100 may be a device configured for use in the "cloud", e.g., a server.

[0045] In this example, the device 100 includes an interface system 105 and a control system 110. In some embodiments, the interface system 105 can be configured to communicate with one or more other devices of the environment. In some examples, the environment can be a home environment. In other examples, the environment can be another type of environment, such as an office environment, a car environment, a train environment, a street or sidewalk environment, a park environment, etc. In some embodiments, the interface system 105 can be configured to exchange control information and associated data with other devices of the environment. In some examples, the control information and associated data can be related to one or more software applications being executed by the device 100.

[0046] In some embodiments, the interface system 105 can be configured to receive a content stream or to provide a content stream. In some examples, the content stream can include video data and audio data corresponding to the video data. The audio data can include but is not limited to an audio signal. In some cases, the audio data can include spatial data such as channel data and / or spatial metadata. For example, metadata can be provided by a device that can be referred to as an "encoder" herein.

[0047] The interface system 105 may include one or more network interfaces and / or one or more external device interfaces (e.g., one or more Universal Serial Bus (USB) interfaces). According to some embodiments, the interface system 105 may include one or more wireless interfaces. The interface system 105 may include one or more devices for implementing a user interface, such as one or more microphones, one or more speakers, a display system, a touch sensor system, a gesture sensor system, or a combination thereof. Thus, although some such devices may be Figure 1A 105 , but in some examples, such devices may correspond to aspects of interface system 105 .

[0048] In some examples, the interface system 105 may include a control system 110 and a memory system (eg, Figure 1A Alternatively or additionally, in some instances, the control system 110 may include a memory system. In some implementations, the interface system 105 may be configured to receive input from one or more microphones in the environment.

[0049] The control system 110 may include, for example, a general purpose single-chip or multi-chip processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or a combination thereof.

[0050] In some embodiments, the control system 110 may reside in more than one device. For example, in some embodiments, a portion of the control system 110 may reside in a device within one of the environments mentioned herein, and another portion of the control system 110 may reside in a device outside the environment, such as a server, a mobile device (such as a smart phone or a tablet computer), etc. In other examples, a portion of the control system 110 may reside in a device within one of the environments depicted herein, and another portion of the control system 110 may reside in one or more other devices of the environment. For example, the functionality of the control system may be shared by an orchestration device (such as a device that may be referred to as a smart home hub herein) and one or more other devices of the environment. In other examples, a portion of the control system 110 may reside in a device (such as a server) that implements a cloud-based service, and another portion of the control system 110 may reside in another device (such as another server, a memory device, etc.) that implements a cloud-based service. In some examples, the interface system 105 may also reside in more than one device.

[0051] In some embodiments, the control system 110 may be configured to at least partially perform the method disclosed herein. According to some examples, the control system 110 may be configured to receive an observation sequence, the observation sequence comprising a plurality of extracted features. Each of the plurality of extracted features may correspond to a sequential signal in a sequential signal sequence. In some examples, the control system 110 may be configured to determine the plurality of extracted features.

[0052] According to some examples, the control system 110 can be configured to determine a posterior likelihood grid. The grid can include the likelihood that each observation sequence corresponds to one of a plurality of label classes.

[0053] In some examples, control system 110 may be configured to apply a loss function to the posterior likelihood grid according to the true values. In some such examples, applying the loss function may involve applying both sequential information and cluster boundary information.

[0054] According to some examples, the control system 110 can be configured to update the parameters used to determine the posterior likelihood grid according to the loss determined by the loss function. In some examples, the control system 110 can be configured to train the neural network by performing the above process until the control system 110 determines that one or more convergence criteria are met.

[0055] As noted elsewhere herein, the control system 110 may reside in a single device or in multiple devices, depending on the particular implementation. In some examples, all of the aforementioned processes may be performed by the same device. In some alternative examples, the aforementioned processes may be performed by two or more devices. For example, an embedding may be generated by one device, and a posterior likelihood grid may be performed by one or more other devices. In some such examples, one or more processes may be performed by one or more servers configured to implement a cloud-based service.

[0056] Some or all of the methods described herein may be performed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include memory devices as described herein, including but not limited to random access memory (RAM) devices, read-only memory (ROM) devices, etc. One or more non-transitory media may reside, for example, on a computer. Figure 1A 1 and / or control system 110. Thus, various innovative aspects of the subject matter described in the present disclosure may be implemented in one or more non-transitory media having software stored thereon. For example, the software may include instructions for controlling at least one device to perform some or all of the methods disclosed herein. For example, the software may be provided by Figure 1A The control system 110 and other control system components are executed.

[0057] In some examples, apparatus 100 may include Figure 1A Optional microphone system 120 shown in . Optional microphone system 120 may include one or more microphones. According to some examples, optional microphone system 120 may include a microphone array. In some examples, the microphone array may be configured to determine direction of arrival (DOA) and / or time of arrival (TOA) information, for example, according to instructions from control system 110. In some instances, the microphone array may be configured to perform receive-side beamforming, for example, according to instructions from control system 110. In some embodiments, one or more microphones may be part of or associated with another device (such as a speaker of a speaker system, a smart audio device, etc.). In some examples, device 100 may not include microphone system 120. However, in some such embodiments, device 100 may still be configured to receive microphone data corresponding to one or more microphones in an environment or corresponding to one or more microphones in another environment via interface system 110. In some such embodiments, a cloud-based implementation of device 100 may be configured to receive microphone data obtained in one or more microphones in an environment or metadata corresponding to microphone data via interface system 110.

[0058] According to some embodiments, the apparatus 100 may include Figure 1A Optional loudspeaker system 125 is shown in FIG. Optional loudspeaker system 125 may include one or more loudspeakers, which may also be referred to herein as “speakers” or more generally as “audio reproduction transducers”. In some examples (e.g., cloud-based implementations), device 100 may not include loudspeaker system 125.

[0059] In some embodiments, the apparatus 100 may include Figure 1A Optional sensor system 130 shown in . Optional sensor system 130 may include one or more touch sensors, gesture sensors, motion detectors, cameras, eye tracking devices, or a combination thereof. In some embodiments, the one or more cameras may include one or more independent cameras. In some examples, one or more cameras, eye trackers, etc. of optional sensor system 130 may reside in a television, a mobile phone, a smart speaker, or a combination thereof. In some examples, device 100 may not include sensor system 130. However, in some such embodiments, device 100 may still be configured to receive sensor data from one or more sensors (such as cameras, eye trackers, etc.) residing in or on other devices in the environment via interface system 110.

[0060] In some embodiments, the apparatus 100 may include Figure 1A Optional display system 135 shown in . Optional display system 135 may include one or more displays, such as one or more light emitting diode (LED) displays. In some instances, optional display system 135 may include one or more organic light emitting diode (OLED) displays. In some examples, optional display system 135 may include one or more displays of a television, a laptop computer, a mobile device, a smart audio device, or another type of device. In some examples where device 100 includes display system 135, sensor system 130 may include a touch sensor system and / or a gesture sensor system proximate to one or more displays of display system 135. According to some such embodiments, control system 110 may be configured to control display system 135 to present one or more graphical user interfaces (GUIs).

[0061] According to some such examples, the apparatus 100 may be or may include a smart audio device, such as a smart speaker. In some such implementations, the apparatus 100 may be or may include a wake-up word detector. For example, the apparatus 100 may be configured to (at least in part) implement a virtual assistant.

[0062] Figure 1B An example of an audio environment is shown. As with the other figures provided herein, Figure 1B The types, quantities, and arrangements of elements shown in the audio environment 150 are provided only as examples. Other implementations of the audio environment 150 may include more, fewer, and / or different types, quantities, or arrangements of elements.

[0063] In this example, person 102 and person 104 are sitting at table 101, and audio device 111 has been placed on the table. According to this example, audio device 111 is configured to capture audio signals via microphones 103A, 103B, and 103C, including, but not limited to, audio signals corresponding to the speech of person 102 and person 104. The speech of person 102 and person 104 generates direct sound waves 105 and 107, as well as sound waves 106 and 108 reflected from ceiling 112 and other parts of audio environment 100, which are captured by audio device 111.

[0064] According to some examples, the control system 110a of the audio device 111 can be configured to provide an acoustic event detection function, an automatic speech recognition function, and / or a speaker log function. Alternatively or additionally, the control system 110b of the server 116 (with which the audio device 111 is configured to communicate via the network 117) can be configured to provide an acoustic event detection function, an automatic speech recognition function, and / or a speaker log function. The present disclosure discloses methods, devices, and systems that can be used to train a neural network to provide such functions and / or other functions (such as a handwriting recognition function). The present disclosure also discloses a trained neural network configured to provide such functions.

[0065] Figure 2 and Figure 3 1 shows an example of a block that may be included in some disclosed fixed state connection order classification (PS-CSC) implementations. As with other figures provided herein, Figure 2 and Figure 3 The types, quantities, and arrangements of the elements shown in the drawings are provided only as examples. Other implementations may include more, fewer, and / or different types, quantities, and / or arrangements of elements.

[0066] According to this example, Figure 2 As shown by reference Figure 1A 1 and 2. The control system 110 of FIG. 1 is a block diagram of an example implementation of the control system 110 described herein. As described elsewhere herein, in some examples, the control system 110 can reside in more than one device. In this example, the control system 110 implements at least a posterior probability generator 204 and a loss calculator 206. According to this example, the loss calculator 206 is configured to apply a loss function that implements PS-CSC. In some alternative examples, the control system 110 can also implement a feature extractor 202.

[0067] In this example, the a posteriori probability generator 204 is configured to generate a posteriori probability grid 205 . Figure 3 The posterior probability grid 305 shown in FIG. Figure 2 A specific example of the posterior probability grid 205 is shown in FIG. Similarly, Figure 3 The true value label sequence 303 [L1, L2, L3] shown in is Figure 2 An example of the truth label sequence 208 is shown in . The truth label sequence 208 may reside, for example, in a data structure stored in a memory of the control system 110 , or in a memory accessible by the control system 110 .

[0068] According to the example, the posterior probability generator 204 is configured to generate a posterior probability grid 205 from the observation sequence 203. In the example, the observation sequence 203 includes a plurality of extracted features extracted by the feature extractor 202. In the example, each extracted feature generated by the feature extractor 202 corresponds to a sequential signal in the sequential signal sequence 201 and has been generated by the feature extractor 202 from the sequential signal.

[0069] The extracted features in the observation sequence 203 may vary depending on the specific implementation. In some examples, the sequential signal sequence may be a time series of an audio signal. In some such examples, the extracted features in the observation sequence 203 may simply be time samples of the audio signal, while in other examples, the extracted features in the observation sequence 203 may be vector representations of time samples of the audio signal. In some examples, the extracted features may be or may include frequency bands or bin energies. According to some examples, the extracted features may be or may include transform bins stacked across multiple channels.

[0070] However, in other examples, the sequential signal sequence may be a spatial sequence, such as a spatial sequence of handwritten images. In some such examples, the extracted features in the observation sequence 203 may be a vector representation of the segmented input image. In some examples, the extracted features in the observation sequence 203 may be a vector representation of the segmented input image with a specific receptive field that defines the length and width of the image segment.

[0071] If the sequence of sequential signals is a time series of audio signals including human speech, then in some examples, the true value label sequence 208 may be or may include the identity of each speaker in the speaker sequence. The true value label sequence 208 may be represented as [L1, L2, ...], such as [L1, L2, L3], [L1, L2, L3, L4], etc., where each "L" value corresponds to a label, such as a speaker identity.

[0072] According to some examples where the sequential signal sequence is a time series of audio signals including audio signals corresponding to human speech, the posterior probability generator 204 can be configured to generate the posterior probability grid 205 by estimating the probability that each extracted feature in the observation sequence 203 belongs to a category indicated by one of the true value labels in the true value label sequence 208. An example of a category is the identity of each speaker in the speaker sequence. In some such examples where the true value label sequence 208 is [L1, L2, L3, and L4], the posterior probability generator 204 can be configured to generate the posterior probability grid 205 by estimating the probability that each extracted feature in the observation sequence 203 belongs to the category [L1, L2, L3, or L4].

[0073] exist Figure 2 In the example shown, the loss calculator 206 is configured to apply a loss function that implements PS-CSC. Various examples and details are described below. Here, the loss calculator 206 is configured to apply the loss function to the values ​​of the posterior probability grid 205 based on the true value label sequence 208. According to this example, the loss calculator 206 is configured to provide updated parameters based on the loss determined by the loss function, and the updated parameters are Figure 2 205. The loss calculator 206 is represented as a parameter update block 207 in the example. The parameter update block 207 can be, for example, a data structure stored in a memory of the control system 110 or in a memory accessible by the control system. In this example, the loss calculator 206 is configured to provide the updated parameters to the posterior probability generator 204 for generating the posterior probability grid 205. According to some examples, the loss calculator 206 can be configured to provide the updated parameters to the feature extractor 202 for determining a plurality of extracted features in the observation sequence 203.

[0074] A group of extracted features with the same label (such as a group of sequential extracted features corresponding to the same speaker) is an example of what is referred to herein as a "cluster." A transition from one class of extracted features to another class of extracted features (in this example, a transition from extracted features corresponding to one speaker to extracted features corresponding to another speaker) indicates or corresponds to what is referred to herein as a "cluster boundary." For example, a cluster of sequential extracted features corresponding to the same speaker (also called a cluster of sequential observations) can be said to have a first cluster boundary just before the first such extracted feature and a second cluster boundary just before the last extracted feature in the sequence corresponding to the same speaker. For example, referring to Figure 3 , it can be observed that observations 302A and 302B correspond to label / speaker L1, while observation 302C corresponds to label / speaker L2. Therefore, there is a cluster boundary 310 between observation 302B and observation 302C.

[0075] As mentioned above, Figure 3 The true value label sequence [L1, L2, L3] shown in is Figure 2 302A to 302N. Each circle represents the probability that the corresponding observation belongs to the category L1, L2, L3 or L4.

[0076] In this example, the base labels are monotonically aligned with the observations. For example, an observation that follows another observation that is aligned with L2 will not be aligned with L1. Instead, the observations can only be aligned with L2 or L3.

[0077] according to Figure 3 In the example shown in , the solid black circles correspond to "fixed states", which means that the state or label of each corresponding observation is known in advance. Conversely, the unfilled circles correspond to "normal states", which means that the state or label of each corresponding observation is unknown in advance. Because the states of sequential observations 302B and 302C are known, and because the known base labels are monotonically aligned with the observations, there is only one valid path between the states of sequential observations 302B and 302C within the posterior probability grid 305. Because the states of observations 302C and 302E are also known, there are only two valid paths between the states of observations 302C and 302E within the posterior probability grid 305. If there is no fixed state information or cluster boundary information for different categories, there will be a greater number of potential valid paths between the states of observations within the posterior probability grid 305.

[0078] Therefore, PS-CSC implementations may result in faster convergence and lower computational overhead. Furthermore, the posterior probability generator 204 does not need to determine the probability of observations with fixed states. Therefore, the fixed states of some PS-CSC implementations may result in lower computational overhead. Figure 3 The potential advantages of PS-CSC implementations are described (or at least demonstrated).

[0079] As mentioned above, in reference Figure 2 and Figure 3 In the described example, the loss calculator 206 is configured to apply a loss function implementing PS-CSC. Some examples are described below.

[0080] PS-CSC loss function example

[0081] Assume that there is only one observation sequence X = [x1, x2, x3, ... x N ](For example, Figure 2), the observation sequence is generated by a feature extractor (such as feature extractor 202) from the corresponding sequential signal. In some examples, there may be multiple observation sequences. In some such examples, the loss function may provide a sum of individual loss values. However, without loss of generality, for simplicity, the following examples involve one observation sequence. In one such example, the corresponding true value label is [L1, L2, L3], which is Figure 3 The true value label sequence 303 shown in . According to the following example, the path through the posterior probability grid is represented as π. One possible path would be π1 = [L1, L1, L2, L2...L3]. By applying the independence assumption (which can be the same independence assumption as the connection time classification (CTC)), the probability of the entire path can be expressed as follows:

[0082]

[0083] In equation (1), s∈[1, N] represents the sequence observation index, represents the sth alignment label in path π1, and Indicates that from x s Become a path when observing In other words, the length of the path π1 is the same as the number of vectors in X, which can be represented as [x1, x2, ..., x N ]. In the path π1, each vector x s With label For example, in the π1 path, x 1 Marked as That is L1. But for another path, for example path π2, x 1 can be labeled differently. These probabilities can be represented, for example, by Figure 2 The posterior probability extractor 204 generates.

[0084] exist Figure 3 In the example, and

[0085] In one example, the loss function can be expressed as the sum of all possible paths as follows:

[0086]

[0087] In equation (2), k represents the index of a valid path.

[0088] Figure 3The fixed state shown means that a specific state must be passed through a valid path. In this example, the fixed state is given as prior knowledge. The number of valid paths will decrease due to the fixed state. One might think of the CTC loss as an extreme instance of PS-CSC, where there is no fixed state.

[0089] For the other extreme case where the alignment of every observation in a given sequence of observations is known, there is only one possible path. For simplicity, we can call this single path π ce The PS-CTC objective function can be expressed as follows:

[0090]

[0091] It can be observed that when y s When is a one-hot vector, Equation 3 is the same as the cross-entropy loss.

[0092] Speaker log example

[0093] Conventionally, in speaker recognition systems (including speaker recognition and verification), an embedding extractor is first trained. This is the de-factoring process disclosed in Kinnunen, Tomi and Haizhou Li, "An overview of text-independent speaker recognition: From features to supervectors", Speech Communications 52, No. 1 (2010), pp. 12-40, and the i-vector system disclosed in Dehak, Najim et al., "Front-end factor analysis for speaker verification". IEEE Transactions on Audio, Speech and Language Processing 19, No. 4 (2010): pp. 788-798, as well as those systems that use neural networks as extractors (such as Snyder, David et al., "X-vectors: Robust DNN embeddings for speaker verification"). recognition[X-Vectors: Robust DNN Embeddings for Speaker Recognition]", 2018 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), pp. 5329-5333. (IEEE, 2018)). Next, cosine distance similarity or probabilistic linear discriminant analysis (PLDA) is usually applied or trained to measure the similarity between two embeddings.

[0094] These embedding extractors are trained using cross entropy loss via an objective function such as cross entropy with SoftMax or its variants (e.g., angular SoftMax, additive marginal SoftMax, etc.), which maps aggregated utterance-level embeddings to their corresponding class labels. This means that the extractor learns to accumulate identity-related information in an utterance.

[0095] In some use cases, such as online speaker logs, the goal is to identify who is speaking and when. This process usually occurs in a limited duration (which can be a few seconds or less), which means that the amount of information to be collected is also limited. The duration is limited by the tolerance interval between the boundaries of two decisions, which is typically 250 to 300 milliseconds. In addition, in the aforementioned example, in the log scenario, the embedding extractor is not trained in a sequential manner.

[0096] Some public examples involve using PS-CSC to learn both speaker identity and temporal alignment. The loss applied by the loss function can allow the neural network to learn how to map sequential embeddings to sequential labels, which can be considered a sequence-to-sequence problem. Unlike the speech recognition use case, if two segments are from the same speaker, no gaps are required because the two segments share the same speaker label. Since the boundaries of some labels may be known in advance, this boundary information can be provided to PS-CSC before the training process, which is not possible in CTC or cross entropy loss.

[0097] Figure 4 Figure 1 shows an example of the blocks involved in training a neural network for the speaker log use case using PS-CSC. As with the other figures provided in this article, Figure 4 The types, quantities, and arrangements of the elements shown in the drawings are provided only as examples. Other implementations may include more, fewer, and / or different types, quantities, and / or arrangements of elements.

[0098] According to this example, Figure 4 As shown by reference Figure 1A 1 and 10. As described elsewhere herein, in some instances, the control system 110 may reside in more than one device. In this example, the control system 110 is configured to implement at least a posterior probability generator 404, a loss calculator 406, and a parameter update block 407, which are respectively Figure 2 2, and the like. In some alternative examples, the control system 110 may also implement an embedding extractor 402, which is an example of a posterior probability generator 204, a loss calculator 206, and a parameter update block 207. According to this example, the loss calculator 406 is configured to apply a loss function that implements PS-CSC. Figure 2 An instance of the feature extractor 202.

[0099] In this example, the input sequential signal 401 is an audio data sample that includes audio signals corresponding to the speech of speakers or speakers spk_684, spk_379, spk_167, spk_1195, and spk_527 in order. Therefore, the true value label sequence 408 includes labels L1, L2, L3, L4, and L5, corresponding to speakers spk_684, spk_379, spk_167, spk_1195, and spk_527, respectively. According to this example, the embedding extractor 402 generates an observation sequence 403 from the input sequential signal 401, which is a speaker embedding corresponding to the speech of speakers spk_684, spk_379, spk_167, spk_1195, and spk_527.

[0100] According to the example, the posterior probability generator 404 is configured to generate a posterior probability grid 405. In the example, the posterior probability generator 404 is configured to generate the posterior probability grid 405 by estimating the probability that each extracted feature in the observation sequence 403 belongs to the category indicated by one of the true value labels in the true value label sequence 408. In the example, the true value label sequence 408 is [L1, L2, L3, L4, L5], and the posterior probability generator 404 is configured to generate the posterior probability grid 405 by estimating the probability that each extracted feature in the observation sequence 403 belongs to the category L1, L2, L3, L4 or L5.

[0101] exist Figure 4 In the example shown, the loss calculator 406 is configured to apply a loss function that implements PS-CSC. Here, the loss calculator 406 is configured to apply the loss function to the values ​​of the posterior probability grid 405 according to the true value label sequence 408. According to this example, the loss calculator 406 is configured to provide updated parameters according to the loss determined by the loss function, and the updated parameters are Figure 4 407. In this example, the loss calculator 406 is configured to provide the updated parameters to the posterior probability generator 404 for generating the posterior probability grid 405. According to some examples, the loss calculator 406 may be configured to provide the updated parameters to the embedding extractor 402 for determining a plurality of extracted features in the observation sequence 403.

[0102] Experimental Results

[0103] The inventors have conducted experiments to show the potential of PS-CSC in a speaker log scenario. Figure 5A , Figure 5B and Figure 5CThe experimental results involving conventional methods in the speaker logging use case are shown. FIG. 5A to FIG. 5C In the examples shown, the embedding extractor has been trained with the cross entropy loss. In these examples, no sequential information was introduced during the training process. Fig. 6A and Figure 6B The experimental results involving the PS-CSC method in the speaker log use case are shown. Fig. 6A and Figure 6B In the example represented by , the sequential information is provided during the training process and the embedding extractor is trained using CTC. FIG. 5A to FIG. 5C as well as Fig. 6A and Figure 6B The methods represented share the same dataset and model. In each case, a SoftMax score is then used to indicate the quality of the extracted speaker embeddings for the log.

[0104] In by FIG. 5A to FIG. 5C In the experiments presented, no sequence boundary information was provided during training. Using PS-CSC, separability is improved and training time is reduced, both of which come at least in part from the sequence boundary information provided during the training process.

[0105] In by FIG. 5A to FIG. 5C In the experiments represented, the true label sequence is [spk_684, spk_379, spk_167, spk_1195, spk_527]. Figure 5A An example input audio clip 503 in the time domain is shown. Figure 5B A graph 502 is shown, including curves 505A, 505B, 505E, 505E, and 505E, which respectively represent the SoftMax scores of the speech of speakers spk_684, spk_379, spk_167, spk_1195, and spk_527 when the neural network is trained with the CE cost function but without order information. Figure 5B As can be seen in Figure 2, a neural network trained without sequential information cannot predict speaker alignment well.

[0106] Figure 5C A graph 501 is shown, including curves 504A, 504B, 504E, 504E, and 504E, which respectively represent the SoftMax scores of the speech of speakers spk_684, spk_379, spk_167, spk_1195, and spk_527 when the neural network is trained with a sequential loss cost function (CTC). By comparing Figure 5C and Figure 5B, it can be observed that the neural network performs much better when sequential information is provided during the training process. For example, it can be clearly seen from graph 501 that frames from time unit 0 to 5 are aligned with spk_684 with high probability. However, in graph 502, there is no clear trend.

[0107] Depend on FIG. 5A to FIG. 5C The presented experiments show that using sequential information when training neural networks is highly useful in the speaker logging use case.

[0108] Fig. 6A and Figure 6B The advantage of using order information and alignment information in a PS-CSC implementation is shown.With some a priori alignment information (also referred to herein as fixed state information), the number of valid paths through the posterior possibility grid that need to be calculated during training will be reduced to a smaller number.

[0109] Fig. 6A An example input audio segment 603A in the time domain is shown, along with alignment information 610A and 610B. Fig. 6A It also shows Figure 2 An example of a posteriori likelihood grid 205, which in this example is a posteriori likelihood grid 605A. According to this example, Figure 3 As shown, the solid black circles in the posterior likelihood grid 605A indicate fixed states, each of which corresponds to one of the instances of alignment information 610A and 610B. In this example, only the alignments corresponding to spk_684 and spk_379 are known before the neural network training process. Fig. 6A Regions 604A and 604B within the a posteriori likelihood grid 605A are shown. In this example, regions 604A and 604B are regions where valid paths through the a posteriori likelihood grid 605A may exist given known alignment information 610A and 610B.

[0110] Figure 6B Also shown is an example input audio segment 603B in the time domain, but without alignment information. Figure 6BA posterior possibility grid 605B is also shown. According to this example, there are no solid black circles indicating a fixed state in the posterior possibility grid 605A because no alignment information is provided. In this example, a valid path through the posterior possibility grid 605B can exist within the region 602 because no prior alignment information is provided before training. It can be observed that even if only alignment information 610A and 610B are given, regions 604A and 604B are much smaller than region 602. Therefore, since no prior alignment information is provided before training, the number of valid paths through the posterior possibility grid 605B is much greater than the number of valid paths through the posterior possibility grid 605A.

[0111] Figure 7 Results of two additional experiments are shown. In this example, graph 700 indicates the loss determined by the loss function on the vertical axis and the number of iterations of the neural network training process on the horizontal axis. Based on this example, curve 705 indicates the loss value corresponding to the CTC-based neural network training process, while curve 710 indicates the loss value corresponding to the PS-CSC-based neural network training process. It can be observed that after approximately 45,000 iterations, the loss corresponding to the PS-CSC-based neural network training process is always less than the loss corresponding to the CTC-based neural network training process.

[0112] Figure 8 800 is a flowchart outlining an example of the disclosed method. As with other methods described herein, the blocks of method 800 need not be executed in the order indicated. According to some examples, one or more blocks may be executed in parallel. In addition, some similar methods may include more or fewer blocks than those shown and / or described.

[0113] Method 800 may be performed as follows: Figure 1A 1 and described above. In some examples, the apparatus 100 includes at least the control system 110 disclosed herein. In some examples, at least some aspects of the method 800 may be performed by one or more devices within the audio environment, for example, by an audio system controller (such as a device that may be referred to as a smart home hub herein) or by another component of the audio system (such as a television, a television control module, a laptop computer, a mobile device (such as a cellular phone), etc.). However, in some embodiments, at least some blocks of the method 800 may be performed by one or more devices (such as one or more servers) configured to implement a cloud-based service.

[0114] In this example, box 805 involves receiving a sequence of observations by a control system, the sequence of observations comprising a plurality of extracted features. According to some examples, each of the plurality of extracted features may correspond to a sequential signal in a sequence of sequential signals. In some examples, the sequence of sequential signals may be a time series comprising an audio signal, which in some instances may include an audio signal corresponding to speech. In some alternative examples, the sequence of sequential signals may be a time series comprising other types of audio signals, signals corresponding to stock market data, signals corresponding to biological data, and the like. According to some alternative examples, the sequence of sequential signals may be a spatial sequence, such as a spatial sequence comprising a handwritten image, a DNA sequence image, or other images.

[0115] According to this example, block 810 involves determining, by the control system, a posterior likelihood grid. In some examples, the grid may include a likelihood that each observation sequence corresponds to one of a plurality of label categories. In some examples, the label category may correspond to the identity of each of a plurality of speakers.

[0116] In this example, box 815 involves applying a loss function to the posterior likelihood grid based on the true values ​​by the control system. According to this example, applying the loss function involves applying both sequential information and cluster boundary information. According to some examples, the sequential information and / or the cluster boundary information of the cost function may be variable. In some examples, the cluster boundary information may be incomplete cluster boundary information. However, in some alternative examples, the cluster boundary information may be complete cluster boundary information. In some examples, the loss function may combine elements of connection time classification and cross entropy loss. According to some examples, applying the loss function may involve determining one or more valid paths between observations in the grid.

[0117] According to this example, block 820 involves updating, by the control system, parameters used to determine the posterior likelihood grid according to the loss determined by the loss function. In some examples, block 820 may involve updating, by the control system, parameters used to determine the posterior likelihood grid according to the parameters of parameter update block 207. Figure 2 Parameters used by the posterior probability generator 204.

[0118] In this example, block 825 involves continuing to execute blocks 805 to 820 until the control system determines that one or more convergence criteria are met. In some examples, the control system 110 can be configured to determine that convergence has been reached when the training process reaches a state where the loss determined by the loss function stabilizes within an error range around the final value, or reaches a state where the loss determined by the loss function no longer decreases or the loss determined by the loss function does not decrease within a predetermined number of steps or epochs.

[0119] In some examples, executing blocks 805 through 825 may provide for a trained neural network to be implemented by control system 110. In some examples, method 800 may involve performing one or more downstream tasks via the trained neural network.

[0120] According to some examples, the sequential signal sequence may be or may include an audio signal sequence.In some such examples, the one or more downstream tasks may include acoustic event detection, speaker logging, automatic speech recognition, or a combination thereof.

[0121] In some examples, method 800 may involve determining, by a control system, a plurality of extracted features. In some such examples, the method may involve updating, by the control system, parameters used to determine the plurality of extracted features based on a loss determined by a loss function.

[0122] In some examples, the sequence of sequential signals may be or may include a sequence of handwritten images. In some such examples, one or more downstream tasks may include handwriting recognition.

[0123] Some aspects of the present disclosure include a system or device configured (e.g., programmed) to perform one or more examples of the disclosed methods, and a tangible computer-readable medium (e.g., a disk) storing code for implementing one or more examples of the disclosed methods or steps thereof. For example, some disclosed systems may be or include a programmable general-purpose processor, a digital signal processor, or a microprocessor that is programmed with software or firmware and / or otherwise configured to perform any of a variety of operations on data, including embodiments of the disclosed methods or steps thereof. Such a general-purpose processor may be or include a computer system that includes an input device, a memory, and a processing subsystem that is programmed (and / or otherwise configured) to perform one or more examples of the disclosed methods (or steps thereof) in response to data asserted thereto.

[0124] Some embodiments may be implemented as a configurable (e.g., programmable) digital signal processor (DSP), which is configured (e.g., programmed and otherwise configured) to perform the required processing on (multiple) audio signals, including the execution of one or more examples of the disclosed method. Alternatively, embodiments of the disclosed system (or its elements) may be implemented as a general-purpose processor (e.g., a personal computer (PC) or other computer system or microprocessor, which may include an input device and a memory), which is programmed with software or firmware and / or otherwise configured to perform any of various operations, including one or more examples of the disclosed method. Alternatively, the elements of some embodiments of the system of the present invention are implemented as a general-purpose processor or DSP configured (e.g., programmed) to perform one or more examples of the disclosed method, and the system also includes other elements (e.g., one or more loudspeakers and / or one or more microphones). The general-purpose processor configured to perform one or more examples of the disclosed method may be coupled to an input device (e.g., a mouse and / or keyboard), a memory, and a display device.

[0125] Another aspect of the present disclosure is a computer-readable medium (e.g., a disk or other tangible storage medium) storing code for executing one or more examples of the disclosed method or its steps (e.g., an encoder executable to execute one or more examples of the disclosed method or its steps).

[0126] Although specific embodiments of the present disclosure and applications of the present disclosure have been described herein, it will be apparent to those skilled in the art that many changes may be made to the embodiments and applications described herein without departing from the scope of the present disclosure as described and claimed herein. It should be understood that although certain forms of the present disclosure have been shown and described, the present disclosure is not limited to the specific embodiments described and shown or the specific methods described.

Claims

1. A method comprising: (a) receiving, by a control system, a sequence of observations, the sequence of observations comprising a plurality of extracted features, each of the plurality of extracted features corresponding to a sequential signal in a sequence of sequential signals; (b) determining, by the control system, a posterior likelihood grid, the grid including the likelihood that each observation sequence corresponds to one of a plurality of label categories; (c) applying, by the control system, a loss function to the posterior likelihood grid based on true values, wherein applying the loss function involves applying both sequential information and cluster boundary information; (d) updating, by the control system, parameters used to determine the posterior likelihood grid according to the loss determined by the loss function; and (e) Continue executing (a) through (d) until the control system determines that one or more convergence criteria are satisfied.

2. The method of claim 1, wherein: Performing (a) through (e) provides a trained neural network implemented by the control system.

3. The method of claim 2, further comprising performing one or more downstream tasks via the trained neural network.

4. The method of claim 3, wherein: The sequential signal sequence comprises an audio signal sequence, and wherein the one or more downstream tasks comprise at least one of acoustic event detection, speaker logging, and automatic speech recognition.

5. The method of claim 3, wherein: The sequence of sequential signals comprises a sequence of handwriting images, and wherein the one or more downstream tasks comprises handwriting recognition.

6. The method according to any one of claims 1 to 5, wherein: The cluster boundary information includes incomplete cluster boundary information.

7. The method according to any one of claims 1 to 6, wherein: The cluster boundary information includes complete cluster boundary information.

8. The method according to any one of claims 1 to 7, wherein: Applying the loss function involves determining one or more valid paths between observations in the grid.

9. The method of any one of claims 1 to 8, further comprising determining, by the control system, the plurality of extracted features.

10. The method of claim 9, further comprising updating, by the control system, parameters used to determine the plurality of extracted features according to the loss determined by the loss function.

11. The method according to any one of claims 1 to 10, wherein: The loss function combines elements of join-time classification and cross entropy loss.

12. The method according to any one of claims 1 to 11, wherein: The order information of the cost function, the cluster boundary information of the cost function, or both are variable.

13. An apparatus configured to implement one or more of the methods of claims 1 to 12.

14. A system configured to implement one or more of the methods of claims 1 to 12.

15. One or more non-statutory media, the one or more non-statutory media having instructions for controlling one or more devices to perform one or more of the methods of claims 1 to 12.

Images