Linearization of nonlinear transformed signals

Abstract

A device includes one or more processors configured to perform signal processing including a linear transformation and a nonlinear transformation of an input signal to generate a reference target signal. The reference target signal has a linear component associated with the linear transformation and a nonlinear component associated with the nonlinear transformation. The one or more processors are further configured to perform linear filtering of the input signal by controlling adaptation of the linear filtering to generate an output signal that substantially matches the linear component of the reference target signal.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This patent application claims priority to non-provisional application No. 16 / 685,987, filed on November 15, 2019, entitled “LINEARIZATION OF NON-LINEARLY TRANSFORMED SIGNALS,” which has been assigned to the assignee of the present application and is expressly incorporated by reference into this application. Technical Field

[0003] The present disclosure relates generally to signal processing. Background Art

[0004] Advances in technology have resulted in smaller and more powerful computing devices. For example, there are currently a variety of portable personal computing devices, including wireless phones such as mobile phones and smart phones, tablet computers, and notebook computers that are small, lightweight, and easy for users to carry. These devices can communicate voice and data packets over wireless networks. In addition, many such devices incorporate additional functionality, such as digital cameras, digital video cameras, digital recorders, and audio file players. In addition, such devices can process executable instructions, including software applications that can be used to access the Internet, such as web browser applications. Therefore, these devices can include significant computing capabilities.

[0005] The computing device may perform a transformation of the input signal for signal enhancement, such as enhancing speech and suppressing noise in an input audio signal. For example, the computing device may perform a transformation of the input signal, which results in an enhanced signal, the enhanced signal being characterized by including a linear component indicating a linear transformation of the input signal and a nonlinear component indicating a nonlinear transformation of the input signal. The enhanced signal is effective for certain implementations, such as a speech enhanced audio signal played to a person via a speaker during a phone call. However, such an enhanced signal typically results in a reduction in performance in a speech recognition or voice verification / authentication implementation. Summary of the invention

[0006] In a particular aspect, a device for performing signal processing includes one or more processors configured to perform signal processing including a linear transformation and a nonlinear transformation of an input signal to generate a reference target signal. The reference target signal has a linear component associated with the linear transformation and a nonlinear component associated with the nonlinear transformation. The one or more processors are further configured to perform linear filtering of the input signal by controlling adaptation of the linear filtering to generate an output signal that substantially matches the linear component of the reference target signal.

[0007] In another specific aspect, a method of signal processing includes performing signal processing including a linear transformation and a nonlinear transformation of an input signal to generate a reference target signal. The reference target signal has a linear component associated with the linear transformation and a nonlinear component associated with the nonlinear transformation. The method also includes performing linear filtering of the input signal by controlling adaptation of the linear filtering to generate an output signal that substantially matches the linear component of the reference target signal.

[0008] In another specific aspect, a computer-readable storage medium stores instructions that, when executed by one or more processors, cause the one or more processors to perform signal processing including a linear transformation and a nonlinear transformation of an input signal to generate a reference target signal. The reference target signal has a linear component associated with the linear transformation and a nonlinear component associated with the nonlinear transformation. The instructions also cause the one or more processors to perform linear filtering of the input signal by controlling adaptation of the linear filtering to generate an output signal that substantially matches the linear component of the reference target signal.

[0009] In another specific aspect, an apparatus includes means for performing signal processing including a linear transformation and a nonlinear transformation of an input signal to generate a reference target signal. The reference target signal has a linear component associated with the linear transformation and a nonlinear component associated with the nonlinear transformation. The apparatus also includes means for performing linear filtering of the input signal by controlling adaptation of the linear filtering to generate an output signal that substantially matches the linear component of the reference target signal.

[0010] Other aspects, advantages, and features of the present disclosure will become apparent after reviewing the entire application, which includes the following sections: a brief description of the drawings, a detailed description, and the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is a block diagram of certain illustrative aspects of a system operable to perform linearization of a nonlinearly transformed signal according to some examples of the present disclosure;

[0012] Figure 2 is a diagram of an illustrative example of a system operable to perform linearization of a non-linearly transformed audio signal according to some examples of the present disclosure;

[0013] Figure 3 is a diagram of illustrative examples of device components operable to perform linearization of a non-linearly transformed audio signal according to some examples of the present disclosure;

[0014] Figure 4 is a diagram of illustrative examples of device components operable to perform linearization of a nonlinearly transformed signal according to some examples of the present disclosure;

[0015] Figure 5 is a diagram of an illustrative example of a system operable to perform linearization of a non-linearly transformed visual input signal according to some examples of the present disclosure;

[0016] Figure 6 is a flow chart illustrating a method of performing linearization of a nonlinearly transformed input signal according to some examples of the present disclosure;

[0017] Figure 7 is an illustration of a vehicle operable to perform linearization of a nonlinearly transformed signal according to some examples of the present disclosure;

[0018] Fig. 8A is an illustration of a virtual reality or augmented reality headset operable to perform linearization of a non-linearly transformed signal according to some examples of the present disclosure;

[0019] Figure 8B is an illustration of a wearable electronic device operable to perform linearization of a nonlinearly transformed signal according to some examples of the present disclosure; and

[0020] Fig. 9 is a block diagram of a specific illustrative example of an apparatus operable to perform linearization of a nonlinearly transformed input signal in accordance with some examples of the present disclosure. DETAILED DESCRIPTION

[0021] Disclosed herein are systems and methods for performing linearization of a nonlinearly transformed input signal. A computing device includes a signal processing unit that performs signal enhancement (e.g., speech enhancement, audio enhancement, noise suppression, image enhancement, or a combination thereof). Signals that include nonlinear components that are enhanced in a conventional manner are effective for certain implementations, such as a speech enhancement audio signal played to a person via a speaker during a telephone call. However, such conventional enhanced signals typically result in reduced performance in speech recognition or voice verification / authentication implementations.

[0022] According to various aspects of the disclosed systems and methods, a signal processing unit performs a linear transformation and a nonlinear transformation of an input signal (e.g., a speech signal, an audio signal, a visual signal, a data signal, etc.), resulting in an enhanced signal including a linear component and a nonlinear component, the linear component being caused by one or more linear transformations and the nonlinear component being caused by one or more nonlinear transformations. A linearizer performs linear filtering of the input signal by controlling the adaptation of the linear filtering to generate an output signal that substantially matches the linear component of the enhanced signal. The output signal is provided to a signal processing component, such as a speech recognition or voice verification component. Because the output signal substantially matches the linear component of the enhanced signal and excludes the nonlinear component of the enhanced signal, processing the output signal (compared to the enhanced signal) improves the accuracy of the signal processing component compared to a signal enhanced in a conventional manner.

[0023] Certain aspects of the present disclosure are described below with reference to the accompanying drawings. In the description, common features are represented by common reference numerals. As used herein, various terms are used only for the purpose of describing a particular implementation and are not intended to limit the implementation. For example, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. In addition, some features described herein are singular in some implementations and plural in other implementations. For example, Figure 1 Depicts a system including one or more processors ( Figure 1 108), indicating that in some implementations, the device 102 includes a single processor 108, while in other implementations, the device 102 includes multiple processors 108. For ease of reference herein, these features are generally introduced as "one or more" features and are subsequently referred to in the singular unless aspects related to multiple of the feature are being described.

[0024] It will be further understood that the terms "some include", "one includes" and "including..." can be used interchangeably with "some include", "one includes" or "including...". In addition, it will be understood that the term "wherein" can be used interchangeably with "herein". As used herein, "exemplary" can indicate examples, implementations and / or aspects, and should not be interpreted as limiting or indicating a preferred or preferred implementation. As used herein, ordinal numbers (e.g., "first", "second", "third", etc.) used to modify elements such as structures, components, operations, etc. do not themselves indicate any priority or order of an element relative to another element, but simply distinguish the element from another element with the same name (but using ordinal terms). As used herein, the term "set" refers to one or more specific elements, while the term "plurality" refers to multiple (e.g., two or more) specific elements.

[0025] As used herein, "coupling" may include "communicatively coupled," "electrically coupled," or "physically coupled," and may also (or alternatively) include any combination thereof. Two devices (or components) may be coupled directly (e.g., communicatively coupled, electrically coupled, or physically coupled) or indirectly coupled via one or more other devices, components, wires, buses, networks (e.g., wired networks, wireless networks, or combinations thereof), etc. As illustrative, non-limiting examples, two electrically coupled devices (or components) may be included in the same device or in different devices, and may be connected via electronic devices, one or more connectors, or inductive coupling. In some implementations, two devices (or components) such as those communicatively coupled in electrical communication may send and receive electrical signals (digital signals or analog signals) directly or indirectly (such as via one or more wires, buses, networks, etc.). As used herein, "direct coupling" may include two devices that are coupled (e.g., communicatively coupled, electrically coupled, or physically coupled) without intervening components.

[0026] In the present disclosure, terms such as "determine", "calculate", "estimate", "shift", "adjust", etc. may be used to describe how to perform one or more operations. It should be noted that these terms should not be interpreted as limiting, but other technologies may be used to perform similar operations. Additionally, as mentioned herein, "generate", "calculate", "estimate", "use", "select", "access", and "determine" may be used interchangeably. For example, "generating", "calculating", "estimating" or "determining" a parameter (or signal) may refer to actively generating, estimating, calculating or determining the parameter (or signal), or may refer to using, selecting or accessing a parameter (or signal) that has already been generated, such as by another component or device.

[0027] refer to Figure 1 , certain illustrative aspects of a system operable to perform linearization of a nonlinearly transformed signal are disclosed, and the system is generally designated 100. The system 100 includes a device 102, which includes one or more processors 108. The processor 108 includes a signal processing unit 120 coupled to a linearizer 130. The signal processing unit 120, the linearizer 130, or both are coupled to one or more components 150. The signal processing unit 120 is configured to perform signal enhancement on an input signal 114 ("y"). For example, the signal processing unit 120 is configured to perform signal processing on the input signal 114 ("y") to generate a reference target signal 122 exist Figure 1 In the example shown, the signal processing unit 120 includes a neural network 124, and the signal processing unit 120 is configured to use the neural network 124 to perform signal processing on an input signal ("y") to generate a reference target signal 122. It should be understood that performing signal processing using neural network 124 is provided as an illustrative example. In some implementations, signal processing unit 120 is configured to perform signal processing independent of any neural network.

[0028] The linearizer 130 includes a linear filter 134 and a filter adapter 136. The linear filter 134 is configured to perform linear filtering of the input signal 114 ("y") to generate an output signal 132 ("h*y", where "h" is the filter response of the linear filter 134 and "*" indicates convolution). The filter adapter 136 is configured to control the adaptation of the linear filtering in response to the reference target signal 122. For example, the filter adapter 136 attempts to control the adaptation of the linear filtering so that the output signal 132 matches the reference target signal 122 as closely as possible. As a result of applying the linear filtering (and omitting the nonlinear filtering), the output signal 132 substantially matches the linear components of the reference target signal 122 and substantially omits the nonlinear components of the reference target signal 122. In particular aspects, the processing performed by the system 100 (e.g., the signal processing unit 120, the linearizer 130, or both) can occur in the time domain, in a transform domain (e.g., the frequency domain), or a combination thereof. For example, the signal processing unit 120, the linearizer 130, or both can process a time domain signal (e.g., the input signal 114) received from a microphone or camera. In another example, the processor 108 can include a first fast Fourier transform (FFT) module (not shown) configured to apply a transform to the time domain signal to generate a transform domain signal, and the signal processing unit 120, the linearizer 130, or both can process the transform domain signal. The processor 108 can include a second FFT module configured to apply an inverse transform to the reference target signal 122, the output signal 132, or both before further processing by one or more components 150. For example, a first component of the components 150 can receive the output signal 132 in the transform domain, and a second component of the components 150 can receive the output signal 132 in the time domain.

[0029] During operation, the processor 108 receives (e.g., accesses) an input signal 114 ("y"). The input signal 114 ("y") includes a portion of interest ("x") and a noise portion ("n"). In a specific example, the input signal 114 ("y") includes an audio signal and the component 150 includes a speech processing component. In this example, the portion of interest ("x") includes a speech portion and the noise portion ("n") includes audio noise, as shown in FIG. Figure 2 The signal processing unit 120 performs signal processing on the input signal 114 (“y”) to generate a reference target signal 122 In a specific example, the signal processing unit 120 performs signal processing on the input signal 114 (“y”) using the neural network 124 to generate a reference target signal 122 Reference target signal 122 The signal enhancement corresponds to the input signal 114 ("y") and may be generated by enhancing the portion of interest ("x"), reducing the noise portion ("n"), or a combination of both.

[0030] Signal processing includes linear transformation 126 and nonlinear transformation 128 that result in signal enhancement. The signal processing unit 120 may generate the reference target signal 122 by: having a linear component associated with (e.g., generated by) the linear transformation 126 and a nonlinear component associated with (e.g., generated by) the nonlinear transformation 128. Although the linear transformation 126 and the nonlinear transformation 128 are depicted as separate components, the signal processing unit 120 may generate the reference target signal 122 The linear and nonlinear components of the reference target signal 122 can be processed by the neural network 124 by performing operations in which the linear transformation and the nonlinear transformation are not clearly distinguished to generate the reference target signal 122. In a specific example, the reference target signal 122 corresponding to the portion of interest (“x”) and the distorted portion (e.g. where “f(x)” corresponds to the distorted part).

[0031] The linearizer 130 receives the reference target signal 122 from the signal processing unit 120. In response to the reference target signal 122 The linearizer 130 generates a reference target signal 122 The linear components of the output signal 132 (“h*y”) are substantially matched. For example, the linear filter 134 performs linear filtering on the input signal 114 (“y”) to generate the output signal 132 (“h*y”). In response to the reference target signal 122, the filter adapter 136 controls the adaptation of the linear filtering (performed by the linear filter 134) so ​​that the output signal 132 substantially matches the linear components of the reference target signal 122. For example, the filter adapter 136 may select or update a set of filtering values ​​of the linear filter 134 so that the output signal 132 (“h*y”) and the reference target signal 122 are substantially matched. The difference between L1 norm In certain aspects, because “h” is the filter response of the linear filter 134, the output signal 132 (“h*y”) and the reference target signal 122 are made The difference between the two is reduced or minimized, resulting in the output signal 132 corresponding to the reference target signal 122 The linear component of .

[0032] In certain aspects, component 150 is configured to process output signal 132 ("h*y") to generate output 153 corresponding to input signal 114. Figure 2 and Figure 5 An example of describing the component 150, since the output signal 132 corresponds to the linear component of the reference target signal 122 and excludes the nonlinear component of the reference target signal 122, is similar to processing the reference target signal 122 In comparison, component 150 can obtain more accurate results by processing output signal 132 ("h*y"). Therefore, system 100 improves signal processing of component 150 by providing the linear component of the nonlinearly transformed signal (e.g., reference target signal 122) to component 150. System 100 enables component 150 to take advantage of signal enhancement performed by signal processing unit 120 while maintaining accuracy by processing the linear component of the enhanced signal.

[0033] refer to Figure 2 , discloses certain illustrative aspects of a system operable to perform linearization of a non-linearly transformed audio signal and is generally designated 200 . In certain aspects, system 100 includes one or more components of system 200 .

[0034] Device 102 includes one or more microphones 212. Microphone 212 receives input sound including speech 206 from user 204 and noise (e.g., background noise). Microphone 212 generates an audio input signal 214 ("y") including a speech portion ("x") and a noise portion ("n"). In certain aspects, audio input signal 214 corresponds to Figure 1 The input signal 114 ("y").

[0035] The signal processing unit 120 performs signal processing on the audio input signal 214 (“y”) to generate a reference target signal 122 In certain aspects, the signal processing performed at the signal processing unit 120 results in audio signal enhancement, such as echo cancellation, noise suppression, dereverberation, or a combination thereof. In certain aspects, the linear transformation 126 and the nonlinear transformation 128 result in speech enhancement. The linear filter 134 performs linear filtering (such as finite impulse response (FIR) filtering) of the audio input signal 214 ("y") to generate an output signal 132 ("h*y").

[0036] Component 150 includes one or more audio signal processing components, such as a speech recognition engine 252, a voice recognition engine 254, a voice activation component 256, a speaker verification component 258, or a combination thereof. In a particular example, speech recognition engine 252 processes output signal 132 to generate output 153. By way of example, output 153 includes speech recognition output, such as text output corresponding to speech detected in output signal 132. In a particular aspect, because output signal 132 includes (e.g., substantially matches) linear components of reference target signal 122 and excludes (e.g., substantially omits) nonlinear components of reference target signal 122, output 153 is a more accurate representation of speech 206 (compared to speech recognition output that can be generated from reference target signal 122). In a particular aspect, speech recognition engine 252 provides output 153 to a display device that displays output 153. For example, in some implementations, device 102 includes a display device.

[0037] In a particular example, the speech recognition engine 254 processes the output signal 132 to generate the output 153. By way of example, the output 153 includes a speech recognition output, such as a user identifier corresponding to a determination that the speech detected in the output signal 132 matches the speech of the user associated with the user identifier. In particular aspects, because the output signal 132 includes (e.g., substantially matches) linear components of the reference target signal 122 and excludes (e.g., substantially omits) nonlinear components of the reference target signal 122, the output 153 is a more accurate identification of the user corresponding to the speech 206 (compared to the user that would be identified based on detecting the speech in the reference target signal 122).

[0038] In a particular example, the voice activation component 256 processes the output signal 132 to generate the output 153. By way of example, the output 153 corresponds to an activation instruction (e.g., a car engine activation instruction) corresponding to the voice command (e.g., "start the car") detected in the output signal 132. In particular aspects, because the output signal 132 includes (e.g., substantially matches) the linear components of the reference target signal 122 and excludes (e.g., substantially omits) the nonlinear components of the reference target signal 122, the output 153 is a more accurate activation instruction corresponding to the voice command in the speech 206 (compared to the activation instruction that would be generated based on detecting the voice command in the reference target signal 122).

[0039] In a particular example, speaker verification component 258 processes output signal 132 to generate output 153. By way of example, output 153 includes a speaker verification output, such as an indicator of whether the speech detected in output signal 132 matches the speech of a particular authorized user. In particular aspects, because output signal 132 includes (e.g., substantially matches) linear components of reference target signal 122 and excludes (e.g., substantially omits) nonlinear components of reference target signal 122, output 153 is a more accurate indication of whether speech 206 corresponds to a particular authorized user (than would be generated by detecting speech in reference target signal 122).

[0040] Although a single linearizer 130 is depicted, in some implementations, the processor 108 includes multiple linearizers 130 corresponding to the multiple microphones 212. The signal processing unit 120 generates multiple reference target signals 122, such as one reference target signal 122 for each active microphone 212. In an example, each linearizer 130 operates in parallel with the other linearizers 130 and performs linear filtering on a particular audio input signal 214 from a particular microphone 212 based on a particular reference target signal 122 corresponding to the particular audio input signal 214. In this regard, the multiple output signals 132 from the multiple linearizers 130 are provided to the component 150.

[0041] It should be understood that the signal processing unit 120 performs speech enhancement is provided as an illustrative example. In other examples, the signal processing unit 120 performs other types of audio signal enhancement, such as noise suppression. For example, the audio input signal 214 ("y") includes an interesting portion ("x") and a noise portion ("n"). In some examples, the interesting portion ("x") corresponds to speech. In other examples, the interesting portion ("x") corresponds to other types of audio (e.g., audio tuning). The system 200 therefore improves the accuracy of processing the audio signal by the audio signal processing component while utilizing the audio enhancement performed by the signal processing unit 120.

[0042] refer to Figure 3 , showing Figure 1 System 100, Figure 2 2, or an example of components of the system 200, and the system is generally designated 300. In the example 300, the processor 108 includes a voice activity detector (VAD) 302 coupled to the signal processing unit 120 and the linearizer 130. The VAD 302 performs voice activity detection on consecutive frames of the reference target signal 122 and generates a voice detection signal 310 that indicates to the linearizer 130 whether each corresponding frame of the audio input signal 214 has voice content. The delay element 320 synchronizes the processing at the VAD 302 and the linearizer 130 so that the frames of the reference target signal 122 processed to generate the speech detection signal 310 correspond to the frames of the audio input signal 214 processed by the linearizer 130.

[0043] Figure 3 Also depicted is an example 350 of operations that may be performed by the linearizer 130 based on the speech detection signal 310. At 352, if the speech detection signal 310 indicates that voice activity is detected (e.g., VAD=1), the speech covariance matrix (e.g., E[yy h ]) and related vectors (e.g., At 354, if no voice activity is detected (eg, VAD=0), the noise covariance matrix of the non-speech frame is updated (eg, E[yy h ]) and related vectors (e.g., ). The speech covariance matrix and the noise covariance matrix are combined using one or more tunable weight factors to generate a combined covariance matrix. The speech related vector and the non-speech related vector are also combined using one or more tunable weight factors to generate a combined correlation vector.

[0044] In a specific example, the combined covariance matrix (e.g., E[yy h ]) corresponds to the speech covariance matrix (e.g., E[yy h ]) and the noise covariance matrix (e.g., E[yy h In certain aspects, the weighted sum of the speech covariance matrix (e.g., E[yy h ]) and the noise covariance matrix (e.g., E[yy h ]) to determine the combined covariance matrix (e.g., E[yy h ]) are based on default values, configured values, user input, or a combination of these.

[0045] In a specific example, the combined correlation vector (e.g., ) corresponds to the correlation vector of the speech frame (e.g., ) and the correlation vectors of non-speech frames (e.g., ). In certain aspects, the correlation vector (e.g., ) and the correlation vectors of non-speech frames (e.g., ) to determine the combined correlation vector (e.g., ) based on default values, configured values, user input, or a combination of these.

[0046] At 356, the combined covariance matrix and the combined correlation vector (e.g., ) to update the linear filter 134. Thus, adaptation of the linear filtering is controlled at least in part based on whether speech activity is detected in the reference target signal 122.

[0047] refer to Figure 4 , an illustrative example of a system operable to perform linearization of a nonlinearly transformed signal is shown and is generally depicted as 400. In a particular example, Figure 1 System 100 includes one or more components of system 400 .

[0048] The processor 108 includes a controller 402 coupled to the signal processing unit 120 and the linearizer 130. The controller 402 is configured to generate a signal based on the reference target signal 122. The associated nonlinearity metric 406 selectively disables the linearizer 130. The component 150 operates using the output signal 132 ("h*y") when the linearizer 130 is enabled, and uses the reference target signal 122 when the linearizer 130 is disabled. to perform the operation.

[0049] The controller 402 includes a nonlinear calculator 404 that processes the reference target signal 122 To generate an indication reference target signal 122 When the nonlinearity metric 406 indicates that the amount of nonlinearity exceeds the threshold, the controller 402 enables operation of the linearizer 130 via the enable / disable signal 410. Otherwise, the controller 402 disables operation of the linearizer 130 via the enable / disable signal 410.

[0050] A threshold value may be set to balance the reference target signal 122 The performance loss of component 150 caused by the nonlinearity of the linearizer 130 outweighs the benefits of reduced power consumption and reduced processing resource utilization resulting from disabling the linearizer 130. Disabling operation of the linearizer 130 reduces power consumption and processing resource utilization of the device 102.

[0051] refer to Figure 5 , discloses certain illustrative aspects of a system operable to perform linearization of a non-linearly transformed visual input signal, and the system is generally designated 500. In certain aspects, the system 100 includes one or more components of the system 500.

[0052] Device 102 is coupled to or includes one or more image sensors 502. In certain aspects, image sensor 502 includes a camera, an infrared sensor, a radar sensor, a night vision sensor, a video camera, a thermal imaging sensor, a sonar sensor, a digital imaging sensor, a medical imaging sensor, or a combination thereof. Image sensor 502 receives an input image of a face 506 of a user 504 and noise (e.g., non-facial regions and “noise” in the input image). Image sensor 502 generates a visual input signal 514 (“y”) that includes a facial portion (“x”) and a noise portion (“n”). In certain aspects, visual input signal 514 corresponds to Figure 1 The input signal 114 ("y").

[0053] The signal processing unit 120 performs signal processing on the visual input signal 514 (“y”) to generate a reference target signal 122 In certain aspects, performing signal processing results in visual signal enhancement, such as image noise suppression. In certain aspects, linear transformation 126 and nonlinear transformation 128 result in facial image enhancement. The signal processing unit 120 performing facial image enhancement is provided as an illustrative example. In other examples, the signal processing unit 120 may perform other types of visual enhancement. The linear filter 134 performs linear filtering of the visual input signal 514 ("y") to generate a signal substantially identical to the reference target signal 122. The linear component of the reference target signal 122 is matched and substantially omitted. An output signal 132 (“h*y”) is generated by removing the nonlinear components (eg, image noise) of the image.

[0054] Component 150 includes one or more visual signal processing components, such as a facial recognition engine 552. In a specific example, facial recognition engine 552 processes output signal 132 to generate output 153. By way of example, output 153 includes facial recognition output corresponding to one or more faces represented by output signal 132. In a specific example, facial recognition output includes an authorized user indicator. In a specific aspect, because output signal 132 includes (e.g., substantially matches) a linear component of reference target signal 122 and excludes (e.g., substantially omits) a nonlinear component of reference target signal 122, output 153 is a more accurate indicator of whether face 506 is associated with an authorized user (compared to facial recognition output that can be generated from reference target signal 122). Processing output signal 132 by facial recognition engine 552 is provided as an illustrative example. In other implementations, output signal 132 is processed by other types of visual processing components. System 500 thus improves the accuracy of processing visual signals by visual signal processing components while utilizing visual enhancement performed by signal processing unit 120. It should be understood that audio signal processing and visual signal processing are provided as illustrative examples. In other examples, the system 100 can process various types of data signals to improve the accuracy of various types of components 150.

[0055] It should be noted that although Figure 3 The example 300 includes a VAD 302 and a delay element 320, while Figure 4 The system 400 includes a controller 402, similar components of VAD 302, delay element 320, and the controller 402 may be combined in other implementations. For example, the system 500 may include the controller 402 and a face detection module. The processor 108 receives Figure 1 The single input signal 114 of FIG. 1 is provided as an illustrative example. In other examples, the processor 108 may receive multiple input signals 114. For example, the processor 108 receives multiple input signals 114 from multiple image sensors, multiple microphones, or a combination thereof. In certain aspects, the processor 108 processes the multiple input signals 114 in parallel to generate multiple output signals 132, as shown in FIG. Figure 2 For example, the signal processing unit 120 generates a plurality of reference target signals 122 corresponding to the plurality of input signals 114. The first linearizer 130 generates a first output signal 132 by processing the first reference target signal 122, while the second linearizer 130 generates a second output signal 132 by processing the second reference target signal 122.

[0056] exist Figure 6, a method for performing linearization of a nonlinearly transformed input signal is shown and generally designated 600. In certain aspects, one or more operations of method 600 are performed by neural network 124, signal processing unit 120, linear filter 134, filter adapter 136, linearizer 130, processor 108, device 102, Figure 1 System 100, Figure 2 System 200, Figure 3 One or more components of example 300, Figure 4 System 400, Figure 5 The system 500 or a combination thereof may be used to perform the above.

[0057] The method 600 includes performing signal processing including a linear transformation and a nonlinear transformation of an input signal at 602 to generate a reference target signal. For example, Figure 1 The signal processing unit 120 performs signal processing including a linear transformation 126 and a nonlinear transformation 128 on the input signal 114 (“y”) to generate a reference target signal 122 As reference Figure 1 Described. Reference target signal 122 has a linear component associated with the linear transformation 126 and a nonlinear component associated with the nonlinear transformation 128. In certain aspects, the signal processing corresponds to signal enhancement. As an example, the input signal 114 ("y") includes an audio signal and the signal processing corresponds to speech enhancement. As another example, the input signal 114 ("y") includes an audio signal and the signal processing corresponds to noise suppression, echo cancellation, dereverberation, or a combination thereof. In a certain example, the input signal 114 ("y") includes a visual input signal and the signal processing corresponds to visual enhancement. In certain aspects, the method 600 includes performing the signal processing using a neural network. For example, Figure 1 The signal processing unit 120 uses the neural network 124 to perform signal processing.

[0058] The method 600 also includes, at 604, performing linear filtering on the input signal by controlling adaptation of the linear filtering to generate an output signal that substantially matches the linear component of the reference target signal. Figure 1 The linear filter 134 performs linear filtering on the input signal 114 (“y”) by controlling the adaptation of the linear filtering to generate a signal substantially identical to the reference target signal 122. The linear component of the matched output signal 132, as referenced Figure 1 In certain aspects, the linear filtering is based on the reference target signal 122 The associated nonlinear metric 406 is optionally performed as described in reference Figure 4In certain aspects, method 600 includes performing voice activity detection on a reference target signal. For example, Figure 3 The VAD302 performs a reference target signal Voice activity detection, such as reference Figure 3 The linearizer 130 controls the adaptation of the linear filter based at least in part on whether speech activity is detected, as described in reference Figure 3 Described.

[0059] In certain aspects, method 600 includes processing the output signal to generate a text output corresponding to speech in the audio input signal. For example, Figure 2 The speech recognition engine 252 processes the output signal 132 to generate an output 153 (e.g., a text output) corresponding to the speech 206 in the audio input signal 214 ("y"), as shown in FIG. Figure 2 The input signal 114 ("y") comprises an audio input signal 214 ("y"), and the signal processing corresponds to speech enhancement.

[0060] In certain aspects, method 600 includes processing the output signal to generate a facial recognition output corresponding to one or more faces represented by the visual input signal. For example, Figure 5 The facial recognition engine 552 processes the output signal 132 to generate an output 153 (e.g., a facial recognition output) corresponding to the face 506 represented by the visual input signal 514 ("y"). The input signal 114 ("y") includes the visual input signal 514 ("y"), and the signal processing corresponds to facial image enhancement.

[0061] The method 600 can thus generate an output signal 132 that corresponds to the linear component of the reference target signal 122 and excludes (eg, substantially omits) the nonlinear component of the reference target signal 122. The method 600 can be performed by Figure 1 The component 150 improves signal processing. The component 150 takes advantage of the signal enhancement performed by the signal processing unit 120 while maintaining accuracy by processing the output signal 132 corresponding to the linear component of the enhanced signal.

[0062] Figure 7 Describes the Figure 1 The device 102 is integrated into an example of an implementation 700 of a vehicle dashboard device, such as a car dashboard device 702. The plurality of sensors 750 may include one or more microphones, cameras, or other sensors, and may include Figure 2 Microphone 212, Figure 5Although illustrated as being in a single location, in other implementations, one or more sensors 750 may be positioned at other locations in the vehicle, such as at various locations distributed within the cabin of the vehicle, such as an array of one or more microphones and one or more cameras located near each seat in the vehicle to detect multimodal input from the vehicle operator and each passenger.

[0063] A visual interface device such as display 720 is mounted or positioned within the vehicle dashboard device 702 (e.g., removably secured to a vehicle phone holder) so as to be visible to the vehicle driver. In a specific example, display 720 is configured to display Figure 1 The output 153 of the display 720 and the sensor 750 is shown. The signal processing unit 120, the linearizer 130, and the assembly 150 are shown with a dashed border to indicate that the signal processing unit 120, the linearizer 130, and the assembly 150 are not visible to the occupants of the vehicle. The signal processing unit 120, the linearizer 130, and the assembly 150 may be implemented in a device that also includes the display 720 and the sensor 750, or may be separate from and coupled to the display 720 and the sensor 750.

[0064] Fig. 8A An example of a signal processing unit 120, a linearizer 130, and an assembly 150 integrated into a headset 802, such as a virtual reality, augmented reality, or mixed reality headset, is depicted. A visual interface device, such as a display 820, is positioned in front of the user's eyes to be able to display augmented reality or virtual reality images or scenes to the user while wearing the headset 802. In a specific example, the display 820 is configured to display Figure 1 Output 153. Sensor 850 may include one or more microphones, cameras, or other sensors, and may include Figure 2 Microphone 212, Figure 5 Although shown in a single location, in other implementations, one or more sensors 850 may be positioned at other locations on the headset 802, such as an array of one or more microphones and one or more cameras distributed around the headset 802 to detect multimodal input.

[0065] Figure 8B An example of a signal processing unit 120, a linearizer 130, and an assembly 150 integrated into a wearable electronic device 804, shown in the figure as a "smart watch," is depicted, which includes a display 820 and a sensor 850. For example, the sensor 850 can detect user input based on modalities such as video, voice, and gestures.

[0066] refer to Fig. 9 , depicts a block diagram of a particular illustrative implementation of a device, and the device is generally designated 900. In various implementations, the device 900 may have more than Fig. 9 In an illustrative implementation, the device 900 may correspond to Figure 1 In an illustrative implementation, the device 900 may execute the reference Figures 1 to 8B Describes one or more operations.

[0067] In certain implementations, the device 900 includes a processor 906 (e.g., a central processing unit (CPU)). The device 900 may include one or more additional processors 910 (e.g., one or more DSPs). The processor 910 may include the signal processing unit 120, one or more of the components 150, the linearizer 130, the voice activity detector 302, the controller 402, or a combination thereof. In certain aspects, Figure 1 The processor 108 corresponds to the processor 906, the processor 910, or a combination thereof.

[0068] The device 900 may include a memory 952 and a CODEC 934. The memory 952 may include instructions 956 that may be executed by one or more additional processors 910 (or processor 906) to implement the reference Figures 1 to 8B One or more operations described. In an example, the memory 952 includes a computer-readable storage device storing instructions 956. The instructions 956, when executed by one or more processors (e.g., the processor 108, the processor 906, or the processor 910 as an illustrative example), cause the one or more processors to perform signal processing including linear transformation and non-transformation of the input signal to generate a reference target signal. The reference target signal has a linear component associated with the linear transformation and a nonlinear component associated with the nonlinear transformation. The instructions 956, when executed, also cause the one or more processors to perform linear filtering of the input signal by controlling the adaptation of the linear filtering to generate an output signal that substantially matches the linear component of the reference target signal.

[0069] Device 900 may include a wireless controller 940 coupled to an antenna 942 via a transceiver 950 .

[0070] The device 900 may include a display 928 coupled to a display controller 926. One or more speakers 936 and one or more microphones 946 may be coupled to the CODEC 934. In certain aspects, the microphone 946 includes the microphone 212. The CODEC 934 may include a digital-to-analog converter (DAC) 902 and an analog-to-digital converter (ADC) 904. In certain implementations, the CODEC 934 may receive an analog signal from the microphone 946, convert the analog signal to a digital signal using the analog-to-digital converter 904, and provide the digital signal to the processor 910. The processor 910 (e.g., a speech and music codec) may process the digital signal, and the digital signal may be further processed by the signal processing unit 120, the linearizer 130, or both. In certain implementations, the processor 910 (e.g., a speech and music codec) may provide the digital signal to the CODEC 934. The CODEC 934 may convert the digital signal to an analog signal using the digital-to-analog converter 902, and may provide the analog signal to the speaker 936. Device 900 may include input device 930. In certain aspects, input device 930 includes Figure 5 An image sensor 502 is provided.

[0071] In a particular implementation, the device 900 may be included in a system-in-package or system-in-chip device 922. In a particular implementation, the memory 952, the processor 906, the processor 910, the display controller 926, the CODEC 934, and the wireless controller 940 are included in the system-in-package or system-in-chip device 922. In a particular implementation, the input device 930 and the power supply 944 are coupled to the system-in-package or system-in-chip device 922. In addition, in a particular embodiment, as Fig. 9 As shown, the display 928, input device 930, speaker 936, microphone 946, antenna 942, and power supply 944 are external to the system-in-package or system-in-chip device 922. In particular embodiments, the display 928, input device 930, speaker 936, microphone 946, antenna 942, and power supply 944 may be coupled to a component of the system-in-package or system-in-chip device 922, such as an interface or a controller.

[0072] Device 900 may include a portable electronic device, an automobile, a vehicle, a computing device, a communication device, an Internet of Things (IoT) device, a virtual reality (VR) device, a smart speaker, a sound bar, a mobile communication device, a smart phone, a cellular phone, a laptop, a computer, a tablet, a personal digital assistant, a display device, a television, a game console, a music player, a radio, a digital video player, a digital video disc (DVD) player, a tuner, a camera, a navigation device, or any combination thereof. In certain aspects, processor 906, processor 910, or a combination thereof is included in an integrated circuit.

[0073] In conjunction with the described implementation, an apparatus includes a component for performing signal processing including linear transformation and nonlinear transformation of an input signal to generate a reference target signal. For example, the component for performing signal processing includes a processor 108, Figure 1 The neural network 124, the signal processing unit 120, the processor 906, the processor 910, one or more other circuits or components configured to perform signal processing including linear transformation and nonlinear transformation of the input signal, or any combination thereof. Figure 1 The reference target signal 122 has a linear component associated with a linear transformation 126 and a nonlinear component associated with a nonlinear transformation 128, as shown in FIG. Figure 1 Described.

[0074] The apparatus further comprises means for performing linear filtering of the input signal by controlling the adaptation of the linear filtering to generate an output signal that substantially matches the linear component of the reference target signal. For example, the means for performing the linear filtering comprises Figure 1 The processor 108, the linear filter 134, the filter adapter 136, the linearizer 130, the processor 906, the processor 910, one or more other circuits or components configured to perform linear filtering of the input signal by controlling the adaptation of the linear filtering to generate an output signal that substantially matches the linear component of the reference target signal, or any combination thereof.

[0075] Those skilled in the art will further appreciate that the various illustrative logic blocks, configurations, modules, circuits, and algorithmic steps described in conjunction with the implementations disclosed herein may be implemented as electronic hardware, computer software executed by a processor, or a combination of the two. Various illustrative components, blocks, configurations, modules, circuits, and steps have been described above generally in terms of their functions. Whether such functions are implemented as hardware or processor executable instructions depends on the specific application and design constraints imposed on the entire system. A skilled person may implement the described functions in different ways for each specific application, and such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

[0076] The steps of the method or algorithm described in conjunction with the implementation disclosed herein can be directly implemented in hardware, in a software module executed by a processor, or in a combination of the two. The software module can reside in a random access memory (RAM), a flash memory, a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a register, a hard disk, a removable disk, a compact disk read-only memory (CD-ROM), or any other form of non-transient storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. In an alternative manner, the storage medium can be integrated into the processor. The processor and the storage medium can reside in an application specific integrated circuit (ASIC). The ASIC can reside in a computing device or a user terminal. In an alternative manner, the processor and the storage medium can reside in a computing device or a user terminal as discrete components.

[0077] The foregoing description of the disclosed aspects is provided to enable those skilled in the art to make or use the disclosed aspects. Various modifications to these aspects will be apparent to those skilled in the art, and the principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but should be given as wide a scope as possible consistent with the principles and novel features defined by the appended claims.

Claims

1. A device for performing signal processing, the device include: One or more processors configured to: performing signal processing including a linear transformation and a nonlinear transformation of an input signal to generate a reference target signal having a linear component associated with the linear transformation and a nonlinear component associated with the nonlinear transformation; as well as A linear filter is used to perform linear filtering of the input signal by controlling adaptation of the linear filtering to generate an output signal, wherein the adaptation of the linear filtering is controlled by updating a filter value of the linear filter with reference to the reference target signal, so that the linear filtering of the input signal based on the filter value reduces the difference between the output signal and the reference target signal.

2. The device according to claim 1, in, The one or more processors are integrated into at least one of a mobile device, a mobile phone, a portable electronic device, an automobile, a vehicle, a computing device, a communication device, an Internet of Things (IoT) device, a virtual reality (VR) device, or an augmented reality (AR) device.

3. The device according to claim 1, in, The signal processing corresponds to signal enhancement.

4. The device according to claim 1, in, The input signal comprises an audio signal, and wherein the signal processing corresponds to speech enhancement. 5 . The device of claim 4 , further comprising one or more microphones coupled to the one or more processors, the one or more processors configured to receive audio signals from the one or more microphones.

6. The device according to claim 1, in, The input signal comprises an audio signal, and wherein the signal processing corresponds to noise suppression, echo cancellation, dereverberation, or a combination thereof.

7. The device according to claim 1, in, The one or more processors are configured to perform the signal processing using a neural network.

8. The device according to claim 1, in, The input signal comprises a visual input signal, and wherein the signal processing corresponds to visual enhancement.

9. The device of claim 8, further comprising an image sensor coupled to the one or more processors, the one or more processors configured to receive the visual input signal from the image sensor.

10. The device according to claim 1, in, The one or more processors are included in an integrated circuit.

11. The device according to claim 1, in, The input signal comprises an audio input signal, wherein the signal processing corresponds to speech enhancement, and wherein the one or more processors are further configured to process the output signal to generate a text output corresponding to speech in the audio input signal.

12. The device of claim 11, further comprising a display device coupled to the one or more processors, the display device configured to display the text output.

13. The device according to claim 1, in, The one or more processors are configured to selectively perform the linear filtering based on a nonlinear metric associated with the reference target signal.

14. The device according to claim 1, in, The one or more processors are further configured to perform voice activity detection on the reference target signal, and wherein the adapting of the linear filtering is controlled at least in part based on whether voice activity is detected.

15. The device according to claim 1, in, The input signal comprises a visual input signal, wherein the signal processing corresponds to facial image enhancement, and wherein the one or more processors are further configured to process the output signal to generate a facial recognition output corresponding to one or more faces represented by the visual input signal.

16. The device of claim 15, further comprising an image sensor coupled to the one or more processors, the one or more processors configured to receive the visual input signal from the image sensor.

17. A method for signal processing, include: performing signal processing including a linear transformation and a nonlinear transformation of an input signal to generate a reference target signal having a linear component associated with the linear transformation and a nonlinear component associated with the nonlinear transformation; as well as Linear filtering of the input signal is performed to generate an output signal by controlling adaptation of the linear filtering, wherein the adaptation of the linear filtering is controlled by updating filter values ​​with reference to the reference target signal, so that the linear filtering of the input signal based on the filter values ​​reduces the difference between the output signal and the reference target signal.

18. The method according to claim 17, in, The signal processing corresponds to signal enhancement.

19. The method according to claim 17, in, The input signal comprises an audio signal, and wherein the signal processing corresponds to speech enhancement.

20. The method according to claim 17, in, The input signal comprises an audio signal, and wherein the signal processing corresponds to noise suppression, echo cancellation, dereverberation, or a combination thereof.

21. The method of claim 17, further comprising using a neural network to perform the signal processing.

22. The method according to claim 17, in, The input signal comprises a visual input signal, and wherein the signal processing corresponds to visual enhancement.

23. The method of claim 17, further comprising processing the output signal to generate a text output corresponding to speech in an audio input signal, wherein the input signal comprises the audio input signal, and wherein the signal processing corresponds to speech enhancement.

24. The method according to claim 17, in, The linear filtering is selectively performed based on a nonlinear metric associated with the reference target signal.

25. The method of claim 17, further comprising performing voice activity detection on the reference target signal, wherein the adaptation of the linear filtering is controlled based at least in part on whether voice activity is detected.

26. The method of claim 17, further comprising processing the output signal to generate a facial recognition output corresponding to one or more faces represented by a visual input signal, wherein the input signal comprises the visual input signal, and wherein the signal processing corresponds to facial image enhancement.

27. A computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to: performing signal processing including a linear transformation and a nonlinear transformation of an input signal to generate a reference target signal having a linear component associated with the linear transformation and a nonlinear component associated with the nonlinear transformation; and Linear filtering of the input signal is performed to generate an output signal by controlling adaptation of the linear filtering, wherein the adaptation of the linear filtering is controlled by updating filter values ​​with reference to the reference target signal, so that the linear filtering of the input signal based on the filter values ​​reduces the difference between the output signal and the reference target signal.

28. The computer readable storage medium of claim 27, in, The linear filtering is selectively performed based on a nonlinear metric associated with the reference target signal.

29. The computer-readable storage medium of claim 27, in, The instructions, when executed by the one or more processors, further cause the one or more processors to perform voice activity detection on the reference target signal, and wherein adaption of the linear filtering is controlled based at least in part on whether voice activity is detected in the reference target signal.

30. An apparatus for performing signal processing, include: means for performing signal processing including a linear transformation and a nonlinear transformation of an input signal to generate a reference target signal having a linear component associated with the linear transformation and a nonlinear component associated with the nonlinear transformation; as well as A component for performing linear filtering of the input signal to generate an output signal by controlling adaptation of the linear filtering, wherein the adaptation of the linear filtering is controlled by updating filter values ​​with reference to the reference target signal so that the linear filtering of the input signal based on the filter values ​​reduces the difference between the output signal and the reference target signal.

31. The device according to claim 30, in, The components for performing signal processing and the components for performing linear filtering are integrated into at least one of a mobile device, a mobile phone, a portable electronic device, an automobile, a vehicle, a computing device, a communication device, an Internet of Things (IoT) device, a virtual reality (VR) device, or an augmented reality (AR) device.

Images