Method and system for Gaussian weighted self-attention for speech enhancement

Abstract

A method and system for Gaussian weighted self-attention for speech enhancement is provided herein. According to one embodiment, the method includes: receiving an input noise signal; generating a score matrix based on the received input noise signal; and applying a Gaussian weighting function to the generated score matrix.

Description

[0001] This application is based upon and claims the benefit of U.S. Provisional Patent Application filed on May 8, 2019 and assigned Serial No. 62 / 844,954 and U.S. Patent Application filed on October 2, 2019 and assigned Serial No. 16 / 591,117, the entire contents of which are incorporated herein by reference. Technical Field

[0002] The present disclosure generally relates to a speech processing system. In particular, the present disclosure relates to a system and method for providing a transformer with Gaussian weighted self-attention for speech enhancement. Background Art

[0003] The transformer uses self-attention to compute symbol-by-symbol correlations in parallel over the entire input sequence, which are used to predict the similarity ratio between the target and adjacent context symbols. The predicted ratio is normalized by a softmax function and used to combine the input context symbols for the next layer output.

[0004] Compared to recurrent networks, such as LSTM or GRU, transformers can be configured to parallelize operations, but also transparently for all context symbols with the same path length. Path length is the number of steps to traverse for an operation, and the shorter the path length, the easier it becomes to learn dependencies between them. Typical recurrent models require path lengths proportional to their symbol distances. In contrast, transformers have a constant path length across context symbols, which is one of the advantages of transformers.

[0005] Transformers have recently replaced recurrent networks (e.g., LSTM, GRU) on many neuro-linguistic programming (NLP) tasks by presenting state of the art performance. However, there has been no report of Transformers showing performance on speech or image denoising problems. The main problem is that the speech denoising problem is different from typical NLP tasks, and the equal path length attention model in Transformers is incompatible with the physical properties of speech signals. For example, noise or signal correlation decreases as the distance between two correlated components becomes larger. Therefore, self-attention can unexpectedly have high correlation with contexts located far away. Summary of the invention

[0006] According to one embodiment, a method includes: receiving an input noise signal; generating a score matrix based on the received input noise signal; and applying a Gaussian weighting function to the generated score matrix.

[0007] According to one embodiment, a system includes: a memory and a processor, the processor configured to: receive an input noise signal; generate a score matrix based on the received input noise signal; and apply a Gaussian weighting function to the generated score matrix. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0009] Figure 1 A flow chart showing a method of Gaussian weighted self-attention for speech enhancement according to an embodiment;

[0010] Figure 2 A diagram showing a system of Gaussian weighted self-attention for speech enhancement according to an embodiment; and

[0011] Figure 3 A block diagram illustrating an electronic device in a network environment according to one embodiment. DETAILED DESCRIPTION

[0012] Hereinafter, embodiments of the present disclosure are described in detail with reference to the accompanying drawings. It should be noted that although the same elements are shown in different drawings, the same elements will be designated by the same reference numerals. In the following description, specific details (such as, detailed configuration and components) are only provided to help the overall understanding of the embodiments of the present disclosure. Therefore, it should be clear to those skilled in the art that various changes and modifications of the embodiments described herein can be made without departing from the scope of the present disclosure. In addition, for clarity and conciseness, the description of well-known functions and configurations is omitted. The terms described below are defined in consideration of the functions in the present disclosure, and may be different according to the user, the user's intention and / or custom. Therefore, the definition of the term should be determined based on the content throughout this specification.

[0013] The present disclosure may have various modifications and various embodiments, and the embodiments thereof are described in detail below with reference to the accompanying drawings. However, it should be understood that the present disclosure is not limited to these embodiments, but includes all modifications, equivalents and substitutes within the scope of the present disclosure.

[0014] Although terms including ordinal numbers (such as, first, second, etc.) can be used to describe various elements, structural elements are not limited by these terms. These terms are only used to distinguish one element from another element. For example, without departing from the scope of the present disclosure, the first structural element may be referred to as the second structural element. Similarly, the second structural element may also be referred to as the first structural element. As used herein, the term "and / or" includes any combination and all combinations of one or more related items.

[0015] The terms used herein are only used to describe various embodiments of the present disclosure and are not intended to limit the present disclosure. Unless the context clearly indicates otherwise, the singular form is intended to include the plural form. In the present disclosure, it should be understood that the term "including" or "having" indicates the presence of features, quantities, steps, operations, structural elements, parts or combinations thereof, and does not exclude the presence or possible addition of one or more other features, quantities, steps, operations, structural elements, parts or combinations thereof.

[0016] Unless defined differently, all terms used herein have the same meaning as understood by a person skilled in the art to which the present disclosure belongs. Unless clearly defined in the present disclosure, terms (such as those defined in a general dictionary) will be interpreted as having the same meaning as the contextual meaning in the relevant technical field and will not be interpreted as having an idealized or overly formal meaning.

[0017] The electronic device according to one embodiment may be one of various types of electronic devices. For example, the electronic device may include a portable communication device (e.g., a smart phone), a computer, a portable multimedia device, a portable medical device, a camera, a wearable device, or a household appliance. According to one disclosed embodiment, the electronic device is not limited to the electronic device described above.

[0018] The terms used in the present disclosure are not intended to limit the present disclosure, but are intended to include various changes, equivalents or alternatives of corresponding embodiments. With respect to the description of the accompanying drawings, similar reference numerals may be used to represent similar or related elements. Unless the relevant context clearly indicates otherwise, the singular form of the noun corresponding to the project may include one or more things. As used herein, each of such phrases as "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C" and "at least one of A, B or C" may include all possible combinations of the project enumerated together with the corresponding phrase in the phrase. As used herein, terms such as "the 1st", "the 2nd", "the first" and "the second" may be used to distinguish a corresponding component from another component, and are not intended to limit the component in other aspects (e.g., importance or order). It is intended that if an element (e.g., a first element) is referred to as being "coupled with" another element (e.g., a second element), "coupled to" another element, "connected with" another element, or "connected to" another element, with or without the term "operably" or "communicatively", it indicates that the element can be coupled with the other element directly (e.g., wired), wirelessly, or via a third element.

[0019] As used herein, the term "module" may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with other terms (e.g., "logic," "logic block," "component," or "circuit"). A module may be a single integrated component or its smallest unit or component adapted to perform one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application specific integrated circuit (ASIC).

[0020] In one embodiment, the present system and method provide Gaussian weighted self-attention for speech denoising. For self-attention, the query and key (or keyword, key) relevance are used to generate attention weights after applying a softmax function.

[0021] Figure 1 A flow chart 100 is shown of a method of Gaussian weighted self-attention for speech enhancement according to an embodiment. At 102, the system receives an input noise signal.

[0022] Figure 2 A diagram of a system for Gaussian weighted self-attention for speech enhancement is shown in accordance with an embodiment. For example, the system 200 receives an input noise signal 202 .

[0023] At 104, the system generates a score matrix based on the received input noise signal. For example, the system 200 processes the input noise signal 202 through three separate batch matrix multiplication operations 204, 206, and 208, which respectively receive trainable parameters W for multiplication with the input noise signal 202. Q , W K , W V V represents the value matrix, K represents the key matrix, and Q represents the query matrix. B represents the batch size, S represents the sequence size, and D represents the input dimension. The system 200 processes the parameter W through respective reshape operations 210, 212, and 214. Q , W K and W V , reshape operations 210, 212, and 214 divide the input dimension according to the number of attention heads H. The system 200 divides the input dimension according to the number of attention heads H. Q The output of the parameter reshape operation 210 and the K The output of the parameter reshape operation 212 is processed into a batch matrix multiplication operation 216, which produces a score matrix as in equation (1):

[0024]

[0025] Among them, Q h is the query matrix, is the key matrix, h is the head index, d is the input dimension, S h is the score matrix. Q is calculated as in equations (2), (3) and (4) h , K h and V h :

[0026] Q h = reshape(W Q V) (2)

[0027] K h = reshape(W K V) (3)

[0028] V h = reshape(W V V) (4)

[0029] Among them, Q h and K h They have the same dimensions (B*H)×S×(D / H), and “*” represents a convolution operation.

[0030] At 106, the system applies a Gaussian weighting function to the generated score matrix.

[0031] For example, the system 200 multiplies the score matrix with a Gaussian weighting (GW) function 218 to fade out the score proportional to its distance to the target frame. The Gaussian weighting function includes a Gaussian weighting matrix, and the Gaussian weighting matrix can be constructed as in equation (5).

[0032]

[0033] Here, i, j = 1, 2, ..., S, S is a positive integer, and wherein σ is a real number greater than 0. The diagonal of the Gaussian matrix has the maximum value, and the weights of the diagonal decay equally from left to right. As in equation (6), the Gaussian matrix in equation (5) is element-wise multiplied with the fractional matrix.

[0034]

[0035] System 200 may apply a Gaussian weighting function as in equation (7), where O i Represents the result or output of applying the Gaussian weighting function.

[0036] O i =(SoftMax(G⊙|S h |))V h (7)

[0037] Equation (7) is the element-wise multiplication of the absolute value of the Gaussian matrix and the score matrix. For equation (7), S h The absolute value of is used as the softmax input, and S h The sign of is compensated after the softmax output. The reason for this two-step approach is that, unlike typical NLP tasks, negative correlations in signal estimation are as important as positive correlations. The Gaussian weighting before applying the softmax function decays the correlation values ​​regardless of their sign. By taking the absolute value of the score, the self-attention will only depend on the score size. Afterwards, when V h When matrices are combined, the system is connected to the sign matrix Sign(S h ) to compensate for the sign of the fraction.

[0038] System 200 may apply a Gaussian weighting function as in equation (8).

[0039] O i =(SoftMax(G⊙|S h |)⊙Sign(S h ))V h (8)

[0040] Equation (8) is the element-wise multiplication of the Gaussian matrix and the absolute value of the score matrix, the sign of which is compensated after applying the softmax function. Equation (8) is then not sign-compensated. Since V h , Q h , K h are trainable matrices, so they can find the right sign even without explicit sign compensation.

[0041] System 200 may apply a Gaussian weighting function as in equation (9).

[0042] O i =(SoftMax(G⊙S h ))V h (9)

[0043] Equation (9) is the element-wise multiplication of the Gaussian matrix and the score matrix. Equation (9) can learn to flip the negative sign by the expected score function instead of using the absolute function of the score matrix. Each of the applications in equations (7), (8) and (9) Figure 2 The system uses the output of the softmax operation 220 and the output of the reshape operation 214 to perform a batch matrix multiplication 222. The system performs a reshape operation 224 on the output of the batch matrix multiplication 222. The system uses the output of the reshape operation 224 and the output of the reshape operation 214 to perform a batch matrix multiplication 222. OUTTo perform batch matrix multiplication operation 226 to produce output 228.

[0044] Optionally, as in equation (10), a Gaussian weighting function 218 may be applied after the softmax operation 220.

[0045]

[0046] In equation (10), positive correlation is used because negative correlation will be ignored after the softmax function is applied.

[0047] Figure 3 FIG. 1 is a block diagram showing an electronic device 301 in a network environment 300 according to one embodiment. Figure 3 , the electronic device 301 in the network environment 300 may communicate with the electronic device 302 via a first network 398 (e.g., a short-range wireless communication network), or communicate with the electronic device 304 or the server 308 via a second network 399 (e.g., a long-range wireless communication network). The electronic device 301 may communicate with the electronic device 304 via the server 308. The electronic device 301 may include a processor 320, a memory 330, an input device 350, a sound output device 355, a display device 360, an audio module 370, a sensor module 376, an interface 377, a haptic module 379, a camera module 380, a power management module 388, a battery 389, a communication module 390, a user identification module (SIM) 396, and / or an antenna module 397. In one embodiment, at least one of the components (e.g., the display device 360 ​​or the camera module 380) may be omitted from the electronic device 301, or one or more other components may be added to the electronic device 301. In one embodiment, some of the components may be implemented as a single integrated circuit (IC). For example, the sensor module 376 (eg, a fingerprint sensor, an iris sensor, or an illumination sensor) may be embedded in the display device 360 ​​(eg, a display).

[0048] The processor 320 may execute, for example, software (e.g., program 340) to control at least one other component (e.g., hardware component or software component) of the electronic device 301 connected to the processor 320, and may perform various data processing or calculations. As at least part of the data processing or calculation, the processor 320 may load a command or data received from another component (e.g., sensor module 376 or communication module 390) into the volatile memory 332, process the command or data stored in the volatile memory 332, and store the result data in the non-volatile memory 334. The processor 320 may include a main processor 321 (e.g., a central processing unit (CPU) or an application processor (AP)) and an auxiliary processor 323 (e.g., a graphics processor (GPU), an image signal processor (ISP), a sensor hub processor, or a communication processor (CP)), and the auxiliary processor 323 may operate independently of the main processor 321, or operate together with the main processor 321. Additionally or alternatively, the auxiliary processor 323 may be adapted to consume less power than the main processor 321, or to perform a specific function. The auxiliary processor 323 may be implemented separately from the main processor 321 , or implemented as part of the main processor 321 .

[0049] The auxiliary processor 323 may replace the main processor 321 when the main processor 321 is in an inactive (e.g., sleeping) state, or control at least some of the functions or states related to at least one of the components of the electronic device 301 (e.g., the display device 360, the sensor module 376, or the communication module 390) together with the main processor 321 when the main processor 321 is in an active state (e.g., executing an application). According to one embodiment, the auxiliary processor 323 (e.g., an image signal processor or a communication processor) may be implemented as part of another component (e.g., a camera module 380 or a communication module 390) that is functionally related to the auxiliary processor 323.

[0050] The memory 330 may store various data used by at least one component (e.g., the processor 320 or the sensor module 376) in the electronic device 301. The various data may include, for example, software (e.g., the program 340) and input data or output data for commands related thereto. The memory 330 may include a volatile memory 332 and / or a nonvolatile memory 334.

[0051] The program 340 may be stored as software in the memory 330 , and may include, for example, an operating system (OS) 342 , middleware 344 , and / or applications 346 .

[0052] The input device 350 may receive commands or data to be used by other components (eg, processor 320) of the electronic device 301 from outside the electronic device 301 (eg, a user). The input device 350 may include, for example, a microphone, a mouse, and / or a keyboard.

[0053] The sound output device 355 can output sound signals to the outside of the electronic device 301. The sound output device 355 may include, for example, a speaker and / or a receiver. The speaker can be used for general purposes (such as playing multimedia or recording), and the receiver can be used to receive incoming calls. According to one embodiment, the receiver can be implemented as a separate speaker or as part of the speaker.

[0054] The display device 360 ​​may visually provide information to the outside of the electronic device 301 (e.g., a user). The display device 360 ​​may include, for example, a display, a hologram device, and / or a projector, and a control circuit for controlling a corresponding one of the display, the hologram device, and the projector. According to one embodiment, the display device 360 ​​may include a touch circuit suitable for detecting a touch, or a sensor circuit (e.g., a pressure sensor) suitable for measuring the strength of a force caused by a touch.

[0055] The audio module 370 may convert sound into an electrical signal, and vice versa. According to one embodiment, the audio module 370 may obtain sound via the input device 350, or output sound via the sound output device 355 or an earphone of an external electronic device 302 directly (e.g., wired) or wirelessly connected to the electronic device 301.

[0056] The sensor module 376 may detect an operating state (e.g., power or temperature) of the electronic device 301 or an environmental state (e.g., a user's state) outside the electronic device 301, and then generate an electrical signal or data value corresponding to the detected state. The sensor module 376 may include, for example, a posture sensor, a gyroscope sensor, an atmospheric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a biometric sensor, a temperature sensor, a humidity sensor, and / or an illumination sensor.

[0057] The interface 377 may support one or more specific protocols to be used for connecting the electronic device 301 directly (e.g., wired) or wirelessly with the external electronic device 302. According to one embodiment, the interface 377 may include, for example, a high-definition multimedia interface (HDMI), a universal serial bus (USB) interface, a secure digital (SD) card interface, and / or an audio interface.

[0058] The connection terminal 378 may include a connector, and the electronic device 301 may be physically connected to the external electronic device 302 via the connector. According to one embodiment, the connection terminal 378 may include, for example, an HDMI connector, a USB connector, an SD card connector, and / or an audio connector (e.g., a headphone connector).

[0059] The haptic module 379 may convert the electrical signal into mechanical stimulation (eg, vibration or movement) or electrical stimulation that can be recognized by the user via tactile or kinesthetic sense. According to one embodiment, the haptic module 379 may include, for example, a motor, a piezoelectric element, and / or an electrical stimulator.

[0060] The camera module 380 may capture still images or moving images. According to one embodiment, the camera module 380 may include one or more lenses, an image sensor, an image signal processor, and / or a flash.

[0061] The power management module 388 may manage power supplied to the electronic device 301. The power management module 388 may be implemented as, for example, at least a portion of a power management integrated circuit (PMIC).

[0062] The battery 389 may supply power to at least one component of the electronic device 301. According to one embodiment, the battery 389 may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0063] The communication module 390 may support establishing a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device 301 and an external electronic device (e.g., electronic device 302, electronic device 304, or server 308), and support communication via the established communication channel. The communication module 390 may include one or more communication processors that can operate independently of the processor 320 (e.g., AP) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module 390 may include a wireless communication module 392 (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 394 (e.g., a local area network (LAN) communication module or a power line communication (PLC) module). The corresponding modules in these communication modules may communicate via a first network 398 (e.g., a short-range communication network (such as Bluetooth TM), Wireless Fidelity (Wi-Fi) Direct, or Infrared Data Association (IrDA) standards)) or a second network 399 (e.g., a long-distance communication network (such as a cellular network, the Internet, or a computer network (e.g., a LAN or a wide area network (WAN)))) to communicate with an external electronic device. These various types of communication modules may be implemented as a single component (e.g., a single IC), or may be implemented as multiple components separated from each other (e.g., multiple ICs). The wireless communication module 392 may use user information (e.g., an International Mobile Subscriber Identity (IMSI)) stored in the user identification module 396 to identify and authenticate the electronic device 301 in a communication network (such as, the first network 398 or the second network 399).

[0064] The antenna module 397 may transmit a signal or power to the outside of the electronic device 301 (e.g., an external electronic device), or receive a signal or power from the outside of the electronic device 301 (e.g., an external electronic device). According to one embodiment, the antenna module 397 may include one or more antennas, whereby at least one antenna suitable for a communication scheme used in a communication network (e.g., the first network 398 or the second network 399) may be selected, for example, by the communication module 390 (e.g., the wireless communication module 392). Then, a signal or power may be transmitted or received between the communication module 390 and the external electronic device via the selected at least one antenna.

[0065] At least some of the above components may be connected to each other via an inter-peripheral communication scheme (e.g., a bus, a general purpose input output (GPIO), a serial peripheral interface (SPI), or a mobile industry processor interface (MIPI)) and transfer signals (e.g., commands or data) therebetween.

[0066] According to one embodiment, commands or data may be sent or received between the electronic device 301 and the external electronic device 304 via the server 308 connected to the second network 399. Each of the electronic devices 302 and 304 may be a device of the same type or a different type as the electronic device 301. All or some of the operations to be performed at the electronic device 301 may be performed at one or more of the external electronic device 302, the external electronic device 304, or the external electronic device 308. For example, if the electronic device 301 should automatically or in response to a request from a user or another device to perform a function or service, the electronic device 301 may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service, or in addition to performing the function or service, the electronic device 301 may also request one or more external electronic devices to perform at least part of the function or service. The one or more external electronic devices that receive the request may perform at least part of the requested function or service or additional functions or additional services related to the request, and transmit the result of the execution to the electronic device 301. The electronic device 301 may provide the result as at least part of the reply to the request with or without further processing the result. To this end, for example, cloud computing, distributed computing, or client-server computing techniques may be used.

[0067] One embodiment may be implemented as software (e.g., program 340), which includes one or more instructions stored in a storage medium (e.g., internal memory 336 or external memory 338) that can be read by a machine (e.g., electronic device 301). For example, the processor of the electronic device 301 may call at least one of the one or more instructions stored in the storage medium, and use or do not use one or more other components under the control of the processor to execute the at least one instruction. Thus, the machine may be operated to perform at least one function according to the at least one instruction called. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. The term "non-transitory" indicates that the storage medium is a tangible device and does not include signals (e.g., electromagnetic waves), but this term does not distinguish between a case where data is semi-permanently stored in a storage medium and a case where data is temporarily stored in a storage medium.

[0068] According to one embodiment, the disclosed method may be included and provided in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disc read-only memory (CD-ROM)), or via an application store (e.g., Play Store TM) online distribution (e.g., downloading or uploading), or directly between two user devices (e.g., smartphones). If distributed online, at least part of the computer program product may be temporarily generated or at least temporarily stored in a machine-readable storage medium (such as a memory of a manufacturer's server, an application store's server, or a relay server).

[0069] According to one embodiment, each of the above-mentioned components (e.g., a module or a program) may include a single entity or multiple entities. One or more of the above-mentioned components may be omitted, or one or more other components may be added. Alternatively or additionally, multiple components (e.g., modules or programs) may be integrated into a single component. In this case, the integrated component may still perform one or more functions of each of the multiple components in the same or similar manner as one or more functions of each of the multiple components were performed by corresponding components of the multiple components before integration. The operations performed by a module, a program or another component may be performed sequentially, in parallel, repeatedly or heuristically, or one or more of the operations may be performed in a different order or one or more of the operations may be omitted, or one or more other operations may be added.

[0070] Although the specific embodiments of the present disclosure have been described in the specific embodiments of the present disclosure, the present disclosure may be modified in various forms without departing from the scope of the present disclosure. Therefore, the scope of the present disclosure should not be determined based on the described embodiments alone, but should be determined based on the attached claims and their equivalents.

Claims

1. A Gaussian weighted self-attention method for speech enhancement, comprising: receiving an input noise signal; generating a score matrix based on a received input noise signal; and A Gaussian weighting function is applied to the resulting score matrix by element-wise multiplying the absolute value of the score matrix with the Gaussian weighting matrix.

2. The method according to claim 1, wherein: The score matrix is ​​generated based on the query matrix and the key matrix.

3. The method according to claim 1, wherein: The step of applying the Gaussian weighting function to the generated score matrix further includes compensating the sign of the score matrix after applying the flexible maximum function to the output generated by element-wise multiplying the absolute value of the score matrix by the Gaussian weighting matrix.

4. The method according to claim 1, wherein: The step of applying the Gaussian weighting function to the generated score matrix includes element-wise multiplication of the Gaussian weighting matrix and the score matrix.

5. The method according to claim 1, further comprising: A flexible maximum operation is applied to the output produced by applying a Gaussian weighting function to the resulting score matrix.

6. The method according to claim 1, further comprising: A softmax function is applied to the resulting score matrix before applying a Gaussian weighting function to the resulting score matrix.

7. The method according to claim 1, wherein: The Gaussian weighting function includes a Gaussian weighting matrix.

8. The method according to claim 7, wherein: The Gaussian weighting matrix is ， Where i, j = 1, 2, ..., S, S is a positive integer, in, is a real number greater than 0.

9. A Gaussian weighted self-attention system for speech enhancement, comprising: Memory; and The processor is configured as: Receive input noise signal, Generate a score matrix based on the received input noise signal, and A Gaussian weighting function is applied to the resulting score matrix by element-wise multiplying the absolute value of the score matrix with the Gaussian weighting matrix.

10. The system according to claim 9, wherein: The score matrix is ​​generated based on the query matrix and the key matrix.

11. The system according to claim 9, wherein: The processor is further configured to apply a Gaussian weighting function to the generated score matrix by compensating a sign of the score matrix after applying the flexible max function to an output generated by element-wise multiplying an absolute value of the score matrix with the Gaussian weighting matrix.

12. The system according to claim 9, wherein: The processor is configured to apply a Gaussian weighting function to the generated score matrix by element-wise multiplying the Gaussian weighting matrix with the score matrix.

13. The system according to claim 9, wherein: The processor is further configured to apply a softmax operation to the output produced by applying the Gaussian weighting function to the generated score matrix.

14. The system according to claim 9, wherein: The processor is further configured to apply a softmax function to the generated score matrix before applying the Gaussian weighting function to the generated score matrix.

15. The system according to claim 9, wherein: The Gaussian weighting function includes a Gaussian weighting matrix.

16. The system of claim 15, wherein: The Gaussian weighting matrix is ， Where i, j = 1, 2, ..., S, S is a positive integer, in, is a real number greater than 0.

Images