Method of generating key information, corresponding system, electronic device, media and final product

Abstract

FIELD: computer engineering.SUBSTANCE: method includes the following stages: performing cross-processing of a tensor characteristic by channel dimension and time dimension with obtaining a first cross-characteristic corresponding to a speech signal, wherein said first cross-characteristic is a characteristic combining said frequency information and said time series; determining a semantic characteristic corresponding to said speech signal based on said first cross-characteristic; using an external attention module of a trained information generation model for weighting the general part of the semantic characteristic corresponding to said speech signal with obtaining a general characteristic corresponding to said speech signal; using a self-attention module of said information generation model for weighting the individual part of the semantic characteristic corresponding to said speech signal with obtaining an individual characteristic corresponding to said speech signal, wherein the number of said self-attention modules is at least one; generating key information corresponding to said speech signal based on said general characteristic and said individual characteristic.EFFECT: increasing the reliability of the generated key information.10 cl, 10 dwg

Description

[0001] Cross references

[0002] This application claims priority over Chinese Patent Application No. 202511171938.4, which was filed on August 21, 2025, with the China National Intellectual Property Administration (CNIPA) under the title "Method for Generating Key Information, Corresponding System, Electronic Device, Storage Medium, and End Product." All material contained in that application is hereby incorporated by reference in its entirety.

[0003] Technical sphere

[0004] This application relates to, but is not limited to, the field of signal processing technologies and specifically discloses a method for generating key information and the corresponding system, electronic device, storage medium and end product. Prior Art

[0005] Speech key information generation is a technology for converting speech signals into summarized text information, widely applied in fields such as customer service conversation summarization, key data generation from video materials, and key information extraction from news reports. In practical implementations, frequency domain characteristics extracted from a speech signal are typically treated as one-dimensional (1D) data; the characteristics of a given speech signal are obtained by applying 1D convolution to this 1D data.

[0006] Essence Revealed

[0007] The following is an overview of the subject matter described in detail in this document. This overview is not intended to limit the scope of legal protection provided by the claims.

[0008] The embodiments of the present disclosure provide a method for generating key information, a corresponding system, an electronic device, a storage medium, and an end product.

[0009] In embodiments of the present disclosure, one method for generating key information is proposed, including the following steps:

[0010] receive speech signal to be processed;

[0011] pre-processing the specified speech signal to obtain the corresponding tensor characteristic, which includes time series information on dimensions and frequency information by channel dimension, wherein the channel dimension of the specified tensor characteristic is a number that allows or does not allow the operation of extracting a square root;

[0012] perform cross-processing of the specified tensor characteristic by channel dimension and dimensions with obtaining a first cross-characteristic corresponding to said speech signal, wherein said first cross-characteristic is a characteristic that combines said frequency information and said time series;

[0013] determining a semantic characteristic corresponding to said speech signal based on said first cross-characteristic;

[0014] using the external attention module of the trained information generation model to weight the general part of the semantic feature corresponding to the specified speech signal, to obtain the general feature corresponding to the specified speech signal;

[0015] using a self-attention module of said information generation model to weight an individual part of a semantic characteristic corresponding to said speech signal, thereby obtaining an individual characteristic corresponding to said speech signal, wherein the number of said self-attention modules is not less than one;

[0016] generating key information corresponding to the specified speech signal based on the specified general characteristic and the specified individual characteristic.

[0017] The described process flow chart involves cross-processing of the tensor characteristic by jointly transforming it in two dimensions: channel and time. This two-dimensional integration allows for the combination of two types of information: frequency information in the channel dimension and time series information in the time dimension. This results in the formation of a first cross-processing characteristic that simultaneously retains the ability to identify semantic relationships over extended sections of the speech signal and integrates multi-frequency components of the original audio stream. This approach enables a more detailed reproduction of the semantic structure of speech by eliminating the limitation of analyzing only local temporal patterns, a characteristic of previous technical solutions.Synthesizing time-frequency characteristics significantly increases the expressiveness of the generated semantic characteristics, which directly contributes to a more accurate perception of the overall semantic content of the speech signal. The net effect of this processing is an increase in the reliability of the generated key information.

[0018] Optionally, the procedure for identifying the semantic characteristic of the speech signal based on the first cross-characteristic is implemented through three successive operations: determining the weighting coefficients of the speech segments corresponding to individual characteristic points within the first cross-characteristic of the speech signal; adjusting the first cross-characteristic of the speech signal taking into account the identified weighting coefficients of the speech segments, as a result of which the second cross-characteristic of the speech signal is formed; semantic encoding of the second cross-characteristic of the speech signal to obtain the corresponding semantic characteristic.

[0019] Optionally, said performing semantic encoding of said second cross-feature corresponding to said speech signal to obtain a semantic characteristic corresponding to said speech signal includes: determining a first union characteristic corresponding to said speech signal based on a tensor characteristic corresponding to said speech signal and said second cross-feature corresponding to said speech signal, wherein said first union characteristic is consistent with time series information of said tensor characteristic; performing semantic encoding of said first union characteristic corresponding to said speech signal to obtain a semantic characteristic corresponding to said speech signal.

[0020] Optionally, the external attention module is structured as three successive convolutional layers - the first, second, and third. The external attention module of the trained information generation model is used to weight the common part of the semantic feature corresponding to the specified speech signal, thereby obtaining the common feature corresponding to the specified speech signal, including: using the common parameters contained in the first and second convolutional layers, weighting the common part of the semantic feature of the speech signal is performed, resulting in a compressed feature corresponding to the given speech signal; using the third convolutional layer, a convolution operation is performed on the compressed feature, resulting in the restoration of the channel structure and obtaining the final common feature of the speech signal.

[0021] Optionally, generating key information for a speech signal based on the common and individual characteristics includes the following steps: combining the common and individual characteristics to obtain a combined characteristic; applying a convolution operation to the combined characteristic to generate a relationship characteristic that reflects the correlation between the common and individual characteristics along the channel and time dimensions; weighting the common and individual characteristics using gate weights based on the relationship characteristic, resulting in a second combination characteristic; decoding the second combination characteristic to obtain the key information corresponding to the given speech signal.

[0022] Optional, cross-processing of tensor characteristic by channel and The process of obtaining the first cross-characteristic of a speech signal by dimensions includes the following steps: applying a one-dimensional convolution to the tensor characteristic with a transformation of its channel structure, resulting in the formation of a first characteristic containing a predetermined number of channels; splitting the first characteristic by dimensions, during which a second characteristic with separated dimensions is obtained; applying a two-dimensional convolution to the second characteristic to form the first cross-characteristic corresponding to the given speech signal.

[0023] Optionally, the described method further includes the following steps: obtaining a set of speech signal samples; pre-processing each speech signal sample using an untrained information generation model, as a result of which a tensor characteristic corresponding to each speech signal sample is formed therein; cross-processing each tensor characteristic along the channel and time dimensions to obtain a first cross-characteristic corresponding to the given speech signal sample; generating key information for each speech signal sample based on its first cross-characteristic; calculating a target value of the loss function based on the generated key information and reference marks for each speech signal sample; repeatedly adjusting the parameters of the information generation model based on the calculated value of the loss function, as a result of which a trained information generation model is obtained.

[0024] According to embodiments of the present disclosure, one key information generation system is proposed, including the following modules: an input data preprocessing module, a combined processing module, an encoding module, a feature extraction module, and a decoding module, including:

[0025] The input data preprocessing module is designed to: obtain the speech signal to be processed; perform preprocessing of this speech signal to obtain the corresponding tensor characteristic, which contains time series information along the time dimension and frequency information along the channel dimension;

[0026] The combined processing module is designed to perform the integration of the tensor characteristic along the channel and time dimensions, as a result of which the first cross-characteristic of the speech signal is formed; the said first cross-characteristic is a characteristic that combines frequency information and time series information;

[0027] said coding module is used to determine a semantic characteristic corresponding to said speech signal based on said first cross-characteristic;

[0028] said feature extraction module is for using an external attention module of a trained information generation model to weight a common portion of a semantic feature corresponding to said speech signal, thereby obtaining a common feature corresponding to said speech signal; and for using a self-attention module of said information generation model to weight an individual portion of a semantic feature corresponding to said speech signal, thereby obtaining an individual feature corresponding to said speech signal, wherein the number of said self-attention modules is not less than one;

[0029] The specified decoding module is used to generate key information corresponding to the specified speech signal based on the specified general characteristic and the specified individual characteristic.

[0030] According to another aspect of the embodiments of the claimed solution, electronic equipment is also proposed, comprising a memory and a processor, wherein a computer program executed on the processor is stored in the memory; when said program is launched, the processor implements any of the above-described methods for generating key information.

[0031] According to another aspect of the embodiments of the claimed solution, a computer storage medium is also proposed, containing executable instructions, when executed by the processor, any of the above methods for generating key information is implemented.

[0032] According to another aspect of the embodiments of the claimed solution, a software product is also proposed that includes a computer program or set of instructions, when executed by a processor, any of the above methods for generating key information is implemented.

[0033] The advantages achieved in the embodiments of the present disclosure are as follows:

[0034] 1. By pre-scaling, splitting, and merging the tensor feature by channel dimension, the one-dimensional feature is converted into a two-dimensional feature integrating time series and frequency information. This enables the combination of channel semantic attributes and temporal data, allowing the first cross-feature to more fully reflect the semantic details of the speech signal. Next, by assigning high weights to features corresponding to key semantic elements and low weights to features associated with redundant information, the first cross-feature is transformed from a feature integrating time series and frequency information into a feature containing semantic significance information. As a result, the adjusted second cross-feature more closely correlates with the actual semantics of the speech signal.Subsequent merging of the tensor feature and the second cross-feature forms the first merging feature, which, while preserving the original temporal and frequency data of the speech signal, additionally includes information about semantic significance. This improves the accuracy of the semantic feature derived from the first merging feature.

[0035] 2. Using shared parameters in the first and second convolutional layers, the common part of the semantic feature is weighted, resulting in a compressed feature with enhanced common components. The third convolutional layer then restores the channel dimension of the compressed feature to its original value. This approach enables efficient identification of common patterns between speech signals while maintaining the computational efficiency of subsequent processing stages. Using individual parameters contained in the self-attention module, the individual characteristics of a specific speech signal are precisely extracted.By weighting the general and individual characteristics, key information is generated that both contains the general information inherent to multiple speech signals and accurately reflects the unique details of an individual speech signal. This weighting mechanism also allows for dynamic adjustment of the ratio of general and individual components. As a result, the decoded key information logically corresponds to the original semantics of the speech signal, fully encompassing the essential information while accurately conveying individual differences.

[0036] 3. By combining common and individual characteristics, a combined characteristic is formed, allowing for the complete preservation of the original information of both characteristic types and eliminating data loss. Subsequent processing of the combined characteristic using convolutional operations allows for the effective identification of deep relationships between common and individual characteristics across the time and channel dimensions. Based on the calculated weighting coefficients, a weighted combination of common and individual characteristics is performed, allowing for flexible adjustment of the component significance. As a result, the generated key information logically corresponds more closely to the original semantics of the speech signal; contains key themes common to multiple speech signals; and accurately conveys the unique details of a specific individual speech signal. Description of the drawings

[0037] The drawings shown and described herein are intended to explain embodiments of the present disclosure and form a part of this application, and the embodiments themselves and their description serve to interpret this application and do not unduly limit its scope. In the accompanying drawings:

[0038] Fig. 1 is a first schematic illustration of a process for implementing a method for generating key information proposed in an embodiment of this application;

[0039] Fig. 2 is a schematic illustration of a method for extracting a tensor characteristic of a speech signal proposed in an embodiment of this application;

[0040] Fig. 3 is a schematic illustration of a tensor characteristic proposed in an embodiment of this application.

[0041] Fig. 4 is a second schematic illustration of a process for implementing the key information generation method proposed in the embodiment of this application;

[0042] Fig. 5 is a schematic illustration of a feature combination process proposed in an embodiment of this application;

[0043] Fig. 6 is a third schematic illustration of a process for implementing the method for generating key information proposed in the embodiment of this application;

[0044] Fig. 7 is a schematic illustration of a feature extraction process proposed in an embodiment of this application;

[0045] Fig. 8 is a schematic illustration of the structural composition of the key information generation system proposed in the embodiment of this application;

[0046] Fig. 9 is a fourth schematic illustration of a process for implementing the method for generating key information proposed in the embodiment of this application;

[0047] Fig. 10 is a schematic illustration of a physical structure of an electronic device proposed in an embodiment of this application;

[0048] Specific implementation methods

[0049] To facilitate a more complete understanding of the proposed solution by those skilled in the art, a clear and comprehensive description of the technical solution in accordance with embodiments of the present application is provided below, accompanied by the corresponding drawings. It should be noted that the described embodiments represent only a portion of all possible implementations of the application, and not a complete list thereof. Based on the embodiments described in this application, all other embodiments that can be obtained by those skilled in the art without creative effort shall fall within the scope of legal protection provided by this application. Within the framework of the presented application, combinations of various embodiments, as well as individual characteristic features described in these embodiments, are permitted, subject to the principle of non-contradiction.

[0050] The terms "first," "second," and similar terms used in the specification, claims, and drawings are intended to distinguish similar objects and do not imply a description of any particular sequence or order. It should be understood that the data used in this manner may be interchangeable, if necessary, so that the embodiments described in this application can be implemented in an order different from that illustrated or described herein. Furthermore, the terms "includes" and "has," as well as any forms thereof, imply a non-exclusive inclusion, that is, a process, method, product, or device comprising a number of steps or elements is not strictly limited to the steps or elements listed, but may also include other steps or elements not explicitly indicated or inherent in the given process, method, product, or device.

[0051] Below, the methods for implementing this application will be explained with reference to the drawings and preferred embodiments; those skilled in the art will be able to readily recognize other advantages and functionalities of the claimed solution based on the information disclosed in the description. This application can also be realized or implemented using various other specific implementation examples. The details provided in this description can be changed or supplemented to take into account different viewpoints and fields of application without departing from the implementation examples of this application. It should be understood that the preferred embodiments are provided solely to clarify the essence of this application and not to limit the scope of its legal protection.

[0052] To provide a clear and structured presentation of the objectives, technical essence, and advantages of the claimed solution, a detailed description of the development is provided below, accompanied by references to the accompanying graphic materials (drawings). It is important to emphasize that the specific implementation options provided in the description serve solely as illustrative examples and in no way limit the scope of legal protection granted by the application. The protected scope of technical solutions includes all possible modifications and alternative embodiments of the claimed concept that can be developed by specialists in the relevant technical field without creative input.

[0053] It should be noted that embodiments of this application are applicable to a number of use cases, including summarizing customer support conversations, generating key information from video footage, extracting key information from news stories, and other similar tasks. Below, using a specific example of key information generation, the technical solution proposed in this application will be further described and its results demonstrated.

[0054] The methods for generating key information presented in the embodiments of this application may be implemented using a speech processing system, wherein such a system may be implemented as a server, an end device, or a specialized hardware module for speech recognition. In other words, the methods for generating key information provided in the embodiments of this application may be implemented either via a server, using an end device, or through interaction between the server and the end device.

[0055] In the process of implementing the embodiments of this application, the frequency characteristics extracted from the speech signal are usually regarded as one-dimensional (1D) data; then, by applying 1D convolution to this 1D data, the characteristics of this speech signal are obtained. However, the characteristics extracted by 1D convolution are only local characteristics extracted by the time dimension and contain only a limited amount of local context information. Such characteristics are unable to cover global information crossing different time intervals, as a result of which it is impossible to accurately interpret the overall semantics of the speech signal, which leads to incorrect generation of key information. In this regard, the embodiments of this application propose a method for generating key information.

[0056] Fig. 1 is a first illustration of the process of implementing the method for generating key information in one of the embodiments described in this application. Below, a description will be given of the steps shown in Fig. 1. It should be noted that the method for generating key information shown in Fig. 1 is described using a speech processing system as an executing example. As can be seen from Fig. 1, this method includes the following steps S101 to S104, where:

[0057] Step S101: Obtain the speech signal to be processed.

[0058] In this case, the speech signal has temporal continuity and can contain several speech segments.

[0059] In this embodiment, the speech signal refers to the original audio data, typically stored in digital form; the speech signal contains waveform information represented as a time sequence. The speech signal may be obtained from sources such as telephone support conversations, meeting recordings, news broadcasts, and other similar scenarios.

[0060] As an illustrative example, consider a scenario for processing audio recordings of customer service sessions: during a telephone conversation between a subscriber (client) and a support representative, the entire conversation is recorded by the system as two separate audio streams. Also, for example, in a meeting recording, the statements of different participants are separated into several separate speech signals.

[0061] In one possible embodiment, after the speech signal is collected, it may be processed to suppress noise, which helps to improve the accuracy of subsequent processing stages.

[0062] Step S102: Perform preprocessing of the speech signal to obtain tensor characteristics corresponding to the given speech signal.

[0063] The speech signal is a continuous waveform in the time domain and can be expressed as a time-varying amplitude. A tensor feature can include time series information in the time dimension and frequency characteristics in the channel dimension. A tensor feature allows one to display the dynamics of speech signal changes across two dimensions: and channel. This term refers to formalized characteristics extracted from the original speech signal during preprocessing; they are typically stored in the form of a tensor. It contains time-series and frequency information, serving as a key link between the original speech signal and high-level semantics.

[0064] Temporal Information characterizes the dynamics of the speech signal in coordinates axis, recording regular changes in its parameters over the course of a sound segment. This concept reflects the sequential evolution of speech characteristics over time. It forms the fundamental basis for representing the content of a speech signal and is essential for identifying semantic logic and contextual relationships in the formation of key information elements.

[0065] In particular, frequency information on the channel dimension of tensor characteristics may include the acoustic-physical characteristics of the speech signal. Acoustic-physical characteristics may include, but are not limited to, the following parameters: fundamental frequency (reflects pitch), spectral energy (reflects intensity / loudness of sound), and formants (reflect the characteristics of vowel sounds). Acoustic-physical characteristics act as the physical carrier of semantics in the speech signal. For example, a speech signal corresponding to the emotion "anger" is characterized by increased energy and a high fundamental frequency.

[0066] In the structure of tensor characteristics, the temporal dimension is represented by a specialized dimension that preserves the chronological order of the original speech signal elements. The time series information in tensor characteristics not only records the sequence of acoustic fragments but also reflects the semantic integrity and logical coherence of speech content.

[0067] At this stage, the speech signal can be preprocessed by converting it into tensor characteristics in the frequency domain. Preprocessing is designed to extract time series and frequency information from each speech signal.

[0068] Pre-processing may include, among other things, converting the speech signal into a Mel Spectrogram, which makes it possible to obtain tensor characteristics of the speech signal in the frequency domain.

[0069] Optionally, if necessary, the speech signal can be converted from the time domain into tensor characteristics in the frequency domain through the specified pre-processing operations, which can extract the time series information by Dimensions and frequency information based on channel dimensions. The implementation allows for both sequential preprocessing of each speech signal and simultaneous preprocessing of multiple speech signals.

[0070] In some implementations, the specified preprocessing may include, among other things, the following operations: splitting into frames, windowing, feature extraction, and fragment concatenation.

[0071] When implementing, you can first split the continuous speech signal into frames, that is, divide the signal into many consecutive short fragments, which in dimensions preserve the original order of sequence. In this case, the operation of splitting into frames consists of selecting segments of a fixed duration from a continuous speech signal; each such selected the segment is usually called a frame speech signal, and each frame speech signal corresponds to a local fragment of the original signal within its interval. This is due to the fact that the speech signal is a continuously changing waveform over time and is, by its nature, a continuous function on the time axis. If we perform the conversion to a mel spectrogram directly for the entire speech signal, we will only be able to obtain the dependence of the signal frequency on its intensity, but will not be able to extract performance

[0072] signal. Therefore, to preserve the time series information, a framing operation is used: the continuous speech signal is divided into a series of sequential short fragments. Then, by converting each individual frame fragment into a mel spectrogram, it is possible to obtain tensor characteristics in the frequency domain, including signal representation.

[0073] Some implementations take into account that after splitting into frames, sharp amplitude jumps may occur at frame boundaries in frame speech signals. Such non-periodic discontinuities lead to energy spillover from true frequencies to adjacent ones during subsequent spectral transformation of the signal, resulting in the appearance of spurious frequency components in the transformed spectrum. Therefore, windowing of the frame speech signal can be additionally performed. Specifically, this is implemented by element-by-element multiplication of the window function by each frame speech signal. The resulting windowed frame signal is such that: the amplitude values ​​of the speech signal within a frame smoothly attenuate at the boundaries; and the signal shape within a frame becomes closer to periodic.This allows us to reduce the effect of spectrum spreading during subsequent conversion and to increase the reliability of the frequency characteristics in the converted spectrum, providing a more accurate representation of the frequency characteristics of the original signal.

[0074] After performing frame division or windowing, MFCC feature extraction can be performed for each frame speech signal, resulting in tensor features corresponding to that speech signal.

[0075] Optionally, a Fast Fourier Transform (FFT) can be performed on each frame speech signal, converting the frame signal from the time domain into an amplitude spectrum in the frequency domain. The amplitude spectrum in the frequency domain displays the energy value of each frequency component present in the frame speech signal. Since the frequency scale in the amplitude spectrum is linear, and the human ear perceives frequency nonlinearly (with increased sensitivity to low frequencies and decreased sensitivity to high frequencies), the Mel Scale is used to model this nonlinear perception feature. By weightedly summing the amplitude spectrum with each Mel filter, the output energy for each filter is calculated; this energy represents the components of the Mel spectrogram.

[0076] After obtaining the mel-spectral vector, cepstral characteristics can be further extracted from it. The resulting cepstral characteristics represent the MFCC characteristics of a given frame speech signal. After combining the MFCC characteristics extracted from several frame speech signals, a final set of MFCC characteristics for the original speech signal is formed, which represents tensor characteristics. The MFCC characteristics corresponding to a speech signal directly reflect the distribution of signal energy across various mel-frequency ranges.

[0077] In one possible embodiment, additional acoustic features—for example, pitch and speech rate features—can be combined with MFCC features. This allows for an expanded channel representation of features. For example, 3-dimensional pitch features can be added to 40-dimensional MFCC features, resulting in a 43-dimensional feature tensor. This feature tensor allows for combining prosody information with frequency data, thereby capturing both the spectral and prosodic properties of the speech signal.

[0078] For example, as shown in Fig. 2, the speech signal 1021 corresponds to the tensor characteristic 1022; and an individual frame speech signal 2021 included in the speech signal 1021 corresponds to one time step in the tensor characteristic 1022 (namely, one row 2022 of the tensor characteristic). The frequency distribution information of the speech signal 1022 across different bands 2023 corresponds to one channel in the tensor characteristic (namely, one column 2024 of the tensor characteristic).

[0079] The speech signal can be represented as [T], where T denotes the duration of the speech signal. The tensor characteristic can be represented as [D, L], where L is the number of frame speech signals obtained by dividing the original speech signal into frames, and D is the number of channels used to describe the frequency distribution across different frequency ranges.

[0080] In step S103, cross-processing of the tensor characteristics by time and channel dimensions is performed, as a result of which it is possible to obtain a first set of cross-characteristics corresponding to the original speech signal.

[0081] The first set of cross-features represents features that combine frequency and time-series information in the speech signal. Cross-processing involves using a convolutional neural network to cross-process tensor features across the time and channel dimensions, allowing for the extraction of more contextual information from the speech signal.

[0082] A Fused Feature is formed by performing a fusion operation (e.g., 2D convolution) on tensor features along the time and channel dimensions, resulting in a new feature that combines the inter-channel differences and Contextual dependencies. This characteristic allows for a more effective description of the overall structure of speech content and the distribution of semantic characteristics.

[0083] Since the obtained tensor characteristics typically have a two-dimensional structure, with information across the two dimensions being relatively independent—the temporal dimension reflects exclusively sequential time data, while the channel dimension contains only frequency distribution information—they do not account for the interrelationships between dimensions. Interdimensional relationships include, but are not limited to, the following types of dependencies: the relationship of frequency characteristics at one point in time with frequency characteristics at another point in time; and patterns of change in a specific frequency characteristic of a speech signal along the time axis.

[0084] Therefore, it is possible to perform cross-processing of the tensor features corresponding to the speech signal along the time and channel dimensions, resulting in a first set of cross-features related to the given speech signal.

[0085] Optionally, the fusion processing of tensor characteristics can be performed by the fusion processing module, which achieves the fusion of speech signal characteristics for different time periods without disrupting the signal time series, realizing the mutual interweaving of information along the time and channel dimensions.

[0086] For example, in a given dialog scenario, user A says: “I placed an order for a product yesterday (pause), but it hasn’t been shipped yet today,” then the corresponding speech signal will be the phrase: “I placed an order for a product yesterday (pause), but it hasn’t been shipped yet today.” The tensor characteristics obtained after preprocessing the speech signal contain Frames corresponding to individual lexical units such as "yesterday," "placed an order," "product," "today," and "sent," distributed across the temporal dimension. Each frame speech signal has its own frequency characteristics in the channel dimension: for example, the word "yesterday" is characterized by a low-frequency stable feature, while the word "sent" is characterized by a high-frequency accented feature. As a result of cross-processing at this stage, it becomes possible to: associate the temporal position of the word "yesterday" with the frequency characteristics of the word "placed an order"; correlate the temporal position of the word "today" with the high-frequency features of the word "sent"; and record patterns of frequency changes before and after the "pause." As a result, the first set of cross-characteristics is formed, including temporal ordering, the dynamics of frequency changes, and logical relationships.This set of characteristics allows us to more accurately reflect the key semantics of user feedback about product shipment delays.

[0087] This fusion processing module may be any suitable module capable of implementing the specified functionality. For example, this fusion processing module may include multiple convolutional and splitting modules.

[0088] In some embodiments, the above step S103 may be implemented by performing the following steps S1031-S1033.

[0089] Step S1031: Perform one-dimensional convolution on the given tensor feature, thereby transforming it along the channel dimension into the first feature containing the specified number of channels.

[0090] At this stage, a 1D convolution module can be used to perform a one-dimensional convolution on the tensor feature. This module transforms the channel dimension of the tensor feature into a first feature comprising a specified number of channels. The preset number of channels can be a number that allows square root calculation, which simplifies subsequent splitting of the first feature into a three-dimensional second feature using a splitting module. Then, a 2D convolution module can be used to perform a two-dimensional convolution on the three-dimensional second feature, combining temporal ordering with spatial frequency distribution. The result is an integrated first set of cross-features that simultaneously preserves the temporal logical sequence over the time dimension and includes frequency-spatial features represented in a grid.

[0091] For example, a tensor characteristic can be represented as [D, L]. Since D is not a fixed number that can be square-rooted, it cannot be directly transformed into a regular two-dimensional spatial grid. In particular, for D=20, it is difficult to uniformly partition it into a symmetrical grid structure.

[0092] Therefore, we can first use the 1D convolution module to transform the channel dimension D of the tensor characteristic into a predetermined fixed number of channels N, resulting in the formation of the first characteristic, which can be represented as [N, L]. Here, N is a number that allows for square root calculation, such as 16, 25, etc.

[0093] Specifically, this is implemented by shifting the convolution kernel of the 1D convolution module along the temporal dimension. This exclusively selects and regroups data by channel dimension, without changing the temporal dimension L of the tensor feature. Regrouping is performed exclusively by channel dimension, ensuring that all time series information is preserved in the resulting first feature. For example, for D=20, the value of N is set to 16. In this case, one-dimensional convolution is performed by training 16 convolution kernels, which compress and regroup frequency information from 20 channels, transforming it into 16 key channels. This transformation results in the first feature. This preserves key frequency characteristics while simultaneously bringing the number of channels in line with the requirements for two-dimensional partitioning.

[0094] Step S1032: Perform dimension splitting of the first characteristic, resulting in the second characteristic in a transformed form.

[0095] At this stage, the partitioning module can be used to decompose the first characteristic [N, L] into a two-dimensional spatial grid by channel dimension, resulting in the formation of the second characteristic. Since N is a number that can be square-rooted, it can be represented as a product for example, N=16=4×4.

[0096] This allows the partitioning process to preserve the temporal dimension L of the first characteristic unchanged, only transforming the channel dimension from a one-dimensional 1D sequence into a two-dimensional 2D grid. In this case, each position [i, j] in the resulting grid corresponds to a specific combination of frequencies from the original channels. As a result, the channel dimension of the first characteristic is transformed into a two-dimensional structure. and the second characteristic can be represented as For example, for N=16, the second characteristic can be represented as [4, 4, L], with each 4x4 grid cell corresponding to frequency information from 16 channels.

[0097] Adjacent positions within the grid reflect frequency characteristics that are semantically related to each other.

[0098] Step S1033: Perform two-dimensional convolution on the second feature to obtain the first cross-feature corresponding to the given speech signal.

[0099] At this stage, the second characteristic Although the channel information has already been transformed into a two-dimensional grid, the time series information in the temporal dimension and the frequency-spatial characteristics represented in the grid remain independent of each other. To obtain the first cross-characteristic, their further integration is required, which simultaneously preserves the temporal logical sequence in the temporal dimension and integrates the frequency-spatial features encoded in the two-dimensional grid structure.

[0100] Channel Information refers to the representation of speech signal characteristics in various dimensions of acoustic-physical properties after processing in the frequency domain. For example, in Mel-Frequency Cepstral Coefficients (MFCC) or Filter Bank Features (Fbank), each channel corresponds to a specific frequency component or acoustic parameter; the combination of these channels forms the multidimensional characteristic space of the speech signal.

[0101] Specifically, this is implemented as follows: using a 2D convolution module, the 3D second characteristic is processed using 2D convolution. The convolution kernels of the 2D convolution module are simultaneously moved along the temporal dimension (L) and the spatial dimensions of the grid (height / width). This allows for the direct extraction of interrelated features reflecting the correlation between the frequency information of a specific time segment and a specific grid region. After convolution, the two-dimensional channel dimension of the second feature is transformed into interrelated features with a one-dimensional channel dimension. The resulting interrelated features can be represented as [N', L], where N' denotes the channel dimension after convolution. Thus, two-dimensional convolution allows for the integration of temporal order and spatial frequency distribution, resulting in the formation of the first cross-feature, which simultaneously preserves the temporal logical sequence across the temporal dimension and integrates the frequency-spatial features encoded in the two-dimensional grid structure.For example, when applying a convolutional kernel of size [2, 2, 3] (time step - 3 frames, grid step - 2x2) to the second characteristic of dimension [4, 4, L], the convolution process trains on frequency change patterns within: a time interval covering 3 frames; a spatial region of the grid of size 2x2. As a result, the first cross-characteristic is formed at the output, which contains both the temporal logical sequence along the temporal dimension and information about the frequency-spatial characteristics.

[0102] Thus, by scaling, partitioning, and combining tensor features by channel dimension, it is possible to transform one-dimensional features into two-dimensional ones that integrate temporal ordering and frequency characteristics. This allows for the unification of the semantic properties of the channel dimension and time series information, allowing the resulting first cross-feature to more fully reflect the semantic details of the speech signal.

[0103] Specifically, as shown in Fig. 3, after the speech signal is processed by the MFCC, the obtained tensor feature 1022 can be represented as [D, L]. Since the subsequent steps require applying the Convolutional Block Attention Module (CBAM) to process attention, and CBAM assumes working with a two-dimensional structure, there is a limitation: the number of channels D in the tensor feature 1022 is not a square number, so it cannot be directly converted into a regular two-dimensional spatial grid. For example, when D=20, it is impossible to uniformly divide the channels into a symmetric grid structure. Therefore, by using the 1D convolution module, the channel dimension D of the tensor feature 1022 can be converted into a predetermined fixed number of channels, where N is a square number, such as 16, 25, etc.Thus, tensor characteristic 1022 is transformed into a second characteristic 1023, represented as a two-dimensional spatial grid. This second characteristic 1023 can be expressed in the form - the length of the side of the square.

[0104] Thus, by scaling, partitioning, and combining tensor features by channel dimension, it is possible to transform one-dimensional features into two-dimensional ones that integrate temporal ordering and frequency characteristics. This allows for the unification of the semantic properties of the channel dimension and time series information, resulting in the resulting first cross-feature that more fully reflects the semantic details of the speech signal.

[0105] Step S104: Based on the first cross-characteristic, generate key information corresponding to the given speech signal.

[0106] In this case, key information is understood to be information automatically extracted from the speech signal, intended for quick familiarization with the main essence of the content of the audio recording.

[0107] At this stage, the cross-feature is fed to the encoder input for further identification of the underlying semantic relationships inherent in the feature structure. In this way, low-level features integrating temporal and frequency representations are transformed into high-level semantic features. The encoder can be any encoder with a suitable architecture; in particular, an encoder based on deep neural networks built on the Conformer or Transformer principles is acceptable. This cross-feature can be either the first cross-feature or a second cross-feature obtained by adjusting the first cross-feature.

[0108] Semantic Feature is a high-level abstract feature extracted by applying the attention mechanism and semantic coding to the cross-feature; it reflects the key semantic information of the speech content and serves as the basis for the formation of key information.

[0109] For example, cross-feature coding can be performed on the speech signal corresponding to the dialogue scenario between a customer and a customer service operator to identify the semantic chain of “customer complaint → operator response → proposed solution”, and extract key semantic information such as “complaint content” and “proposed solution”.

[0110] After encoding, the decoder can transform the semantic features into natural language text. The decoder can be any decoder with a suitable architecture; in particular, it is acceptable to use a decoder based on deep neural networks implemented using the Confonner or Transformer principles.

[0111] The said decoder may include a multi-level structure of stacked (sequentially superimposed) submodules and an output linear layer unit; each submodule may include a multi-head self-attention mechanism (multi-head self-attention unit), a cross-attention mechanism (cross-attention unit), and linear fully-connected feedforward layers (linear feedforward layers); the output linear layer unit is designed to generate the final key information.

[0112] At this stage, when a semantic characteristic is fed to the decoder input, its channel characteristic sequence, corresponding to each time frame along the time axis, is used as the decoder input sequence and fed to its input in chronological order. In this case, the channel characteristic sequence corresponding to each time frame along the temporal dimension of the semantic characteristic is associated with a separate frame of the original speech signal.

[0113] Specifically, a given sequence of features is fed as an input sequence to the first layer of the decoding layer of the decoder; after passing through the blocks located therein—the multi-head self-attention mechanism, the cross-attention mechanism, and the linear fully-connected feedforward layers—a multivariate feature optimized for the first time is formed. Then, the obtained optimized multivariate feature is used as an input sequence to the second layer of the decoding layer; similarly, after processing by the blocks of the second layer—the multi-head self-attention mechanism, the cross-attention mechanism, and the linear fully-connected feedforward layers—a multivariate feature optimized for the second time is formed at the output. After multiple multi-layer iterations, the multivariate feature obtained at the output of the last decoding layer is fed to the input of the output linear layer unit.This linear layer unit transforms the incoming multidimensional feature into a vocabulary space, after which the probabilities for each word are calculated using an activation function; as a result, a text sequence of words is formed that corresponds to the original input sequence.

[0114] During implementation, the specified steps are repeated multiple times for the sequence of features corresponding to each time stamp of the semantic feature along the time axis. This procedure is performed until all text words corresponding to the encoded semantic feature are obtained.

[0115] The set of generated text words represents the final key information.

[0116] In one possible embodiment, after the output linear unit layer transforms the multidimensional feature obtained at the output of the last decoding layer into the vocabulary dimensional space, and the activation function calculates probabilities for each word, a multi-branch decoding strategy may be applied to generate text words corresponding to each input sequence.

[0117] For example, after the final decoding layer of the decoder generates a multidimensional feature, the output linear unit layer transforms it into the vocabulary space, and the activation function calculates the probability distribution for each word, the multi-branch decoding module can independently use each of its branches to generate key information based on the resulting probability distribution. For example, if the multi-branch decoding module contains 20 branches, 20 possible key information variants can be generated at the output; then, through a majority voting mechanism, the optimal key information variant is selected and adopted as the final generation result.

[0118] For example, if the probability distribution obtained at the output of the final decoder layer corresponds to the semantic information "phone charges slowly," then each of the 20 branches of the multi-branch decoding module can independently generate its own version of the phrase, such as "phone charges very slowly," "phone charging speed is extremely slow," or "phone charges slowly." If the phrase "phone charges slowly" appears in 12 of the 20 generated phrases (i.e., is the predominant one), then this phrase is selected as the final key information at the system's output.

[0119] Thus, through multi-path sampling, it is possible to reduce the random error that occurs during single generation and increase the reliability of the generated key information.

[0120] In some implementations, dialogues between a customer and a support operator (within a support dialogue scenario) may be used as the speech signal to be processed. Specifically, during a conversation between a support operator and a customer, using dual-channel recording, the dialogue between the operator and the customer is combined along the time axis into a single channel, resulting in the speech signal to be processed. This speech signal is then processed to obtain its corresponding first cross-characterization. Based on this first cross-characterization, a text summary is generated that systematizes the content of the current dialogue between the user and the operator. The generated text summary represents the key information generated within the framework of this disclosure.

[0121] Optionally, in large language model application scenarios, a summary text can be extracted from the user's speech information before feeding it to the model. This summary text can then be passed to the large language model for further processing. The extracted summary text then constitutes the key information generated within this disclosure.

[0122] In another embodiment, audio files from a knowledge base may be used as the speech signal to be processed. Specifically, in a scenario for generating labels for audio files from a knowledge base, an audio file uploaded by a user is received as the speech signal to be processed; after processing this audio file, a first cross-feature corresponding to it is extracted. Based on this cross-feature, a label corresponding to this audio file is generated. The generated label constitutes the key information generated within the framework of this disclosure.

[0123] Meanwhile, the tag generated based on the audio file can be used as key information about this audio file in the process of searching and getting answers by the user.

[0124] The embodiment presented in this application implements cross-processing of a tensor characteristic by jointly transforming it in two dimensions: channel and time. This two-dimensional integration allows for the combination of two types of information: frequency information in the channel dimension and time series information in the time dimension. This results in the formation of a first cross-characteristic that simultaneously retains the ability to identify semantic relationships over extended sections of the speech signal and integrates multi-frequency components of the original audio stream. This approach enables a more detailed reproduction of the semantic structure of speech by eliminating the limitation of analyzing only local temporal patterns, which was characteristic of previous technical solutions.Synthesizing time-frequency characteristics significantly increases the expressiveness of the generated semantic characteristics, which directly contributes to a more accurate perception of the overall semantic content of the speech signal. The net effect of this processing is an increase in the reliability of the generated key information.

[0125] In existing technical solutions, after obtaining semantic characteristics, either exclusively unique information is extracted from a single speech recording or multiple speech characteristics are combined to form key information. However, given that extracting unique information from a single speech recording (e.g., a user's specific need in a specific dialogue with a support operator) results in recurring common information across multiple speech signals (e.g., typical complaints characteristic of a given product type) being dispersed or lost, this prevents the formation of consistent generalizations for an entire class of speech data.At the same time, the forced combination of several speech characteristics leads to the blurring of the unique details of an individual speech recording (for example, a specific proposal put forward by a particular participant during a meeting), as a result of which key information loses its focus and is unable to reflect individual differences.

[0126] Thus, in a number of embodiments, as shown in Fig. 4, the above step S104 may be implemented by executing steps S1041-S1044.

[0127] Step S1041: Based on the first cross-feature, a semantic feature corresponding to the speech signal is determined.

[0128] Although the first cross-feature already integrates time-series and frequency data, it should be noted that this combined information still only reflects local relationships at the signal level, not semantic connections. At its core, it remains an encoding of the physical characteristics of the speech signal and cannot directly relate to high-level semantic categories such as "user complaint" or "decision made at a meeting."

[0129] Thus, at this stage, the encoder can encode the first cross-feature to further reveal the underlying semantic relationships within the feature. In this way, low-level features integrating temporal and frequency representations are transformed into high-level semantic features. Any suitable encoder can serve as the encoder, for example, an encoder based on deep neural networks such as Conformer or Transformer.

[0130] In some embodiments, the encoder is configured to determine a semantic characteristic corresponding to a speech signal. The encoder includes at least one encoding module. For example, the encoder may include multiple encoding modules; when operating in parallel, they simultaneously detect semantic characteristics corresponding to a plurality of speech signals. In the implementation, the number of encoding modules may be consistent with the number of speech signals. For example, the number of encoding modules may coincide with the number of speech signals. Alternatively, the number of encoding modules may exceed the number of speech signals. Furthermore, the number of encoding modules may be less than the number of speech signals; in this case, the first cross-features corresponding to each of the plurality of speech signals are encoded incrementally.

[0131] Step S1042: Using the external attention module of the trained information generation model, the common components of semantic features extracted from the original speech signal are weighted. This operation allows the common feature corresponding to a given speech signal to be identified.

[0132] The common part of semantic characteristics corresponding to a speech signal may represent those characteristics that are ubiquitous in the semantic representation of this speech signal and other speech signals and possess the property of universality. These may include universal semantic patterns and generally recognized characteristics that remain after individual differences (e.g., speaker characteristics), environmental noise factors (e.g., external interference), and subtle differences in the mode of expression (e.g., word choice preferences) are excluded from the speech signal.

[0133] When generating key information from a speech signal, its semantic characteristics often contain a significant amount of redundant data (e.g., meaningless pauses, repeated modal particles) or interfering noise (e.g., background sounds, equipment noise). At the same time, the common part of the semantic characteristics corresponding to the speech signal represents a reliable signal, confirmed across multiple samples. Therefore, weighting the common part of the semantic characteristics allows for enhancing the reliable signal and suppressing noise components. In the context of dialogue systems, when processing speech signals containing digits, such as "1," "2," and "3," pronunciation variability is observed among different users, particularly in intonation contours and relative duration.At the same time, all pronunciation variants retain an invariant part: a set of acoustic characteristics that directly encode the semantic meaning of a digit. By deliberately enhancing these common, meaning-forming characteristics, reliable identification of numeric values ​​becomes possible even from a single speech signal.

[0134] The common part of semantic characteristics can be used to reflect a unified thematic component inherent in multiple speech signals. The individual part of semantic characteristics can be used to reflect the individual characteristics of each speech signal. In the context of customer support dialogues, a situation may arise where three speech signals from customers are linked by a single issue—a delivery delay. Each customer formulates a specific request: in the first signal, User A insists on compensation, in the second signal, User B requests a resend of the product, and in the third signal, User C demands expedited delivery. Thus, the common component of all three signals is the delivery delay itself, while the individual components are the specific requests of each customer, reflecting their personal needs and expectations.

[0135] In this case, the external attention module can be used to weight the common part of the semantic features corresponding to the speech signal. The external attention module can include multiple convolutional layers.

[0136] At this stage, the semantic features extracted from the speech signal are fed to the external attention module. Using the multiple convolutional layers included in this module, the incoming semantic features are convolved. This allows for filtering out individual differences and identifying key features—the very basic characteristics that are common to both the analyzed speech signal and the reference samples used in the model training process.

[0137] In some embodiments, the external attention module is structured as a sequence of three convolutional layers: a first convolutional layer, a second convolutional layer, and a third convolutional layer. The actions in step S1042 can be implemented through the following steps A1 and A2.

[0138] Sub-step A1: performing weighting on the common part of the semantic feature corresponding to the speech signal by means of the common parameters included in the first convolutional layer and the second convolutional layer, to obtain a compressed feature corresponding to the speech signal;

[0139] Here, the common parameters refer to the weights of the convolutional kernels that the external attention module of the information generation model learns during training. These weights demonstrate a high correlation with the common characteristics of the entire training set.

[0140] Optionally, when feeding semantic features to the model's input, the first and second convolutional layers, using their built-in common parameters, can extract and enhance those features that correspond to intersample regularities, i.e., invariant characteristics inherent to a set of samples. This achieves targeted enhancement of the common component of semantic features by assigning it a higher weighting coefficient.

[0141] For example, the first convolutional layer can be used to perform channel-dimensional scaling of the semantic characteristics of a speech signal. This allows for a reduction in the channel dimensionality of the semantic characteristics, thereby reducing the computational complexity of subsequent operations; pre-extracting key information from the semantic characteristics; and preparing the data for subsequent extraction of relationships between samples at subsequent processing stages. Then, using the second convolutional layer, 1D convolution can be performed on the semantic characteristics pre-compressed by the channel dimension.

[0142] The parameters of the convolutional kernels of the first convolutional layer are considered shared parameters—they represent weights learned by the model during training and are highly correlated with common characteristics shared across multiple samples. When semantic features are input to the model, these parameters assign a higher weight to the shared component of the features. This enables two-stage processing: preliminary filtering and emphasis on significant shared features, as well as effective suppression of irrelevant individual variations that act as noise components. The parameters of the convolutional kernels of the second convolutional layer are the inverse parameters of the first convolutional layer (i.e., a weight matrix rotated / inverted relative to the original) and are also considered shared parameters. This organization enables cross-selection of features across both horizontal and vertical dimensions.

[0143] Specifically, the first convolutional layer is primarily focused on identifying general patterns in the transverse (i.e., temporal) dimension, for example, discovering universal patterns of change in semantic characteristics over time. Meanwhile, the second convolutional layer, using inverse parameters, is primarily aimed at identifying complementary general patterns in the longitudinal (i.e., channel) dimension, in particular, capturing universal relationships between different frequency channels. Using the common parameters contained in the first and second convolutional layers, the common part of the semantic characteristics corresponding to the speech signal is weighted. The intersection region of their cross-influence corresponds to the most stable common part of the semantic characteristics (for example, the specific frequency characteristic of a certain semantic element at a specific point in time).Such overlapping characteristics are significantly amplified by additional weighting.

[0144] Sub-step A2: Perform convolution of the compressed feature through the third convolutional layer to obtain the overall feature after channel reconstruction.

[0145] As a result of joint processing by two convolutional layers, semantic features with enhanced common components undergo further dimensionality reduction (e.g., from 32 to 16 dimensions). The output is a result in which the common components have been weighted. Next, a third convolutional layer uses 1D convolution to restore the channel dimensionality of the compressed features to a value suitable for subsequent processing (e.g., from 16 to the original 128 dimensions of semantic features). The restored common features: fully preserve the enhanced inter-sample common information extracted in the first two layers; and have channel dimensionality consistent with subsequent processing stages (e.g., with the individual feature fusion module).

[0146] Thus, using the common parameters contained in the first and second convolutional layers, the common portion of the semantic features is weighted, resulting in compressed features with enhanced common components. Next, the third convolutional layer restores the channel dimensionality of the compressed features, returning them to their original dimensionality. This enables: efficient identification of common patterns inherent in a given speech signal and other speech signals; simultaneously ensuring the computational efficiency of subsequent processing stages; thereby improving the performance of key information generation.

[0147] Step S1043: Using the external attention module of the trained information generation model, the common components of semantic features extracted from the original speech signal are weighted. This operation allows the common feature corresponding to a given speech signal to be identified.

[0148] The self-attention module is designed to weight the individual semantic characteristics corresponding to a speech signal, resulting in the formation of individual characteristics corresponding to a given speech signal. There can be at least one self-attention module.

[0149] The individual part of the semantic characteristics corresponding to a speech signal represents the characteristics unique to this signal, distinguishing it from other speech signals. These are specific markers of the speech signal, reflecting the speaker's characteristics, the context of the situation, and nuances of expressiveness. These characteristics serve as key distinguishing features that allow us to differentiate between different speech signals.

[0150] When generating key information based on a speech signal, the semantic characteristics of that signal also include its unique individual characteristics that distinguish it from other speech signals. For example, for the same semantic content, differences in individual characteristics may be observed depending on the nuances of its wording. For example, the phrases "I'll go tomorrow" and "I plan to go tomorrow" have similar basic semantics but differ in individual characteristics reflecting the details of their presentation. Consequently, when multiple speech signals have similar basic semantic content (common characteristics), the lack of enhancement of individual components will lead to the formation of formulaic, monotonous key information.

[0151] Thus, at this stage, it is possible to: weight the general portion of semantic characteristics corresponding to a speech signal, obtaining the general characteristics of a given speech signal; and simultaneously weight the individual portion of semantic characteristics corresponding to a speech signal, generating the individual characteristics of a given speech signal. This approach allows for a balance between compactness and completeness when generating key information.

[0152] When processing a speech signal using the self-attention module, the module's individual parameters can direct its operation to assign higher weight to the individual portion of the input semantic features (features similar to the characteristic individual features from the training samples, such as specific frequency or sequence characteristics). This enables targeted enhancement of individual feature components.

[0153] In some embodiments, the self-attention module may include multiple linear layers.

[0154] For example, semantic features corresponding to a speech signal can be fed to the input of a self-attention module. This module calculates the weights of the connections between different time samples and different channels within a given semantic feature. Regions with high weight values ​​correspond to key unique information contained within a given speech signal.

[0155] For example, if a certain characteristic (e.g., “compensation”) is particularly prominent (has a high weight of internal attentiveness / internal attention) in the current speech signal, while it is absent or has a low weight in other signals, then it will be identified as an individual characteristic.

[0156] Thus, the external attention module weights the general portion of semantic characteristics corresponding to a speech signal, resulting in the formation of general characteristics. This enables the effective identification of recurring general patterns present in the speech signal. Simultaneously, the self-attention module weights the individual portion of semantic characteristics, ensuring the precise extraction of the personalized characteristics of a given speech signal. Generating key information based on both general and individual characteristics achieves a dual effect: it reflects the general trends inherent in multiple speech signals and preserves the unique characteristics of each individual speech signal. This ultimately ensures the completeness of the key information, simultaneously encompassing all key information and accurately reflecting the individual differences between signals.

[0157] Step S1044: Based on the general and individual characteristics, key information corresponding to the speech signal is generated.

[0158] At this stage, the general and individual characteristics of a given speech signal can be merged, resulting in the formation of a combined characteristic. This characteristic includes: general characteristics—information reflecting the similarities of a given speech signal with other speech signals; individual characteristics—information characterizing the unique properties of a given speech signal. The combined characteristics can then be encoded. Upon completion of encoding, the underlying semantic relationships within the combined characteristic are revealed, thereby forming semantic characteristics corresponding to a given speech signal.

[0159] Finally, the semantic features can be fed to the input of the decoder, which maps them into natural language text.

[0160] In particular, to map semantic characteristics into natural language text, you can refer to the previously described method for implementing step S104.

[0161] Thus, by identifying the common characteristics of multiple speech signals and the individual characteristics specific to each individual speech signal, the formation of key information based on these common and individual characteristics achieves a dual effect: it reflects the general trends observed across multiple speech signals and preserves the unique features of each individual speech signal. This ensures that the key information is simultaneously: complete – encompassing all key information; and accurate – reliably reflecting the individual differences between signals.

[0162] In some embodiments, the above step S1044 may be implemented by performing the following steps B1 and B2.

[0163] Step B1: Perform weighted aggregation of common and individual characteristics to obtain the second aggregation characteristic.

[0164] In this case, general and individual characteristics carry different types of key information: general characteristics represent information reflecting the similarities of a given speech signal with other speech signals, and individual characteristics contain unique information inherent only to a given speech signal.

[0165] At this stage, it is possible to carry out a step-by-step combination of general and individual characteristics by dynamically weighing them, resulting in the formation of the second combination characteristic.

[0166] Specifically, based on the degree of correlation between common and individual characteristics, corresponding weights can be determined: one weight corresponding to the common characteristic, and one weight corresponding to the individual characteristic. Once the weights for the common and individual characteristics are obtained, the common and individual characteristics can be summed element by element using these weights, resulting in a second fusion characteristic. This second fusion characteristic simultaneously preserves the common key information inherent in this speech signal and other speech signals, and includes unique details characteristic only of this speech signal. This ensures a balance between common and individual information within the speech signal.

[0167] In some embodiments, the above step B1 may be performed by performing the following sub-steps - B11 - B13.

[0168] Step B11: Combine the common and individual characteristics to obtain a combined characteristic.

[0169] In this case, common characteristics serve to reflect the overall theme of a given speech signal in comparison with other speech signals, while individual characteristics reflect the unique information inherent to a specific speech signal. In the context of customer support dialogues, a situation may arise where three speech signals from customers are linked by a single issue—a delivery delay. Each customer formulates a specific request: in the first signal, User A insists on compensation, in the second signal, User B requests a resend of the product, and in the third signal, User C demands expedited delivery. Thus, the common component of all three signals is the delivery delay itself, while the individual components are the specific requests of each customer, reflecting their personal needs and expectations.

[0170] Common and individual characteristics carry different types of key information: common characteristics form the "basic structure" of key information (e.g., "this dialogue addresses a logistics issue"), while individual characteristics represent the "differentiating details" of key information (e.g., "user A's specific requirement"). The concatenation operation, through simple and direct dimensionality expansion, ensures the complete preservation of the key information contained in common and individual characteristics and avoids the blurring of characteristics that can occur when combining them in a weighted manner.

[0171] This combines common and individual characteristics to produce a combined characteristic. The combination can be performed directly across the time or channel dimension.

[0172] The resulting unified characteristic allows for the general and individual information to be placed in a single characteristic space, forcing the subsequent processing module to simultaneously consider the relationship between them. The subsequent processing module may also include a decoder.

[0173] Thus, when generating key information, general and individual characteristics are simultaneously considered, preventing the omission of information during the key information generation process. This approach ensures a dual result: the generated key information fully reflects the overall thematic focus while accurately preserving individual details.

[0174] Step B12: Perform a convolution operation on the combined characteristic to obtain a relationship characteristic.

[0175] The obtained relationship characteristic serves to formalize the description of the nature of the interaction between the general and individual components of the characteristics, both in terms of channel dimension and time dimension.

[0176] In particular, by performing the convolution operation, it is possible to identify the deep relationships between general and individual characteristics, based on which a relationship characteristic is formed; this characteristic allows one to simultaneously reflect the correlation of semantic attributes along the channel dimension and temporal logic along the temporal dimension.

[0177] This is due to the fact that the preceding concatenation operation merely physically combines common and individual characteristics without revealing their internal relationships. In contrast, the relationship characteristic is formed through convolution, which specifically examines these relationships. This approach avoids misalignment between common and individual characteristics in subsequent merging stages and transforms the physically combined characteristic into a structured characteristic with semantic relationships. This creates the necessary foundation for the precise alignment of common and individual information in subsequent processing, guaranteeing the logical coherence and semantic consistency of the generated key information.

[0178] By applying the convolution operation to the combined characteristic, a relationship characteristic is formed that allows one to identify spatiotemporal correlations: in particular, to capture channel interactions (the relationship between common and individual characteristics in the channel dimension) and temporal logic (sequential dependencies in the temporal dimension). For example, if a speech signal contains the utterance, "Due to a delivery delay, the customer filed a complaint, after which User A demanded compensation in the amount of 200 yuan," the identified channel relationship will represent the semantic connection between the phrases "delivery delay" and "compensation of 200 yuan," and the established temporal relationship will reflect the chronological order of events—"first the complaint, then the compensation demand."

[0179] Step B13: Based on the gate weights corresponding to the relationship characteristic, perform weighting of the general and individual characteristics to obtain the second association characteristic.

[0180] For example, a two-dimensional convolution operation can be applied to the combined feature to reveal the relationships between the global and individual features of a speech signal in the time and channel dimensions. The convolution result can then be mapped onto the interval [0, 1] using an activation function to generate gate weights. Once the gate weights are obtained, they are used to weight the combined feature and the individual feature, resulting in a second combined feature. This allows the contribution of the global and individual features to be dynamically adjusted.

[0181] Thus, firstly, by combining common and individual characteristics, a combined characteristic is formed, allowing for the complete preservation of the original information of both characteristic types and eliminating data loss. Secondly, subsequent processing of the combined characteristic using convolutional operations allows for the effective identification of deep relationships between common and individual characteristics in the channel and over time. Thirdly, based on the calculated weighting coefficients, a weighted combination of common and individual characteristics is performed, enabling flexible adjustment of the component significance. As a result, the generated key information logically corresponds more closely to the original semantics of the speech signal; it contains key themes common to multiple speech signals; and it accurately conveys the unique details of a specific individual speech signal.

[0182] Step B2: Perform decoding of the second combination characteristic to obtain the key information corresponding to the given speech signal.

[0183] At this stage, a decoder can be used to decode the second union characteristic, converting this characteristic into natural language text. At this stage, a decoder can be used to convert the second union characteristic into natural language text. The decoder can be any suitable decoder, for example, a decoder based on deep neural networks such as Conformer or Transformer. Moreover, the specified decoder can include a multi-layer structure of stacked (sequentially superimposed) submodules and an output linear layer-unit; each submodule can include: a multi-head self-attention mechanism (multi-head self-attentive unit), a cross-attention mechanism (cross-attentive unit), as well as linear fully-connected feedforward layers (linear layers with signal transmission); the output linear layer-unit is designed to form the final key information.

[0184] Thus, the weighted combination of common and individual characteristics ensures that the generated key information simultaneously contains common information shared by multiple speech signals and accurately reflects the unique details of a single speech signal. Moreover, the weighting mechanism enables dynamic adjustment of the ratio of the contribution of common and individual characteristics, ensuring that the decoded key information not only logically corresponds to the original semantics of the speech signal but also fully encompasses the key information and accurately conveys individual differences.

[0185] In some implementations, as shown in Fig. 5, for a semantic characteristic 501, using an external attention module 502, a common characteristic 504 is extracted from the semantic characteristic 501 corresponding to the speech signal, reflecting the common features of this speech signal with other speech signals; while for the same semantic characteristic 501, using a self-attention module 503, an individual characteristic 505 is extracted from the semantic characteristic 501 corresponding to the speech signal, characterizing the features of this particular speech signal.

[0186] Then, the common characteristics 504 and the individual characteristics 505 are combined by concatenation to obtain a combined characteristic 506; after that, a convolution operation is applied to the combined characteristic 506, resulting in a relationship characteristic 507.

[0187] After obtaining the relationship characteristic 507, based on the gate weights corresponding to the relationship characteristic 507, the general characteristic 504 and the individual characteristic 505 are weighted, resulting in the formation of the second association characteristic 508. In this case, the gate weights include weight 1 and weight 2, where weight 1 corresponds to the weight coefficient for the general characteristic 504, and weight 2 corresponds to the weight coefficient for the general characteristic 505.

[0188] Thus, the embodiment of the claimed solution under consideration enables dynamic adjustment of the ratio of contributions from general and individual characteristics. This allows the key information generated during decoding not only to logically correspond to the original semantics of the speech signal, but also to simultaneously fully capture the key information and accurately reflect individual differences.

[0189] In some implementations, performing step S1041 may be performed by performing steps T1-T3 to determine a semantic characteristic corresponding to the speech signal.

[0190] Step T1: Determine the weights of speech segments corresponding to different feature points in the first cross-feature associated with the analyzed speech signal.

[0191] In this case, the characteristic value at each intersection point of the time axis for individual time steps and the channel axis for individual channels within the first cross-characteristic can be taken as its individual characteristic point. Such a characteristic point corresponds to the speech signal characteristic at the intersection point, determined by the time index t and the channel index c.

[0192] For example, if the first cross-characteristic is represented in the form F1∈R B×L×D , then each point of the characteristic in its composition can be designated as f t, с, where t is the time step index and c is the channel index. In this case, the element f 100, 2 corresponds to the energy value of the speech signal at the intersection point corresponding to the 100th millisecond (time step) and the 2nd channel index (high-frequency region).

[0193] A speech segment is a continuous fragment of a speech signal that carries actual semantic content. This continuous fragment can be a single word, a phrase, or an entire phrase. For example, in the sentence "I'm going to a meeting tomorrow," the lexical units "tomorrow" and "meeting" are key speech segments.

[0194] The speech segment weight corresponding to a feature point quantifies the degree of association between a given feature point and a specific speech segment. The higher the speech segment weight, the stronger the correlation between the temporal and frequency information contained in a given feature point and the semantic content of the corresponding speech segment. The lower the speech segment weight, the higher the probability that a given feature point corresponds to an uninformative signal, such as background noise or modal particles.

[0195] The first cross-feature contains a large number of feature points, but not all of them contribute to the semantics. Therefore, by calculating weighting coefficients for each feature point, it is possible to filter out insignificant feature points, directing subsequent processing to work with highly significant feature points, i.e., with acoustic signals closely related to the semantics of the speech segment.

[0196] At this stage, the weighting coefficients of the speech segments corresponding to each characteristic point in the first cross-characterization can be first determined in the process of its initial spatio-temporal weighting.

[0197] The initial space-time weighing process can be realized as follows:

[0198] First, the global statistical features are extracted from the first cross-feature using the global pudding operation. Specifically, the averaging pudding operation and the maximizing pudding operation can be used to extract the global statistical features from the first cross-feature, resulting in two types of global statistical features: the averaging pooling feature and the maximizing pudding feature. The averaging pooling operation calculates the average value for each channel dimension in the first cross-feature. This average value reflects the average energy level of the corresponding channel over the entire duration of the speech signal and makes it possible to capture the overall trend of the first cross-feature.For example, a high average value in low-frequency channels may correspond to a stable narrative tone of speech, which serves as a guide to determining the basic semantics of a speech segment. Maximizing pooling calculates the maximum value for each channel dimension in the first cross-feature. Maximizing pooling identifies the most significant signals in each channel, thereby identifying key characteristics of the first cross-feature. For example, a sudden high-frequency peak may correspond to an "accent word" or "keyword."

[0199] After obtaining the global statistical characteristics of the first cross-feature using a multilayer perceptron (MLP), the average pooling characteristics and the maximizing pooling characteristics are processed separately. As a result of applying the MLP, the transformed pooling characteristics, both average pooling and maximizing pooling, are mapped into vector characteristics of the same dimension as the original first cross-feature. The outputs are: a global average vector characteristic and a global key vector characteristic.

[0200] After obtaining the global mean feature vector and the global key feature vector, their elements are summed element-by-element to form an integrated result that combines both global representations. An activation function is then applied to the resulting combined result, compressing the values ​​into the range [0, 1], resulting in the generation of an attention matrix. Each attention value in this attention matrix represents a speech segment weight corresponding to a single feature point in the first cross-feature.

[0201] Step T2: Based on the weight coefficients of the speech segments corresponding to the individual characteristic points, adjust the first cross-characteristic associated with the speech signal to obtain a second cross-characteristic also associated with the given speech signal.

[0202] In this embodiment, after obtaining the attention matrix, the first cross-feature can be element-wise multiplied by the attention weights with element-wise multiplication, resulting in a second cross-feature corresponding to the original speech signal. This allows features with high weight coefficients to be highlighted and features with low weight coefficients to be suppressed. Features with high weight coefficients correspond to semantically key fragments of the speech signal, while features with low weight coefficients represent noise components and redundant information in the speech signal. Thus, the features obtained through preliminary weighting enable the identification of semantically key content elements from a global analysis perspective.

[0203] In one possible embodiment, after adjusting the first cross-feature corresponding to the speech signal based on the weighting coefficients of the speech segments associated with individual points of the characteristics, it is possible to repeatedly apply the above-described procedures of steps T1 and T2 to the adjusted first cross-feature in order to perform a detailed spatio-temporal weighting of this adjusted characteristic.

[0204] Then, the pre-weighted features are successively subjected to pooling, MLP, feature fusion, and reweighting operations, resulting in the formation of a second cross-feature, which can strengthen the semantic relationships from the perspective of local analysis.

[0205] Thus, by double-weighting the first cross-feature, the following is achieved: after the first weighting, globally significant features are enhanced; after the second weighting, based on the pre-weighted features, local relationships are enhanced. The result is a second cross-feature in which both significant channels and important time sequences are simultaneously enhanced.

[0206] Step TZ: Perform semantic encoding of the second cross-feature corresponding to the speech signal to obtain a semantic feature also associated with this speech signal.

[0207] Thus, by determining the weights of the speech segments corresponding to each feature point in the first cross-feature, each feature element is assigned a weight reflecting its semantic significance: feature points corresponding to key semantic information in the first cross-feature are assigned high weights, while points associated with redundant information are assigned low weights. This transforms the first cross-feature, which integrates temporal and frequency representations, into a feature containing information about semantic significance. As a result, the adjusted second cross-feature demonstrates a closer relationship with the actual semantics of the speech signal.

[0208] However, during the two described stages of dynamic weighting, aimed at extracting global key features of the speech signal, redundant information, such as modal particles, background noise, and gibberish, is deliberately suppressed. In this process, along with redundant data, necessary local details are also suppressed, which may lead to under-recognition of some key features. Furthermore, the tensor features obtained as a result of speech signal preprocessing are formed using a sliding window that extracts characteristic signal features. This approach enables the identification of local sequential patterns without relying on global statistical regularities. Due to this, these features demonstrate increased resilience to noise, are capable of effectively processing complex input data, and are better adapted to rare semantic constructions.Thus, at this stage it is possible to combine two types of characteristics, which allows us to simultaneously take into account both the globally significant characteristics of the speech signal and its local sequential features.

[0209] In some embodiments, the above step T3 may be implemented by performing the following sub-steps - T31-T32.

[0210] Substep T31: Based on the tensor characteristic corresponding to the speech signal and the second cross-characteristic also associated with this speech signal, determine the first fusion characteristic associated with this speech signal. The first fusion characteristic preserves time series information identical to the temporal information parameters of the tensor characteristic.

[0211] Since the tensor characteristic contains speech signal information across both time and channel dimensions, it encompasses a rich set of the original acoustic details present in the speech signal. For example, the speech signal contains temporal variations in frequency characteristics and energy components, but the semantic coherence of such data remains weak, with noise interference and redundant information present. The second cross-characteristic, although enhanced by weighting speech segments and emphasizing semantic-related features, is distinguished by a clear semantic focus and minimal redundancy. However, it may lose some subtle but critical original details, such as important commands spoken in a quiet voice.Thus, one can combine the tensor feature and the second cross-feature to obtain the first merge feature, which combines both the focus on semantics and the preservation of the original details.

[0212] In one of the possible embodiments, taking into account the discrepancy in the number of time steps and the number of channels between the tensor characteristic and the second cross-characteristic, at the initial stage their spatio-temporal unification is carried out - the coordination of dimensions along the time axis and along the channel axis.

[0213] Specifically: in the case where the quantity the number of steps of the tensor characteristic is T, and the number If the number of steps of the second cross-characteristic is equal to T / 4, a one-dimensional convolution with a step of 4 and a downsampling operation can be applied to the tensor characteristic. This will reduce the number of time steps of the tensor characteristic from T to T / 4, thereby ensuring that the tensor characteristic and the second cross-characteristic are aligned along the time axis.

[0214] When the number of channels of the tensor characteristic is D and the number of channels of the second cross-characteristic is 64, the number of channels of the tensor characteristic D can be mapped to 64 by means of one-dimensional convolution. This ensures that the tensor characteristic and the second cross-characteristic are consistent in channel dimension.

[0215] After this, a weighted residual fusion can be performed on the tensor feature and the second cross-feature, provided that they are already matched on both the time and channel dimensions. This transformation produces the first fusion feature.

[0216] For example, one can first train and obtain the fusion coefficients αβ using a lightweight MLP-type neural network, which determine the weighting relationships between a tensor feature and a second cross-feature. The ranges of the fusion coefficients α and β are [0, 1], and the condition α+β=1 is satisfied. The fusion coefficient α can dynamically reflect the significance of a tensor feature. The fusion coefficient β can dynamically reflect the significance of the second cross-feature.

[0217] Then, the two characteristics, the tensor characteristic and the second cross characteristic, can be combined, taking into account their corresponding combination coefficients; as a result of this transformation, the first combination characteristic is formed.

[0218] Specifically, each feature value in the tensor feature can be first elementwise scaled using the fusion coefficient α, resulting in an α-adjusted tensor feature preserving local details. Then, each feature value in the second cross-feature can be similarly scaled elementwise using the fusion coefficient β, resulting in a β-adjusted second cross-feature emphasizing key semantic features. A "bridging" operation can then be performed between the second cross-feature emphasizing key semantic features and the tensor feature preserving local details. This process results in the first fusion feature with completed feature bridging.

[0219] The procedure for bridging the characteristics may include, but is not limited to, the following processing methods: element-by-element addition of the second cross-characteristic and the tensor characteristic; combining the channels of the second cross-characteristic and the tensor characteristic.

[0220] Step T32: Perform semantic encoding of the first association characteristic corresponding to the speech signal to obtain a semantic characteristic also associated with the speech signal.

[0221] At this stage, the first union characteristic is fed to the input of the encoder for further identification of the deep semantic relationships embedded in the structure of the first union characteristic.

[0222] Thus, by combining the tensor feature and the second cross-feature to obtain the first fusion feature, it is possible to ensure that the first fusion feature, while preserving the original temporal and frequency representations of the speech signal, additionally includes information about the semantic significance of this speech signal. This allows the first fusion feature to simultaneously contain global semantics and strictly preserve the temporal sequence of the original speech signal; as a result, when obtaining a semantic feature based on the first fusion feature, the accuracy of this semantic feature is increased.

[0223] Fig. 6 is a third schematic illustration of the process of implementing the method for generating key information proposed in this embodiment of the application. As shown in Fig. 6, the method includes steps S501-S506.

[0224] Step S501: Obtain a plurality of speech signal samples.

[0225] At this stage, speech samples can be obtained from various scenarios. Both the speakers corresponding to the speech samples and the tasks associated with these samples can include several types. Examples include scenarios such as everyday conversations, audio recordings of meetings, and speech communication in noisy environments. Speakers can vary in gender, age, and accent, and target tasks can involve detecting a wide range of keywords.

[0226] This allows the model to be provided with a variety of input data for training and ensures that the model learns the universal patterns of speech signals that appear in different scenarios.

[0227] Step S502: Using the learnable information generation model, preprocess a plurality of speech signal samples to obtain a corresponding tensor feature for each speech signal sample. For each tensor feature associated with a speech signal sample, perform cross-processing across the time and channel dimensions to obtain a first cross-feature corresponding to the speech signal sample. Based on the first cross-feature associated with each speech signal sample, generate key information corresponding to that sample.

[0228] The information generation model can be implemented using a neural network. In practice, multiple speech signal samples can be fed to the input of the information generation model to be trained, resulting in the corresponding key information being obtained for each sample.

[0229] At this stage, multiple speech signal samples can be preprocessed, converting them into tensor characteristics represented in the frequency domain. Preprocessing is designed to extract time series and frequency information from each speech signal. Preprocessing can also include converting the speech signal into a Mel Spectrogram, which allows one to obtain the tensor characteristics of the speech signal in the frequency domain.

[0230] Optionally, if necessary, the specified preprocessing operations can be used to transform the speech signal from the time domain into tensor characteristics in the frequency domain, allowing for the extraction of time series information along the time dimension and frequency information along the channel dimension. Preprocessing can be performed either individually for each speech signal sample or simultaneously for a group of speech signal samples.

[0231] Since the obtained tensor characteristics typically have a two-dimensional structure, with information across the two dimensions being relatively independent—the temporal dimension reflects exclusively sequential time data, while the channel dimension contains only frequency distribution information—they do not account for the interrelationships between dimensions. Thus, it is possible to perform combined processing of multiple tensor characteristics using a combined processing module, which enables the fusion of speech signal characteristics for different time intervals without disrupting the signal's time series, implementing the interweaving of information across the temporal and channel dimensions.

[0232] As a result of cross-processing, the first cross-feature is formed, which simultaneously contains interrelated data on temporal dynamics and frequency distribution. Compared to features limited to a single dimension, it more fully reflects the essential properties of the speech signal. This feature can be fed to the encoder input for subsequent identification of the underlying semantic relationships inherent in its structure. Thus, low-level features integrating temporal and frequency representations are transformed into high-level semantic features.

[0233] During the initial training process, random initialization of the parameters of the external attention and self-attention modules in the architecture of the information generation model to be trained is allowed. The key element of the external attention module is the convolutional layer, designed to extract common components in semantic characteristics. The initialized parameters include the weights of the convolutional kernels (including numerical values ​​for 3x3 kernels) and bias parameters. These parameters define the mechanism by which the convolutional operator extracts common patterns, including universal patterns of tonal variations in the speech signal and the frequency characteristics of frequently used lexical units. The core of the self-attention module consists of three linear matrices—Q (query), K (key), and V (value)—designed to calculate attention weights.The initialized parameters include the weighting coefficients of the three matrices mentioned, as well as their corresponding biases. These parameters determine how individual differences between elements are estimated at different time steps and in different channels. For example, the speaker's individual speech rate, fluctuations in their emotional state, and so on.

[0234] During the feedforward phase of training, the external attention and self-attention modules process speech samples using the current parameters, generating input data for subsequent loss function calculation. In the external attention module, a convolutional layer weights semantic features to extract common features; in the self-attention module, the matrices Q, K, and V are used to calculate attention weights, which are used to generate individual features.

[0235] After combining the general and individual characteristics, the key information is generated by the decoder included in the information generation model.

[0236] Step S503: Based on the key information corresponding to each speech signal sample and the label data of each speech signal sample, the target value of the loss function is determined.

[0237] In this case, the target loss function value serves to quantify the discrepancy between the label data of each speech signal sample and its corresponding key information. This metric allows us to evaluate how accurately the external attention module identified common features, whether the self-attention module correctly identified individual features, and whether the result of their fusion matches the expected values ​​specified in the labels.

[0238] Optionally, the key information generated by the model and its corresponding label data can be obtained for each speech signal sample. An individual loss value is then calculated for each sample using a predefined loss function. This value quantitatively reflects the discrepancy between the key information and labels for a specific speech signal sample. Individual loss values ​​across all samples are then aggregated to obtain an overall target loss value, which characterizes the overall model error on the entire training dataset.

[0239] The loss function may include, but is not limited to, the following: cross-entropy loss function, binary cross-entropy loss function, negative log-likelihood loss function, and edit distance loss function.

[0240] At this stage, the target loss function value is calculated based on the discrepancy between the generated key information and the corresponding sample labels. The larger the loss function value, the less reasonable the parameter settings of both modules are, requiring their adjustment via backpropagation.

[0241] Step S504: Based on the target value of the loss function, at least one iteration of updating the parameters of the information generation model is performed, resulting in a trained information generation model.

[0242] At this stage, the updating of the parameters of the information generation model can be performed under the condition that the target value of the loss function satisfies the pre-set criteria.

[0243] In this case, the predetermined conditions may include, among other things, meeting convergence criteria and similar requirements. Moreover, the convergence conditions may include, among other things, a situation where the target value of the loss function exceeds a predetermined threshold. During implementation, those skilled in the art are free to independently determine the predetermined conditions based on practical needs; in the presented embodiment of the present disclosure, no specific limitations are set.

[0244] The parameters of the information generation model may include, among others, the parameters of the convolution kernels in each convolutional layer of the external attention module of the model, and the parameters of the linear matrices Q, K, and V in the self-attention module of the information generation model.

[0245] Optionally, partial derivatives (gradients) of the loss function with respect to each parameter of the external attention module of the information generation model can be calculated, as well as partial derivatives (gradients) of the loss function with respect to each parameter of the self-attention module of this model; then, the parameters are adjusted in the direction opposite to the gradient to minimize the value of the loss function.

[0246] In the external attention module, the gradient source is the partial derivatives (gradients) of the loss function with respect to the convolutional layer parameters—that is, the convolution kernel weights and bias values. If key patterns are missed during feature extraction, the value of the loss function increases; in this case, the gradients of the convolutional kernel parameters indicate the need to enhance the extraction of features of a certain type, in particular, characteristic frequency or temporal features. Based on the gradient magnitude, the convolutional kernel weights can be adjusted so that subsequent feature extractions better match the universal patterns embedded in the labels. For example, sensitivity to the characteristics of high-frequency lexical words can be increased.

[0247] In the self-attention module, the gradient source is the partial derivatives of the loss function with respect to the parameters of the Q, K, and V matrices. If, during the extraction of individual features, attention is mistakenly focused on stylistic nuances of the text that are irrelevant to the task, the value of the loss function increases; in this case, the gradients of the Q, K, and V matrices indicate the need to reduce the attentional weights for these tonal features. The weights of the Q, K, and V matrices can be adjusted; in particular, it is permissible to modify the matrix elements associated with interfering stylistic features so that the attentional weights more accurately focus on the individual differences required by the labels.

[0248] These processes—forward propagation, loss function calculation, and backpropagation—are repeated iteratively until the target loss function satisfies predetermined conditions. At each iteration, the parameters of both modules are fine-tuned based on the loss function gradients calculated for the current batch of samples. As the iterations continue, the external attention module gradually learns to more accurately detect general patterns, while the self-attention module learns to more accurately recognize individual differences.

[0249] In some embodiments, if the target value of the loss function does not satisfy predetermined conditions, the current information generation model is adopted as the trained information generation model. If the target value of the loss function is less than a set threshold, the current information generation model can be adopted as the trained information generation model.

[0250] When the loss function converges, the parameters of the external attention module and the self-attention module can reach an optimal state, where the shared characteristics fully reflect the universal patterns of the data, and the individual characteristics eliminate irrelevant noise. As a result of the fusion of both components, the generated key information achieves a high degree of label matching. If the target value of the loss function is less than the set threshold, the current information generation model can be accepted as the trained information generation model.

[0251] Step S505: Obtain the speech signal to be processed.

[0252] Step S506: Process the speech signal using the trained information generation model to obtain key information corresponding to the signal.

[0253] Steps S505-S506 correspond to the previously described steps S101-S104; the implementation should refer to the specific implementation methods described for steps S101-S104.

[0254] In this embodiment of the present application, the information generation model is trained based on a predefined set of samples containing labeled label data, enabling end-to-end learning. Thus, the trained information generation model is capable of integrating time series information over the temporal dimension and frequency information over the channel dimension for each speech signal. Based on the combined first-order cross-features, the model recognizes key information in the speech signal without the need for manual speech feature assignment. This not only significantly reduces labor costs but also improves the quality of key information recognition.

[0255] In some embodiments, as shown in Fig. 7, each speech signal is considered as a separate speech sample; the duration of each sample is T. If there are B speech samples, the set of signals can be represented as an array [B, T]. After processing a plurality of speech signals using the MFCC method, tensor characteristics 2071 corresponding to them are formed. Each such tensor characteristic 2071 is represented in the form of [T, D], and the set of tensor characteristics is represented as a three-dimensional array [B, T, D].

[0256] Then, a one-dimensional convolution is applied to each tensor feature of the shape [T, D], resulting in the formation of folded features 2072. The resulting folded features 2072 are represented as [B, T / S, D']. This allows: to reduce the dimension along the time dimension T, which directly reduces the volume of subsequent calculations; to ensure pre-focusing of features on long-term key fragments, avoiding masking of useful information by short-term redundant components; to change the number of channels of each tensor feature from D to D', while it is possible to increase the number of channels to a fixed value that allows square root extraction, for example, to transform a 43-dimensional feature into a 49-dimensional one.

[0257] After obtaining the collapsed characteristics 2072, a downsampling operation is additionally applied to compress the temporal dimension. In this case, the temporal dimension of the collapsed characteristics 2072 T / S is reduced to T / S2, resulting in the compressed characteristics 2073. The compressed characteristics 2073 are represented as [B, TVS2, D']. This approach allows: to reduce the volume of subsequent calculations; by forcibly focusing on long-term key fragments, to filter out short-term redundant data from the set of speech signals [B, T].

[0258] In this context, redundant information includes modal particles and microvariations in background noise—irrelevant components. Eliminating these elements allows us to focus feature extraction on key speech segments that carry significant information load for generating the target key information. The kernel of the second 1D convolution then reslides over the previously extracted convolved features 2072 (i.e., local features), revealing temporal dependencies over a longer time range (e.g., relationships between adjacent syllables or phrases). The second 1D convolution, by expanding the receptive field (e.g., by using a larger kernel or sequentially superimposing multiple layers) while preserving temporal relationships, allows us to generate features that integrate contextual information from a wider range.This lays the foundation for the subsequent application of 2D convolution in the channel and time domain, which allows encoding more extensive temporal dependencies in a single feature.

[0259] After obtaining the tensor features 2071, one-dimensional convolution is applied to each of them, resulting in features 2074. In this case, the features 2074 are represented as [B, T / S,M].

[0260] Based on tensor characteristics 2071, represented in the form [B, T, D], the following is performed: direct compression of the temporal dimension from T to T / S; dimension transformation by channels from D to M. As a result of these transformations, characteristics 2074 are formed. Thus, compression of the temporal dimension allows us to simultaneously achieve two goals: reduce the computational load on subsequent processing stages and forcibly focus the model's attention on key characteristics that extend over time. In parallel, a transformation of the channel dimension is performed - the number of channels of the characteristic 2074 is reduced to a given value M, which ensures scaling of the dimension to a fixed parameter convenient for further mathematical operations, in particular, allowing the calculation of a square root. As an example: a 43-dimensional vector representation of a characteristic is transformed into a 49-dimensional one.

[0261] After receiving the characteristics 2074, they are transformed, the characteristics are divided according to the channel dimension into two-dimensional spatial mesh characteristics 2075. In this case, the two-dimensional spatial mesh characteristics 2075 are represented as [B, T / S, U, V].

[0262] After obtaining two-dimensional spatial mesh characteristics 2075, each feature point within these characteristics is weighted, resulting in weighted characteristics 2076. The weighted characteristics 2076 are represented as [B, T / S, U', V]. This approach allows: filtering out insignificant feature points; focusing subsequent processing stages on points with a high weighting coefficient, that is, on acoustic signals closely related to the semantics of the speech segment.

[0263] Then, two-dimensional convolution is applied to the weighted features 2076, resulting in features 2077 after two-dimensional convolution. The features 2077 are represented as [B, T / S 2 , U', V]. This approach allows for additional integration of the temporal characteristics and frequency characteristics of the speech signal.

[0264] After obtaining the characteristics 2077 after two-dimensional convolution, their dimensionality is further reduced, resulting in the formation of reduced-dimensional characteristics 2078. In this case, the reduced-dimensional characteristics 2078 are represented in the form [B, U', V].

[0265] After that, the reduced-size characteristics 2078 are combined with the compressed characteristics 2073, resulting in the formation of the combination characteristic 2079. The combination characteristic 2079 is represented as [B, T / S 3 , O d]. This approach allows us to integrate two types of characteristics, ensuring simultaneous consideration of both the global key characteristics of the speech signal and its local sequential features.

[0266] Thus, by applying the feature fusion process described in Fig. 7, it is possible to transform one-dimensional features into two-dimensional ones that integrate temporal and frequency representations: by scaling, separating, and combining tensor features along the channel dimension. As a result, it is possible to combine semantic attributes along the channel and temporal ordering along the time axis. This ensures a more complete reflection of the semantic details of the speech signal in the resulting first integrated feature.

[0267] Since the dimensionality of the first union characteristic remains quite high, using it directly for semantic encoding would lead to excessive computational costs in subsequent stages. Therefore, one possible implementation utilizes 1D downsampling convolution, which allows for a reduction in the dimensionality of the first union characteristic.

[0268] For example, if the first union feature is [B, L / 4, D2] [B, L / 4, D2], 1D downsampling convolution can be applied to it, resulting in the first union feature being [B, L / 8, D0][B, L / 8, D0]

[0269] Optionally, after obtaining the first union feature, an encoder can be applied to it for multi-stage semantic extraction. This allows the low-level representations of the first union feature to be transformed into semantic features containing high-level semantic properties. The encoder can be any suitable encoder, for example, an encoder based on deep neural networks such as Conformer or Transformer. This encoder can be a Conformer encoder, comprising several sequentially connected, stacked modules, with each modular layer containing a multi-head self-attention mechanism and linear, fully-connected feedforward layers.

[0270] In one possible embodiment, this encoder also includes a residual connection module and a normalization module. The residual connection module sums the input and output data of each modular layer of the encoder. The normalization module then applies a layer-level normalization operation to the resulting sum. This approach prevents bias in feature values ​​that occurs when data sequentially passes through multiple layers of the neural network.

[0271] Thus, by determining the weights of the speech segments corresponding to each feature point in the first cross-feature, each feature element is assigned a weight reflecting its semantic significance: feature points corresponding to key semantic information in the first cross-feature are assigned high weights, while points associated with redundant information are assigned low weights. This transforms the first cross-feature, which integrates temporal and frequency representations, into a feature containing information about semantic significance. As a result, the adjusted second cross-feature demonstrates a closer relationship with the actual semantics of the speech signal.

[0272] Based on the described embodiment examples, the present claimed solution also provides a key information generation system. Fig. 8 shows a block diagram of such a key information generation system according to an embodiment example. As shown in Fig. 8, the system includes: an input data preprocessing module 301, a combined processing module 302, and a processing module 303.

[0273] In this case, the input data preprocessing module 301 is intended for: obtaining the speech signal to be processed; performing preprocessing of this speech signal with obtaining the corresponding tensor characteristic, which contains time series information by time dimension and frequency information by channel dimension;

[0274] The combined processing module 302 is used to perform integration of the tensor characteristic along the channel and time dimensions, as a result of which a first cross-characteristic of the speech signal is formed; said first cross-characteristic is a characteristic that combines frequency information and time series information;

[0275] The processing module 303 is used to generate key information corresponding to the speech signal based on the first cross-characteristic.

[0276] In this case, the input data preprocessing module 301 may be any suitable module capable of implementing the specified functionality. For example, an audio signal preprocessing chip. When implemented, the input data preprocessing module 301 and the combined processing module 302 can be connected via a communication channel or in another way, for example, via a common storage medium. For example, the input data preprocessing module 301 stores the generated tensor characteristics in RAM, on a hard disk or in a distributed storage system. The combined processing module 302 obtains these tensor characteristics by reading data from the specified storage medium, thereby ensuring the transfer of information between the modules. When implemented, the input data preprocessing module 301 performs: receiving a speech signal to be processed; performing preprocessing of the speech signal to obtain the corresponding tensor characteristics.These operations can be implemented in accordance with the previously described methods for performing steps S101 and S102.

[0277] The fusion processing module 302 may be any suitable module capable of implementing the specified functionality, such as an application-specific integrated circuit (ASIC) or a field-programmable gate array (FPGA). By implementing computational logic in hardware, including two-dimensional convolutions and attention mechanisms (e.g., a CBAM module), such solutions provide highly efficient cross-processing of tensor features across the temporal and channel dimensions. The fusion processing module 302 may integrate a pre-trained feature fusion model containing a plurality of nested fusion units. Using these nested fusion units, the tensor features are cross-processed across the temporal and channel dimensions, resulting in the formation of a first cross-feature corresponding to the speech signal.When implemented, the combined processing module 302 cross-processes the tensor characteristics across the time and channel dimensions to obtain a first cross-characteristic corresponding to the speech signal. This procedure can be implemented in accordance with the previously described method for implementing step SI03.

[0278] Processing module 303 may be any suitable module capable of implementing the specified functionality. For example, it may be a specialized processor or a chip for accelerating deep learning operations. When implemented, processing module 303 generates key information corresponding to the speech signal based on the first cross-feature of this signal. This operation may be performed in accordance with the implementation method of step S104 described in detail previously.

[0279] In some embodiments, the processing module 303 may include: an encoding module, a feature extraction module, and a decoding module.

[0280] The coding module is designed to determine the semantic characteristics corresponding to the speech signal based on the first cross-characteristic.

[0281] The feature extraction module is designed to: perform weighting of the general part of the semantic characteristic of the speech signal using the external attention module integrated into the trained information generation model, resulting in the formation of a general characteristic corresponding to the given speech signal; perform weighting of the individual part of the semantic characteristic of the speech signal using self-attentive modules integrated into the trained information generation model, resulting in the formation of an individual characteristic corresponding to the given signal;

[0282] The decoding module is designed to generate key information corresponding to the speech signal based on the general and individual characteristics.

[0283] The encoding module may be any suitable module capable of implementing the specified functionality. For example, it may be a Transformer encoder. When implemented, the encoding module determines semantic characteristics corresponding to the speech signal based on its first cross-feature. This procedure may be performed in accordance with the implementation method of step S1041 described in detail previously.

[0284] The feature extraction module may be any suitable module capable of implementing the specified functionality. For example, it may be a hardware circuit or an algorithmic module integrating attention mechanisms. When implemented, the feature extraction module weights the general part of the semantic feature of the speech signal using an external attention module integrated into the trained information generation model, resulting in the formation of a general feature corresponding to the given speech signal; weights the individual part of the semantic feature of the speech signal using self-attentive modules integrated into the trained information generation model, resulting in the formation of an individual feature corresponding to the given signal. This procedure can be performed in accordance with the previously described detailed implementation method of step S1042.

[0285] The decoding module may be any suitable module capable of implementing the specified functionality. For example, it may be a hardware decoder implementing the key information generation logic or a software generation model. When implemented, the decoding module generates key information corresponding to the speech signal based on its general and individual characteristics. This procedure may be performed in accordance with the previously described implementation method of step S1043.

[0286] The description of the given system implementation examples is similar to the description of the previously described method implementation examples, demonstrating an identical set of beneficial effects. Technical details not disclosed in the system implementation examples of this application should be interpreted based on the description of the corresponding method implementation examples provided within the framework of this application.

[0287] Fig. 9 is an illustration of the process of implementing the method for generating key information #4 in one of the embodiments described in this application. Below, a description will be given of the steps shown in Fig. 9. It should be noted that the method for generating key information shown in Fig. 9 is described using the speech processing system as an executing example. As can be seen from Fig. 9, this method includes the following steps from S201 to S213, where:

[0288] Step S201: Obtain the speech signal to be processed.

[0289] This procedure can be implemented in accordance with the method for performing step S101 described in detail earlier.

[0290] Step S202: Perform preprocessing on the speech signal to obtain tensor characteristics corresponding to the speech signal.

[0291] This procedure can be implemented in accordance with the method for performing step S102 described in detail earlier.

[0292] In step S2023, cross-processing of the tensor characteristics according to the time and channel dimensions is performed, as a result of which it is possible to obtain a first set of cross-characteristics corresponding to the original speech signal.

[0293] For example, using the following formulas (1-1), (1-2) tensor characteristics are transformed by one-dimensional convolution in the channel domain into the first characteristic with a given number of channels:

[0294]

[0295] In this case - input tensor characteristic with dimension , where: B is the packet size (the number of speech samples processed in parallel in one pass; for example, 16 or 32); T is the number of time steps (the duration of the speech signal in discrete samples; for example, if a speech fragment contains 1000 samples, then T = 1000); D is the channel dimension (the dimension of the acoustic characteristics; for example, for a 40-dimensional vector of MFCC characteristics, D = 40). - the output characteristic obtained after applying the operation “one-dimensional convolution + Relu activation”; - one-dimensional convolution operation; - linear rectifying activation function (linear rectifier); M is the channel dimension after convolution (usually chosen as the closest square number to the original number of channels D; for example, for D=40, it is taken - for the convenience of subsequent two-dimensional operations); - the length of the time step after convolution (determined by the size of the convolution kernel and the step; usually for example, at step 2, downsampling is performed along the time axis: T'≈T / 2).

[0296] Thus, the input tensor characteristic X is first subjected to a one-dimensional convolution operation , which involves: adjusting the channel dimension to a value of M; compressing the number of time steps to a value of T'. Then, the ReLU activation function is applied to the result to introduce nonlinearity. As a result, the first characteristic X' with the new dimension is formed. This allows us to preserve key frequency characteristics while simultaneously bringing the number of channels into line with the requirements for two-dimensional partitioning.

[0297] Optional, first characteristic The data obtained after one-dimensional convolution (for example, with M=36) can be transformed by partitioning by the number of channels into a two-dimensional spatial array, the second dimension characteristic being U×V. For example, one can choose U=V=6, since 6×6=36.

[0298] The specific implementation can be achieved using the following formula (1-3):

[0299]

[0300] In this case: - the first characteristic for the t'-th time step and the m-th channel in batch b; - a function of transforming characteristics, designed to change the order of the dimensions of a characteristic;

[0301] - the second characteristic; U and V are values ​​equal to the square root of the original number of channels D, rounded up to the nearest integer.

[0302] The second characteristic has already transformed the channel information into a two-dimensional grid, but the time series information in the temporal dimension and the frequency information in the channel dimension remain independent of each other. To obtain the first cross-characteristic, their further combination is required, which simultaneously preserves the temporal logical sequence in the temporal dimension and integrates the frequency-spatial features encoded in the two-dimensional grid structure.

[0303] Specifically, a two-dimensional convolution can be applied to the three-dimensional second characteristic using a 2D convolution module, resulting in the first cross-characteristic. In this case, convolution over the channel grid V allows one to identify "joint changes in different frequency ranges"; and convolution over the temporal dimension T' allows one to identify "semantic relationships between successive time steps."

[0304] One possible implementation allows for dynamic adjustment of the channel dimensionality of the output characteristics using two-dimensional convolution parameters. This ensures that the convolution characteristics are balanced across three dimensions: number of channels, grid dimensionality, and number of time steps. This approach both preserves key semantic information and ensures compliance with the input data requirements of subsequent models.

[0305] The specific implementation can be achieved using the following formula (1-4):

[0306]

[0307] In this case: - new number of characteristic channels after two-dimensional convolution; - the specified final number of output channels; S - the convolution step; V - the number of columns in the channel grid before applying two-dimensional convolution; p - the size of the padding during two-dimensional convolution; k - the size of the two-dimensional convolution kernel.

[0308] Thus, cross-processing achieves the interweaving of information in the time and channel dimensions. This allows us to identify patterns in the changes of single-frequency characteristics of a speech signal at different points in time and to capture the semantic combination of multi-frequency characteristics of a speech signal at the same point in time. This approach breaks down the barrier between the time and channel dimensions, allowing the characteristics to more fully reflect the semantic information of the speech signal.

[0309] Step S204: Determine the weights of speech segments corresponding to different feature points in the first cross-feature associated with the analyzed speech signal.

[0310] In this case, the characteristic value at each intersection point of the time axis for individual time steps and the channel axis for individual channels within the first cross-characteristic can be taken as its individual characteristic point. Such a characteristic point corresponds to the speech signal characteristic at the intersection point, determined by the time index t and the channel index c.

[0311] For example, if the first cross-over characteristic is represented as F1∈RB×L×D, then each characteristic point within it can be designated as f, t, c, where t is the time step index and c is the channel index. The element f100, 2 corresponds to the energy value of the speech signal at the intersection point corresponding to the 100th millisecond (time step) and the 2nd channel index (high-frequency region).

[0312] Step S205: Based on the weight coefficients of the speech segments corresponding to the individual characteristic points, adjust the first cross-characteristic associated with the speech signal to obtain a second cross-characteristic also associated with the speech signal.

[0313] For steps S204-S205, the first cross-characterization contains a large number of feature points, but not all of them contribute to the semantics. Therefore, by calculating weighting coefficients for each feature point, it is possible to filter out insignificant feature points, directing subsequent processing to work with highly significant feature points, that is, with acoustic signals closely related to the semantics of the speech segment.

[0314] Specific adjustment of the first cross-feature corresponding to the speech signal can be performed using the following formula (1-5) to obtain a second cross-feature also corresponding to the given speech signal.

[0315]

[0316] In this case: F is the first cross-characteristic; and - output characteristics after double weighing; - global averaging pudding operation; - global maximizing pudding operation; - multilayer perceptron; - convolution operation; - Sigmoid activation function (produces values ​​in the range [0, 1], used as weighting coefficients); - element-wise multiplication; - element-by-element addition.

[0317] The first cross-characteristic spatio-temporal pre-weighting can be performed first.

[0318] The global averaging pudding operation and the global maximizing pudding operation can be applied to the first cross characteristic, resulting in two global statistical characteristics. The global statistical characteristics include: the characteristic obtained by the averaging pudding; the characteristic obtained by the maximizing pudding. Both characteristics are represented in the form [B, C], where C is the number of channels; B is the batch size (the number of samples in the batch).

[0319] After obtaining the global statistical features of the first cross-feature using a multilayer perceptron (MLP), the averaging pudding features and maximizing pudding features are processed separately. As a result of applying MLP, the transformed pudding features, both average-pooling and maximizing, are mapped to vector features of the same dimension as the original first cross-feature. The resulting outputs are a global mean vector feature and a global key vector feature. After obtaining the global mean vector feature and the global key vector feature, their elements are summed element-by-element, producing an integrated result combining both global representations.

[0320] Finally, the combined result is compressed to the range [0, 1] using the Sigmoid function, forming an attention matrix with values ​​in the interval [0, 1]. Then, the obtained attention weights are multiplied element-by-element by the first cross-feature, resulting in the output feature after the first weighting. This approach allows us to highlight features with high weights and suppress features with low weights. Features with high weights correspond to semantically key fragments of the speech signal, while features with low weights represent noise components and redundant information in the speech signal.

[0321] Thus, the characteristics obtained as a result of preliminary weighing allow us to identify semantically key elements of content from the perspective of global analysis.

[0322] Then a more detailed spatial-temporal weighting can be performed again for the previously weighted characteristic.

[0323] The global averaging pudding and global maximizing pudding operations are repeatedly applied to the output feature after the first weighting; the results are combined element-wise using addition, resulting in a local pooling feature. A convolution operation is then performed on this local pooling feature. Afterwards, local weights are generated using an activation function. These local weights are then multiplied element-wise by the output feature after the first weighting, yielding a second cross-feature.

[0324] Thus, after applying weighting using channel-based and temporal-based attention, it becomes possible to: extract semantically significant acoustic characteristics (e.g., energy, frequency, etc.) from the channel dimension; and extract semantically significant temporal fragments (e.g., stages of filing a complaint, making a decision, etc.) from the time sequence. This results in a second cross-feature, in which both significant channels and significant temporal sequences are simultaneously enhanced.

[0325] Step S206: based on the tensor characteristic corresponding to the speech signal and the second cross characteristic also associated with this speech signal, determine the first fusion characteristic associated with this speech signal.

[0326] The first unification characteristic stores time series information identical to the time information parameters of the tensor characteristic.

[0327] At this stage, the tensor feature contains time series information in the temporal dimension and frequency information in the channel dimension of the speech signal. It preserves rich original acoustic details, particularly information about temporal changes in frequency and energy in the speech signal. However, it is characterized by weak semantic coherence and may include noise interference and redundant information. Meanwhile, the second cross-feature, although already enhanced by segment weighting coefficients and containing a clearly defined semantic focus with minimal redundancy, may potentially lose some weak but critical original details, such as quiet meaningful commands. Therefore, it is advisable to combine the tensor feature and the second cross-feature to form the first fusion feature, which combines a focus on semantics and the preservation of original details.

[0328] The specific implementation can be achieved using the following formula (1-6):

[0329]

[0330] In this case: - the second cross-characteristic; F - the alignment function; - weight coefficient of the second cross-characteristic; - tensor characteristic corresponding to the speech signal; - weighting coefficient of the tensor characteristic; the first characteristic of the union.

[0331] This allows us to combine the local, detailed information contained in the tensor feature with the global key data presented in the second cross-feature. This allows us to obtain a more complete semantic feature based on the subsequently generated first union feature.

[0332] Step S207: perform semantic encoding of the first association characteristic corresponding to the speech signal to obtain a semantic characteristic also associated with the speech signal.

[0333] Optionally, after obtaining the first union feature, an encoder can be applied to it for multi-stage semantic extraction. This allows the low-level representations of the first union feature to be transformed into semantic features containing high-level semantic properties.

[0334] The encoder can be any suitable encoder, such as an encoder based on deep neural networks such as Conformer or Transformer. This encoder can be a Conformer encoder, comprising several sequentially connected, stacked modules; each modular layer contains a multi-head self-attention mechanism and linear, fully connected feedforward layers.

[0335] At this stage, the feature sequence corresponding to each time step within the time dimension of the first union feature can be respectively used as the input sequence for this encoder.The processing algorithm: the sequence of features is fed to the input of the first decoding layer in the encoder, after passing through the multi-head self-attention mechanism and linear fully connected feedforward layers in the first coding layer, a multidimensional feature is formed after the first optimization; the optimized multidimensional feature is passed to the input of the second coding layer as an input sequence, similarly, after processing by the multi-head self-attention mechanism and linear fully connected feedforward layers in the second decoding layer, a multidimensional feature is formed after the second optimization; after multiple iterations through successive decoding layers, the multidimensional feature obtained at the output of the last decoding layer is adopted as the final semantic feature.

[0336] In one possible embodiment, this encoder also includes a residual connection module and a normalization module. The residual connection module sums the input and output data of each modular layer of the encoder. The normalization module then applies a layer-level normalization operation to the resulting sum. This approach prevents bias in feature values ​​that occurs when data sequentially passes through multiple layers of the neural network.

[0337] In an embodiment of this application, generating a semantic characteristic based on the first association characteristic provides for an increase in the accuracy of semantic coding.

[0338] Step S208: Using the external attention module of the trained information generation model, the common components of semantic features extracted from the original speech signal are weighted. This operation allows the common feature corresponding to a given speech signal to be identified.

[0339] At this stage, the external attention module uses the formulas (1-7) below to weigh the general part of the semantic characteristics corresponding to the speech signal, resulting in the formation of a general characteristic corresponding to the speech signal.

[0340]

[0341] In this case: the output characteristic of the encoder is designated as - the general characteristic formed at the output, the dimension of which remains identical to the dimension of the input semantic characteristic, is designated as And - two linear transform layers: serves to project the original semantic characteristic into a new space of characteristics; is intended for the reconstruction of characteristics after calculating the relationships back into the dimension D of the semantic characteristic; - a normalization function that performs row-by-row normalization of the input matrix of features. As a result, the output values ​​lie in the interval (0, 1); the sum of the values ​​in each row is equal to 1; the function is used to calculate global weighting coefficients of relationships.

[0342] Step S209: Using the self-attention module of the information generation model, individual components of semantic features extracted from the original speech signal are weighted. This operation allows the individual feature corresponding to a given speech signal to be identified.

[0343] At this stage, the self-attention module uses the formulas (1-8) below to weigh the individual part of the semantic characteristics corresponding to the speech signal, resulting in the formation of an individual characteristic corresponding to the speech signal.

[0344]

[0345] In this case: - extracted individual characteristic; - linear transformation layer; - normalization function; Q, K, V - matrix for calculating queries from semantic characteristics, matrix for calculating keys from semantic characteristics, matrix for calculating values ​​from semantic characteristics, are used to calculate the strength of relationships between time steps; C - the dimension of each head of the attention mechanism, used to scale the weighting coefficients of attention; this dimension of C is used to prevent excessive growth of numerical values, which can lead to the disappearance of the gradient in the softmax function.

[0346] The self-attention module may contain multiple linear layers and attention layers.

[0347] For example, the semantic characteristic of a single input speech signal can be transformed into the computation matrices Q, K, and V using three independent linear layers. In this case, Q (query) reflects "what information needs to be taken into account at the current time step"; K (key) reflects "what information is contained at other time steps"; V (value) reflects "the specific content of information to be aggregated". By multiplying the query matrix Q by the transposed key matrix K, the degree of relationship between the sequences of characteristics of different time frames in the semantic characteristic is calculated. This is because the individual characteristics of a single speech signal often manifest themselves in unique internal relationships, for example, in a temporal relationship such as "refund of 200 yuan + crediting within 3 days" that is characteristic of a particular speech signal.

[0348] The obtained degree of interrelationship is then scaled and normalized, resulting in an attention weight matrix for which the sum of the weights is equal to 1. This attention weight coefficient represents the distribution of weights corresponding to the feature sequences of different time frames in a semantic feature. After obtaining the attention weight coefficients, the attention layer performs weighted summation of the value matrix V using these coefficients. This operation is intended to aggregate information about the relationships between feature sequences of different time frames. As a result, a feature obtained through attention weighting is formed; this feature contains aggregated information about the relationships between feature sequences of different time frames.This allows for the identification of unique details within a single speech signal: for example, if the phrase "User A demands a refund of 200 yuan" is unique in a given speech signal (other speech signals lack such a statement), then the weight of the relationship between the time frame "refund" and the time frame "200 yuan" will be significantly higher than that of other relationships. As a result of this weighting, the feature "200 yuan" is strengthened and becomes the key element of the individual feature.

[0349] Finally, a linear layer is used to adjust the dimensionality of the feature obtained after attention weighting. This results in the formation of an individual feature corresponding to each semantic signal. The dimensionality of this individual feature remains identical to the dimensionality of the input semantic feature.

[0350] Step S210: Combine the common characteristics and individual characteristics to obtain a combined characteristic.

[0351] In this case, common characteristics serve to reflect the general theme of several speech signals, while individual characteristics reflect the unique information inherent to each specific speech signal. In the context of customer support dialogues, a situation may arise where three speech signals from customers are linked by a single issue—a delivery delay. Each customer formulates a specific request: in the first signal, User A insists on compensation, in the second signal, User B requests a resend of the product, and in the third signal, User C demands expedited delivery. Thus, the common component of all three signals is the delivery delay itself, while the individual components are the specific requests of each customer, reflecting their personal needs and expectations.

[0352] Common and individual characteristics carry different types of key information: common characteristics form the "basic structure" of key information (e.g., "this dialogue addresses a logistics issue"), while individual characteristics represent the "differentiating details" of key information (e.g., "user A's specific requirement"). Concatenation, through simple and direct dimensionality expansion, ensures the complete preservation of the key information contained in common and individual characteristics and avoids the blurring of characteristics that can occur during their weighted merging. Thus, at this stage, common and individual characteristics can be combined to obtain a combined characteristic. The merging operation can be performed directly along the time or channel dimension.

[0353] The resulting unified characteristic allows for the integration of general and individual information into a single characteristic space, forcing the subsequent processing module to simultaneously consider the relationship between them. This subsequent processing module can also include a decoder. This approach ensures a dual result: the generated key information fully reflects the overall thematic focus while accurately preserving individual details.

[0354] Step S211: Perform a convolution operation on the combined characteristic to obtain a relationship characteristic.

[0355] The obtained relationship characteristic serves to formalize the description of the nature of the interaction between the general and individual components of the characteristics, both in terms of channel dimension and time dimension.

[0356] At this stage, the convolution operation reveals the deep relationships between common and individual characteristics, which is used to form a relationship characteristic; this characteristic allows for the simultaneous reflection of the correlation of semantic attributes along the channel dimension and temporal logic along the temporal dimension.

[0357] Specifically, this can be implemented by moving a sliding window simultaneously across the time dimension and the channel dimension of the combined feature. This approach allows us to capture the relationship between the semantic attributes of the features in the channel dimension and the temporal logic of the features in the time dimension.

[0358] This is due to the fact that the concatenation operation performed in step 210 merely physically combines common and individual characteristics without revealing their internal relationships. In contrast, the relationship characteristic is formed through convolution, which specifically examines these relationships. This approach avoids misalignment between common and individual characteristics in subsequent merging steps and transforms the physically combined characteristic into a structured characteristic with semantic relationships. This creates the necessary foundation for the precise alignment of common and individual information in subsequent processing, guaranteeing the logical coherence and semantic consistency of the generated key information.

[0359] Step S212: Based on the gate weights corresponding to the relationship characteristic, perform weighting of the common and individual characteristics to obtain the second fusion characteristic.

[0360] At this stage, the second characteristic of the union is formed using the formulas (1-9) below.

[0361]

[0362] In this case: - the combined characteristic obtained in step 210 by concatenation over the channel dimension;

[0363] - a convolution operation (usually two-dimensional) applied to a combined characteristic to extract relationship characteristics; in this case, the relationships between common and individual characteristics are recorded both along the channel dimension and along the time dimension; - activation function that maps the convolution output to the range [0, 1] and forms gate weights; - the output second characteristic of the union, which is the result of the weighted union of the general characteristic and the individual characteristic.

[0364] Step 213: Perform decoding of the second combination characteristic to obtain the key information corresponding to the given speech signal.

[0365] This procedure can be implemented in accordance with the method for performing step S104 described in detail earlier.

[0366] The embodiment presented in this application utilizes cross-processing of tensor characteristics across the channel and temporal dimensions to combine frequency information across the channel dimension and sequential-temporal information across the temporal dimension. As a result, the first cross-processing characteristic, while retaining the ability to identify semantic relationships in the speech signal over extended time intervals, integrates data on various frequency components of the speech signal. This allows for a more complete reflection of the semantic details of the speech signal, reduces the likelihood of being limited to local temporal-sequential information, as in existing approaches that employ 1D convolution, and significantly improves the expressiveness of semantic characteristics. This processing facilitates a precise understanding of the overall semantics of the speech signal, which ultimately contributes to an increase in the accuracy of the generated key information.

[0367] It should be noted that in the embodiment of this application, if the described methodology can be implemented in the form of software functional modules and released as an independent product for sale or use; it can also be stored on a machine-readable storage medium. In view of the above, the technical solution proposed in the embodiment of this application, or, in other words, that part thereof that contributes to the development of the relevant technical field, can essentially be presented in the form of a software product that is stored on a storage medium and contains one or more instructions intended for a computing device (be it a personal computer, a server or network equipment) to perform all or part of the steps of the method described in the embodiments of this application.The storage medium may be a USB drive, external hard drive, read-only memory (ROM), magnetic disk, or optical disk—that is, any medium suitable for storing program code. Therefore, this embodiment is not limited to any specific combination of hardware and software.

[0368] The present application also provides electronic equipment comprising a memory and a processor, wherein the memory stores a computer program executable on the processor; when said program is run, the processor implements any of the above-described methods.

[0369] This application also provides a machine-readable storage medium on which an executable program is stored; when this program is executed by the processor, the above-described methodologies are implemented. The machine-readable storage medium may be either temporary or permanent.

[0370] This application also provides a software product comprising a computer program or set of instructions that, when executed by a processor, implement some or all of the steps of any of the above methods. This software product may be implemented using hardware, software, or a combination of both. In one possible embodiment, the software product is implemented as a computer storage medium; in another possible embodiment, it is implemented as a software product, such as a Software Development Kit (SDK), etc.

[0371] It should be noted that Fig. 10 shows a diagram of the hardware implementation of the electronic device according to an embodiment of the present application. As shown in Fig. 10, the hardware implementation of the electronic device 400 includes: a processor 401, a communication interface 402 and a memory device 403, wherein:

[0372] The processor 401 generally controls the overall operation of the electronic device 400.

[0373] The 402 communication interface enables the electronic device to communicate with other end devices or servers via a network.

[0374] Memory device 403 is designed to store instructions and applications executed by processor 401, as well as to buffer data to be processed or already processed by processor 401 and other modules of electronic device 400 (including image data, audio data, voice communication data, and video communication data). Implementation is possible based on flash memory (FLASH) or random access memory (RAM). Processor 401, communication interface 402, and memory device 403 transfer data among themselves via bus 404.

[0375] It should be noted that the provided description of the storage media and device embodiments is similar to the description of the previously described method implementation examples and provides comparable useful technical results. Technical details not disclosed in the storage media and device embodiments of this application should be interpreted based on the description of the corresponding method implementation examples provided within the framework of this application.

[0376] The embodiments given herein serve only to comprehensively explain this application and represent the most preferred embodiments; however, the scope of legal protection provided by the application is not limited to these embodiments. Those skilled in the art who make equivalent replacements or modifications based on this application remain within the scope of legal protection provided by this application.

Claims

1. A method for generating key information, including: obtaining a speech signal to be processed; pre-processing of said speech signal to obtain a tensor characteristic corresponding thereto, wherein said tensor characteristic includes time series information along the time dimension and frequency information along the channel dimension, wherein the channel dimension of said tensor characteristic is a number that allows or does not allow the operation of extracting a square root; performing cross-processing of said tensor characteristic according to the channel dimension and the time dimension to obtain a first cross-characteristic corresponding to said speech signal, wherein said first cross-characteristic is a characteristic that combines said frequency information and said time series; determining a semantic characteristic corresponding to said speech signal based on said first cross-characteristic; using an external attention module of a trained information generation model to weight the general portion of a semantic feature corresponding to a specified speech signal to obtain a general feature corresponding to the specified speech signal; using the self-attention module of the said information generation model to weigh the individual part of the semantic characteristic corresponding to the said speech signal, with the result that an individual characteristic corresponding to the said speech signal is obtained, wherein the number of the said self-attention modules is not less than one; generating key information corresponding to the specified speech signal based on the specified general characteristic and the specified individual characteristic.

2. The method according to paragraph 1, characterized in that said determination of a semantic characteristic corresponding to said speech signal, based on said first cross-characteristic, includes: determining weights of speech segments corresponding to different characteristic points in said first cross-characteristic associated with said speech signal; determining weights of speech segments corresponding to different feature points in said first cross-feature associated with the analyzed speech signal; adjusting said first cross-characteristic corresponding to said speech signal based on the weighting coefficients of the speech segments corresponding to different points of the characteristic, to obtain a second cross-characteristic corresponding to said speech signal; performing semantic encoding of said second cross-characteristic corresponding to said speech signal, with obtaining a semantic characteristic corresponding to said speech signal.

3. The method according to paragraph 2, characterized in that said execution of semantic coding of said second cross-characteristic corresponding to said speech signal, with the receipt of a semantic characteristic corresponding to said speech signal, includes: determining a first combination characteristic corresponding to said speech signal based on a tensor characteristic corresponding to said speech signal and said second cross-characteristic corresponding to said speech signal, wherein said first combination characteristic is consistent with time series information of said tensor characteristic; performing semantic encoding of said first characteristic of the association corresponding to said speech signal, with obtaining a semantic characteristic corresponding to said speech signal.

4. The method according to claim 1, characterized in that said external attention module includes a first convolutional layer, a second convolutional layer and a third convolutional layer, wherein said use of the external attention module of the trained information generation model for weighting the general part of the semantic characteristic corresponding to said speech signal, with obtaining the general characteristic corresponding to said speech signal, includes: performing weighting of a common part of a semantic characteristic corresponding to said speech signal by means of common parameters included in said first convolutional layer and said second convolutional layer, thereby obtaining a compressed characteristic corresponding to said speech signal; performing a convolution of said compressed characteristic by said third convolutional layer to obtain said overall characteristic after reconstructing the channels.

5. The method according to paragraph 1, characterized in that said generation of key information corresponding to said speech signal, based on said general characteristic and said individual characteristic, includes: performing a concatenation of the specified general characteristic and the specified individual characteristic to obtain a combined characteristic after concatenation; performing a convolution of the said combined characteristic to obtain a relationship characteristic, wherein the said relationship characteristic is intended to reflect the relationship of the said general characteristic and the said individual characteristic in the channel dimension and the time dimension; performing a weighting of the said general characteristic and the said individual characteristic, respectively, based on the control weighting coefficients corresponding to the said relationship characteristic, to obtain a second association characteristic; performing decoding of said second combination characteristic to obtain key information corresponding to said speech signal.

6. The method according to any one of paragraphs 1–5, characterized in that said cross-processing of said tensor characteristic according to the channel dimension and the time dimension to obtain the first cross-characteristic corresponding to said speech signal includes: performing a one-dimensional convolution of the specified tensor characteristic in order to transform the specified tensor characteristic by channel into a first characteristic containing a specified number of channels; performing a partition of the dimension of the specified first characteristic to obtain the second characteristic in a partitioned form; performing a two-dimensional convolution of said second characteristic to obtain a first cross-characteristic corresponding to said speech signal.

7. The method according to any one of paragraphs 1–5, further comprising: obtaining multiple samples of speech signals; using a trainable information generation model to pre-process a plurality of said speech signal samples to obtain a tensor characteristic corresponding to each said speech signal sample, cross-processing said tensor characteristic by channel dimension and time dimension for the tensor characteristic corresponding to each speech signal sample to obtain a first cross-characteristic corresponding to said speech signal sample, and generating key information corresponding to each said speech signal sample based on the first cross-characteristic corresponding to each said speech signal sample; determining a target value of the loss function based on key information corresponding to each specified speech signal sample and labeling data of each specified speech signal sample; performing at least one iteration of updating the parameters of the specified information generation model based on the specified target value of the loss function to obtain a trained information generation model.

8. A key information generation system, characterized in that said system includes: an input data processing module, a combined processing module, an encoding module, a feature extraction module, and a decoding module, where said input data processing module is intended for receiving a speech signal to be processed, performing pre-processing of said speech signal with the receipt of a tensor characteristic corresponding thereto, wherein said tensor characteristic includes time series information by time dimension and frequency information by channel dimension, wherein the channel dimension of said tensor characteristic is a number that allows or does not allow the operation of extracting a square root; said combined processing module is intended to perform cross-processing of said tensor characteristic according to channel dimension and time dimension with the receipt of a first cross-characteristic corresponding to said speech signal, wherein said first cross-characteristic is a characteristic combining said frequency information and said time series; said coding module is intended to determine a semantic characteristic corresponding to said speech signal based on said first cross-characteristic; said feature extraction module is designed to use an external attention module of a trained information generation model to weight a common portion of a semantic feature corresponding to said speech signal, thereby obtaining a common feature corresponding to said speech signal; and to use a self-attention module of said information generation model to weight an individual portion of a semantic feature corresponding to said speech signal, thereby obtaining an individual feature corresponding to said speech signal, wherein the number of said self-attention modules is not less than one; said decoding module is designed to generate key information corresponding to said speech signal based on said general characteristic and said individual characteristic.

9. An electronic device comprising a memory and a processor, wherein said memory stores a computer program intended for execution on the processor, wherein said processor is configured to execute said computer instructions for the purpose of implementing the method described in any of paragraphs 1–7.

10. A computer storage medium on which computer instructions are stored, characterized in that said instructions are configured to be executed for the purpose of implementing the method described in any of paragraphs 1–7.

Images