Cross-language speech translation method, device and equipment based on AI big model

Abstract

The present invention provides a cross-language speech translation method, device and equipment based on an AI big model, including: performing frame division and spectrum analysis on cross-language speech data to obtain a spectrum sequence; inputting the spectrum sequence into a pre-trained AI big model; based on a feature capture path, vectorizing spectrum features of different frequency bands in the spectrum sequence through a capsule network to obtain a package feature; based on a global association path, constructing each spectrum feature in the spectrum sequence into a graph structure through a graph neural network, and mining the global semantic association features between each spectrum feature; weighted fusion of the package features corresponding to the spectrum features of different frequency bands to obtain a fused package feature, aggregating the fused package feature with the global semantic association feature to obtain an aggregate feature; based on the aggregate feature, translating the speech data of the target language. In the present invention, the accuracy of cross-language translation is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of speech processing, and in particular to a cross-language speech translation method, device and equipment based on an AI big model. Background Art

[0002] Voice translation in cross-language communication is in great demand in many fields such as international communication, cross-border business, and tourism.

[0003] When processing speech data in different languages, traditional speech translation methods have significant differences in speech characteristics, including phonemes, syllables, intonation, rhythm, etc. Previous methods based on statistical models or simple neural networks often find it difficult to fully capture these complex speech features, resulting in low translation accuracy.

[0004] Semantic understanding and conversion between languages ​​is a key issue. Simply translating based on the surface features of speech can easily overlook the deep semantic connections between different languages, resulting in low accuracy in translation results. Summary of the invention

[0005] The main purpose of the present invention is to provide a cross-language speech translation method, device and equipment based on an AI big model, aiming to overcome the defect of low accuracy of speech translation results.

[0006] To achieve the above object, the present invention provides a cross-language speech translation method based on an AI large model, comprising the following steps:

[0007] Perform frame division and spectrum analysis on cross-language speech data to obtain spectrum sequences;

[0008] Inputting the spectrum sequence into a pre-trained AI big model; the AI ​​big model includes a feature capture path based on a capsule network and a global association path based on a graph neural network;

[0009] Based on the feature capture path, the spectrum features of different frequency bands in the spectrum sequence are vectorized and encapsulated by a capsule network to obtain encapsulated features;

[0010] Based on the global association path, each spectrum feature in the spectrum sequence is constructed into a graph structure through a graph neural network, and global semantic association features between the spectrum features are mined;

[0011] Performing weighted fusion on the encapsulation features corresponding to the spectrum features of the different frequency bands to obtain a fused encapsulation feature, and aggregating the fused encapsulation feature with the global semantic association feature to obtain an aggregated feature;

[0012] Based on the aggregated features, speech data in a target language is obtained by translation.

[0013] Furthermore, based on the aggregated features, the speech data in the target language is translated, including:

[0014] Processing the aggregated features based on a pyramid neural network to obtain abstract features;

[0015] The abstract features are converted into target language abstract features; based on the target language abstract features, speech data of the target language is translated.

[0016] Furthermore, based on the feature capture path, the spectrum features of different frequency bands in the spectrum sequence are vectorized and encapsulated through a capsule network to obtain encapsulated features, including:

[0017] The spectrum sequence is input into a capsule network having multiple capsule layers; wherein a spectrum window of a preset size in the first capsule layer slides on the spectrum sequence, and vector encoding is performed on the spectrum features within the spectrum window covered by the first capsule layer;

[0018] The next capsule layer receives the vector output of the previous capsule layer and updates the connection weight according to the dynamic routing algorithm; wherein the dynamic routing algorithm adjusts the weight according to the activation strength of each capsule layer and the correlation between the spectrum characteristics of different frequency bands;

[0019] After encapsulation processing by multiple capsule layers, the final output is the encapsulation features obtained by encapsulating the spectrum features of different frequency bands.

[0020] Furthermore, based on the global association path, each spectrum feature in the spectrum sequence is constructed into a graph structure through a graph neural network, and global semantic association features between the spectrum features are mined, including:

[0021] Each spectrum feature in the spectrum sequence is used as each node of the graph structure; wherein the initial feature vector of each node is the original numerical representation of the corresponding spectrum feature; the weight of the edge in the graph structure is the similarity between each spectrum feature;

[0022] The constructed graph structure is input into the graph neural network, which uses a multi-layer graph convolution layer. In each graph convolution layer, each node updates its own feature vector by weighted summation based on its own feature vector and the feature vectors of adjacent nodes, and the weight is the weight of the edge in the graph structure.

[0023] After information transmission and aggregation through multiple layers of graph convolutional layers, the feature vector of each node integrates the information of other related nodes in the global scope, and the global semantic association features between the spectral features are mined.

[0024] Furthermore, weighted fusion is performed on the encapsulation features corresponding to the spectrum features of the different frequency bands to obtain fused encapsulation features, and the fused encapsulation features are aggregated with the global semantic association features to obtain aggregated features, including:

[0025] For the encapsulation features corresponding to the spectrum features of different frequency bands, their importance in the expression of speech semantics is analyzed to assign weights, and the encapsulation features are weighted summed according to the assigned weights to obtain the fused encapsulation features;

[0026] The fused encapsulation features are dimensionally aligned with the global semantic association features, the aligned features are spliced ​​to obtain a splicing vector, the splicing vector is input into a multi-layer perceptron, and the aggregated features are output after nonlinear transformation and information fusion.

[0027] Furthermore, after the translation obtains the speech data in the target language, the translation includes:

[0028] Extracting the spectrum features of the specified frequency band in the spectrum sequence, and performing singular value decomposition to obtain a spectrum singular value matrix;

[0029] Extracting suprasegmental features of speech data of the target language and converting them into symbol sequences;

[0030] Inputting the spectrum singular value matrix and the symbol sequence into a key generator based on a chaotic system;

[0031] Determine the initial parameter range of the chaotic system, convert the initial parameter range into high-dimensional discrete chaotic mapping initial parameters based on a mapping algorithm, and perform high-dimensional discrete chaotic iteration to obtain the chaotic system parameters of the final key generator;

[0032] The spectrum singular value matrix and the symbol sequence are processed based on the final key generator to output an encryption key for encrypting the speech data of the target language.

[0033] The present invention also provides a cross-language speech translation device based on an AI large model, comprising:

[0034] An analysis unit, used for performing frame division and spectrum analysis on the cross-language speech data to obtain a spectrum sequence;

[0035] An input unit, used to input the spectrum sequence into a pre-trained AI big model; the AI ​​big model includes a feature capture path based on a capsule network and a global association path based on a graph neural network;

[0036] An encapsulation unit, configured to perform vector encapsulation on the spectrum features of different frequency bands in the spectrum sequence through a capsule network based on the feature capture path to obtain encapsulated features;

[0037] A mining unit, configured to construct each spectrum feature in the spectrum sequence into a graph structure through a graph neural network based on the global association path, and mine global semantic association features between the spectrum features;

[0038] an aggregation unit, configured to perform weighted fusion on the encapsulation features corresponding to the spectrum features of the different frequency bands to obtain a fused encapsulation feature, and aggregate the fused encapsulation feature with the global semantic association feature to obtain an aggregate feature;

[0039] A translation unit is used to translate the speech data into a target language based on the aggregated features.

[0040] The present invention also provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any one of the above methods when executing the computer program.

[0041] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned methods are implemented.

[0042] The cross-language speech translation method, device and equipment based on AI big model provided by the present invention include: framing and spectrum analysis of cross-language speech data to obtain a spectrum sequence; inputting the spectrum sequence into a pre-trained AI big model; the AI ​​big model includes a feature capture path based on a capsule network and a global association path based on a graph neural network; based on the feature capture path, vectorizing spectrum features of different frequency bands in the spectrum sequence through a capsule network to obtain encapsulation features; based on the global association path, constructing each spectrum feature in the spectrum sequence into a graph structure through a graph neural network, and mining the global semantic association features between each spectrum feature; weighted fusion of the encapsulation features corresponding to the spectrum features of different frequency bands to obtain fused encapsulation features, aggregating the fused encapsulation features with the global semantic association features to obtain aggregated features; based on the aggregated features, translating the speech data of the target language. In the present invention, a feature capture path based on a capsule network is utilized to vectorize the spectral features of different frequency bands in a spectral sequence through a capsule network, which can effectively process the features of different frequency bands and capture the spectral features from local to wider areas, thereby accurately obtaining the unique features of various language speech in different frequency bands; based on the global association path of a graph neural network, each spectral feature in the spectral sequence is constructed into a graph structure; the global semantic association features between each spectral feature can be mined, and the complex semantic relationship between speech spectral features can be well captured, so that the translation process is no longer limited to the surface speech feature matching, but goes deep into the semantic level for conversion, thereby improving the accuracy of cross-language translation. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a flowchart of a cross-language speech translation method based on an AI big model in one embodiment of the present invention;

[0044] Figure 2 It is a schematic diagram of the steps of a cross-language speech translation method based on an AI big model in one embodiment of the present invention;

[0045] Figure 3 It is a structural block diagram of a cross-language speech translation device based on an AI big model in one embodiment of the present invention;

[0046] Figure 4 It is a schematic block diagram of the structure of a computer device according to an embodiment of the present invention.

[0047] The implementation, functional features and advantages of the present invention will be further described in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0049] Reference Figure 1 as well as Figure 2 In one embodiment of the present invention, a cross-language speech translation method based on an AI large model is provided, comprising the following steps:

[0050] Step S1, performing frame division and spectrum analysis on the cross-language speech data to obtain a spectrum sequence;

[0051] Step S2, inputting the spectrum sequence into a pre-trained AI big model; the AI ​​big model includes a feature capture path based on a capsule network and a global association path based on a graph neural network;

[0052] Step S3, based on the feature capture path, using a capsule network to vectorize spectrum features of different frequency bands in the spectrum sequence to obtain packaged features;

[0053] Step S4, based on the global association path, constructing each spectrum feature in the spectrum sequence into a graph structure through a graph neural network, and mining the global semantic association features between the spectrum features;

[0054] Step S5, performing weighted fusion on the encapsulation features corresponding to the spectrum features of the different frequency bands to obtain a fused encapsulation feature, and aggregating the fused encapsulation feature with the global semantic association feature to obtain an aggregated feature;

[0055] Step S6: based on the aggregated features, translate and obtain speech data in the target language.

[0056] In this embodiment, as described in step S1 above, framing the cross-language speech data is to divide the continuous speech signal into smaller time segments. The purpose is to facilitate the subsequent analysis of the characteristics of the speech in a short period of time, because the characteristics of the speech may change at different times, and framing can handle these changes more finely. The selection of frame length and frame shift needs to comprehensively consider factors such as the characteristics of the speech and computing resources. The appropriate frame length and frame shift can reduce the amount of calculation and data redundancy while ensuring sufficient information.

[0057] After framing, spectrum analysis is performed on each frame. Spectral analysis converts the speech signal in the time domain into a frequency domain representation. Commonly used methods include Fast Fourier Transform (FFT). Through spectrum analysis, we can obtain information such as the energy distribution of speech signals at different frequencies. These spectrum sequences provide a basis for the subsequent extraction and processing of speech features in the frequency domain, and help to explore the differences and commonalities in the frequency characteristics of speech in different languages.

[0058] As described in step S2 above, the obtained spectrum sequence is input into the pre-trained AI big model. The above AI big model has a unique architecture, including a feature capture path based on capsule network and a global association path based on graph neural network. This dual-path design can process the spectrum sequence from different angles and fully mine the information therein. The model has been trained using a large amount of cross-language speech data during the training phase to learn the features and semantic relationships of speech spectra in different languages, so that it can accurately analyze and translate when a new spectrum sequence is input.

[0059] As described in step S3 above, the spectral features of different frequency bands in the spectral sequence are vectorized based on the capsule network in the feature capture path. The capsule network is different from the traditional neural network in that it can represent features in a hierarchical manner. Each capsule can be regarded as a group of neurons with a specific function. For spectral features of different frequency bands, the capsule can be vectorized according to its frequency band range and the local or global properties of the features. This vector encapsulation not only contains the intensity information of the spectral features, but also reflects the spatial and hierarchical relationship between the features, which helps to more accurately describe the speech features of different frequency bands and provide richer information for subsequent fusion and translation.

[0060] As described in step S4 above, based on the global association path, the graph neural network is used to construct each spectral feature in the spectral sequence into a graph structure. In this graph structure, each spectral feature is a node, and the edges between the nodes represent a certain association between the spectral features. Through the information transmission and aggregation capabilities of the graph neural network, the global semantic association features between the spectral features can be mined. This global semantic association reflects the interaction between different spectral features in the entire speech semantic expression. For example, some spectral feature combinations may correspond to specific semantic units, which helps to understand the semantic information behind the speech and overcome the limitations of translation based solely on local spectral features.

[0061] As described in step S5 above, the encapsulation features corresponding to the spectral features of different frequency bands are weighted and fused. The weighting here is determined according to the importance of different frequency bands in the semantic expression of speech. For example, frequency bands related to key phonemes or syllables in a language may be given higher weights. Through this weighted fusion, the features of important frequency bands can be highlighted, while the information of other frequency bands is comprehensively considered to obtain fused encapsulation features.

[0062] Aggregate the fusion encapsulation features with the global semantic association features. This aggregation operation combines the frequency band feature information obtained based on the capsule network and the semantic association information mined based on the graph neural network to form a more comprehensive aggregate feature. This aggregate feature contains both detailed information on the spectral level of the speech and the association on the semantic level, providing sufficient basis for accurate translation.

[0063] As described in step S6 above, the target language speech data is obtained by translation based on the aggregated features. In this process, the model uses the learned cross-language mapping relationship and semantic conversion rules to convert the aggregated features corresponding to the source language speech into the feature representation of the target language speech. Then, through related technologies such as inverse spectrum transformation, the features of the target language speech are restored to the target language speech data, achieving the ultimate goal of cross-language speech translation.

[0064] In this embodiment, a feature capture path based on a capsule network is utilized to vectorize the spectral features of different frequency bands in the spectral sequence through the capsule network, which can effectively process the features of different frequency bands and capture the spectral features from local to wider areas, thereby accurately obtaining the unique features of various language speech in different frequency bands; based on the global association path of the graph neural network, each spectral feature in the spectral sequence is constructed into a graph structure; the global semantic association features between each spectral feature can be mined, and the complex semantic relationship between the speech spectral features can be well captured, so that the translation process is no longer limited to the surface speech feature matching, but goes deep into the semantic level for conversion, thereby improving the accuracy of cross-language translation.

[0065] In one embodiment, based on the aggregated features, translating speech data into a target language includes:

[0066] Processing the aggregated features based on a pyramid neural network to obtain abstract features;

[0067] The abstract features are converted into target language abstract features; based on the target language abstract features, speech data of the target language is translated.

[0068] In the present embodiment, the pyramid neural network has a multi-layer structure, and the number of neurons in different layers shows a decreasing trend, similar to the shape of a pyramid. The design of this structure enables the network to gradually abstract and compress the input aggregate features. In each layer, neurons transmit and process information through specific connection weights and activation functions. When the aggregate features are input to the first layer of the pyramid neural network, a large number of neurons can perform preliminary analysis and integration of the rich information in the aggregate features. As information is transmitted in the network, the reduction in the number of neurons in the subsequent layers prompts the network to abstract the features at a higher level. For example, the lower layers may capture local patterns of spectral features and semantic associations, and as they are transmitted to higher levels, the network begins to extract more universal and representative features, gradually ignoring those details, and finally obtaining abstract features. These abstract features can more concisely represent the core information of the source language speech at the semantic and feature levels, and prepare for cross-language conversion.

[0069] Then, use the pre-trained cross-language mapping model or knowledge base. These resources are built in the process of training a large amount of multilingual speech data, and they contain the correspondence and conversion rules between speech features of different languages. By matching the obtained abstract features with the cross-language mapping model, the abstract features are adjusted and converted according to the commonalities and differences between the languages ​​learned by the model. For example, if the source language and the target language have similar semantic expressions in certain speech spectral features, but there are subtle differences in the specific spectral patterns, then in this step, the abstract features will be modified accordingly according to the mapping relationship to obtain the target language abstract features. The above conversion is based on an in-depth understanding of the semantic relationship between the speech of multiple languages ​​to ensure that the converted target language abstract features can accurately reflect the semantic content of the target language speech.

[0070] After obtaining the abstract features of the target language, it is necessary to convert them into speech data of the target language. This process involves inverse processing of some operations performed in the previous analysis. For example, if certain mathematical transformations (such as Fourier transform, etc.) have been performed on the speech in the steps of spectrum analysis, then the corresponding inverse transformations need to be performed here. According to the speech feature information contained in the abstract features of the target language, such as the phonemes, syllables, intonations and other features of the target language, these features are combined through a preset speech synthesis algorithm. The above algorithm can restore the abstract features to actual speech waveforms according to the speech rules and patterns of the target language, and finally obtain the speech data of the target language, completing the entire process of cross-language speech translation. The above speech synthesis algorithm can adopt a general algorithm, which will not be described here.

[0071] In one embodiment, based on the feature capture path, spectrum features of different frequency bands in the spectrum sequence are vectorized and encapsulated through a capsule network to obtain encapsulated features, including:

[0072] The spectrum sequence is input into a capsule network having multiple capsule layers; wherein a spectrum window of a preset size in the first capsule layer slides on the spectrum sequence, and vector encoding is performed on the spectrum features within the spectrum window covered by the first capsule layer;

[0073] The next capsule layer receives the vector output of the previous capsule layer and updates the connection weight according to the dynamic routing algorithm; wherein the dynamic routing algorithm adjusts the weight according to the activation strength of each capsule layer and the correlation between the spectrum characteristics of different frequency bands;

[0074] After encapsulation processing by multiple capsule layers, the final output is the encapsulation features obtained by encapsulating the spectrum features of different frequency bands.

[0075] In this embodiment, the capsule network is a special neural network structure that can better preserve the hierarchical structure and spatial information of features compared to traditional convolutional neural networks (CNNs). In this solution, multiple capsule layers are used to process the spectrum sequence. This multi-layer structure can gradually abstract and encapsulate the spectrum features at a higher level.

[0076] The operations in the first capsule layer are crucial to the entire feature capture process. It slides over the spectrum sequence with a spectral window of preset size, and the size of this spectral window is determined based on prior knowledge and experiments of speech spectral features. For example, a smaller window may be used to capture local high-frequency features, while a larger window is more suitable for obtaining a wider spectrum trend in the low-frequency range. The spectral features covered by each spectral window will be vector encoded. This vector encoding does not simply list the spectral values, but converts them into a vector with specific dimensions and semantic information based on the intrinsic relationship between the spectral features. This vector can represent the comprehensive information of the spectral features within the spectral window, such as the frequency distribution pattern, the degree of energy concentration, etc.

[0077] The next capsule layer receives the vector output of the previous capsule layer and updates the connection weights according to the dynamic routing algorithm. The dynamic routing algorithm is one of the core mechanisms of the capsule network. In traditional neural networks, the connection weights between layers are usually gradually adjusted during the training process through a fixed learning algorithm (such as the back propagation algorithm). In capsule networks, the dynamic routing algorithm can dynamically adjust the connection weights between capsule layers according to the characteristics of the input data.

[0078] Each capsule has an activation strength, which reflects the prominence of the feature represented by the capsule in the current input. For example, if a capsule detects a strong energy signal in a certain frequency band when processing the spectral features of the frequency band, then its activation strength will be higher. This activation strength information will be used by the dynamic routing algorithm. When the activation strength is high, the weight of the capsule when passing information to the next layer may increase accordingly, so that important features can be more effectively passed to subsequent layers.

[0079] There are complex correlations between spectral features in different frequency bands. The dynamic routing algorithm analyzes these correlations. For example, spectral features in certain frequency bands always appear together or have a specific correlation pattern in a certain semantic unit of speech (such as a specific phoneme or syllable). When this correlation is reflected in the output of the capsule layer, the algorithm adjusts the connection weights and strengthens the connection between related capsules, so that these semantically related spectral features can be better processed in the network, thereby more accurately encapsulating the spectral features.

[0080] After encapsulation processing by multiple layers of capsule layers, the final output is the encapsulated features obtained by encapsulating the spectral features of different frequency bands. After being processed by multiple layers of capsule layers, the spectral features are gradually converted from the original original spectral sequence to encapsulated features with high semantic and structural information. Each layer of capsule layer further abstracts and integrates the features of the previous layer, so that the final encapsulated features can fully and accurately represent the spectral features of different frequency bands. The above encapsulated features are not just simple spectral feature vectors, they contain the hierarchical relationship, semantic association and position information between different frequency bands in the entire speech spectrum. For example, in a speech spectrum containing multiple frequency bands, the encapsulated features can clearly show how the high-frequency band features and the low-frequency band features interact to form a specific speech pattern. This information is of great value for subsequent speech processing (such as combining with graph neural networks to mine global semantic associations, final speech translation, etc.).

[0081] In one embodiment, based on the global association path, each spectrum feature in the spectrum sequence is constructed into a graph structure through a graph neural network, and global semantic association features between the spectrum features are mined, including:

[0082] Each spectrum feature in the spectrum sequence is used as each node of the graph structure; wherein the initial feature vector of each node is the original numerical representation of the corresponding spectrum feature; the weight of the edge in the graph structure is the similarity between each spectrum feature;

[0083] The constructed graph structure is input into the graph neural network, which uses a multi-layer graph convolution layer. In each graph convolution layer, each node updates its own feature vector by weighted summation based on its own feature vector and the feature vectors of adjacent nodes, and the weight is the weight of the edge in the graph structure.

[0084] After information transmission and aggregation through multiple layers of graph convolutional layers, the feature vector of each node integrates the information of other related nodes in the global scope, and the global semantic association features between the spectral features are mined.

[0085] In this embodiment, a graph structure is constructed based on a spectrum sequence, and the core idea is to regard each spectrum feature as a node in the graph. The advantage of this is that the characteristics of the graph can be used to mine the complex relationship between spectrum features. For example, spectrum features in different frequency bands may be interrelated in speech semantic expression, and these relationships can be more intuitively presented and analyzed through the graph structure.

[0086] The initial feature vector of each node is set to the original numerical representation of the corresponding spectrum feature. That is, the original numerical information of the spectrum feature is directly used to initialize the node's attributes. For example, for the spectrum feature of a certain frequency band, its energy value, frequency value and other raw data in the frequency domain constitute the initial feature vector of the node. This initial setting retains the most basic information of the spectrum feature and provides basic data for subsequent processing in the graph structure.

[0087] The weight of the edge in the graph structure is defined as the similarity between each spectral feature. It is very important to use similarity to measure the degree of association between spectral features. Generally, a variety of methods can be used to calculate similarity, such as cosine similarity, Euclidean distance, etc. Taking cosine similarity as an example, if the cosine similarity of two spectral feature vectors is high, it means that they are close in direction, which means that the two spectral features are similar to some extent. Determining the weight of the edge in this way can accurately reflect the closeness between different spectral features, laying the foundation for subsequent graph neural networks to transfer and aggregate information based on edge weights.

[0088] Graph Neural Network (GNN) is a type of neural network specifically designed to process graph structure data. In this solution, the constructed graph structure is input into a graph neural network using multiple layers of graph convolutional layers. The graph convolutional layer is the core component of the graph neural network. It can perform convolution operations on the graph structure similar to the convolutional neural network on the image, thereby effectively processing the node features in the graph. The design of multiple layers of graph convolutional layers enables the network to gradually and deeply explore the relationship between nodes in the graph structure. Each layer of graph convolutional layer can further refine and integrate node features, just like multiple layers of hidden layers in traditional neural networks gradually extract more abstract features.

[0089] In each graph convolution layer, each node must update its own feature vector based on its own feature vector and the feature vectors of adjacent nodes. Specifically, the update is performed in a weighted summation manner, and the weight here is the weight of the edge in the graph structure determined previously. The principle of the above update mechanism is that the characteristics of a node not only depend on its own original characteristics, but also are affected by its adjacent nodes. Because in the graph structure, adjacent nodes are connected to the node through edges, and the weight of the edge represents the degree of association between them. Therefore, when a node updates its own feature vector, it will perform a weighted summation of the feature vectors of the adjacent nodes according to the degree of association between the adjacent nodes and itself (that is, the weight of the edge), and combine it with its own original feature vector to obtain an updated feature vector. In this way, each node can continuously absorb relevant information from adjacent nodes in each graph convolution layer, and realize the gradual enrichment and optimization of features.

[0090] As multiple layers of graph convolutional layers are processed in sequence, information is continuously transmitted and aggregated between nodes in the graph structure. In each layer of graph convolutional layer, the node updates its own feature vector according to the above update mechanism. This process is actually the process of information transmission. Each node transmits its own information to adjacent nodes, and also receives and integrates information from adjacent nodes. After repeated operations of multiple layers of graph convolutional layers, information gradually spreads throughout the entire graph structure. From a local perspective, each node absorbs information from surrounding adjacent nodes at each layer; from a global perspective, after multiple layers of processing, the feature vectors of each node eventually integrate the information of other related nodes in the global scope. This means that the spectral features represented by each node are no longer limited to its initial, isolated feature vector, but are integrated with the information of other related spectral features in the entire graph structure.

[0091] When the feature vectors of each node have integrated the information of other related nodes in the global scope, the global semantic association features between the spectral features are mined. These features reflect the semantic relationship between different spectral features in the entire speech spectrum. For example, it may be found that the spectral features of certain frequency bands always have a specific association pattern with the spectral features of other frequency bands when the speech expresses specific semantics (such as a word, phrase, or speech intonation pattern). By mining these global semantic association features, a deeper and more comprehensive semantic information foundation is provided for subsequent operations such as speech translation based on these features, which helps to improve the accuracy of speech translation and the depth of understanding of speech semantics.

[0092] In one embodiment, weighted fusion is performed on the encapsulation features corresponding to the spectrum features of the different frequency bands to obtain a fused encapsulation feature, and the fused encapsulation feature is aggregated with the global semantic association feature to obtain an aggregated feature, including:

[0093] For the encapsulation features corresponding to the spectrum features of different frequency bands, their importance in the expression of speech semantics is analyzed to assign weights, and the encapsulation features are weighted summed according to the assigned weights to obtain the fused encapsulation features;

[0094] The fused encapsulation features are dimensionally aligned with the global semantic association features, the aligned features are spliced ​​to obtain a splicing vector, the splicing vector is input into a multi-layer perceptron, and the aggregated features are output after nonlinear transformation and information fusion.

[0095] In this embodiment, in the speech data, the spectral features of different frequency bands play different roles in the expression of speech semantics. For example, some frequency bands may be closely related to the pronunciation features of vowels, while other frequency bands may be more critical to the expression of consonants, or may have different focuses on the intonation, rhythm, etc. of the speech. Therefore, it is necessary to deeply analyze the importance of the spectral features of each frequency band in the expression of speech semantics, and use this as the basis for allocating weights. The above analysis can be carried out in a variety of ways, such as statistical analysis of a large number of speech samples annotated with speech semantic content, observing the frequency of occurrence and change rules of spectral features of different frequency bands in different semantic contexts; it is also possible to use existing phonetic knowledge and related research results, combined with specific application scenarios to determine the degree of correlation between spectral features of different frequency bands and speech semantics.

[0096] Once the importance of spectral features of different frequency bands in speech semantic expression is determined, corresponding weights can be assigned to the encapsulation features corresponding to each frequency band. For example, if the spectral features of a certain frequency band are determined to be very important for expressing a key speech semantic element (such as a specific phoneme or syllable), then a relatively high weight will be assigned to its corresponding encapsulation features. After the weights are assigned, a weighted summation operation is performed on the encapsulation features corresponding to the spectral features of different frequency bands. Through weighted summation, the encapsulation features of different frequency bands are fused according to their importance. The resulting fused encapsulation features integrate the information of each frequency band and highlight the frequency band features that are more important to the speech semantic expression.

[0097] The fused encapsulation features and the global semantic association features are usually obtained under different processing paths, and their dimensions may be inconsistent. For example, the fused encapsulation features may be a vector of a specific dimension obtained after encapsulation processing of the spectral features, while the global semantic association features are vectors of another dimension obtained by constructing a graph structure and mining the spectral features through a graph neural network. In order to be able to effectively process these two features in the subsequent process, the dimensions need to be aligned first. This can be achieved through a variety of linear transformation methods, such as using a fully connected layer for dimensional mapping, transforming the lower-dimensional feature vectors to the same dimension as the higher-dimensional feature vectors through a suitable weight matrix, ensuring that they have matching dimensional structures in subsequent splicing operations and further processing.

[0098] After the dimension alignment is completed, the aligned fusion encapsulation features and global semantic association features are spliced. Splicing is to connect two feature vectors in a certain order to form a new splicing vector. For example, if the fusion encapsulation feature is a vector of dimension m after dimension alignment, and the global semantic association feature is a vector of dimension n after dimension alignment, then the dimension of the spliced ​​splicing vector is m + n. Through the splicing operation, two features from different sources but both related to speech features and semantics are integrated together to form a vector containing richer information.

[0099] The concatenated vector is input into a multi-layer perceptron (MLP). The multi-layer perceptron is a common neural network structure, which consists of multiple hidden layers and can perform nonlinear transformation and information fusion on the input feature vector. In each hidden layer, neurons process the input vector through specific activation functions (such as ReLU, tanh, etc.) to achieve nonlinear transformation. The above nonlinear transformation can dig out the deeper relationships and information hidden in the input vector, so that the output feature vector processed by the multi-layer perceptron can better reflect the comprehensive characteristics and semantic information of the speech. At the same time, through the information transmission and fusion between the layers, the final output is the fully processed and fused aggregated feature, which contains both the information from the fusion of spectral features in different frequency bands and the information of global semantic association features, providing a more comprehensive and accurate feature basis for subsequent operations such as speech translation.

[0100] In one embodiment, after the translation obtains the speech data in the target language, the translation includes:

[0101] Extracting the spectrum features of the specified frequency band in the spectrum sequence, and performing singular value decomposition to obtain a spectrum singular value matrix;

[0102] Extracting suprasegmental features of speech data of the target language and converting them into symbol sequences;

[0103] Inputting the spectrum singular value matrix and the symbol sequence into a key generator based on a chaotic system;

[0104] Determine the initial parameter range of the chaotic system, convert the initial parameter range into high-dimensional discrete chaotic mapping initial parameters based on a mapping algorithm, and perform high-dimensional discrete chaotic iteration to obtain the chaotic system parameters of the final key generator;

[0105] The spectrum singular value matrix and the symbol sequence are processed based on the final key generator to output an encryption key for encrypting the speech data of the target language.

[0106] In this embodiment, in the entire speech spectrum sequence, the spectrum features of different frequency bands contribute to various characteristics of speech (such as timbre, phoneme composition, semantic expression, etc.) to varying degrees. The extraction of the spectrum features of the specified frequency band here is based on an in-depth study and analysis of the relationship between speech characteristics and frequency bands. For example, some frequency bands are closely related to the vowel pronunciation characteristics of a specific language, or play a key role in reflecting changes in speech intonation. By determining these specified frequency bands that are of great significance in speech semantics or acoustic characteristics in advance, key information related to subsequent encryption and speech characteristics can be extracted more specifically. Singular value decomposition (SVD) is performed on the extracted spectrum features of the specified frequency band. Singular value decomposition is an important matrix decomposition method that can decompose a matrix into the product form of three matrices. In this embodiment, a spectrum singular value matrix is ​​obtained by performing singular value decomposition on the spectrum feature matrix. This spectrum singular value matrix contains the importance information of the original spectrum features in different dimensions, and presents the structure and characteristics of the spectrum features of the specified frequency band in a more concise and more reflective way, which provides basic data for subsequent generation of encryption keys in combination with other features.

[0107] Suprasegmental features refer to features of speech involving intonation, stress, rhythm, etc. in addition to basic speech units such as phonemes. These features play an important role in the semantic expression of speech and the embodiment of language style. Suprasegmental features are extracted from the speech data of the target language through specific speech analysis techniques and algorithms. For example, audio processing software or special speech analysis tools can be used to analyze the changing rules of speech signals in the time domain and frequency domain to determine suprasegmental features such as the fluctuation of intonation, the position of stress, and the pattern of rhythm. The extracted suprasegmental features are converted into symbol sequences to facilitate subsequent fusion processing with other features and operations in the encryption generator. This conversion is usually based on preset encoding rules, mapping different suprasegmental feature values ​​(such as different intonation patterns, stress intensity, etc.) to specific symbols. For example, different symbols can be set to represent the ups and downs of intonation, and the stress position can be marked with another symbol. Through such encoding, complex suprasegmental features are presented in a concise and standardized symbol sequence form, so that they can better participate in the subsequent encryption key generation process.

[0108] Chaotic systems have characteristics such as being highly sensitive to initial conditions and unpredictable long-term behavior. The key generator based on chaotic systems uses these characteristics to generate encryption keys with high randomness and high security. When the spectrum singular value matrix and the symbol sequence are input into the key generator, it will generate an encryption key that is closely related to these inputs and has good randomness through internal complex calculations and processing mechanisms based on the characteristic information of these two inputs.

[0109] The spectral singular value matrix and the symbol sequence reflect the characteristics of speech data from different perspectives. The spectral singular value matrix carries the structure and importance information of the speech spectrum in the specified frequency band, while the symbol sequence reflects the suprasegmental feature information of the target language speech data. These two different types of features are input into the key generator, so that the generated encryption key can comprehensively consider the spectral structure and suprasegmental features of the speech data, thereby improving the relevance and pertinence of the encryption key to the speech data, and enhancing the encryption effect and security.

[0110] The output result of the chaotic system depends on the setting of its initial parameters. In this embodiment, it is necessary to first determine the initial parameter range of the chaotic system. This determination process is based on a comprehensive analysis of the spectrum singular value matrix and the symbol sequence. For example, the value range of the initial parameters of the chaotic system is roughly determined based on factors such as the size distribution of the singular values ​​in the spectrum singular value matrix, the types and frequency of occurrence of symbols in the symbol sequence, etc. In this way, the initial parameter range can be associated with the input voice data characteristics, laying the foundation for the subsequent generation of suitable encryption keys. Based on the mapping algorithm, the determined initial parameter range is converted into high-dimensional discrete chaotic mapping initial parameters. The mapping algorithm is a method for accurately converting information related to the input voice data characteristics (such as various information in the spectrum singular value matrix and the symbol sequence) into the initial parameters of the chaotic system. It needs to take into account the characteristics of the chaotic system and the relationship between the characteristics of the voice data, and through carefully designed mathematical formulas and operation rules, the information in the initial parameter range is converted into the initial parameters of the high-dimensional discrete chaotic mapping in a way that meets the requirements of the chaotic system, ensuring that the chaotic system can perform reasonable operations and behavior evolution according to the characteristics of the voice data in the subsequent iteration process.

[0111] After obtaining the initial parameters of the high-dimensional discrete chaotic mapping, high-dimensional discrete chaotic iteration is performed. High-dimensional discrete chaotic iteration is a series of complex calculation processes performed by the chaotic system based on its own mathematical model and initial parameters. In each iteration, the chaotic system calculates the value of the next state according to the current parameter values ​​and internal operation rules. Due to the high sensitivity of the chaotic system to the initial conditions, even a small difference in the initial parameters will lead to completely different results in the iteration process. Through multiple iterations, the chaotic system continues to evolve, and its state value is also constantly changing, and finally the chaotic system parameters of the final key generator are obtained. These parameters are the results of multiple iterations that are closely related to the characteristics of the voice data and have high randomness and unpredictability, providing an important basis for generating high-quality encryption keys.

[0112] Based on the chaotic system parameters of the key generator finally obtained, the input spectrum singular value matrix and symbol sequence are processed. In this process, the key generator performs complex operations and combinations on the two input features according to its internal operation mechanism and chaotic system parameters, and finally outputs the encryption key. This encryption key is generated based on the characteristics of the chaotic system and the operation results closely related to the characteristics of the voice data. It has high randomness and unpredictability, can effectively prevent unauthorized access and data leakage, and ensure the security of the target language voice data. Once the encryption key is generated, the target language voice data can be encrypted using this encryption key. In this way, even if the voice data is intercepted during transmission or storage, unauthorized personnel cannot obtain its true content. Only authorized personnel with the correct decryption key can restore the ciphertext to the original voice data, thereby achieving the security protection of the target language voice data.

[0113] In one embodiment, processing the spectrum singular value matrix and the symbol sequence based on a final key generator to output an encryption key includes:

[0114] The spectrum singular value matrix and the symbol sequence are embedded into the state space of the chaotic system of the final key generator. For the spectrum singular value matrix, each singular value is mapped to a specific dimension of the chaotic system according to the rules, so that the spectrum characteristics can be reflected in the chaotic system. For the symbol sequence, each symbol is converted into a vector through encoding, and then these vectors are embedded into the chaotic system in a specific order and manner. In this way, the spectrum singular value matrix and the symbol sequence become part of the initial state of the chaotic system.

[0115] Start the chaotic iteration process. In each iteration, the chaotic system updates its state according to the current input and the chaotic mapping rules. In this process, the information carried by the spectral singular value matrix and the symbol sequence is continuously fused and diffused in the chaotic system. For example, the information between different dimensions interacts with each other through the nonlinear operation of chaotic mapping, so that the spectral features and suprasegmental features are deeply fused in the chaotic system to form complex mixed information.

[0116] During the iteration process, intermediate monitoring points are set to monitor the changes in the state of the chaotic system. According to the monitoring results, the iteration process is adjusted. After a preset number of iterations, the encryption key is extracted from the state of the chaotic system. There are many ways to extract it, such as selecting elements at specific positions in the state vector of the chaotic system as part of the key, or performing mathematical transformations (such as hash operations, discrete cosine transforms, etc.) on the entire state vector to generate the key.

[0117] Optimize the extracted encryption key. Check the randomness and uniformity of the key. If the key is found to have local regularity or unevenness, eliminate these problems through some operations (such as XOR operation with pseudo-random numbers, rearrange the key, etc.) to improve the quality of the key.

[0118] Reference Figure 3 In another embodiment of the present invention, a cross-language speech translation device based on an AI large model is provided, comprising:

[0119] An analysis unit, used for performing frame division and spectrum analysis on the cross-language speech data to obtain a spectrum sequence;

[0120] An input unit, used to input the spectrum sequence into a pre-trained AI big model; the AI ​​big model includes a feature capture path based on a capsule network and a global association path based on a graph neural network;

[0121] An encapsulation unit, configured to perform vector encapsulation on the spectrum features of different frequency bands in the spectrum sequence through a capsule network based on the feature capture path to obtain encapsulated features;

[0122] A mining unit, configured to construct each spectrum feature in the spectrum sequence into a graph structure through a graph neural network based on the global association path, and mine global semantic association features between the spectrum features;

[0123] an aggregation unit, configured to perform weighted fusion on the encapsulation features corresponding to the spectrum features of the different frequency bands to obtain a fused encapsulation feature, and aggregate the fused encapsulation feature with the global semantic association feature to obtain an aggregate feature;

[0124] A translation unit is used to translate the speech data into a target language based on the aggregated features.

[0125] In this embodiment, for the specific implementation of each unit in the above device embodiment, please refer to the above method embodiment, which will not be repeated here.

[0126] Reference Figure 4 In an embodiment of the present invention, a computer device is also provided. The computer device may be a server, and its internal structure may be as follows: Figure 4As shown. The computer device includes a processor, a memory, a display screen, an input device, a network interface and a database connected through a system bus. Among them, the processor designed by the computer is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store the corresponding data in this embodiment. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the above method is implemented.

[0127] Those skilled in the art will understand that Figure 4 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present invention, and does not constitute a limitation on the computer device to which the solution of the present invention is applied.

[0128] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the above method is implemented. It can be understood that the computer-readable storage medium in this embodiment can be a volatile readable storage medium or a non-volatile readable storage medium.

[0129] In summary, the cross-language speech translation method, apparatus and device based on the AI ​​big model provided in the embodiments of the present invention include: framing and spectral analysis of cross-language speech data to obtain a spectrum sequence; inputting the spectrum sequence into a pre-trained AI big model; the AI ​​big model includes a feature capture path based on a capsule network and a global association path based on a graph neural network; based on the feature capture path, vectorizing the spectrum features of different frequency bands in the spectrum sequence through a capsule network to obtain encapsulation features; based on the global association path, constructing each spectrum feature in the spectrum sequence into a graph structure through a graph neural network, and mining the global semantic association features between each spectrum feature; weighted fusion of the encapsulation features corresponding to the spectrum features of different frequency bands to obtain fused encapsulation features, aggregating the fused encapsulation features with the global semantic association features to obtain aggregated features; based on the aggregated features, translating the speech data of the target language. In the present invention, a feature capture path based on a capsule network is utilized to vectorize the spectral features of different frequency bands in a spectral sequence through a capsule network, which can effectively process the features of different frequency bands and capture the spectral features from local to wider areas, thereby accurately obtaining the unique features of various language speech in different frequency bands; based on the global association path of a graph neural network, each spectral feature in the spectral sequence is constructed into a graph structure; the global semantic association features between each spectral feature can be mined, and the complex semantic relationship between speech spectral features can be well captured, so that the translation process is no longer limited to the surface speech feature matching, but goes deep into the semantic level for conversion, thereby improving the accuracy of cross-language translation.

[0130] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media provided by the present invention and used in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double-speed data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM.

[0131] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the presence of other identical elements in the process, device, article or method including the element.

[0132] The above description is only a preferred embodiment of the present invention, and does not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A cross-language speech translation method based on an AI large model, characterized in that: The following steps are involved: Perform frame division and spectrum analysis on cross-language speech data to obtain spectrum sequences; Inputting the spectrum sequence into a pre-trained AI big model; the AI ​​big model includes a feature capture path based on a capsule network and a global association path based on a graph neural network; Based on the feature capture path, the spectrum features of different frequency bands in the spectrum sequence are vectorized and encapsulated by a capsule network to obtain encapsulated features; Based on the global association path, each spectrum feature in the spectrum sequence is constructed into a graph structure through a graph neural network, and global semantic association features between the spectrum features are mined; Performing weighted fusion on the encapsulation features corresponding to the spectrum features of the different frequency bands to obtain a fused encapsulation feature, and aggregating the fused encapsulation feature with the global semantic association feature to obtain an aggregated feature; Based on the aggregated features, translating to obtain speech data in a target language; Based on the global association path, each spectrum feature in the spectrum sequence is constructed into a graph structure through a graph neural network, and global semantic association features between the spectrum features are mined, including: Each spectrum feature in the spectrum sequence is used as each node of the graph structure; wherein the initial feature vector of each node is the original numerical representation of the corresponding spectrum feature; the weight of the edge in the graph structure is the similarity between each spectrum feature; The constructed graph structure is input into the graph neural network, which uses a multi-layer graph convolution layer. In each graph convolution layer, each node updates its own feature vector by weighted summation based on its own feature vector and the feature vectors of adjacent nodes, and the weight is the weight of the edge in the graph structure. After information transmission and aggregation through multiple layers of graph convolutional layers, the feature vector of each node integrates the information of other related nodes in the global scope, and the global semantic correlation features between the spectral features are mined; The encapsulation features corresponding to the spectrum features of the different frequency bands are weightedly fused to obtain fused encapsulation features, and the fused encapsulation features are aggregated with the global semantic association features to obtain aggregated features, including: For the encapsulation features corresponding to the spectrum features of different frequency bands, their importance in the expression of speech semantics is analyzed to assign weights, and the encapsulation features are weighted summed according to the assigned weights to obtain the fused encapsulation features; The fused encapsulation features are dimensionally aligned with the global semantic association features, the aligned features are spliced ​​to obtain a splicing vector, the splicing vector is input into a multi-layer perceptron, and the aggregated features are output after nonlinear transformation and information fusion.

2. The cross-language speech translation method based on the AI ​​big model according to claim 1 is characterized in that: Based on the aggregated features, the speech data in the target language is translated, including: Processing the aggregated features based on a pyramid neural network to obtain abstract features; The abstract features are converted into target language abstract features; based on the target language abstract features, speech data of the target language is translated.

3. The cross-language speech translation method based on AI big model according to claim 1 is characterized in that: Based on the feature capture path, the spectrum features of different frequency bands in the spectrum sequence are vectorized and encapsulated through a capsule network to obtain encapsulated features, including: The spectrum sequence is input into a capsule network having multiple capsule layers; wherein a spectrum window of a preset size in the first capsule layer slides on the spectrum sequence, and vector encoding is performed on the spectrum features within the spectrum window covered by the first capsule layer; The next capsule layer receives the vector output of the previous capsule layer and updates the connection weight according to the dynamic routing algorithm; wherein the dynamic routing algorithm adjusts the weight according to the activation strength of each capsule layer and the correlation between the spectrum characteristics of different frequency bands; After encapsulation processing by multiple capsule layers, the final output is the encapsulation features obtained by encapsulating the spectrum features of different frequency bands.

4. The cross-language speech translation method based on the AI ​​big model according to claim 1 is characterized in that: After the translation obtains the speech data in the target language, the translation includes: Extracting the spectrum features of the specified frequency band in the spectrum sequence, and performing singular value decomposition to obtain a spectrum singular value matrix; Extracting suprasegmental features of speech data of the target language and converting them into symbol sequences; Inputting the spectrum singular value matrix and the symbol sequence into a key generator based on a chaotic system; Determine the initial parameter range of the chaotic system, convert the initial parameter range into high-dimensional discrete chaotic mapping initial parameters based on a mapping algorithm, and perform high-dimensional discrete chaotic iteration to obtain the chaotic system parameters of the final key generator; The spectrum singular value matrix and the symbol sequence are processed based on the final key generator to output an encryption key for encrypting the speech data of the target language.

5. A cross-language speech translation device based on an AI large model, characterized in that: include: An analysis unit, used for performing frame division and spectrum analysis on the cross-language speech data to obtain a spectrum sequence; An input unit, used to input the spectrum sequence into a pre-trained AI big model; the AI ​​big model includes a feature capture path based on a capsule network and a global association path based on a graph neural network; An encapsulation unit, configured to perform vector encapsulation on the spectrum features of different frequency bands in the spectrum sequence through a capsule network based on the feature capture path to obtain encapsulated features; A mining unit, configured to construct each spectrum feature in the spectrum sequence into a graph structure through a graph neural network based on the global association path, and mine global semantic association features between the spectrum features; an aggregation unit, configured to perform weighted fusion on the encapsulation features corresponding to the spectrum features of the different frequency bands to obtain a fused encapsulation feature, and aggregate the fused encapsulation feature with the global semantic association feature to obtain an aggregate feature; A translation unit, configured to translate speech data into a target language based on the aggregated features; Based on the global association path, each spectrum feature in the spectrum sequence is constructed into a graph structure through a graph neural network, and global semantic association features between the spectrum features are mined, including: Each spectrum feature in the spectrum sequence is used as each node of the graph structure; wherein the initial feature vector of each node is the original numerical representation of the corresponding spectrum feature; the weight of the edge in the graph structure is the similarity between each spectrum feature; The constructed graph structure is input into the graph neural network, which uses a multi-layer graph convolution layer. In each graph convolution layer, each node updates its own feature vector by weighted summation based on its own feature vector and the feature vectors of adjacent nodes, and the weight is the weight of the edge in the graph structure. After information transmission and aggregation through multiple layers of graph convolutional layers, the feature vector of each node integrates the information of other related nodes in the global scope, and the global semantic correlation features between the spectral features are mined; The encapsulation features corresponding to the spectrum features of the different frequency bands are weightedly fused to obtain fused encapsulation features, and the fused encapsulation features are aggregated with the global semantic association features to obtain aggregated features, including: For the encapsulation features corresponding to the spectrum features of different frequency bands, their importance in the expression of speech semantics is analyzed to assign weights, and the encapsulation features are weighted summed according to the assigned weights to obtain the fused encapsulation features; The fused encapsulation features are dimensionally aligned with the global semantic association features, the aligned features are spliced ​​to obtain a splicing vector, the splicing vector is input into a multi-layer perceptron, and the aggregated features are output after nonlinear transformation and information fusion.

6. A computer device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 4 are implemented.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images