Speech data processing method based on AI large model and related equipment

Abstract

The application relates to the technical field of data processing, and specifically provides a voice data processing method based on an AI large model and related equipment, which comprises the following steps: after obtaining voice data to be processed, the voice data is divided into a plurality of voice segments; at least one acoustic feature is extracted from each voice segment to construct an acoustic feature set; all the voice segments are input into a pre-trained voice semantic artificial intelligence large model to generate a semantic feature vector corresponding to the voice segments, then mapping is performed to obtain a mapping feature vector; after the mapping feature vector and the acoustic feature set are subjected to feature fusion, the fused features are subjected to voice data processing to obtain a voice data processing result. The voice data is divided into a plurality of voice segments to construct an acoustic feature set and a mapping feature vector for feature fusion, and then voice data processing is performed to obtain a voice data processing result, thereby improving the problem of insufficient stability of the voice data processing result in a complex or continuous voice scene.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and more specifically, to a method and related equipment for voice data processing based on large AI models. Background Technology

[0002] With the continuous development of voice interaction technology and intelligent information processing technology, voice data has been widely used in various fields such as voice recognition, voice understanding, intelligent customer service, in-vehicle interaction, and human-computer dialogue. In practical application scenarios, voice data often has characteristics such as long duration, complex environment, and diverse speaking styles, which places higher demands on the accuracy and stability of voice data processing methods.

[0003] In existing speech data processing technologies, the speech data is usually preprocessed before being directly input into a pre-trained speech processing model. The model then analyzes the speech data and outputs the corresponding processing results, thereby improving the automation and efficiency of speech data processing.

[0004] Although automatic analysis and processing of speech information can be achieved by inputting speech data into a speech processing model, the processing results output by the model are prone to fluctuations or discontinuities when faced with long continuous speech, speech content that changes frequently, or application scenarios with complex environmental noise. This results in insufficient stability of the speech data processing results, which in turn affects the overall effect of subsequent speech applications. Summary of the Invention

[0005] The embodiments of this application provide a speech data processing method and related equipment based on AI large models, which can improve the problem of insufficient stability of speech data processing results in complex or continuous speech scenarios.

[0006] Other features and advantages of this application will become apparent from the following detailed description, or may be learned in part from practice of this application.

[0007] According to one aspect of the embodiments of this application, a speech data processing method based on an AI large model is provided, comprising: acquiring speech data to be processed, and dividing the speech data into multiple speech segments based on a preset speech segmentation rule; extracting at least one acoustic feature from each speech segment, and constructing an acoustic feature set based on the acoustic features; inputting all the speech segments into a pre-trained speech semantic AI large model to generate semantic feature vectors corresponding to the speech segments; mapping the semantic feature vectors based on a preset semantic stability constraint rule to obtain mapped feature vectors; fusing the mapped feature vectors with the acoustic feature set to obtain fused features; and performing speech data processing on the fused features to obtain a speech data processing result.

[0008] In some embodiments of this application, based on the aforementioned scheme, the step of acquiring the speech data to be processed and dividing the speech data into multiple speech segments based on a preset speech segmentation rule includes: acquiring speech data through an audio acquisition interface and preprocessing the speech data to obtain the speech data to be processed; performing short-time energy calculation on the speech data to be processed, sliding a preset time window on the speech data to be processed, and calculating the short-time energy value within each time window; marking time windows with short-time energy values ​​lower than a preset silence energy threshold as silence segments, and marking time windows with short-time energy values ​​higher than the preset silence energy threshold as valid speech segments; using silence segments with a duration exceeding a preset silence duration threshold as segmentation boundaries, and using the valid speech segments between adjacent segmentation boundaries as a speech segment; wherein the duration is less than the time window.

[0009] In some embodiments of this application, based on the foregoing scheme, the step of extracting at least one acoustic feature from each of the speech segments and constructing an acoustic feature set based on the acoustic features includes: performing a short-time Fourier transform on each of the speech segments to obtain a corresponding time-spectrum diagram, and extracting Mel-frequency cepstral coefficients as a first acoustic feature based on the time-spectrum diagram; calculating the fundamental frequency trajectory for each of the speech segments, extracting the fundamental frequency period of the speech segment based on the fundamental frequency trajectory, obtaining the fundamental frequency value according to the fundamental frequency period, and using the fundamental frequency value and its time-domain rate of change as a second acoustic feature; concatenating the first acoustic feature and the second acoustic feature of each of the speech segments to form an acoustic feature vector corresponding to the speech segment, and arranging the acoustic feature vectors of all the speech segments in order of speech segment number to construct an acoustic feature set.

[0010] In some embodiments of this application, based on the aforementioned scheme, the pre-trained speech-semantic AI large model includes a convolutional coding layer, a multi-head self-attention layer, and a feedforward decoding layer; the step of inputting all the speech segments into the pre-trained speech-semantic AI large model to generate semantic feature vectors corresponding to the speech segments includes: dividing the speech segments into frames using a preset frame length and frame shift, and applying a Hamming window function to each frame to obtain a frame sequence; extracting linear frequency cepstral coefficients from the frame sequence, and using the extraction result as the input features of the speech-semantic AI large model; inputting the input features into the speech-semantic AI large model, and in the speech-semantic AI large model, using the convolutional coding layer to extract local temporal patterns from the input features to obtain an encoded representation; performing global context dependency modeling on the encoded representation through the multi-head self-attention layer to obtain an attention representation; performing a nonlinear transformation on the attention representation according to the feedforward decoding layer, and outputting a fixed-dimensional semantic feature vector corresponding to each speech segment.

[0011] In some embodiments of this application, based on the aforementioned scheme, the step of mapping the semantic feature vectors based on preset semantic stability constraint rules to obtain mapped feature vectors includes: calculating the mean and standard deviation of each dimension component in each semantic feature vector; performing constraint verification on the semantic feature vectors based on preset semantic stability constraint rules, the semantic stability constraint rules including a mean upper limit threshold, a standard deviation upper limit threshold, and a cross-segment cosine similarity lower limit threshold; truncating and normalizing the dimension components in the semantic feature vectors whose mean and standard deviation exceed the mean upper limit threshold and the standard deviation upper limit threshold to obtain normalized semantic feature vectors; performing weighted smoothing on semantic feature vectors whose cosine similarity with semantic feature vectors corresponding to adjacent speech segments is lower than the cross-segment cosine similarity lower limit threshold to obtain weighted smoothed semantic feature vectors; and inputting the normalized semantic feature vectors and the weighted smoothed semantic feature vectors into a preset linear mapping matrix for linear transformation to obtain mapped feature vectors.

[0012] In some embodiments of this application, based on the aforementioned scheme, the step of fusing the mapped feature vector with the acoustic feature set to obtain fused features, and performing speech data processing on the fused features to obtain speech data processing results includes: concatenating the mapped feature vector corresponding to each speech segment with the corresponding acoustic feature vector in the acoustic feature set along the feature dimension to obtain a concatenated feature vector; inputting the concatenated feature vector into a preset cross-modal attention fusion module to calculate an attention weight matrix; performing weighted aggregation on the concatenated feature vector according to the attention weight matrix to obtain fused features; and inputting the fused features into a task processing layer to output the speech data processing results.

[0013] In some embodiments of this application, based on the foregoing scheme, in the cross-modal attention fusion module, the mapped feature vector is used as the query vector, the acoustic feature vector is used as the key vector and the value vector, and the attention weight matrix is ​​calculated based on the query vector, the key vector and the value vector.

[0014] According to another aspect of the embodiments of this application, a speech data processing system based on an AI large model is provided, comprising: a data acquisition module, configured to acquire speech data to be processed and divide the speech data into multiple speech segments based on a preset speech segmentation rule; a feature extraction module, configured to extract at least one acoustic feature from each of the speech segments and construct an acoustic feature set based on the acoustic features; a vector generation module, configured to input all the speech segments into a pre-trained speech semantic AI large model to generate semantic feature vectors corresponding to the speech segments; a vector mapping module, configured to map the semantic feature vectors based on a preset semantic stability constraint rule to obtain mapped feature vectors; and a data processing module, configured to fuse the mapped feature vectors with the acoustic feature set to obtain fused features, and perform speech data processing on the fused features to obtain speech data processing results.

[0015] According to another aspect of the embodiments of this application, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the processor executes the computer program to implement the above-described AI-based large model-based voice data processing method.

[0016] According to another aspect of the embodiments of this application, a computer-readable storage medium is provided having a computer program stored thereon, which, when run by a processor, causes the processor to perform the speech data processing method based on an AI large model as described above.

[0017] Compared with existing technologies, this application has the following beneficial effects: It acquires speech data to be processed, divides the speech data into multiple speech segments based on preset speech segmentation rules, extracts at least one acoustic feature from each speech segment, and constructs an acoustic feature set based on the acoustic features to form a basic acoustic representation of the speech data. All speech segments are input into a pre-trained speech semantic artificial intelligence model to generate semantic feature vectors corresponding to the speech segments, enabling the speech data to obtain semantic-level representation information. Based on preset semantic stability constraint rules, the semantic feature vectors are mapped to obtain mapped feature vectors, which are then constrained and adjusted. The mapped feature vectors are then fused with the acoustic feature set to obtain fused features. The fused features are then processed to obtain the speech data processing results, thereby improving the problem of insufficient stability of speech data processing results in complex or continuous speech scenarios. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating the voice data processing method and related equipment based on an AI large model provided in an embodiment of the present invention. Figure 2 This is a schematic block diagram of the structure of the AI-based large model-based voice data processing method and related equipment provided in the embodiments of the present invention; Figure 3 This is a schematic block diagram of the structure of the electronic device provided in the embodiment of the present invention. Detailed Implementation

[0019] Exemplary embodiments will now be described in a more comprehensive manner with reference to the accompanying drawings. However, the exemplary embodiments can be implemented in various forms and should not be construed as limited to these examples; rather, these embodiments are provided so that this application will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art.

[0020] Furthermore, the features, structures, or characteristics described in this application can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to provide a full understanding of the embodiments of this application. However, those skilled in the art will recognize that when implementing the technical solutions of this application, not all the detailed features in the embodiments may be used, one or more specific details may be omitted, or other methods, elements, devices, steps, etc., may be employed.

[0021] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0022] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0023] It should be noted that "multiple" in this article refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0024] The technical solution of the present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0025] like Figure 1 As shown, this application provides a speech data processing method based on a large AI model, which can be applied to speech recognition, speech understanding, speech emotion analysis, or speech event detection scenarios to improve the stability of speech data processing results in continuous speech or complex environments. The processing method mainly includes the following steps S100 to S500: Step S100: Obtain the voice data to be processed, and divide the voice data into multiple voice segments based on preset voice segmentation rules.

[0026] The speech data consists of continuous speech signals acquired by an audio acquisition device. The speech data is stored using pulse code modulation (PCM) format, with a sampling rate of 16000Hz and a quantization precision of 16 bits. The speech data is stored in a single-channel format. The preset speech segmentation rules are defined based on speech energy distribution characteristics. These rules include four parameters: time window length, time window sliding step, silence energy threshold, and silence duration threshold. The time window length is defined as the number of sampling points covered in each short-time energy calculation; the time window sliding step is defined as the number of displacement sampling points between two adjacent time windows; the silence energy threshold is defined as the energy boundary value for determining whether a time window belongs to a silent segment; and the silence duration threshold is defined as the minimum continuous silence duration for determining the segment boundary.

[0027] For example, the time window length is set to 400 sampling points, the time window sliding step is set to 160 sampling points, the silence energy threshold is set to 0.3 times the average short-time energy of all time windows, and the silence duration threshold is set to greater than or equal to 5 consecutive time windows.

[0028] Furthermore, in step S100, the following scheme may also be preferred: Speech data is acquired through an audio acquisition interface and preprocessed, including noise reduction filtering and amplitude normalization, to obtain the speech data to be processed.

[0029] The denoising filtering employs the Wiener filtering algorithm, which is based on the minimum mean square error criterion. It performs frequency domain filtering on the speech data by estimating the power spectral density difference between the speech data and the noise. Amplitude normalization uses a linear normalization method, mapping the amplitude of the speech data to the [-1, 1] interval. The normalization formula is as follows: in These are the original sampled values. It represents the amplitude of the sampling point with the largest absolute value in the speech data.

[0030] For example, in speech data with a sampling rate of 16000Hz, Wiener filtering uses a 512-point Fast Fourier Transform window for frequency domain transformation, the noise power spectrum is estimated through the first 200ms silence segment, and after amplitude normalization, the maximum amplitude of the speech data is 1 and the minimum amplitude is -1.

[0031] Short-time energy calculation is performed on the speech data to be processed by sliding a preset time window over the speech data and calculating the short-time energy value within each time window.

[0032] The formula for calculating short-time energy is: in, For the first The short-time energy value within a time window. The time window length, This represents the sampled values ​​of the speech data to be processed within the current time window. The time window slides through the data in increments until it covers the entire speech data to be processed.

[0033] When the time window length is 400 sampling points and the time window sliding step is 160 sampling points, approximately 990 short-time energy values ​​can be obtained for speech data with a duration of 10 seconds.

[0034] The time window with a short-term energy value lower than the preset silence energy threshold is marked as a silence segment, and the time window with a short-term energy value higher than the preset silence energy threshold is marked as a valid speech segment.

[0035] The preset silence energy threshold is defined by multiplying the average short-time energy of all time windows by a preset proportional coefficient, which ranges from 0.2 to 0.5. The short-time energy values ​​of each time window are compared and judged. When E(n) < the silence energy threshold, the time window number is marked as a silence segment number; when E(n) ≥ the silence energy threshold, the time window number is marked as a valid speech segment number.

[0036] For example, if the average short-term energy value for all time windows is 0.015, then the silence energy threshold is set to 0.015 × 0.3 = 0.0045, and time windows with short-term energy values ​​less than 0.0045 are marked as silence segments.

[0037] The speech data is divided into multiple speech segments by using silence segments whose duration exceeds a preset silence duration threshold as segment boundaries and the effective speech segments between adjacent segment boundaries as a single speech fragment; wherein the duration is less than the time window.

[0038] The preset silence duration threshold is counted based on the number of time windows. If the number of consecutive silence segments in time windows is greater than or equal to the preset silence duration threshold, then the consecutive silence segments are determined to be segment boundaries. Consecutive valid speech segment time windows between segment boundaries are concatenated in the order of sampling points to form a speech segment. Each speech segment retains its start and end sampling point indices from the original speech data.

[0039] When six consecutive time windows are marked as silent segments and the preset silence duration threshold is 5, the time intervals corresponding to these six time windows are used as segment boundaries, and the speech data between adjacent segment boundaries is used as a speech segment. The length of the speech segment is the number of effective speech segment time windows multiplied by the time window sliding step.

[0040] Step S200: Extract at least one acoustic feature from each speech segment and construct an acoustic feature set based on the acoustic features.

[0041] Each speech segment undergoes acoustic feature extraction processing, which includes spectral features and fundamental frequency features. The acoustic feature set is a sequence of acoustic feature vectors arranged in order of speech segment number, with each acoustic feature vector corresponding one-to-one with a speech segment.

[0042] When the speech data is divided into 8 speech segments, the acoustic feature set contains 8 acoustic feature vectors.

[0043] Furthermore, in step S200, the following scheme may also be preferred: A short-time Fourier transform is performed on each speech segment to obtain the corresponding time-spectrum graph. Based on the time-spectrum graph, the Mel frequency cepstral coefficients are extracted as the first acoustic feature. The Mel frequency cepstral coefficients include static coefficients, first-order difference coefficients, and second-order difference coefficients.

[0044] The Short-Time Fourier Transform (SFT) employs the Fast Fourier Transform (FFT) algorithm with a window length of 512 sampling points and a frame shift of 256 sampling points. The number of Mel filter banks is set to 40. The spectrum is weighted and summed based on the Mel scale, and the logarithm is followed by a Discrete Cosine Transform (DCT) to obtain 13-dimensional static coefficients. First-order difference coefficients are obtained by calculating the difference between the static coefficients of adjacent frames, and second-order difference coefficients are obtained by calculating the difference between the first-order difference coefficients. Ultimately, the first acoustic feature dimension is 39 dimensions.

[0045] When a certain speech segment contains 120 frames, a 120×39-dimensional Mel frequency cepstral coefficient matrix is ​​extracted, and the average value of each dimension is used to obtain a 39-dimensional feature vector.

[0046] For each speech segment, the fundamental frequency trajectory is calculated. The fundamental frequency period of the speech segment is extracted based on the fundamental frequency trajectory using the autocorrelation function method. The fundamental frequency value is obtained based on the fundamental frequency period. The fundamental frequency value and its time-domain change rate are used as the second acoustic feature.

[0047] The autocorrelation function method is a fundamental frequency detection algorithm that calculates the autocorrelation coefficients of speech segments at different delays and finds the delay corresponding to the maximum peak as the fundamental frequency period. The fundamental frequency value is calculated as F0 = sampling rate / fundamental frequency period. The rate of change in the time domain is calculated by the difference between the fundamental frequency values ​​of adjacent frames.

[0048] For example, with a sampling rate of 16000Hz, when the pitch period is 160 sampling points, the fundamental frequency is 100Hz. If the fundamental frequencies of adjacent frames are 100Hz and 105Hz respectively, then the time domain change rate is 5Hz.

[0049] The first acoustic feature and the second acoustic feature of each speech segment are concatenated to form the acoustic feature vector corresponding to that speech segment. The acoustic feature vectors of all speech segments are then arranged in order of speech segment number to construct an acoustic feature set.

[0050] The first acoustic feature is a 39-dimensional vector, the second acoustic feature is a 2-dimensional vector, and the concatenated acoustic feature vector has a dimension of 41. The acoustic feature set is stored in the form of a two-dimensional matrix, with the number of rows equal to the number of speech segments and the number of columns being 41.

[0051] When the number of speech segments is 8, the dimension of the acoustic feature set is 8×41.

[0052] Step S300: Input all speech segments into the pre-trained speech semantic artificial intelligence big data model to generate semantic feature vectors corresponding to the speech segments.

[0053] Each speech segment is sequentially input into a pre-trained speech-semantic AI model for feature encoding. This pre-trained model is a speech representation learning model built on a deep neural network, trained using a combination of supervised and self-supervised pre-training. Self-supervised pre-training constructs the training task based on a mask prediction mechanism, which employs the Masked Language Model (MLM) algorithm to zero out or replace a specific proportion of frame features in consecutive speech frames with random noise vectors, thus training the model to recover the original feature representation. Supervised pre-training uses labeled speech sentiment datasets or speech recognition datasets, optimizing parameters using cross-entropy loss or Connectionist Temporal Classification (CTC) loss. During model training, stochastic gradient descent or adaptive moment estimation (Adam optimization algorithm) is used for parameter updates. The model parameters obtained through the above pre-training process remain fixed in step S300 and are used for semantic representation extraction of speech segments.

[0054] For example, a training corpus containing 10,000 hours of Mandarin speech data is used as self-supervised pre-training data, with a masking ratio of 15% and a masking span of 3 frames. In the supervised training phase, a speech dataset containing sentiment tags is used, and the model parameters are iteratively updated by minimizing the cross-entropy loss function. The number of iterations is set to 50, the initial learning rate is set to 0.001, and the batch size is set to 32, thus obtaining a pre-trained large-scale speech semantic AI model. In step S300, a single speech segment is input into the model, and the output is a 256-dimensional semantic feature vector.

[0055] Furthermore, in step S300, the following scheme is also preferred: The pre-trained speech and semantic AI large model includes a convolutional coding layer, a multi-head self-attention layer, and a feedforward decoding layer.

[0056] The convolutional coding layer extracts local temporal pattern features from speech segments, the multi-head self-attention layer models global dependencies between different time frames within a speech segment, and the feedforward decoding layer maps the attention representation to a fixed-dimensional semantic feature vector. The convolutional coding layer consists of multiple sets of one-dimensional convolutional units, each set including a convolutional operation layer, a batch normalization layer, and a ReLU activation function layer. The multi-head self-attention layer is built based on the Transformer algorithm structure. The feedforward decoding layer includes two fully connected neural network layers and an intermediate ReLU activation function layer.

[0057] For example, the convolutional coding layer includes three one-dimensional convolutional units, each with a kernel size of 3, a stride of 1, and output channels of 128, 256, and 256 respectively; the multi-head self-attention layer is set to four attention heads, each with a feature dimension of 64; the feedforward decoding layer includes a first fully connected layer (input dimension 256, output dimension 512) and a second fully connected layer (input dimension 512, output dimension 256), thereby outputting a 256-dimensional semantic feature vector.

[0058] The speech segment is divided into frames by setting the frame length and frame shift, and a Hamming window function is applied to each frame of data to obtain a frame sequence.

[0059] Each speech segment obtained in step S100 is processed into frames in chronological order. The preset frame length is defined as the number of sampling points in each frame, set based on the sampling rate; the preset frame shift is defined as the sampling point interval between the start positions of two adjacent frames. The values ​​of frame length and frame shift are determined based on the short-time stationarity principle of the speech signal. A Hamming window function is applied to each frame of data. The Hamming window function is a windowing algorithm, and its mathematical expression is: in, For frame length, This is the index of the current sampling point. The windowed frame sequence is obtained by multiplying the original data of each frame by the Hamming window function point by point.

[0060] For example, with a sampling rate of 16000Hz, the frame length is set to 25ms, corresponding to 400 sampling points, and the frame shift is set to 10ms, corresponding to 160 sampling points. A Hamming window function of length 400 is applied to each frame of 400 sampling points, and a windowed frame sequence is generated by multiplying them point by point.

[0061] Linear frequency cepstral coefficients are extracted from the frame sequence, and the extraction results are used as input features for a large-scale speech and semantic artificial intelligence model.

[0062] The windowed frame sequence is subjected to a Fast Fourier Transform (FFT) algorithm, converting the time-domain signal into a frequency-domain amplitude spectrum. A linear triangular filter bank is constructed in the frequency domain, with the center frequencies of the filter bank uniformly distributed along the linear frequency axis. The amplitude spectrum is weighted and summed with the linear triangular filter bank to obtain the filter energy. The logarithm of the filter energy is then taken. A Discrete Cosine Transform (DCT-II) algorithm is applied to the logarithmic energy sequence to obtain the linear frequency cepstral coefficients. The dimension of the linear frequency cepstral coefficients is set according to the number of filters and the order of the cepstral coefficients to be retained.

[0063] For example, if the number of linear triangular filter banks is set to 40, and the first 20 order cepstral coefficients are retained as linear frequency cepstral coefficients, then a 20-dimensional feature vector is generated for each frame; the 20-dimensional feature vectors of all frames are arranged in chronological order to form a two-dimensional input feature matrix.

[0064] The input features are fed into a large-scale speech and semantic AI model. In this model, a convolutional coding layer is used to extract local temporal patterns from the input features to obtain a coded representation.

[0065] A two-dimensional input feature matrix is ​​fed into a convolutional coding layer. The convolutional coding layer extracts local patterns between consecutive frames by sliding the convolution kernel along the temporal dimension through one-dimensional convolution operations. The convolution operation employs a discrete convolution algorithm, performing element-wise multiplication and addition operations between the convolution kernel weights and the local regions of the input features. The convolution output is then standardized to a mean of 0 and a variance of 1 by a batch normalization layer. The standardized result undergoes a non-linear transformation using the ReLU activation function to obtain the encoded representation.

[0066] For example, for a feature matrix with an input dimension of T×20, the convolution kernel size is set to 3×20, and the number of convolution kernels is 128. A T×128 encoded representation is generated by sliding along the time dimension.

[0067] The attention representation is obtained by modeling the global context dependency of the encoded representation through a multi-head self-attention layer.

[0068] The encoded representation is mapped to a query matrix Q, a key matrix K, and a value matrix V, respectively. The mapping method is to perform matrix multiplication operations through three sets of independent linear transformation matrices. The attention weight matrix is ​​calculated using the scaled dot product attention algorithm, and the formula is as follows: in, The dimension of the key vector; The multi-head mechanism divides the encoded representation into multiple subspaces, performs scaling dot product attention calculations in parallel in each subspace, concatenates the outputs of each attention head, and obtains the attention representation through a linear transformation.

[0069] For example, in a 4-head attention structure, the 256-dimensional encoding representation is divided into four 64-dimensional subspaces. In each subspace, the attention weight matrix is ​​calculated to obtain four 64-dimensional output vectors, which are then concatenated into a 256-dimensional vector as the attention representation.

[0070] The attention representation is nonlinearly transformed by the feedforward decoding layer, and a fixed-dimensional semantic feature vector corresponding to each speech segment is output.

[0071] The attention representation is input to the feedforward decoding layer, which consists of two fully connected layers. The first fully connected layer maps the input features to a high-dimensional space through matrix multiplication and performs a non-linear transformation using the ReLU activation function. The second fully connected layer maps the high-dimensional features to a semantic feature vector of a preset dimension. The output vector is then subjected to average pooling in the time dimension. The average pooling operation is a global average pooling algorithm, which calculates the arithmetic mean of the features in each frame in the time dimension to obtain a semantic feature vector of a fixed dimension.

[0072] For example, for an attention representation matrix (T×256) containing T frames, a 256-dimensional vector is computed as a semantic feature vector through global average pooling, thereby generating a 256-dimensional semantic feature vector for each speech segment.

[0073] Step S400: Map the semantic feature vector based on the preset semantic stability constraint rules to obtain the mapped feature vector.

[0074] The semantic feature vectors corresponding to each speech segment output in step S300 are subjected to stability analysis and constraint processing. The semantic stability constraint rules are constructed based on the continuous characteristics of the speech segment sequence in the time dimension. They suppress abnormal fluctuations in the semantic feature vectors by combining statistical distribution constraints and cross-segment similarity constraints. The construction process of the semantic stability constraint rules includes: based on the training corpus from the pre-training stage, statistically analyzing the distribution range of each dimension component of the semantic feature vector under normal speech data, calculating its mean distribution interval and standard deviation distribution interval, and statistically analyzing the cosine similarity distribution interval of semantic feature vectors between adjacent speech segments; and setting upper thresholds for the mean, standard deviation, and cross-segment cosine similarity based on the statistical results, thereby forming the semantic stability constraint rules.

[0075] For example, in a statistical analysis of 5000 consecutive speech samples, the mean distribution range of each dimension of the 256-dimensional semantic feature vector is [-0.8, 0.8], the standard deviation distribution range is [0.05, 0.6], and the cosine similarity distribution range between adjacent speech segments is [0.65, 0.98]. Based on the above statistical results, the upper limit threshold of the mean is set to 0.8, the upper limit threshold of the standard deviation is set to 0.6, and the lower limit threshold of the cross-segment cosine similarity is set to 0.65.

[0076] Furthermore, in step S400, the following scheme may also be preferred: Calculate the mean and standard deviation of each dimension component in each semantic feature vector.

[0077] The semantic feature vector corresponding to each speech segment is represented as follows: ,in This represents the number of dimensions of the semantic feature vector.

[0078] For each dimension Construct a component set for this dimension across all speech segments of the current speech data. ,in The number of speech segments; the mean of this dimension is calculated using the arithmetic mean formula. The calculation formula is: Calculate the standard deviation of this dimension using the standard deviation calculation formula. The calculation formula is: .

[0079] When the number of speech segments N is 20 and the semantic feature vector dimension D is 256, the values ​​of the 10th dimension component in the 20 speech segments are statistically analyzed to obtain the mean. =0.92, standard deviation =0.75, and the above formula is used to calculate the mean and standard deviation of all 256 dimensions.

[0080] The semantic feature vectors are constrained and verified based on the preset semantic stability constraint rules, which include the upper limit threshold of the mean, the upper limit threshold of the standard deviation, and the lower limit threshold of the cross-segment cosine similarity.

[0081] Mean of each dimension Compare the standard deviations of each dimension to the upper limit threshold of the mean. Compare with the upper limit threshold of standard deviation; simultaneously, analyze the semantic feature vectors corresponding to adjacent speech segments. and The cosine similarity is calculated using the cosine similarity algorithm, and the formula is as follows: Where "·" represents the vector dot product operation. Represent the L2 norm of a vector; calculate the L2 norm. The semantic mutation is determined by comparing the cross-segment cosine similarity lower limit threshold.

[0082] When the 10th dimension mean =0.92 exceeds the upper limit threshold of 0.8 for the mean, and the standard deviation is... When the cosine similarity between the 5th and 6th speech segments is 0.75, which exceeds the upper limit of the standard deviation threshold of 0.6, the semantic feature vector corresponding to the 6th speech segment is marked as a semantic mutation vector.

[0083] The semantic feature vector is truncated and normalized for the dimensional components whose mean and standard deviation exceed the upper thresholds for mean and standard deviation, respectively, to obtain the normalized semantic feature vector.

[0084] For components marked as anomalous dimensions The processing method combines interval truncation and linear normalization; the interval truncation rule is: when... When the upper limit of the mean is reached, all components of this dimension are uniformly truncated to the upper limit of the mean; when When the upper limit of the standard deviation is reached, a scaling transformation is performed on the component of that dimension, with the scaling factor... = Upper limit of standard deviation / Update all components of this dimension to After truncation and scaling, the component of this dimension is subjected to min-max normalization. The min-max normalization algorithm is the same as the Min-Max normalization algorithm, and the calculation formula is as follows: ,in and These are the minimum and maximum values ​​of this dimension in the current speech data.

[0085] When the 10th dimension When the standard deviation is 0.75 and the upper limit threshold is 0.6, calculate the scaling factor. Scaling is performed on all components of this dimension, and then Min-Max normalization is performed to map the components of this dimension to the [0,1] interval.

[0086] For semantic feature vectors whose cosine similarity with the semantic feature vectors corresponding to adjacent speech segments is lower than the lower limit threshold of cross-segment cosine similarity, a weighted smoothing process is performed to obtain the weighted smoothed semantic feature vector.

[0087] For those labeled as semantic mutation vectors A sliding window weighted average algorithm is used for smoothing, with the sliding window length set to 3, including... , and The weighted average algorithm is a weighted moving average algorithm, where the weight coefficients α, β, and γ are set to satisfy α + β + γ = 1, and β > α and β > γ; the smoothing vector is calculated. The weighting coefficient is determined based on the difference in cosine similarity between adjacent components. When the cosine similarity is lower, the value of γ is smaller, thereby enhancing the smoothing effect.

[0088] when When α=0.3, β=0.5, γ=0.2, calculate... We obtain the weighted smoothed semantic feature vector.

[0089] The normalized semantic feature vector and the weighted smoothed semantic feature vector are input into a preset linear mapping matrix for linear transformation. The linear transformation result is then normalized through a batch normalization layer to obtain the mapped feature vector.

[0090] The semantic feature vector that has undergone truncation, normalization, and weighted smoothing is represented as follows: A pre-defined linear mapping matrix W is constructed, with dimensions D×M, where D is the dimension of the input semantic feature vector and M is the dimension of the mapped features. The linear transformation employs a matrix multiplication algorithm, calculated using the following formula: , where b is the bias vector; The data is input to the batch normalization layer, which uses the BatchNormalization algorithm to standardize the data by calculating the mean and variance within each batch. The calculation formula is as follows: ,in To prevent constants with zero denominators; then a trainable scaling factor is used. and offset factor A linear adjustment is performed to obtain the final mapped feature vector.

[0091] When D=256 and M=128, a 256×128 linear mapping matrix W is constructed to map the 256-dimensional semantic feature vector to a 128-dimensional feature vector. After batch normalization, the 128-dimensional mapped feature vector is output.

[0092] Step S500: Perform feature fusion between the mapped feature vector and the acoustic feature set to obtain the fused features. Then, perform speech data processing on the fused features to obtain the speech data processing result.

[0093] The acoustic feature set constructed in step S200 and the mapped feature vector obtained in step S400 are matched one-to-one according to the speech segment number, and cross-feature space fusion processing is performed on each speech segment. The acoustic feature set includes the acoustic feature vector corresponding to each speech segment, and the mapped feature vector is the semantic representation after semantic stability constraints and linear mapping. The two types of features are jointly modeled in the same feature space through a feature-level fusion algorithm to construct the fused feature representation. The fused features are used to perform preset speech data processing tasks, including speech emotion recognition, speech speaker recognition, or speech content classification. The preset speech data processing task type is defined based on task configuration parameters, which include a task category identifier field, the number of output categories, and the output dimension format.

[0094] For example, in a speech emotion recognition task, the task category identifier field is set to "emotion_classification", the number of output categories is set to 6 emotion labels, the fused features are input into the emotion classification model, and a 6-dimensional probability distribution vector is output. The speech data processing result is determined by the maximum probability principle.

[0095] Furthermore, in step S500, the following scheme may also be preferred: The mapping feature vector corresponding to each speech segment is concatenated with the corresponding acoustic feature vector in the acoustic feature set along the feature dimension to obtain the concatenated feature vector.

[0096] For the i-th speech segment, let its mapping feature vector be... The acoustic feature vector is ,in For mapping feature dimensions, For acoustic feature dimensions, a vector concatenation operation is performed along these dimensions. The concatenation algorithm is a vector concatenation algorithm, which connects two vectors end-to-end to form a new vector. The dimension is obtained as The concatenated feature vectors of all speech segments are arranged in the order of speech segment numbers to form a concatenated feature sequence matrix.

[0097] When the dimension M of the mapping feature vector is 128 and the dimension K of the acoustic feature vector is 64, a 192-dimensional concatenated feature vector is obtained through the vector concatenation algorithm; if the number of speech segments is 20, a 20×192 concatenated feature sequence matrix is ​​constructed.

[0098] The concatenated feature vector is input into a pre-defined cross-modal attention fusion module to calculate the attention weight matrix.

[0099] The cross-modal attention fusion module is built upon the scaled dot product attention algorithm and includes a linear mapping unit, an attention computation unit, and a weight normalization unit. The linear mapping unit maps the concatenated feature vectors to a query matrix Q and a key matrix. Sum matrix The attention calculation unit calculates the attention score matrix according to the scaled dot product attention formula; the weight normalization unit normalizes the attention score using the Softmax function. The structural parameters of the cross-modal attention fusion module include the number of attention heads H and the subspace dimension of each attention head. and output dimensions The above parameters are set based on the splicing feature dimension (M+K), where .

[0100] When the concatenation feature dimension is 192 and the number of attention heads H is set to 3, the subspace dimension of each attention head is... The value is 64; it is generated through three sets of linear mapping matrices respectively. The attention weight matrix is ​​computed in parallel across three subspaces.

[0101] The concatenated feature vectors are weighted and aggregated based on the attention weight matrix to obtain the fused features.

[0102] Let the attention weight matrix be... The calculation formula is as follows: in, The scaling factor is used; the fused representation matrix is ​​calculated using a matrix multiplication algorithm. After splicing the multiple outputs, a linear transformation matrix is ​​applied. The mapping is performed, and the calculation formula is as follows: The fused feature representation is obtained. The fused feature dimension is determined by the linear transformation matrix. The number of columns determines the sequence.

[0103] When each attention head outputs a dimension of 64, the concatenated three attention heads have a dimension of 192, which is achieved through a 192×128 linear transformation matrix. The mapping is a 128-dimensional fused feature vector.

[0104] The fused features are input into the task processing layer, which performs classification or regression operations on the fused features according to the preset speech data processing task type, and outputs the speech data processing results.

[0105] The task processing layer consists of a fully connected layer and an output layer. When the task type is classification, the output layer uses the Softmax classification algorithm and is trained using the cross-entropy loss function. When the task type is regression, the output layer uses the linear regression algorithm and is trained using the mean squared error loss function. The parameters of the task processing layer are trained under supervised supervision using labeled training data, and the optimization algorithm is the adaptive moment estimation algorithm (Adam optimization algorithm).

[0106] For example, in a speaker recognition task, the number of output categories is set to 100 speaker labels. The fused 128-dimensional features are input to a fully connected layer (128×100) and output as a 100-dimensional vector. After normalization by the Softmax function, the probability value of each category is obtained, and the category with the highest probability is taken as the result of speech data processing.

[0107] Furthermore, in the cross-modal attention fusion module, the attention weight matrix is ​​calculated based on the query vector, key vector, and value vector by using the mapped feature vector as the query vector and the acoustic feature vector as the key vector and value vector.

[0108] For the i-th speech segment, the mapping feature vector will be... Through linear transformation matrix Mapped to query vector acoustic feature vectors Through linear transformation matrix and Mapped to key vectors respectively Sum value vector Calculate attention score ,in The dimension is the key vector; the Softmax function is used to... Normalization is performed to obtain attention weights According to the weighted summation formula The fused feature vector is obtained. The above linear transformation matrix... Determined through supervised training methods.

[0109] When the dimension of the mapped feature vector is 128, the dimension of the acoustic feature vector is 64, and the dimensions of the query vector, key vector, and value vector are all set to 64, the 128×64 matrix... Matrix, 64×64 Matrix and 64×64 The matrix performs a linear mapping and calculates the attention weights. The 64-dimensional fusion feature vector is obtained as the fused feature of the speech segment.

[0110] In this embodiment, the speech data is segmented based on short-time energy and silence duration thresholds. Mel-frequency cepstral coefficients are extracted using short-time Fourier transform, and fundamental frequency and pitch period are extracted using autocorrelation function to construct an acoustic feature set. Simultaneously, speech segments are input into a pre-trained speech-semantic AI model containing convolutional coding layers, multi-head self-attention layers, and feedforward decoding layers to obtain fixed-dimensional semantic feature vectors. Semantic stability constraints are constructed based on statistically obtained upper limits for mean, standard deviation, and cross-segment cosine similarity. After truncation, normalization, and weighted smoothing, the semantic feature vectors are processed by a linear mapping matrix and batch normalization to obtain mapped feature vectors. A cross-modal attention fusion module, constructed using vector concatenation and scaled dot product attention algorithms, achieves deep fusion of the mapped feature vectors and acoustic feature vectors. The speech data processing results are output through a classification or regression task processing layer. This process suppresses semantic mutations and abnormal fluctuations while enhancing the collaborative expression of semantic and acoustic details, improving the accuracy and stability of the speech data processing results, and addressing the issue of insufficient stability in complex or continuous speech scenarios.

[0111] like Figure 2 As shown, this application also provides a speech data processing system 10 based on an AI large model, including: The data acquisition module 11 is used to acquire the voice data to be processed and divide the voice data into multiple voice segments based on preset voice segmentation rules.

[0112] The feature extraction module 12 is used to extract at least one acoustic feature from each speech segment and construct an acoustic feature set based on the acoustic features.

[0113] The vector generation module 13 is used to input all speech segments into a pre-trained speech semantic artificial intelligence big data model to generate semantic feature vectors corresponding to the speech segments.

[0114] The vector mapping module 14 is used to map semantic feature vectors based on preset semantic stability constraint rules to obtain mapped feature vectors.

[0115] Data processing module 15 is used to fuse the mapped feature vector with the acoustic feature set to obtain the fused features, and to perform speech data processing on the fused features to obtain the speech data processing result.

[0116] In this embodiment, the speech data processing flow is divided into a data acquisition module 11, a feature extraction module 12, a vector generation module 13, a vector mapping module 14, and a data processing module 15. This creates a clear functional division of labor for speech segmentation, acoustic feature construction, semantic feature generation, semantic stability constraint mapping, and cross-modal feature fusion processing. Specifically, the data acquisition module 11 accurately segments speech segments based on short-time energy and silence duration thresholds; the feature extraction module 12 extracts Mel-frequency cepstral coefficients and fundamental frequency features using short-time Fourier transform and autocorrelation function methods to construct an acoustic feature set; and the vector generation module 15... 3. Based on a pre-trained speech and semantic AI big model containing convolutional coding layers, multi-head self-attention layers, and feedforward decoding layers, fixed-dimensional semantic feature vectors are generated. The vector mapping module 14 performs truncation normalization and weighted smoothing processing based on the upper limit threshold of mean, the upper limit threshold of standard deviation, and the lower limit threshold of cross-segment cosine similarity, and completes linear mapping. The data processing module 15 realizes cross-modal fusion of the mapped feature vector and the acoustic feature vector through the scaling dot product attention algorithm and outputs the speech data processing results. Thus, under the modular structure, the collaborative modeling of semantic information and acoustic information is realized, improving the accuracy, stability and scalability of speech data processing.

[0117] It should be noted that although several modules or units of the system for executing actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0118] like Figure 3 As shown, this application also provides an electronic device 20, including a memory 21 and a processor 22. The memory 21 stores a computer program that can run on the processor 22. When the processor 22 executes the computer program, it implements the above-described AI-based large model-based voice data processing method.

[0119] In this embodiment, by setting up a memory 21 and a processor 22 in the electronic device 20, and storing a computer program in the memory 21 for performing speech segmentation, acoustic feature extraction, semantic feature generation, semantic stability constraint mapping, and cross-modal attention fusion processing, the processor 22 sequentially completes short-time energy calculation, Mel-frequency cepstral coefficient extraction, autocorrelation function fundamental frequency calculation, convolutional coding and multi-head self-attention modeling, mean and standard deviation statistics, cosine similarity calculation, and scaled dot product attention fusion operation within a single electronic device 20. This enables end-to-end joint modeling of speech features, reduces manual feature selection intervention, improves processing efficiency, and enhances the stability and consistency of speech data processing results.

[0120] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when run by processor 22, causes processor 22 to perform the speech data processing method based on the AI ​​big model described above.

[0121] In this embodiment, by storing a computer program in a computer-readable storage medium for implementing speech segmentation rule execution, acoustic feature set construction, pre-trained speech semantic AI large model inference, semantic stability constraint rule verification, linear mapping and batch normalization processing, and cross-modal attention fusion operation, the processor 22 can reproduce the complete AI large model-based speech data processing method when reading and running the computer program. This solidifies the semantic stability constraint mechanism and cross-modal fusion mechanism in the form of program code, ensuring the consistency and portability of the algorithm logic, and facilitating deployment and execution on different hardware platforms, thereby improving the application flexibility and engineering implementation reliability of the speech data processing solution.

[0122] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored on a non-volatile storage medium (such as a CD). The method is contained in or on a ROM, USB flash drive, external hard drive, etc., or on a network, and includes several instructions to cause an electronic device 20 (which may be a personal computer, server, touch terminal, or network device, etc.) to execute the method according to the embodiments of this application.

[0123] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0124] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A speech data processing method based on a large AI model, characterized in that, include: Acquire the voice data to be processed, and divide the voice data into multiple voice segments based on preset voice segmentation rules; Extract at least one acoustic feature from each of the aforementioned speech segments, and construct an acoustic feature set based on the acoustic features; All the aforementioned speech segments are input into a pre-trained speech semantic artificial intelligence large model to generate semantic feature vectors corresponding to the speech segments; The semantic feature vector is mapped based on a preset semantic stability constraint rule to obtain a mapped feature vector; The mapped feature vector is fused with the acoustic feature set to obtain fused features. The fused features are then subjected to speech data processing to obtain speech data processing results.

2. The speech data processing method based on a large AI model according to claim 1, characterized in that, The step of acquiring the speech data to be processed and dividing the speech data into multiple speech segments based on preset speech segmentation rules includes: Voice data is acquired through an audio acquisition interface, and the voice data is preprocessed to obtain voice data to be processed. Short-time energy calculation is performed on the speech data to be processed, and a preset time window is slid across the speech data to be processed, and the short-time energy value within each time window is calculated. The time window in which the short-term energy value is lower than the preset silence energy threshold is marked as a silence segment, and the time window in which the short-term energy value is higher than the preset silence energy threshold is marked as a valid speech segment; The silence segment whose duration exceeds a preset silence duration threshold is used as the segment boundary, and the effective speech segment between adjacent segment boundaries is used as a speech fragment; wherein, the duration is less than the time window.

3. The speech data processing method based on a large AI model according to claim 1, characterized in that, The step of extracting at least one acoustic feature from each of the aforementioned speech segments and constructing an acoustic feature set based on the acoustic features includes: A short-time Fourier transform is performed on each of the aforementioned speech segments to obtain the corresponding time-spectrum diagram, and the Mel frequency cepstral coefficients are extracted as the first acoustic feature based on the time-spectrum diagram. For each of the speech segments, a fundamental frequency trajectory is calculated, the fundamental frequency period of the speech segment is extracted based on the fundamental frequency trajectory, and the fundamental frequency value is obtained according to the fundamental frequency period. The fundamental frequency value and its time-domain rate of change are used as the second acoustic feature. The first acoustic feature and the second acoustic feature of each speech segment are concatenated to form the acoustic feature vector corresponding to the speech segment. The acoustic feature vectors of all the speech segments are arranged in order of speech segment number to construct an acoustic feature set.

4. The speech data processing method based on a large AI model according to claim 1, characterized in that, The pre-trained speech and semantic AI big model includes a convolutional coding layer, a multi-head self-attention layer, and a feedforward decoding layer; The step of inputting all the speech segments into a pre-trained speech semantic artificial intelligence model to generate semantic feature vectors corresponding to the speech segments includes: The speech segment is divided into frames by a preset frame length and frame shift, and a Hamming window function is applied to each frame of data to obtain a frame sequence; Linear frequency cepstral coefficients are extracted from the frame sequence, and the extraction results are used as input features of the speech semantic artificial intelligence big model; The input features are input into the speech and semantic AI model. In the speech and semantic AI model, the convolutional coding layer is used to extract local temporal patterns from the input features to obtain an encoded representation. The multi-head self-attention layer is used to perform global context dependency modeling on the encoded representation to obtain the attention representation; The attention representation is nonlinearly transformed by the feedforward decoding layer, and a fixed-dimensional semantic feature vector corresponding to each speech segment is output.

5. The speech data processing method based on a large AI model according to claim 1, characterized in that, The step of mapping the semantic feature vector based on a preset semantic stability constraint rule to obtain the mapped feature vector includes: Calculate the mean and standard deviation of each dimension component in each of the semantic feature vectors; The semantic feature vector is constrained and verified based on preset semantic stability constraint rules, which include upper limit threshold for mean, upper limit threshold for standard deviation, and lower limit threshold for cross-segment cosine similarity. The semantic feature vector is truncated and normalized for the dimension components whose mean and standard deviation exceed the upper threshold of the mean and the upper threshold of the standard deviation, to obtain the normalized semantic feature vector. For semantic feature vectors whose cosine similarity with the semantic feature vectors corresponding to adjacent speech segments is lower than the lower limit threshold of cross-segment cosine similarity, a weighted smoothing process is performed to obtain a weighted smoothed semantic feature vector. The normalized semantic feature vector and the weighted smoothed semantic feature vector are input into a preset linear mapping matrix for linear transformation to obtain the mapped feature vector.

6. The speech data processing method based on a large AI model according to claim 1, characterized in that, The step of fusing the mapped feature vector with the acoustic feature set to obtain fused features, and then processing the fused features to obtain speech data processing results includes: The mapping feature vector corresponding to each speech segment is concatenated with the corresponding acoustic feature vector in the acoustic feature set along the feature dimension to obtain the concatenated feature vector. The concatenated feature vector is input into a preset cross-modal attention fusion module to calculate the attention weight matrix; The concatenated feature vectors are weighted and aggregated according to the attention weight matrix to obtain the fused features. The fused features are input into the task processing layer, and the voice data processing results are output.

7. The speech data processing method based on a large AI model according to claim 6, characterized in that, In the cross-modal attention fusion module, the mapped feature vector is used as the query vector, the acoustic feature vector is used as the key vector and the value vector, and the attention weight matrix is ​​calculated based on the query vector, the key vector and the value vector.

8. A speech data processing system based on a large AI model, characterized in that, include: The data acquisition module is used to acquire the voice data to be processed and divide the voice data into multiple voice segments based on preset voice segmentation rules; The feature extraction module is used to extract at least one acoustic feature from each of the speech segments and construct an acoustic feature set based on the acoustic features; The vector generation module is used to input all the speech segments into a pre-trained speech semantic artificial intelligence big data model to generate semantic feature vectors corresponding to the speech segments; The vector mapping module is used to map the semantic feature vector based on a preset semantic stability constraint rule to obtain a mapped feature vector; The data processing module is used to fuse the mapped feature vector with the acoustic feature set to obtain fused features, and to perform speech data processing on the fused features to obtain speech data processing results.

9. An electronic device, characterized in that, The system includes a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the processor executes the computer program to implement the speech data processing method based on an AI large model as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores a computer program that, when run by a processor, causes the processor to perform the speech data processing method based on an AI large model as described in any one of claims 1 to 7.

Images