LED display screen interaction method and system supporting voice interaction

Abstract

The invention provides an LED display screen interaction method and system supporting voice interaction, and the method comprises the steps: carrying out the voice feature extraction processing through obtaining a continuous voice data stream sent by a user, and obtaining an acoustic feature sequence and a semantic association feature set of a voice segment; and calling a pre-trained voice semantic understanding model to perform joint semantic analysis processing on the acoustic feature sequence and the semantic association feature set, generating a semantic understanding result of the voice segment, and determining a user interaction intention type corresponding to the continuous voice data stream and key information positioning features of the user interaction intention in the voice content based on the semantic understanding result. And generating an LED display control instruction containing an interaction content identifier based on the user interaction intention type and the key information positioning feature, and sending the LED display control instruction to the target LED display screen to execute an interaction display operation. The real intention and key information in the continuous voice of the user can be accurately understood, the accuracy and naturalness of voice interaction between the LED display screen and the user are remarkably improved, and the interaction experience of the user is effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of speech recognition, in particular to an LED display screen interaction method and system supporting voice interaction. BACKGROUND

[0002] With the development of intelligent interaction technology, LED display screen interaction methods supporting voice interaction are widely used in information display, public service and other scenarios. Such methods control the display content of the LED display screen by recognizing user voice instructions to achieve more convenient human-computer interaction. At present, the common implementation method is usually to obtain a single sentence voice input of the user, extract the key words or simple semantic features therein, and then generate corresponding display instructions based on preset rules. However, this processing method only for discrete voice instructions cannot effectively cope with the interaction requirements of continuous voice data stream in user natural conversation. Continuous voice often contains multiple voice segments with time continuity, and there is semantic continuity and intention change between segments. The existing method cannot capture the semantic association and intention evolution between segments, which easily leads to the mismatch between the display content of the LED display screen and the core information of the user's real intention, reduces the accuracy and naturalness of the interaction, and affects the user experience. SUMMARY

[0003] The present application provides an LED display screen interaction method and system supporting voice interaction.

[0004] In a first aspect, the present application embodiment provides an LED display screen interaction method supporting voice interaction, which comprises:

[0005] obtaining a continuous voice data stream issued by a user, the continuous voice data stream containing multiple voice segments with timestamp markers;

[0006] performing voice feature extraction processing on the continuous voice data stream to obtain an acoustic feature sequence and a semantic association feature set of the voice segments;

[0007] calling a pre-trained voice semantic understanding model to perform joint semantic analysis processing on the acoustic feature sequence and the semantic association feature set to generate a semantic understanding result of the voice segments;

[0008] determining a user interaction intention type corresponding to the continuous voice data stream and a key information positioning feature of the user interaction intention in voice content according to the semantic understanding result;

[0009] generating an LED display control instruction containing an interaction content identifier based on the user interaction intention type and the key information positioning feature, and sending the LED display control instruction to a target LED display screen to perform an interaction display operation.

[0010] In a second aspect, the embodiment of the present application provides a computer system, comprising:

[0011] a memory, wherein a computer program is stored in the memory;

[0012] a processor configured to load the computer program to implement the LED display screen interaction method supporting voice interaction as described above.

[0013] The LED display screen interaction method supporting voice interaction provided by the present application can effectively process the time continuity characteristics of continuous voice by obtaining continuous voice data streams issued by a user and segmenting the continuous voice data streams into voice segments with timestamp markers, and provides a structured basis for subsequent semantic analysis; the acoustic feature sequence and the semantic association feature set are obtained by performing voice feature extraction processing on the continuous voice data streams, so that the subsequent semantic analysis can simultaneously use the acoustic expression features of the voice and the semantic association features between the segments, thereby breaking through the limitations of traditional solutions; the pre-trained voice semantic understanding model is called to perform joint semantic analysis on the dual-modal features to generate a semantic understanding result containing an intention evolution track and semantic core content, which can capture the intention change trend between continuous voice segments and the core semantics in colloquial expressions; the user interaction intention type and the key information positioning feature are determined according to the semantic understanding result, which not only clarifies the overall intention of the user, but also locates the core information area in the voice content that is strongly related to the intention, thereby solving the problem that the traditional solution can only identify the intention but cannot match the display focus; the LED display control instruction containing the interaction content identifier is generated based on the intention type and the key information, so that the display screen can accurately adjust the display content according to the core intention and the key information expressed by the user in continuous voice, rather than generating the instruction based on the intention type only. Through the synergistic effect of the above steps, the present method can more accurately understand the real intention and key information in the continuous voice of the user, significantly improves the accuracy and naturalness of the voice interaction between the LED display screen and the user, and effectively improves the user's interaction experience. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 is a flowchart of an LED display screen interaction method supporting voice interaction provided by the embodiment of the present application.

[0015] Figure 2 is a composition schematic diagram of a computer system provided by the embodiment of the present application. DETAILED DESCRIPTION

[0016] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0017] Referring to Figure 1 , Figure 1 A flowchart of an LED display screen interaction method supporting voice interaction is provided for an embodiment of the present application. The method can be executed by a computer system, which can be a computer system embedded in an LED display screen. The method provided by the embodiment of the present application includes the following steps:

[0018] Step S100: Obtain a continuous voice data stream issued by a user. The continuous voice data stream contains a plurality of voice segments with timestamp markers.

[0019] The continuous voice data stream is a data stream formed by continuous voice signals issued by a user during voice expression. It is a continuous and uninterrupted sequence of voice information. The timestamp marker is an identifier added to the voice segment to indicate the time position of the voice segment in the continuous voice data stream. Through the timestamp, the time point of each voice segment in the entire voice expression process can be accurately located. The continuous voice data stream issued by the user can be obtained using a microphone or other audio acquisition device. For example, in a conference room scenario, a high-performance microphone installed on the conference table can collect continuous voice issued by the participants in real time. During the collection process, the audio acquisition device converts the received voice signals into digital signals and records them in chronological order, while adding a timestamp marker to each voice segment. These voice segments with timestamp markers constitute the continuous voice data stream, facilitating subsequent processing and analysis.

[0020] Step S200: Perform voice feature extraction processing on the continuous voice data stream to obtain an acoustic feature sequence of the voice segment and a semantic association feature set.

[0021] Voice feature extraction processing is a process of extracting information that can reflect the essential features of voice from the continuous voice data stream. The acoustic feature sequence is a feature set used to describe the acoustic characteristics of the voice segment, which contains information such as the distribution pattern of voice stress and the pitch contour, which can help understand the emotion and emphasis of the voice. The semantic association feature set is a feature set used to represent the semantic association between voice segments, which contains information such as cross-segment semantic continuity and intra-segment semantic concentration, which can reflect the coherence and closeness of the semantic content of the voice. By performing voice feature extraction processing on the continuous voice data stream, acoustic and semantic information can be comprehensively utilized to more accurately understand the user's voice intent.

[0022] As an implementation, step S200, performing voice feature extraction processing on the continuous voice data stream to obtain an acoustic feature sequence of the voice segment and a semantic association feature set, can be implemented as the following steps S210-S240:

[0023] Step S210: boundary detection processing is performed on the continuous speech data stream, natural segmentation boundaries of the continuous speech data stream are determined based on the abrupt points of speech energy and semantic pause markers, and the continuous speech data stream is segmented into speech segments with semantic integrity.

[0024] The boundary detection processing is a process of finding suitable segmentation points in the continuous speech data stream to divide it into different speech segments. The abrupt point of speech energy is a position where the energy of the speech signal changes significantly in a short time, which is usually related to the pause, stress, etc. of the speech. The semantic pause marker is a pause position determined according to the semantic logic of the speech content, such as the pause between sentences, the pause between paragraphs, etc. The natural segmentation boundary is a segmentation position determined based on the abrupt point of speech energy and the semantic pause marker, which conforms to the natural expression of speech and the understanding of semantics. The speech segment with semantic integrity is a speech unit that can express a relatively complete meaning in semantics, such as a complete sentence, an independent topic, etc. When performing boundary detection processing on the continuous speech data stream, the energy of the speech signal can be calculated first to find the abrupt point of energy. For example, the sliding window method is used to calculate the energy of the speech signal in each window, and when the energy change exceeds a certain threshold, the position is marked as an abrupt point of energy. At the same time, combined with natural language processing technology, the semantic pause markers are determined through analysis of the speech text. The natural segmentation boundary is determined by combining the abrupt point of energy and the semantic pause marker, so as to segment the continuous speech data stream into speech segments with semantic integrity.

[0025] Step S220: intent-oriented acoustic feature focusing processing is performed on each speech segment, and a subset of acoustic features in the speech segment that is strongly associated with the user intent is extracted as an acoustic feature sequence, the subset of acoustic features including a stress distribution pattern and a tone rise-fall trajectory.

[0026] The intent-oriented acoustic feature focusing processing is a process of screening acoustic features closely related to the intent from acoustic features of the voice segment. The acoustic feature subset strongly associated with the user intent is a set of acoustic features that can directly reflect the user intent. The stress distribution pattern is the position and distribution rule of the stress in the voice, which can reflect the user's emphasis on certain words or content. The intonation rising and falling trajectory is the change of the voice intonation over time, which can convey the user's emotions, questions, statements and different tones. When performing the intent-oriented acoustic feature focusing processing on each voice segment, the user's interactive intent needs to be determined first, and then the relevant acoustic features are screened according to the intent. For example, in a voice for querying information, the user may emphasize the key information with stress, and the key information can be determined by analyzing the stress distribution pattern. At the same time, the rising and falling of the intonation can also reflect the user's tone, such as the rising intonation usually has a questioning tone. By extracting the acoustic features such as stress distribution pattern and intonation rising and falling trajectory that are strongly associated with the user intent, the acoustic feature sequence is formed.

[0027] As an implementation, in step S220, the intent-oriented acoustic feature focusing processing is performed on each voice segment to extract the acoustic feature subset strongly associated with the user intent as the acoustic feature sequence, which can be implemented as steps S221-S224 as follows:

[0028] In step S221, multi-resolution time-frequency analysis processing is performed on the voice segment to generate a set of time-frequency spectrograms containing different time resolutions.

[0029] The multi-resolution time-frequency analysis processing is a process of analyzing the voice signal in both time and frequency dimensions and using different resolutions to obtain more comprehensive time-frequency information. The time-frequency spectrogram is a graph used to show the distribution of the voice signal in time and frequency, which can intuitively reflect the change of the frequency component of the voice signal over time. The set of time-frequency spectrograms with different time resolutions is a set of time-frequency spectrograms obtained by performing time-frequency analysis on the voice segment using different time resolutions. The multi-resolution time-frequency analysis processing on the voice segment can use methods such as wavelet transform. For example, for a voice segment, wavelet transform can be performed using coarse resolution and fine resolution respectively to obtain two time-frequency spectrograms with different time resolutions. The time-frequency spectrogram with coarse resolution can show the overall time-frequency characteristics of the voice signal, while the time-frequency spectrogram with fine resolution can display the local time-frequency characteristics of the voice signal in more detail. By generating a set of time-frequency spectrograms containing different time resolutions, the time-frequency characteristics of the voice signal can be observed from multiple angles.

[0030] In step S222, the energy concentration points of the stress region are extracted from the set of time-frequency spectrograms, the position distribution rule of the energy concentration points in the voice segment is counted, and the stress distribution pattern is generated.

[0031] The energy concentration points of the accent region are positions corresponding to regions with relatively concentrated energy in the time-frequency spectrogram, which are usually related to the accent part in the speech. The position distribution rule of the energy concentration points in the speech segment is to statistically analyze the positions of the energy concentration points in the speech segment, and find out the characteristics and rules of the distribution. The accent distribution pattern is a pattern of the distribution of the accent in the speech segment according to the position distribution rule of the energy concentration points. When extracting the energy concentration points of the accent region from the set of time-frequency spectrograms, the time-frequency spectrogram can be first thresholded to filter out regions with energy values below a certain threshold and retain regions with higher energy. Then, the points with concentrated energy in these high-energy regions are found. For example, a local maximum value detection algorithm can be used to find the energy maximum value points of each local region in the time-frequency spectrogram, and these points are taken as the energy concentration points. When statistically analyzing the position distribution rule of the energy concentration points in the speech segment, statistical characteristics such as the occurrence frequency and distribution density of the energy concentration points at different time positions can be calculated. The accent distribution pattern can be generated according to these statistical characteristics, for example, the distribution of the accent in the speech segment can be represented in the form of a graph or data.

[0032] Step S223: Perform trend analysis processing on the pitch curve of the speech segment, identify the rising inflection points and falling inflection points in the pitch curve, calculate the slope change rate between adjacent inflection points, and generate the intonation rising and falling trajectory.

[0033] The pitch curve is a curve of the change of the pitch of a speech signal with time, and the pitch is the basic frequency of the speech signal, which is related to the pitch of the speech. The trend analysis processing is a process of analyzing the change trend of the pitch curve. The rising inflection point and the falling inflection point are points in the pitch curve where the slope changes from positive to negative or from negative to positive, which are related to the rising and falling changes of the intonation of the speech. The slope change rate between adjacent inflection points is the change of the slope of the pitch curve between two adjacent inflection points, which can reflect the speed and amplitude of the intonation rising and falling. The intonation rising and falling trajectory is a trajectory of the change of the intonation over time formed according to the rising inflection points, the falling inflection points and the slope change rate of the pitch curve. When performing trend analysis processing on the pitch curve of the speech segment, numerical differentiation and other methods can be used to calculate the slope of the pitch curve. By comparing the slopes of adjacent points, the rising inflection points and the falling inflection points are found. For example, when the slope of the pitch curve changes from positive to negative, the point is the falling inflection point; when the slope changes from negative to positive, the point is the rising inflection point. When calculating the slope change rate between adjacent inflection points, the change amount of the pitch between two adjacent inflection points can be divided by the time interval. The intonation rising and falling trajectory is generated according to the rising inflection points, the falling inflection points and the slope change rate, for example, the rising and falling changes of the intonation in the speech segment can be represented in the form of a curve or data.

[0034] Step S224: Based on the accent distribution pattern and the intonation rise-fall trajectory, filter the parts overlapping with the general speech features through the pre-trained intent-acoustic feature mapping relationship, retain the feature components strongly associated with the target interactive intent, and form an acoustic feature sequence.

[0035] The pre-trained intent-acoustic feature mapping relationship is a model or rule trained in advance, which describes the correspondence between different acoustic features and specific interactive intents. The general speech features refer to the features that exist universally in various speeches and do not have a specific interactive intent direction. The target interactive intent is the intent of a user to express through speech to achieve a certain specific interaction, such as querying information, controlling equipment, etc.

[0036] In this step, first, the accent distribution pattern and the intonation rise-fall trajectory generated in the previous steps need to be obtained. For example, in a user query weather speech scenario, after processing in steps S221-S223, the accent distribution pattern of the speech segment is obtained, and it is found that the position of the word "weather" is the accent concentration area; at the same time, the intonation rise-fall trajectory is obtained, which shows that the intonation rises at "how" when asking "how is the weather today". Then, the pre-trained intent-acoustic feature mapping relationship is used to analyze the accent distribution pattern and the intonation rise-fall trajectory. The pre-trained intent-acoustic feature mapping relationship can be implemented using a deep learning model, such as a Multilayer Perceptron (MLP). The model can have an input layer, a hidden layer, and an output layer. The input layer receives the feature vectors of the accent distribution pattern and the intonation rise-fall trajectory, the hidden layer performs nonlinear transformation on the input features through multiple neurons, and the output layer outputs the probability of each possible interactive intent. When training this MLP model, a large amount of labeled data is required. These data include the accent distribution pattern, the intonation rise-fall trajectory, and the corresponding interactive intent label of different speech segments. The training steps are as follows: first, divide the labeled data into training set, validation set, and test set. Then, initialize the parameters of the MLP model, such as weights and biases. Next, use the training set data to iteratively train the model, calculate the gradient of the loss function through the backpropagation algorithm, and update the parameters of the model to minimize the error between the predicted interactive intent and the true label. During training, the validation set is used to adjust the hyperparameters of the model, such as learning rate, number of hidden layer neurons, etc., to avoid overfitting. Finally, use the test set to evaluate the trained model to ensure that the model has good generalization ability.

[0037] In this embodiment, the accent distribution pattern and the intonation contour are input into the trained MLP model, and the model outputs the probability of each interactive intent. According to this probability, combined with the knowledge of general speech features, the part overlapping with the general speech features is filtered. The general speech features can be obtained by statistical analysis of a large amount of non-specific intent speech data, such as speech rhythm at normal speed, common speech pauses, etc.

[0038] For example, the slight pause commonly existing in speech, which appears in many speeches and does not have specific interactive intent directionality, belongs to general speech features. The model can determine which features belong to general speech features according to the output probability, and filter them. After filtering the general speech features, the feature components strongly associated with the target interactive intent are retained. Finally, the retained feature components are combined to form an acoustic feature sequence, which can more accurately reflect the target interactive intent of the user. In this way, the accuracy of the speech interaction system in understanding the user's intent can be improved, the misjudgment caused by the interference of general speech features can be reduced, and the robustness and reliability of the system can be enhanced.

[0039] Step S230: Perform cross-segment semantic association modeling processing on each speech segment, contextually fuse the text conversion results of the speech segment and the previous speech segment, and generate an associated feature matrix containing semantic continuity.

[0040] The cross-segment semantic association modeling processing is a process of modeling and analyzing the semantic association between different speech segments. The text conversion result is the result of converting the speech segment into text form, which can be achieved by general speech recognition technology. Contextual fusion is a comprehensive processing of the text information of the current speech segment and the text information of the previous speech segment, considering their semantic relationship and context information. The associated feature matrix is a matrix used to represent the semantic association relationship between speech segments, containing related information such as semantic continuity. When performing cross-segment semantic association modeling processing on each speech segment, first convert the speech segment into text form. Then, contextually fuse the text conversion result of the current speech segment with the text conversion result of the previous speech segment. Semantic vector encoding and other methods in natural language processing can be used to convert text information into vector representation, and then context fusion is achieved through vector operation and fusion. According to the context fusion result, an associated feature matrix is generated, and the elements in the matrix can represent the semantic association strength, semantic continuity, and other information between different speech segments. For example, a row and a column in the matrix correspond to two different speech segments, and the value of the matrix element represents the semantic association degree between the two speech segments.

[0041] In an implementation, step S230, a cross-fragment semantic association modeling processing is performed on each speech segment, the text conversion result of the speech segment is contextually fused with the text conversion result of the previous speech segment, and an associated feature matrix containing semantic continuity is generated. The associated feature matrix can be generated by implementing the following steps S231-S235.

[0042] Step S231: Perform semantic vector encoding processing on the text conversion result of the current speech segment and the text conversion result of the previous speech segment, and generate a current segment semantic vector and a previous segment semantic vector.

[0043] The semantic vector encoding processing is to convert the text information into a vector form, so that the semantic information of the text can be embodied in the vector space. The current segment semantic vector is a vector obtained by performing semantic vector encoding on the text conversion result of the current speech segment. The previous segment semantic vector is a vector obtained by performing semantic vector encoding on the text conversion result of the previous speech segment. When performing semantic vector encoding processing on the text conversion result of the current speech segment and the text conversion result of the previous speech segment, a word embedding model such as Word2Vec, GloVe, etc. can be used. These models can convert each word into a fixed-length vector, and then obtain the semantic vector of the entire text by summing or averaging the word vectors in the text. The same method can be used to obtain the previous segment semantic vector. The semantic vector can represent the semantic information of the text in the form of a vector, which facilitates subsequent semantic association analysis and calculation.

[0044] Step S232: Calculate the cosine similarity between the current segment semantic vector and the previous segment semantic vector, and generate a cross-fragment semantic association strength value.

[0045] The cosine similarity is an index for measuring the similarity between two vectors, which represents the similarity between two vectors by calculating the cosine value of the included angle between the two vectors. The cross-fragment semantic association strength value is a value obtained according to the cosine similarity between the current segment semantic vector and the previous segment semantic vector, which represents the semantic association strength between the current speech segment and the previous speech segment. The obtained cosine similarity value is the cross-fragment semantic association strength value, and the value closer to 1 indicates that the semantic association between the two speech segments is stronger, and the value closer to 0 indicates that the semantic association is weaker.

[0046] Step S233: Adjust the fusion weight of the current segment semantic vector and the previous segment semantic vector according to the cross-fragment semantic association strength value, and the fusion weight is positively correlated with the semantic association strength value.

[0047] The fusion weight is a weight value given to each vector when fusing the current segment semantic vector and the previous segment semantic vector. The semantic association strength value is a cosine similarity value between the current segment semantic vector and the previous segment semantic vector. The fusion weight and the semantic association strength value are in a positive correlation relationship, that is, the greater the semantic association strength value, the greater the corresponding fusion weight, and vice versa. When adjusting the fusion weight of the current segment semantic vector and the previous segment semantic vector according to the cross-segment semantic association strength value, a linear mapping method can be used. For example, a weight adjustment function f(x) = ax + b can be set, where x represents the cross-segment semantic association strength value, a and b are pre-set parameters, and a > 0. When the cross-segment semantic association strength value is x1, the calculated fusion weight is f(x1). If the semantic association strength value is large, it means that the semantic association between the current speech segment and the previous speech segment is strong, so the previous segment semantic vector should be given a larger weight during fusion to better reflect the context information. By adjusting the fusion weight, more reasonable context fusion can be performed to generate more accurate association features.

[0048] Step S234: Perform context fusion processing on the current segment semantic vector and the previous segment semantic vector through a weighted summation operation to generate a fused semantic vector containing semantic continuity.

[0049] The weighted summation operation is an operation of multiplying the current segment semantic vector and the previous segment semantic vector by the corresponding fusion weight, respectively, and then adding them. The context fusion processing is a comprehensive processing of the text information of the current speech segment and the text information of the previous speech segment, considering the semantic relationship and context information between them. The fused semantic vector is a vector obtained by fusing the current segment semantic vector and the previous segment semantic vector through the weighted summation operation, containing context information such as semantic continuity.

[0050] Step S235: Perform dimension concatenation processing on the fused semantic vector, the current segment semantic vector, and the previous segment semantic vector to form an association feature matrix.

[0051] The dimension concatenation processing is an operation of concatenating multiple vectors in dimensions, that is, arranging their elements in sequence to form a higher-dimensional vector or matrix. The association feature matrix is a matrix formed by concatenating the fused semantic vector, the current segment semantic vector, and the previous segment semantic vector in dimensions, used to represent the semantic association relationship between speech segments. Through the dimension concatenation processing, the fused semantic vector, the current segment semantic vector, and the previous segment semantic vector are combined together to form an association feature matrix, which contains more semantic association information.

[0052] Step S240: performing semantic strength attenuation calculation on the correlation feature matrix, adjusting the weight value of the correlation feature according to the time interval between the speech segments, and generating a semantic correlation feature set, which includes the semantic continuity across segments and the semantic concentration within the segment.

[0053] Semantic strength decay calculation involves decaying the correlation features in the correlation feature matrix, taking into account the impact of the time interval between speech segments on the strength of semantic association. The time interval between speech segments is the temporal separation between two adjacent speech segments. The longer the time interval, the weaker the semantic association between them. Adjusting the weight of the correlation features involves adjusting the weight of each element in the correlation feature matrix based on the length of the time interval, so that the weight of the correlation features between speech segments with longer time intervals is relatively small. The semantic correlation feature set is the feature set obtained by performing semantic strength decay calculation on the correlation feature matrix. It contains information such as cross-segment semantic continuity and intra-segment semantic concentration. Cross-segment semantic continuity is the degree of semantic continuity between different speech segments, reflecting the semantic coherence of the speech content. Intra-segment semantic concentration is the degree of semantic concentration within a speech segment, reflecting the semantic closeness of the speech segments. When performing semantic strength decay calculation on the correlation feature matrix, the time interval between speech segments can be first calculated. Then, a preset decay function table can be queried based on the time interval to obtain the corresponding weight decay coefficient. Each element in the correlation feature matrix is ​​multiplied by the weight decay coefficient to obtain the decayed correlation feature vector. The cross-segment semantic continuity and intra-segment semantic concentration are calculated based on the attenuated association feature vectors, and they are arranged in the chronological order of the speech segments to form a semantic association feature set.

[0054] As an implementation method, step S240 performs semantic strength attenuation calculation on the correlation feature matrix, adjusts the weight values ​​of the correlation features according to the time intervals between the speech segments, and generates a semantic correlation feature set. Specifically, the following steps S241 to S246 may be implemented:

[0055] Step S241: Calculate the time interval length between the current speech segment and the previous speech segment, and generate a time interval feature value.

[0056] The time interval length is the time difference between the current speech segment and the previous speech segment, reflecting the temporal distance between the two segments. The time interval feature value is a feature value generated based on the time interval length and used to represent the time interval. The time interval length between the current speech segment and the previous speech segment can be calculated by comparing their timestamp information.

[0057] Step S242: According to the time interval characteristic value, the preset attenuation function table is queried to obtain the corresponding weight attenuation coefficient. The attenuation function table defines a negative correlation between the time interval length and the weight attenuation coefficient.

[0058] The preset attenuation function table is predefined and is a table representing the relationship between the time interval length and the weight attenuation coefficient. The weight attenuation coefficient is a coefficient used for attenuating the associated features in the associated feature matrix. The longer the time interval, the smaller the weight attenuation coefficient, so as to reflect the attenuation effect of the time interval on the semantic association strength. When querying the preset attenuation function table according to the time interval characteristic value, the time interval characteristic value is used as an index to find the corresponding weight attenuation coefficient in the attenuation function table.

[0059] Step S243: Apply the weight attenuation coefficient to each associated feature dimension in the associated feature matrix to generate an attenuated associated feature vector.

[0060] Applying the weight attenuation coefficient is an operation of multiplying each element in the associated feature matrix by the corresponding weight attenuation coefficient. The attenuated associated feature vector is a vector obtained by applying the weight attenuation coefficient to the associated feature dimensions in the associated feature matrix. For example, assuming that the associated feature matrix is M=(m ij ), where i represents the row index and j represents the column index, and the weight attenuation coefficient is k, then the attenuated associated feature vector V=(k×m ij ). Applying the weight attenuation coefficient to each associated feature dimension in the associated feature matrix makes the associated feature weight between the speech segments with a longer time interval relatively smaller, reflecting the attenuation effect of the time interval on the semantic association strength. The generated attenuated associated feature vector more accurately reflects the semantic association relationship between the speech segments, providing more reasonable data for subsequent semantic analysis.

[0061] Step S244: Perform semantic concentration calculation on the attenuated associated feature vector to count the distribution density of the high-weight dimensions in the associated feature vector to generate an intra-segment semantic concentration.

[0062] The semantic concentration calculation is to analyze the decayed associated feature vector and calculate the concentration degree of semantics therein. The high-weight dimension is a dimension in the associated feature vector with a relatively high weight value, which usually contains more important semantic information. The distribution density is the distribution of the high-weight dimension in the associated feature vector, such as the frequency of the high-weight dimension in the vector, the degree of aggregation, etc. The intra-fragment semantic concentration is an index generated according to the distribution density of the high-weight dimension, which represents the concentration degree of semantics within a speech segment. When performing the semantic concentration calculation on the decayed associated feature vector, a weight threshold can be first set, and the dimensions in the associated feature vector with weight values greater than the threshold are regarded as high-weight dimensions. Then, the distribution density of the high-weight dimensions in the associated feature vector is calculated. For example, the length of the continuous appearance of the high-weight dimension in the vector, the proportion of the number of high-weight dimensions to the total number of dimensions, etc. can be calculated. The intra-fragment semantic concentration is generated according to these statistical indicators, which reflects the closeness and concentration of the semantics within a speech segment.

[0063] Step S245: Perform a cross-fragment continuity evaluation process on the decayed associated feature vector, compare the high-weight dimension overlap degree of the current speech segment and the previous speech segment, and generate a cross-fragment semantic continuity degree.

[0064] The cross-fragment continuity evaluation process is to analyze the decayed associated feature vector and evaluate the semantic continuity degree between different speech segments. The high-weight dimension overlap degree is the overlap of the high-weight dimensions in the decayed associated feature vector of the current speech segment and the decayed associated feature vector of the previous speech segment. The higher the overlap degree, the stronger the semantic continuity between the two speech segments. The cross-fragment semantic continuity degree is an index generated according to the high-weight dimension overlap degree, which represents the semantic continuity degree between different speech segments. When performing the cross-fragment continuity evaluation process on the decayed associated feature vector, the high-weight dimensions need to be determined first. A weight threshold can be set, and the dimensions in the associated feature vector with weight values greater than the threshold are regarded as high-weight dimensions. Then, the high-weight dimensions of the current speech segment and the previous speech segment are compared, and their overlap degree is calculated. For example, the proportion of the intersection to the union can be used to calculate the overlap degree, i.e. where A represents the set of high-weight dimensions of the current speech segment, and B represents the set of high-weight dimensions of the previous speech segment. The cross-fragment semantic continuity degree is generated according to the calculated high-weight dimension overlap degree, which reflects the semantic coherence of the speech content between different segments.

[0065] Step S246: Arrange the intra-fragment semantic concentration and the cross-fragment semantic continuity degree in the time sequence of the speech segments to form a semantic associated feature set.

[0066] The in-segment semantic concentration degree and the cross-segment semantic continuation degree are arranged in time sequence of the voice segments, that is, the in-segment semantic concentration degree and the cross-segment semantic continuation degree corresponding to each voice segment are arranged in time sequence in the continuous voice data stream. The semantic association feature set is a feature set containing information such as cross-segment semantic continuation degree and in-segment semantic concentration degree formed by this arrangement mode.

[0067] Step S300: calling a pre-trained voice semantic understanding model to perform joint semantic analysis processing on the acoustic feature sequence and the semantic association feature set to generate a semantic understanding result of the voice segment.

[0068] The pre-trained voice semantic understanding model is a model pre-trained by a large amount of data and used for understanding voice semantics. The model can learn the mapping relationship between the acoustic features and semantic features of the voice and the semantic meaning. The joint semantic analysis processing is to combine the acoustic feature sequence and the semantic association feature set, analyze and parse them through the pre-trained voice semantic understanding model, and mine the semantic information of the voice segment. The semantic understanding result is a result containing information such as the intention evolution track and the semantic core content of the voice segment obtained by the joint semantic analysis processing. When calling the pre-trained voice semantic understanding model to perform joint semantic analysis processing on the acoustic feature sequence and the semantic association feature set, the acoustic feature sequence and the semantic association feature set are input into the pre-trained voice semantic understanding model as inputs. The model will analyze and process the input features according to its internal structure and learned mapping relationship, and output the semantic understanding result of the voice segment. For example, the pre-trained voice semantic understanding model can adopt a deep learning model such as a recurrent neural network (RNN), a long short-term memory network (LSTM), etc. These models can process sequence data and well capture the time sequence features and semantic association information of the voice. Through joint semantic analysis processing, the acoustic and semantic features are comprehensively considered to generate more accurate semantic understanding results of the voice segment.

[0069] As an implementation mode, step S300, calling a pre-trained voice semantic understanding model to perform joint semantic analysis processing on the acoustic feature sequence and the semantic association feature set to generate a semantic understanding result of the voice segment, can be implemented as steps S310-S340 as follows:

[0070] Step S310: inputting the acoustic feature sequence and the semantic association feature set into a context fusion module of the voice semantic understanding model, determining a context window length based on the cross-segment semantic continuation degree in the semantic association feature set, and generating an extended feature sequence containing historical segment association information.

[0071] The context fusion module is a module in the speech semantic understanding model, which is used to fuse the features of the current speech segment with the features of the historical speech segment, and consider the context information. The cross-segment semantic continuation degree is the degree of semantic continuation between different speech segments, which reflects the strength of the semantic association between the historical segment and the current segment. The context window length is the number or time range of historical speech segments considered when performing context fusion. The extended feature sequence is a feature sequence containing more historical segment association information obtained by fusing the features of the current speech segment with the features of the historical speech segment through the context fusion module. When the acoustic feature sequence and the semantic association feature set are input into the context fusion module of the speech semantic understanding model, the cross-segment semantic continuation degree is first extracted from the semantic association feature set. According to the strength of the cross-segment semantic continuation degree, the context window length is determined. The stronger the continuation degree, the closer the semantic association between the historical segment and the current segment, and the context window length can be set larger to consider more historical speech segments. The weaker the continuation degree, the context window length can be set smaller to consider fewer historical speech segments. For example, assuming that the cross-segment semantic continuation degree is represented by a numerical value, when the cross-segment semantic continuation degree is greater than 0.8, the context window length is set to 3, that is, three historical speech segments before the current speech segment are considered; when the cross-segment semantic continuation degree is between 0.5 and 0.8, the context window length is set to 2; when the cross-segment semantic continuation degree is less than 0.5, the context window length is set to 1. According to the determined context window length, the acoustic feature sequence and the semantic association feature set of the selected historical speech segment are concatenated and spliced with the corresponding features of the current speech segment to generate a cross-segment feature splicing vector. The cross-segment feature splicing vector is processed for feature dimension alignment to unify the feature representation dimensions of different timestamp speech segments, and an extended feature sequence containing historical segment association information is generated.

[0072] In specific implementation, the context fusion module can be implemented by a neural network architecture based on attention mechanism. Taking the Transformer architecture as an example, it contains multiple encoder layers, each of which is composed of a multi-head self-attention mechanism and a feedforward neural network. The multi-head self-attention mechanism can capture the dependency between different positions in the input sequence, and the feedforward neural network performs nonlinear transformation on the output of the attention mechanism.

[0073] As an implementation, in step S310, the acoustic feature sequence and the semantic association feature set are input into the context fusion module of the speech semantic understanding model, the context window length is determined based on the cross-segment semantic continuation degree in the semantic association feature set, and an extended feature sequence containing historical segment association information is generated. Specifically, the following steps S311-S314 can be implemented:

[0074] Step S311: Extract the cross-segment semantic continuation degree of the current speech segment and the historical speech segment from the semantic association feature set, which reflects the strength of the semantic association between the historical segment and the current segment.

[0075] When extracting the cross-segment semantic continuation degree from the semantic association feature set, according to the storage structure and organization method of the semantic association feature set, the cross-segment semantic continuation degree information corresponding to the current speech segment and the historical speech segment is found. For example, if the semantic association feature set is a list, each element of which contains the intra-segment semantic concentration of a speech segment and the cross-segment semantic continuation degree with the adjacent speech segment, then the cross-segment semantic continuation degree with the historical speech segment can be extracted from the list according to the index position of the current speech segment. The cross-segment semantic continuation degree reflects the degree of semantic association between the historical segment and the current segment. The greater the value, the stronger the semantic association between the historical segment and the current segment; the smaller the value, the weaker the semantic association.

[0076] Step S312: Select the historical speech segment associated with the current segment according to the strength of the cross-segment semantic continuation degree. The stronger the continuation degree, the more associated historical segments are selected, and the weaker the continuation degree, the fewer associated historical segments are selected.

[0077] When selecting the historical speech segment according to the strength of the cross-segment semantic continuation degree, different thresholds can be set for judgment. For example, set three thresholds T1, T2 (T1 < T2), when the cross-segment semantic continuation degree is greater than T2, select three historical speech segments before the current speech segment; when the cross-segment semantic continuation degree is between T1 and T2, select two historical speech segments before the current speech segment; when the cross-segment semantic continuation degree is less than T1, only select one historical speech segment before the current speech segment. The stronger the continuation degree, the closer the semantic association between the historical segment and the current segment, and the more historical speech segments selected can better utilize the context information and improve the accuracy of semantic understanding.

[0078] Step S313: Cascade and splice the acoustic feature sequence and the semantic association feature set of the selected historical speech segment with the corresponding features of the current speech segment to generate a cross-segment feature splicing vector.

[0079] The cascading concatenation processing is an operation of concatenating multiple vectors according to dimensions, that is, arranging the acoustic feature sequence and the semantic association feature set of the selected historical speech segment and the corresponding features of the current speech segment in turn to form a higher-dimensional vector. The cross-segment feature concatenation vector is a vector containing feature information of multiple speech segments obtained through the cascading concatenation processing. Through the cascading concatenation processing, the features of multiple speech segments are combined together to form a cross-segment feature concatenation vector, which contains more historical segment association information and provides richer data for subsequent semantic analysis.

[0080] Step S314: performing feature dimension alignment processing on the cross-segment feature concatenation vector to unify the feature representation dimensions of different timestamp speech segments and generating an extended feature sequence containing historical segment association information.

[0081] The feature dimension alignment processing is an operation of processing different vectors in the cross-segment feature concatenation vector to make their dimensions consistent. The feature representation dimensions of different timestamp speech segments may be different. For example, due to differences in the lengths of speech segments, differences in feature extraction methods, etc., the dimensions of the acoustic feature sequence and the semantic association feature set of different speech segments may be inconsistent. Unifying the feature representation dimensions can make the data in the cross-segment feature concatenation vector consistent, facilitating subsequent model processing. When performing feature dimension alignment processing on the cross-segment feature concatenation vector, a padding or truncation method can be used. For example, if the dimension of the feature vector of a certain speech segment is smaller than the dimensions of other vectors, zero values can be filled after the vector to make its dimension consistent with the dimensions of other vectors; if the dimension of the feature vector of a certain speech segment is larger than the dimensions of other vectors, part of the elements of the vector can be truncated to make its dimension the same as the dimensions of other vectors. After the feature dimension alignment processing, an extended feature sequence containing historical segment association information is generated, which is more suitable as an input of a speech semantic understanding model and can improve the speech semantic understanding ability of the model.

[0082] Step S320: performing time-dependent relationship modeling processing on the extended feature sequence by an intent evolution modeling layer of the speech semantic understanding model to capture the trend of intent change between continuous speech segments and generating an intent evolution feature vector.

[0083] The intention evolution modeling layer is a network layer in the speech semantic understanding model, and is used for modeling and analyzing intention changes of the speech segments. The time dependency modeling processing is to model the extended feature sequence by considering the sequence and mutual relationship of the speech segments in time, and to mine time dependency information in the extended feature sequence. The intention change trend between continuous speech segments is the change of the user's interactive intention with time in continuous speech expression, such as gradual change, sudden change or stable maintenance of the intention, and the like. The intention evolution feature vector is a vector obtained by performing the time dependency modeling processing on the extended feature sequence, and contains feature information of the intention change between continuous speech segments. When the intention evolution modeling layer of the speech semantic understanding model performs the time dependency modeling processing on the extended feature sequence, the intention evolution modeling layer can adopt a recurrent neural network (RNN) or a variant thereof, such as a long short-term memory network (LSTM), a gated recurrent unit (GRU), and the like. These network structures can process sequence data and well capture time dependency information in the sequence. Taking the LSTM as an example, the LSTM contains an input gate, a forget gate and an output gate, and through these gating mechanisms, the LSTM can selectively retain or forget historical information, thereby better capturing the intention change trend between continuous speech segments. The extended feature sequence is input into the LSTM, the LSTM updates the hidden state at the current time according to the current input and the hidden state at the last time, and finally outputs the hidden state at the current time, which is the intention evolution feature vector, and the vector reflects the intention change between continuous speech segments.

[0084] As an implementation, in step S320, the intention evolution modeling layer of the speech semantic understanding model performs time dependency modeling processing on the extended feature sequence, captures the intention change trend between continuous speech segments, and generates an intention evolution feature vector. Specifically, the following steps S321-S325 can be implemented:

[0085] In step S321, the extended feature sequence is input into a time gating unit of the intention evolution modeling layer, and the time gating unit generates forget gate and update gate parameters based on the semantic association strength between the historical segment and the current segment in the extended feature sequence.

[0086] The time gating unit is a unit in the intention evolution modeling layer, which is used to control the flow and update of information, and can generate a forgetting gate and an update gate parameter according to the input feature information. The forgetting gate and the update gate parameter are important parameters in the time gating unit, the forgetting gate parameter is used to control whether to forget the historical information, and the update gate parameter is used to control whether to update the current information. When the extended feature sequence is input into the time gating unit of the intention evolution modeling layer, the time gating unit generates the forgetting gate and the update gate parameter according to the semantic association strength of the historical segment and the current segment in the extended feature sequence. If the semantic association strength of the historical segment and the current segment is strong, the forgetting gate parameter may be small, which means that more historical information is retained; if the semantic association strength is weak, the forgetting gate parameter may be large, which means that more historical information is forgotten. By generating the forgetting gate and the update gate parameter, the time gating unit can better manage the flow of information and capture the intention change trend between continuous speech segments.

[0087] Step S322: filtering the historical feature information irrelevant to the current intention in the extended feature sequence through the forgetting gate parameter, and retaining the key feature components related to the intention evolution.

[0088] When filtering the historical feature information in the extended feature sequence through the forgetting gate parameter, the forgetting gate parameter is multiplied with the historical feature information in the extended feature sequence element by element. The forgetting gate parameter is a vector whose value is between 0 and 1, and the closer the value is to 0, the more the corresponding historical feature information is forgotten; the closer the value is to 1, the more the corresponding historical feature information is retained. In this way, the historical feature information irrelevant to the current intention is filtered out, and only the key feature components related to the intention evolution are retained. These key feature components can better reflect the intention change trend between continuous speech segments, and provide more valuable information for subsequent intention analysis and processing.

[0089] Step S323: performing weighted fusion processing on the core feature of the current speech segment and the retained historical key feature components through the update gate parameter to generate an intention state update vector.

[0090] The update gate parameter is a vector with a value between [0, 1], which is used to control whether to update the current intention state. The core feature of the current speech segment is a feature that best represents the intention of the current speech segment. The weighted fusion processing is an operation of weighted sum of the core feature of the current speech segment and the reserved historical key feature component according to the update gate parameter. The intention state update vector is a vector obtained after the weighted fusion processing, which reflects the intention state at the current moment. When the core feature of the current speech segment and the reserved historical key feature component are weighted and fused by the update gate parameter, the update gate parameter determines the weight of the core feature of the current speech segment and the reserved historical key feature component in the intention state update vector. If the value of the update gate parameter is larger, it means that more attention is paid to the core feature of the current speech segment; if the value of the update gate parameter is smaller, it means that more attention is paid to the reserved historical key feature component. The intention state update vector is generated by the weighted fusion processing, which integrates the intention information of the current speech segment and the historical speech segment, and better reflects the intention change between continuous speech segments.

[0091] Step S324: performing trend analysis processing on the intention state update vector, identifying the change mode of the intention category between continuous speech segments, and generating an evolution trend feature reflecting the intention change direction.

[0092] The trend analysis processing is to analyze the intention state update vector to find the change trend and rule. The change mode of the intention category between continuous speech segments is the change of the user's interactive intention category over time in continuous speech expression, such as gradual change mode, sudden change mode and stable change mode, etc. The evolution trend feature is a feature generated according to the trend analysis result of the intention state update vector, which reflects the direction and degree of intention change. When the trend analysis processing is performed on the intention state update vector, various methods can be used. For example, time series difference processing can be performed on the intention state update vector to calculate the difference value of the intention state between adjacent speech segments to generate an intention change amount sequence. The intention change amount sequence is matched with a preset typical intention change mode library, which includes gradual change mode, sudden change mode and stable change mode, etc. The specific change mode of the intention category between continuous speech segments is determined according to the matching result. The sign feature and amplitude feature of the intention change amount sequence are extracted to generate a direction feature reflecting the intention change direction and an amplitude feature reflecting the change degree. The specific change mode, direction feature and amplitude feature are fused to form the evolution trend feature. Through the trend analysis processing, the change mode of the intention category between continuous speech segments can be more accurately identified, and the evolution trend feature reflecting the intention change direction is generated.

[0093] As an implementation, step S324, trend analysis processing is performed on the intention state update vector to identify the change pattern of the intention category between continuous speech segments, and an evolution trend feature reflecting the change direction of the intention is generated. Specifically, the following steps S3241-S3245 can be implemented:

[0094] Step S3241: Time series difference processing is performed on the intention state update vector to calculate the difference value of the intention state between adjacent speech segments, and an intention change amount sequence is generated.

[0095] The time series difference processing is an operation of difference calculation on the intention state update vector in the time dimension. The difference value of the intention state between adjacent speech segments is the difference value of the corresponding elements of the intention state update vector of the current speech segment and the intention state update vector of the previous speech segment. The intention change amount sequence is a sequence obtained by time series difference processing, which reflects the change of the intention state between adjacent speech segments. The intention change amount sequence can help analyze the change trend of the intention between continuous speech segments. For example, if the value in the intention change amount sequence is large, it means that the intention change is obvious; if the value is small, it means that the intention is relatively stable.

[0096] Step S3242: The intention change amount sequence is matched with a preset typical intention change pattern library, and the typical intention change pattern library includes gradual change patterns, sudden change patterns, and stable change patterns.

[0097] The preset typical intention change pattern library is predefined and includes a collection of various typical intention change patterns. The matching processing is to compare the intention change amount sequence with the patterns in the typical intention change pattern library to find the most similar pattern. The gradual change pattern is a pattern in which the intention gradually changes over a period of time, and the intention change amount is relatively small and stable; the sudden change pattern is a pattern in which the intention changes greatly in a short period of time, and the intention change amount is large and sudden; the stable change pattern is a pattern in which the intention remains basically unchanged over a period of time, and the intention change amount is close to zero. When matching the intention change amount sequence with the preset typical intention change pattern library, a similarity calculation method can be used. For example, the Euclidean distance or cosine similarity between the intention change amount sequence and each typical intention change pattern can be calculated. The smaller the Euclidean distance or the greater the cosine similarity, the more similar the intention change amount sequence and the typical intention change pattern. Through the matching processing, the specific change pattern of the intention category between continuous speech segments is determined, which provides a basis for subsequent intention analysis and processing.

[0098] Step S3243: According to the matching result, the specific change pattern of the intention category between continuous speech segments is determined.

[0099] If the similarity between the intent change amount sequence and the gradual change pattern is the highest, it is determined that the intent change pattern between the continuous speech segments is gradual; if the similarity between the intent change amount sequence and the abrupt change pattern is the highest, it is determined that the intent change pattern is abrupt; and if the similarity between the intent change amount sequence and the stable change pattern is the highest, it is determined that the intent change pattern is stable. For example, it is assumed that through the calculation of the similarity, it is found that the cosine similarity between the intent change amount sequence and the gradual change pattern is 0.9, the cosine similarity between the intent change amount sequence and the abrupt change pattern is 0.2, and the cosine similarity between the intent change amount sequence and the stable change pattern is 0.3, and it is determined that the specific change pattern of the intent category between the continuous speech segments is gradual. Determining the specific intent change pattern helps better understand the change of the user's interactive intent over time, and provides more accurate information for subsequent intent analysis and processing.

[0100] Step S3244: Extract the sign feature and the amplitude feature of the intent change amount sequence, generate the direction feature reflecting the direction of the intent change and the amplitude feature reflecting the change degree.

[0101] The sign feature of the intent change amount sequence is the sign of each element in the intent change amount sequence, and the sign is positive, indicating that the intent state is increasing, and the sign is negative, indicating that the intent state is decreasing. The direction feature is a feature generated according to the sign feature of the intent change amount sequence, which reflects the direction of the intent change. The amplitude feature of the intent change amount sequence is the absolute value of each element in the intent change amount sequence, indicating the degree of the intent change. The amplitude feature is a feature generated according to the amplitude feature of the intent change amount sequence, which reflects the size of the intent change. When extracting the sign feature and the amplitude feature of the intent change amount sequence, it is assumed that the intent change amount sequence is The sign feature is Where sgn(x) is a sign function, sgn(x)=1 when x>0, sgn(x)=0 when x=0, and sgn(x)=-1 when x<0. The amplitude feature is The direction feature is generated according to the sign feature, and the amplitude feature is generated according to the amplitude feature.

[0102] Step S3245: Perform feature fusion processing on the specific change pattern, the direction feature and the amplitude feature to form the evolution trend feature.

[0103] The feature fusion processing is a comprehensive processing of the specific change pattern, direction feature and amplitude feature, which combines them into a more comprehensive feature. The evolution trend feature is a feature obtained through the feature fusion processing, which contains the pattern, direction and degree of the intention change between continuous speech segments. When performing the feature fusion processing on the specific change pattern, direction feature and amplitude feature, the splicing or weighted combination method can be used. For example, if the specific change pattern is represented by a one-hot encoding vector, and the direction feature and amplitude feature are represented by vectors respectively, then the three vectors can be spliced in order to form a new vector as the evolution trend feature. Through the feature fusion processing, multiple features are integrated to form the evolution trend feature, which more comprehensively reflects the trend of the intention change between continuous speech segments.

[0104] Step S325: Perform dimension splicing processing on the intention state update vector and the evolution trend feature to form an intention evolution feature vector.

[0105] The dimension splicing processing is an operation of splicing the intention state update vector and the evolution trend feature in the dimension, that is, arranging their elements in order to form a higher-dimensional vector. The intention evolution feature vector is a vector obtained through the dimension splicing processing, which contains the information of the intention state update vector and the evolution trend feature, and more comprehensively reflects the intention change between continuous speech segments. Through the dimension splicing processing, the intention state update vector and the evolution trend feature are combined to form the intention evolution feature vector, which provides more abundant information for subsequent speech semantic understanding and interactive intention analysis, and helps to more accurately grasp the user's interactive intention.

[0106] Step S330: Perform expression standardization correction processing on the intention evolution feature vector through the colloquial interference calibration module of the speech semantic understanding model, eliminate the non-standard expression interference in the speech content, and generate the calibrated semantic core feature.

[0107] The colloquial interference calibration module is a module in the speech semantic understanding model, used for correcting and processing non-standard expressions in speech expressions. The expression standardization correction processing is to analyze and adjust the intention evolution feature vector, remove the interference information caused by colloquial expressions, and make the expression of speech content more standardized and accurate. Non-standard expression interference is a situation that does not conform to standard grammar or vocabulary usage in speech expression due to colloquial habits, casual expression, etc. For example, catchwords, repeated words, and incorrect pronunciation. The calibrated semantic core feature is the feature obtained by processing the intention evolution feature vector through the colloquial interference calibration module, which more accurately reflects the semantic core of the speech content. When the colloquial interference calibration module of the speech semantic understanding model performs expression standardization correction processing on the intention evolution feature vector, the colloquial interference calibration module can use a combination of rule matching and machine learning. For example, some common catchwords and non-standard expression rules are defined in advance. When the speech content corresponding to the intention evolution feature vector matches these rules, the corresponding correction is performed. At the same time, a machine learning model such as a neural network is used to train a large amount of standard and non-standard speech data, so that the model learns the mapping relationship between non-standard expressions and standard expressions. The intention evolution feature vector is input into the colloquial interference calibration module, which processes it according to the rules and the trained model, eliminates non-standard expression interference, and generates calibrated semantic core features.

[0108] Step S340: Bind and associate the calibrated semantic core feature with the timestamp information of the speech segment to generate a semantic understanding result containing the intention evolution track and the semantic core content.

[0109] The binding association processing is to combine the calibrated semantic core features with the timestamp information of the voice segments, so as to associate them with each other. The timestamp information of the voice segments is the time position of each voice segment in the continuous voice data stream, which can accurately record the occurrence time of the voice segment. The intention evolution trajectory is the trajectory of the change of the user's interactive intention over time in the continuous voice expression, and the relevant information can be obtained through the previous intention evolution modeling layer. The semantic core content is the most critical and most semantic information extracted from the voice content, and the calibrated semantic core features reflect this part of information. When the calibrated semantic core features are bound and associated with the timestamp information of the voice segments, the timestamp information can be added as a dimension of the calibrated semantic core features to form a new vector. In this way, the time information and the semantic information are combined to generate a semantic understanding result containing the intention evolution trajectory and the semantic core content. Such a semantic understanding result not only contains the core semantics of the voice content, but also records the position of the semantics in time, providing more comprehensive information for subsequent determination of the user interactive intention type and the key information positioning according to the semantic understanding result.

[0110] Step S400: determining the user interactive intention type corresponding to the continuous voice data stream and the key information positioning feature of the user interactive intention in the voice content according to the semantic understanding result.

[0111] The semantic understanding result contains the intention evolution trajectory and the semantic core content, and through further analysis of these information, the user interactive intention type corresponding to the continuous voice data stream and the key information positioning feature of the intention in the voice content can be determined. The user interactive intention type is the interactive target expected to be achieved by the user through voice expression, such as querying information, issuing instructions, expressing emotions, etc. The key information positioning feature is a feature used to identify the position of the key information of the user interactive intention in the voice content, which can help accurately extract and understand the important information related to the intention. When determining these information according to the semantic understanding result, the change of the intention evolution trajectory, the meaning of the semantic core content and their matching relationship with different interactive intention types need to be considered comprehensively.

[0112] As an implementation manner, step S400, according to the semantic understanding result, determining the user interactive intention type corresponding to the continuous voice data stream and the key information positioning feature of the user interactive intention in the voice content, can be specifically implemented as steps S410-S440:

[0113] Step S410: extracting the time sequence feature of the intention evolution trajectory in the semantic understanding result, and generating an evolution parameter reflecting the intention change rate.

[0114] The time sequence feature of the intention evolution trajectory is a feature of change of the intention in the time dimension, which contains information such as change trend and amplitude of the intention over time. The evolution parameter reflecting the change rate of the intention is obtained by analyzing the time sequence feature of the intention evolution trajectory, and is used to describe the parameter of the degree of fast or slow change of the intention. When extracting the time sequence feature of the intention evolution trajectory in the semantic understanding result, a plurality of methods can be used to analyze the intention evolution trajectory. For example, the intention evolution trajectory can be smoothed to remove noise and fluctuations therein, so that the change trend is more obvious. Then, the change amount of the intention at different time points is calculated, and the change of the intention is reflected through the change amount. When generating the evolution parameter reflecting the change rate of the intention, the average change rate can be calculated according to the ratio of the change amount of the intention to the time interval, or factors such as acceleration of change can be considered. By generating the evolution parameter, the change rate of the intention can be more accurately grasped, which provides an important basis for subsequent determination of the type of user interaction intention.

[0115] As an implementation manner, step S410: extracting the time sequence feature of the intention evolution trajectory in the semantic understanding result, and generating the evolution parameter reflecting the change rate of the intention, can be implemented as steps S411-S415:

[0116] Step S411: performing time window division processing on the intention evolution trajectory, and dividing the intention evolution trajectory into a plurality of sub-trajectory units with time continuity.

[0117] The time window division processing is to divide a plurality of windows according to a certain time interval on the time axis of the intention evolution trajectory, and the intention evolution trajectory in each window forms a sub-trajectory unit. The sub-trajectory unit with time continuity is a section of the intention evolution trajectory that is continuous in time, which contains change information of the intention in the time period. When performing the time window division processing on the intention evolution trajectory, a suitable time window size can be selected according to actual requirements and characteristics of the voice data. For example, if the intention changes frequently in the voice content, a smaller time window can be selected to observe the change of the intention more carefully; if the intention changes relatively slowly, a larger time window can be selected to reduce the amount of calculation. By dividing the intention evolution trajectory into a plurality of sub-trajectory units, the change of the intention in each time period can be more conveniently analyzed, which provides a basis for subsequent calculation of the change rate of the intention.

[0118] Step S412: calculating the slope of each sub-trajectory unit, extracting the change rate of the intention evolution value with time in the sub-trajectory unit, and generating the sub-trajectory change rate.

[0119] The slope is calculated by analyzing the relationship between the intention evolution value and time in each sub-trajectory unit, and calculating the slope thereof. The rate of change of the intention evolution value over time is the proportion of the change of the intention evolution value over time within the sub-trajectory unit, and reflects the speed of the change of the intention in the time period. The sub-trajectory change rate is the change rate obtained by calculating the slope of each sub-trajectory unit. When calculating the slope of each sub-trajectory unit, the intention evolution value in the sub-trajectory unit can be regarded as a function of time, and the slope thereof is calculated using the numerical differentiation method. For example, for a sub-trajectory unit, two adjacent time points t1 and t2 are selected, and the corresponding intention evolution values are y1 and y2, respectively. The slope of the sub-trajectory unit between the two time points can be approximately calculated as By performing such calculation on each sub-trajectory unit, the sub-trajectory change rate of each sub-trajectory unit is obtained, which can reflect the change of the intention in different time periods.

[0120] Step S413: Statistics of the distribution characteristics of the sub-trajectory change rate in the time dimension, including the mean, variance and extreme value of the change rate, to generate the statistical characteristics of the intention change rate.

[0121] The statistics of the distribution characteristics of the sub-trajectory change rate in the time dimension is a statistical analysis of the change rates of all sub-trajectory units to obtain their distribution law in time. The mean of the change rate is the average value of all sub-trajectory change rates, which reflects the overall level of the intention change rate. The variance is the dispersion degree of the sub-trajectory change rate relative to the mean, and the greater the variance, the greater the fluctuation of the change rate. The extreme value is the maximum and minimum value of the sub-trajectory change rate, which can reflect the maximum and minimum amplitude of the intention change. When generating the statistical characteristics of the intention change rate, first, the sum of all sub-trajectory change rates is calculated, and then the mean is obtained by dividing the number of sub-trajectory units. Next, the square of the difference between each sub-trajectory change rate and the mean is calculated, and the average of these square values is obtained to obtain the variance. Finally, the maximum and minimum values of all sub-trajectory change rates are found as the extreme values. These statistical characteristics can more comprehensively describe the distribution of the intention change rate.

[0122] Step S414: Frequency domain conversion processing is performed on the intention evolution trajectory to convert the intention evolution value in the time domain into an energy distribution spectrum in the frequency domain, and the energy proportion of the main frequency component is extracted as the periodicity characteristic of the intention change.

[0123] The frequency domain conversion processing is to convert the signal in time domain into the representation in frequency domain. Through this conversion, the frequency components and energy distribution of the signal can be analyzed. The energy distribution spectrum in frequency domain is the distribution of the energy of the signal with frequency in the frequency domain. The energy proportion of main frequency components is the proportion of the energy of the higher energy frequency components in the total energy in the frequency domain. The periodicity feature of intention change is a parameter obtained by frequency domain conversion of the intention evolution trajectory, which is used to describe whether the intention change has periodicity and the periodicity feature. When performing frequency domain conversion processing on the intention evolution trajectory, Fourier transform and other methods can be used. Fourier transform can decompose the signal in time domain into a combination of sine and cosine waves of different frequencies, thereby obtaining the representation of the signal in frequency domain. After Fourier transform of the intention evolution trajectory, the energy distribution spectrum in frequency domain is obtained. Then, the main frequency components with higher energy are found, and their energy proportion in the total energy is calculated. This proportion can reflect whether the intention change has periodicity. If there are obvious main frequency components and the energy proportion is high, it means that the intention change may have certain periodicity. By extracting this periodicity feature, the change rule of intention can be further understood, and more basis for determining the user interaction intention type can be provided.

[0124] Step S415: Dimension combination processing of the statistical feature and the periodicity feature is performed to generate an evolution parameter reflecting the intention change rate.

[0125] The dimension combination processing is to combine the statistical feature and the periodicity feature to form a feature vector containing more information. The evolution parameter reflecting the intention change rate is obtained by this combination, which is a parameter more comprehensively describing the intention change rate. When performing dimension combination processing of the statistical feature and the periodicity feature, the statistical feature (mean, variance, extreme value, etc.) and the periodicity feature (energy proportion of main frequency components) can be arranged in order into a vector. For example, assuming that the statistical feature vector is S=(s1, s2, s3), and the periodicity feature is p, then the combined evolution parameter vector E=(s1, s2, s3, p). Through this dimension combination processing, multiple features reflecting intention change are integrated together to form a more comprehensive evolution parameter, which can more accurately reflect the rate and rule of intention change.

[0126] Step S420: Association analysis processing of the evolution parameter and the cross-fragment semantic continuity degree in the semantic association feature set is performed to generate an intention stability evaluation vector.

[0127] The correlation analysis processing is to analyze the evolution parameters and the cross-fragment semantic continuity, and find the correlation between them. The cross-fragment semantic continuity is the degree of semantic continuity between different speech fragments, reflecting the semantic coherence of the speech content. The intention stability evaluation vector is obtained through the correlation analysis processing, and is used to evaluate the stability of the user interaction intention. When performing the correlation analysis processing on the evolution parameters and the cross-fragment semantic continuity, the evolution parameters and the cross-fragment semantic continuity can be normalized first to make them in the same numerical range. Then, the correlation between them is calculated, for example, using the Pearson correlation coefficient method. According to the result of the correlation, combined with the values of the evolution parameters and the cross-fragment semantic continuity, the intention stability evaluation vector is generated. If the evolution parameters indicate that the intention change rate is small and the cross-fragment semantic continuity is high, it means that the intention is relatively stable, and the corresponding element value in the intention stability evaluation vector may be larger; on the contrary, if the intention change rate is large and the cross-fragment semantic continuity is low, it means that the intention is unstable, and the corresponding element value in the intention stability evaluation vector may be smaller. By generating the intention stability evaluation vector, the stability of the user interaction intention can be more accurately evaluated, providing an important reference for determining the intention type.

[0128] Step S430: performing intention type matching processing based on the intention stability evaluation vector and the semantic core content in the semantic understanding result, to determine the overall user interaction intention type of the continuous speech data stream.

[0129] The intention type matching processing is to match the intention stability evaluation vector and the semantic core content in the semantic understanding result with the pre-set different user interaction intention types, and find the most suitable intention type. The overall user interaction intention type of the continuous speech data stream is determined through this matching processing, which is the main interaction goal that the user expects to achieve in the entire speech expression process. When performing the intention type matching processing based on the intention stability evaluation vector and the semantic core content in the semantic understanding result, a pre-set intention type library needs to be established first, which contains various common user interaction intention types and their characteristic descriptions. Then, the intention stability evaluation vector and the semantic core content are compared with each intention type in the intention type library. The similarity calculation method can be used, for example, to calculate the semantic similarity between the semantic core content and the description of each intention type, while considering the matching degree of the intention stability evaluation vector and the intention type stability requirement. According to the results of the similarity and the matching degree, the most matched intention type is selected as the overall user interaction intention type of the continuous speech data stream.

[0130] As an implementation manner, in step S430, the intention type matching processing is performed based on the intention stability evaluation vector and the semantic core content in the semantic understanding result, to determine the overall user interaction intention type of the continuous speech data stream, which can be implemented as the following steps S431-S435:

[0131] Step S431: Input the intent stability evaluation vector into a preset intent confidence calculation model. The model generates a confidence score for the user interaction intent type based on the correlation between the intent evolution parameters and the cross-segment semantic continuity.

[0132] The preset intent confidence calculation model is a pre-trained model used to calculate the confidence score of the user interaction intention type based on the intent stability evaluation vector and related associations. The correlation between the intent evolution parameter and the cross-segment semantic continuity is the correlation and mutual influence between the two obtained by analysis in the previous steps. The confidence score of the user interaction intention type is the credibility score of each user interaction intention type calculated by this model. When the intent stability evaluation vector is input into the preset intent confidence calculation model, the model will perform calculations based on its internal structure and trained parameters, combined with the correlation between the intent evolution parameter and the cross-segment semantic continuity. For example, the model can adopt a neural network structure, take the intent stability evaluation vector as the input of the input layer, perform feature transformation and calculation through the middle hidden layer, and finally output the confidence score of each user interaction intention type at the output layer. The higher the confidence score, the more closely the intent type matches the user's voice expression, and the higher the credibility.

[0133] Step S432: Perform keyword extraction processing on the semantic core content in the semantic understanding result to obtain a set of intention keywords that are strongly associated with the semantic core content.

[0134] Keyword extraction processing is the process of extracting keywords that best represent the meaning from the semantic core content in the semantic understanding results. The set of intent keywords that are strongly associated with the semantic core content is a set of keywords obtained through this extraction that are closely related to the user's interaction intention. When performing keyword extraction processing on the semantic core content in the semantic understanding results, a variety of methods can be used. For example, a statistical-based method can be used to count the frequency of occurrence of each word in the semantic core content, and select words with higher frequency of occurrence as keywords. Semantic-based methods can also be used, such as using a word vector model to calculate the semantic similarity between words, and find words with higher overall semantic similarity to the semantic core content as keywords. It is also possible to combine domain knowledge and grammatical rules to screen out important words related to intent. Through keyword extraction processing, a set of intent keywords is obtained, and these keywords can more accurately reflect the key information of the semantic core content.

[0135] Step S433: Calculate semantic similarity between the intent keyword set and the preset intent type tag library to generate an intent tag matching degree.

[0136] The preset intent type label library is predefined and contains various user interaction intent types and their corresponding label keywords. The semantic similarity calculation is a calculation of the semantic similarity between the keywords in the intent keyword set and the label keywords in the intent type label library. The intent label matching degree is a parameter obtained by the semantic similarity calculation, used to measure the matching degree of the intent keyword set and each intent type label library. When performing semantic similarity calculation on the intent keyword set and the preset intent type label library, methods such as word vector models can be used. For example, the intent keyword set and the label keywords in the intent type label library are both converted into word vector representations, and then the cosine similarity between them is calculated. For each intent type label library, the average cosine similarity between all keywords in the intent keyword set and all label keywords in the label library is calculated as the intent label matching degree. The higher the matching degree, the more matched the intent keyword set is to the intent type label library, and the more likely the intent type is the user's interaction intent type.

[0137] Step S434: fuse the confidence score and the intent label matching degree to generate a comprehensive matching index.

[0138] Fusing the confidence score and the intent label matching degree is to comprehensively process the two and generate a more comprehensive matching index by combining their information. The comprehensive matching index is obtained by such fusion and is used to more accurately evaluate the matching degree of each user interaction intent type and the user's speech expression. When fusing the confidence score and the intent label matching degree, a weighted summation method can be used. For example, different weights are assigned to the confidence score and the intent label matching degree, and then they are multiplied by their respective weights and added to obtain the comprehensive matching index. The allocation of weights can be adjusted according to actual conditions and experience. If the confidence score is considered more important, a larger weight can be assigned to it; if the intent label matching degree is considered more critical, a larger weight can be assigned to it. By generating the comprehensive matching index, the stability of the intent and the semantic matching degree can be more comprehensively considered, providing a more accurate basis for determining the user interaction intent type.

[0139] Step S435: select the intent type with the highest comprehensive matching index as the overall user interaction intent type of the continuous speech data stream.

[0140] After obtaining the comprehensive matching index of each user interaction intent type, the intent type with the highest comprehensive matching index is selected as the overall user interaction intent type of the continuous speech data stream by comparing the sizes of these indexes. The highest comprehensive matching index indicates that the intent type is most consistent with the user's speech expression in terms of intent stability and semantic matching degree, and therefore is most likely to be the user's actual interaction intent.

[0141] Step S440: Perform bimodal saliency joint analysis processing on the semantic core content in the semantic understanding result and the stress distribution pattern in the acoustic feature sequence, identify the overlapping region in the speech content that simultaneously satisfies the semantic concentration threshold and the acoustic saliency threshold, and generate a key information positioning feature.

[0142] The bimodal saliency joint analysis processing is to comprehensively analyze the information of the semantic core content in the semantic understanding result and the stress distribution pattern in the acoustic feature sequence, which are two different modalities, and consider their saliency features. The semantic concentration threshold is pre-set and used to determine the boundary of the semantic concentration degree in the semantic core content. The region exceeding the threshold is considered to have a high semantic concentration degree. The acoustic saliency threshold is pre-set and used to determine the boundary of the acoustic saliency in the stress distribution pattern. The region exceeding the threshold is considered to be more prominent in acoustics. The overlapping region is the intersection of the region whose semantic concentration degree exceeds the threshold in the semantic core content and the region whose acoustic saliency exceeds the threshold in the stress distribution pattern. The key information positioning feature is obtained through this joint analysis and used to identify the location of the key information related to the user interaction intent in the speech content. When performing bimodal saliency joint analysis processing on the semantic core content in the semantic understanding result and the stress distribution pattern in the acoustic feature sequence, first, perform saliency analysis on the semantic core content and the stress distribution pattern respectively. For the semantic core content, calculate the semantic concentration degree of each region and find the region exceeding the semantic concentration threshold. For the stress distribution pattern, calculate the acoustic saliency of each region and find the region exceeding the acoustic saliency threshold. Then, determine the overlapping part of the two regions. Finally, according to the position and range of the overlapping region, generate the key information positioning feature. This feature can be represented by parameters such as position coordinates and length, which can help accurately locate and extract the key information related to the user interaction intent in the speech content.

[0143] Step S500: Based on the user interaction intent type and the key information positioning feature, generate an LED display control instruction containing an interaction content identifier, and send the LED display control instruction to the target LED display screen to perform an interactive display operation.

[0144] When generating the LED display control instruction based on the user interaction intention type and the key information positioning feature, the appropriate interactive content needs to be selected according to the user's intention, and the display mode and time of these contents on the LED display screen are determined. The interactive content identifier is a label or number used to uniquely identify different interactive contents, which can help to accurately select and display the corresponding contents. The LED display control instruction is an instruction containing the interactive content identifier and the display control parameter, which can control the target LED display screen to perform interactive display operations as required. After sending the generated LED display control instruction to the target LED display screen, the display screen will execute the corresponding display operation according to the content of the instruction, realizing the interaction with the user.

[0145] As an implementation, step S500, based on the user interaction intention type and the key information positioning feature, an LED display control instruction containing an interactive content identifier is generated, which can be implemented as steps S510-S560:

[0146] Step S510: According to the user interaction intention type, query the preset interactive content knowledge graph to obtain the candidate content chain associated with the intention type, and the candidate content chain contains a plurality of content identifiers sorted by semantic correlation degree.

[0147] The preset interactive content knowledge graph is pre-constructed and contains the knowledge graph of the association relationship between various user interaction intention types and related interactive contents. It is represented in the form of a graph, where the nodes represent interactive intention types and interactive contents, and the edges represent the association relationship between them. The candidate content chain is a sequence of content identifiers related to the user interaction intention type obtained by querying the knowledge graph. The semantic correlation degree is the semantic relevance between the interactive content and the user interaction intention type, and the higher the correlation degree, the more matched the content and the intention. When querying the preset interactive content knowledge graph according to the user interaction intention type, find the node corresponding to the intention type in the knowledge graph, then find the interactive content nodes related to it along the associated edges, and extract the content identifiers corresponding to these nodes. Sort these content identifiers from high to low according to the semantic correlation degree to form the candidate content chain.

[0148] Step S520: Extract the time distribution feature of the overlapping area in the voice content in the key information positioning feature, and generate a time weight sequence of the information area.

[0149] The overlapping regions in the key information positioning feature are obtained by the bimodal saliency joint analysis in the previous step, which are the regions in the speech content that satisfy both the semantic concentration threshold and the acoustic saliency threshold. The time distribution feature is the time position and duration of these overlapping regions in the speech content. The time weight sequence of the information region is obtained by analyzing the time distribution feature, which is a weight sequence representing the importance of the information region at different time points. When extracting the time distribution feature of the overlapping regions in the key information positioning feature in the speech content, first, the start time and end time of the overlapping region in the speech content are determined. Then, the duration of each overlapping region is calculated. According to these time information, the time weight sequence of the information region is generated. For example, the time points corresponding to the overlapping regions with longer duration can be assigned higher weights, because these regions may contain more important information. By generating the time weight sequence, the time distribution of the key information in the speech content can be more accurately grasped, providing a basis for subsequent content display priority adjustment.

[0150] Step S533: Align the candidate content chain with the time weight sequence, and adjust the display priority of the content identifier in the candidate content chain according to the time order of the information region.

[0151] The alignment process is to correspond the content identifier in the candidate content chain with the time point in the time weight sequence, so that they are related to each other in time. The display priority is the order and importance of each content identifier in the candidate content chain displayed on the LED display screen. When adjusting the display priority of the content identifier in the candidate content chain according to the time order of the information region, first, the content identifier in the candidate content chain is matched with the time point in the time weight sequence. For example, if a content identifier is associated with a time point with a higher time weight, it means that the content corresponds to the time of the appearance of the key information in the speech content, so the display priority of the content identifier can be improved. Then, the candidate content chain is reordered according to the adjusted priority. In this way, when displaying the interactive content on the LED display screen, the content related to the key information can be displayed preferentially according to the time order of the appearance of the key information in the speech, improving the pertinence and effectiveness of the display.

[0152] Step S540: Redundant content filtering processing is performed on the adjusted candidate content chain, and the target content identifier with the highest semantic association degree with the key information positioning feature is retained.

[0153] The redundant content filtering process is to filter the content in the adjusted candidate content chain, and remove those with low semantic relevance to the key information positioning feature or repeated content. The target content with the highest semantic relevance to the key information positioning feature is obtained through this filtering, and the content that can most accurately reflect the user's interactive intention and key information. When filtering the adjusted candidate content chain, a semantic similarity calculation method can be used. Calculate the semantic similarity between the content corresponding to each content identifier and the key information identified by the key information positioning feature, and select the content identifier with high semantic similarity as the target content identifier. The content can also be filtered according to its importance and uniqueness, and repeated or unimportant content can be removed. Through the redundant content filtering process, the most relevant target content identifier is retained, unnecessary display content is reduced, and the accuracy and conciseness of the LED display screen display are improved.

[0154] Step S550: Bind and associate the target content identifier with the peak time point in the time weight sequence to generate display control parameters containing content display timing and key highlight positions.

[0155] The binding and association process is to combine the target content identifier with the peak time point in the time weight sequence, so that they are associated with each other. The peak time point is the time point with the highest weight value in the time weight sequence, which usually corresponds to the moment when the key information in the voice content is most prominent. The content display timing is the time sequence of the target content identifier displayed on the LED display screen, which can be determined according to the association relationship with the peak time point. The key highlight position is the part that needs to be highlighted in the displayed content, which can be determined according to the key information positioning feature. When generating display control parameters containing content display timing and key highlight positions, first, each target content identifier is bound to the corresponding peak time point. For example, if a target content identifier is associated with a peak time point, the content will be displayed on the LED display screen at the time corresponding to this time point. Then, determine the part of the content that needs to be highlighted according to the key information positioning feature. Finally, combine these information to form the display control parameters. These parameters can be represented by time stamps, position coordinates, font sizes, etc., which can accurately control the display method and time of the target content on the LED display screen.

[0156] Step S560: Encapsulate the display control parameters into LED display control instructions.

[0157] The encapsulation is to combine the display control parameters according to a certain format and protocol to form a complete instruction. The LED display control instruction is obtained by the encapsulation and is used for controlling the target LED display screen to perform the interactive display operation. When the display control parameters are encapsulated into the LED display control instruction, the control protocol and the interface specification of the LED display screen need to be followed. For example, the instruction can contain the identification of the display content, the starting time of the display, the duration of the display, the position of the display, the font size, the color and the like. The information is encoded according to the format specified in the protocol to form a binary or text format instruction. Finally, the encapsulated LED display control instruction is sent to the target LED display screen, and the display screen will perform the corresponding interactive display operation according to the content of the instruction to realize the voice interaction with the user.

[0158] It can be understood that the various algorithms involved in the above introduction of the embodiments of the present application, such as the Euclidean distance algorithm, the cosine distance algorithm, the Pearson correlation coefficient algorithm, the local maximum value detection algorithm and the like, can be known from the related content in the prior art. In order to save space, the above algorithms are not expanded in the embodiments of the present application. In addition, the person skilled in the art can supplement the details according to the common knowledge in the art when implementing the scheme of the present application. For example, according to the common knowledge in the art, the normalization can be used to eliminate the dimensional conflict before feature fusion, the interpolation can be used to eliminate the dimensional difference, the threshold can be reasonably set based on the historical data, experience or business scene demand, the model can be trained based on the general model training method, the number of layers in the model structure can be set based on the actual needs, the activation function can be selected, and the like. The present application does not introduce the redundant implementation process in too much detail.

[0159] Please refer to Figure 2 , Figure 2A structural schematic diagram of a computer system provided by the embodiment of the present application is provided, and the computer system at least includes a processor 101, a communication interface 102 and a memory 103. The processor 101, the communication interface 102 and the memory 103 can be connected through a bus or other manners. The processor 101 (or a central processing unit (CPU)) is a calculation core and a control core of the computer system, which can parse various instructions in the computer system and process various data of the computer system. The communication interface 102 can optionally include a standard wired interface, a wireless interface (such as WI-FI, a mobile communication interface and the like), and can be used for transmitting and receiving data under the control of the processor 101; the communication interface 102 can also be used for transmitting and interacting data in the computer system. The memory 103 is a memory device in the computer system, which is used for storing programs and data. It can be understood that the memory 103 can include a built-in memory of the computer system, and of course can also include an extended memory supported by the computer system. The memory 103 provides a storage space, and the storage space stores an operating system of the computer system, and the present application is not limited to this. In an embodiment, the processor 101 executes the computer program in the memory 103 to execute the LED display screen interaction method supporting voice interaction provided by the embodiment of the present application.

Claims

1. A method for LED display screen interaction supporting voice interaction, characterized in that: The method comprises: Acquire a continuous voice data stream sent by a user, wherein the continuous voice data stream includes a plurality of voice segments marked with time stamps; Performing speech feature extraction processing on the continuous speech data stream to obtain an acoustic feature sequence and a semantic association feature set of the speech segment; Calling a pre-trained speech semantic understanding model to perform joint semantic parsing processing on the acoustic feature sequence and the semantic association feature set to generate a semantic understanding result of the speech segment; Determine the user interaction intention type corresponding to the continuous voice data stream and the key information positioning features of the user interaction intention in the voice content according to the semantic understanding result; An LED display control instruction including an interactive content identifier is generated based on the user interaction intention type and the key information positioning feature, and the LED display control instruction is sent to a target LED display screen to perform an interactive display operation.

2. The LED display screen interactive method supporting voice interaction according to claim 1, characterized in that: The performing speech feature extraction processing on the continuous speech data stream to obtain an acoustic feature sequence and a semantic association feature set of the speech segment includes: Performing boundary detection processing on the continuous voice data stream, determining natural segmentation boundaries of the continuous voice data stream based on mutation points of voice energy and semantic pause marks, and segmenting the continuous voice data stream into voice segments with semantic integrity; Performing intent-guided acoustic feature focusing processing on each of the voice segments, extracting a subset of acoustic features in the voice segment that are strongly associated with the user's intent as an acoustic feature sequence, wherein the acoustic feature subset includes an accent distribution pattern and an intonation trajectory; Performing cross-segment semantic association modeling on each of the speech segments, contextually fusing the speech segment with the text conversion result of the preceding speech segment, and generating an association feature matrix containing semantic continuity; A semantic strength attenuation calculation is performed on the association feature matrix, and the weight value of the association feature is adjusted according to the time interval between the speech segments to generate the semantic association feature set, which includes the semantic continuity across segments and the semantic concentration within segments.

3. The LED display screen interactive method supporting voice interaction according to claim 2, characterized in that: The performing intent-oriented acoustic feature focusing processing on each of the voice segments and extracting a subset of acoustic features in the voice segment that are strongly associated with the user's intent as an acoustic feature sequence includes: Performing multi-resolution time-frequency analysis on the speech segment to generate a set of time-frequency spectrograms with different time resolutions; Extracting energy concentration points of the accent area from the set of time-spectrograms, and calculating the position distribution regularity of the energy concentration points in the speech segment to generate the accent distribution pattern; Performing trend analysis on the fundamental frequency curve of the speech segment, identifying rising inflection points and falling inflection points in the fundamental frequency curve, calculating the slope change rate between adjacent inflection points, and generating the intonation rise and fall trajectory; Based on the stress distribution pattern and the intonation trajectory, the pre-trained intent-acoustic feature mapping relationship is used to filter out parts that overlap with common speech features, retain feature components that are strongly associated with the target interaction intent, and form the acoustic feature sequence; The performing semantic strength attenuation calculation on the correlation feature matrix, adjusting the weight values ​​of the correlation features according to the time intervals between the speech segments, and generating the semantic correlation feature set includes: Calculate the time interval between the current speech segment and the previous speech segment to generate a time interval feature value; Querying a preset attenuation function table according to the time interval characteristic value to obtain a corresponding weight attenuation coefficient, wherein the attenuation function table defines a negative correlation between the time interval length and the weight attenuation coefficient; Applying the weight attenuation coefficient to each associated feature dimension in the associated feature matrix to generate an attenuated associated feature vector; Performing semantic concentration calculation on the attenuated associated feature vector, counting the distribution density of high-weight dimensions in the associated feature vector, and generating the semantic concentration within the segment; Performing a cross-segment continuity evaluation process on the attenuated associated feature vector, comparing the high-weight dimension overlap between the current speech segment and the previous speech segment, and generating the cross-segment semantic continuity; The intra-segment semantic concentration and the inter-segment semantic continuity are arranged in the time sequence of the speech segments to form the semantic association feature set.

4. The LED display screen interactive method supporting voice interaction according to claim 1, characterized in that: The calling of the pre-trained speech semantic understanding model to perform joint semantic parsing processing on the acoustic feature sequence and the semantic association feature set to generate a semantic understanding result of the speech segment includes: Inputting the acoustic feature sequence and the semantic association feature set into the context fusion module of the speech semantic understanding model, determining the context window length based on the cross-segment semantic continuity in the semantic association feature set, and generating an extended feature sequence containing historical segment association information; Performing time dependency modeling on the extended feature sequence through the intention evolution modeling layer of the speech semantic understanding model to capture the intention change trend between consecutive speech segments and generate an intention evolution feature vector; Performing expression standardization correction processing on the intention evolution feature vector through the spoken language interference calibration module of the speech semantic understanding model, eliminating non-standard expression interference in the speech content, and generating calibrated semantic core features; The calibrated semantic core features are bound and associated with the timestamp information of the voice segment to generate a semantic understanding result including the intention evolution trajectory and the semantic core content.

5. The LED display screen interactive method supporting voice interaction according to claim 4, characterized in that: The step of inputting the acoustic feature sequence and the semantic association feature set into the context fusion module of the speech semantic understanding model, determining the context window length based on the cross-segment semantic continuity in the semantic association feature set, and generating an extended feature sequence containing historical segment association information includes: Extracting the cross-segment semantic continuity between the current speech segment and the historical speech segment from the semantic association feature set, wherein the cross-segment semantic continuity reflects the strength of the semantic association between the historical segment and the current segment; Selecting historical speech segments that are semantically associated with the current segment based on the strength of the cross-segment semantic continuity, wherein the stronger the continuity, the more associated historical segments are selected, and the weaker the continuity, the fewer associated historical segments are selected; The acoustic feature sequence and semantic association feature set of the selected historical speech segment are cascade-concatenated with the corresponding features of the current speech segment to generate a cross-segment feature concatenation vector; The cross-segment feature concatenation vector is subjected to feature dimension alignment processing to unify the feature representation dimensions of speech segments with different timestamps, thereby generating the extended feature sequence containing historical segment association information.

6. The LED display screen interactive method supporting voice interaction according to claim 4, characterized in that: The intention evolution modeling layer of the speech semantic understanding model performs time dependency modeling on the extended feature sequence, captures the intention change trend between consecutive speech segments, and generates an intention evolution feature vector, including: Inputting the extended feature sequence into the time gating unit of the intention evolution modeling layer, the time gating unit generating forget gate and update gate parameters based on the semantic association strength between the historical segments and the current segment in the extended feature sequence; Filtering historical feature information irrelevant to the current intent in the extended feature sequence through the forget gate parameter, and retaining key feature components related to the intent evolution; The core features of the current speech segment and the retained historical key feature components are weightedly fused by the update gate parameters to generate an intention state update vector; Performing trend analysis on the intention state update vector to identify the change pattern of the intention category between consecutive speech segments and generate an evolution trend feature reflecting the direction of the intention change; The intention state update vector and the evolution trend feature are dimensionally spliced ​​to form the intention evolution feature vector.

7. The LED display screen interactive method supporting voice interaction according to claim 1, characterized in that: The determining, based on the semantic understanding result, the type of user interaction intention corresponding to the continuous voice data stream and key information positioning features of the user interaction intention in the voice content includes: Extracting time series features of the intent evolution trajectory in the semantic understanding result to generate an evolution parameter reflecting the rate of change of the intent; Performing correlation analysis on the evolution parameter and the cross-segment semantic continuity in the semantic association feature set to generate an intention stability evaluation vector; Performing intent type matching processing based on the intent stability evaluation vector and the semantic core content in the semantic understanding result to determine the overall user interaction intent type of the continuous voice data stream; A bimodal saliency joint analysis is performed on the semantic core content in the semantic understanding result and the accent distribution pattern in the acoustic feature sequence to identify overlapping areas in the speech content that simultaneously meet the semantic concentration threshold and the acoustic saliency threshold, thereby generating the key information positioning feature.

8. The LED display screen interactive method supporting voice interaction according to claim 7, characterized in that: The performing intent type matching processing based on the intent stability evaluation vector and the semantic core content in the semantic understanding result to determine the overall user interaction intent type of the continuous voice data stream includes: Inputting the intention stability evaluation vector into a preset intention confidence calculation model, the model generates a confidence score for the user interaction intention type based on the correlation between the intention evolution parameter and the cross-segment semantic continuity; Performing keyword extraction processing on the semantic core content in the semantic understanding result to obtain a set of intention keywords that are strongly associated with the semantic core content; Calculate the semantic similarity between the intent keyword set and the preset intent type tag library to generate the intent tag matching degree; Combining the confidence score with the intent label matching degree to generate a comprehensive matching index; The intention type with the highest comprehensive matching index is selected as the overall user interaction intention type of the continuous voice data stream.

9. The LED display screen interactive method supporting voice interaction according to claim 1, characterized in that: The generating of the LED display control instruction including the interactive content identifier based on the user interaction intention type and the key information positioning feature includes: Querying a preset interactive content knowledge graph according to the user interaction intention type to obtain a candidate content chain associated with the intention type, wherein the candidate content chain includes multiple content identifiers sorted by semantic relevance; Extracting the time distribution features of the overlapping areas in the key information positioning features in the speech content to generate a time weight sequence of the information areas; Aligning the candidate content chain with the time weight sequence, and adjusting the display priority of the content identifiers in the candidate content chain according to the time sequence of appearance of the information regions; Performing redundant content filtering on the adjusted candidate content chain, and retaining the target content identifier with the highest semantic relevance to the key information positioning feature; Binding and associating the target content identifier with the peak time point in the time weight sequence to generate display control parameters including content display timing and key highlight positions; The display control parameters are encapsulated as the LED display control instructions.

10. A computer system, characterized in that: include: a memory storing a computer program; A processor, configured to load the computer program to implement the LED display screen interactive method supporting voice interaction as described in any one of claims 1 to 9.

Images