Streaming processing method and device of voice dialogue, electronic equipment and storage medium

Abstract

The invention discloses a streaming processing method and device for voice dialogues, electronic equipment and a storage medium, and relates to the technical field of audios and videos or other related technical fields, and the streaming processing method for the voice dialogues comprises the steps: receiving continuous voice streams, and obtaining historical dialogue texts corresponding to the voice streams; the voice stream and the historical dialogue text are input into a streaming processing model, a streaming voice fusion feature vector is output, and the streaming processing model carries out feature extraction on the voice stream based on a state space model and a dynamic selection mechanism; inputting the streaming voice fusion feature vector into a language processing model, and outputting a response text; and generating a response audio based on the response text, and returning the response audio to the user side. According to the invention, the technical problem of low voice conversation processing efficiency under a cascade architecture in the prior art is solved.

Description

Technical Field

[0001] The present invention relates to the field of audio and video technology or other related technical fields, and in particular to a method for streaming voice conversations, a device thereof, an electronic device, and a storage medium. Background Art

[0002] As a key achievement in the field of artificial intelligence, voice dialogue systems have rapidly emerged in recent years in a variety of scenarios, including intelligent customer service, smart homes, and in-vehicle intelligent systems, demonstrating enormous application potential and market value. In the intelligent customer service sector, voice dialogue systems not only provide 24 / 7 uninterrupted service but also simulate human-like communication to enhance customer satisfaction. Leveraging deep learning and natural language processing technologies, these systems can understand customer verbal expressions, quickly identify issues, and provide precise feedback or solutions, reducing the burden on human customer service representatives while significantly shortening response times and resolution cycles. The smart home industry is one of the most widely used areas for voice dialogue systems. By integrating voice recognition and voice control capabilities, users can control various home devices, such as lighting, temperature, and music, with simple verbal commands, eliminating the need for complex user interfaces, enabling true "smart living." Voice dialogue systems in in-vehicle intelligent systems not only provide navigation and entertainment functions but also enhance driving safety and comfort. With the rapid development of artificial intelligence and deep learning, voice dialogue systems are moving towards greater intelligence, efficiency, and user-friendliness. Low-latency, high-accuracy processing capabilities, as well as support for complex scenarios and diverse languages, will become the core competitiveness of future voice dialogue systems.

[0003] In the financial sector, due to the large customer base and high business complexity, the introduction of voice dialogue systems can effectively solve user problems and improve business processing efficiency. Related technologies generally use a cascaded architecture to build voice dialogue systems. These systems require waiting for complete voice input before analyzing and responding. This results in long input delays and response times, making it difficult to quickly respond to user-side voice dialogues and resulting in a poor communication experience. Furthermore, in cascaded voice dialogue systems, there is accumulated delay and error transmission between multiple modules, resulting in low accuracy in the final output, making it difficult to meet user-side conversational needs.

[0004] To address the above-mentioned problems, no effective solutions have been proposed so far. Summary of the Invention

[0005] The embodiments of the present invention provide a method for streaming voice conversation processing, an apparatus thereof, an electronic device, and a storage medium, so as to at least solve the technical problem of low voice conversation processing efficiency under a cascade architecture in the related art.

[0006] According to one aspect of an embodiment of the present invention, a streaming processing method for voice conversations is provided, comprising: receiving a continuous voice stream and obtaining a historical conversation text corresponding to the voice stream, wherein the voice stream is real-time voice data based on financial services; inputting the voice stream and the historical conversation text into a streaming processing model, and outputting a streaming voice fusion feature vector, wherein the streaming processing model performs feature extraction on the voice stream based on a state space model and a dynamic selection mechanism, and the dynamic selection mechanism is used to dynamically select model parameters of the state space model; inputting the streaming voice fusion feature vector into a language processing model, and outputting a response text, wherein the language processing model is a pre-built model for making response decisions based on text data; generating a response audio based on the response text, and returning the response audio to a user terminal.

[0007] Furthermore, the step of inputting the voice stream and the historical conversation text into the streaming processing model and outputting the streaming voice fusion feature vector includes: inputting the voice stream into the streaming processing model, compressing and segmenting the voice stream through the streaming processing model to obtain N voice feature segments, where N is a positive integer; performing feature extraction on the N voice feature segments based on the state space model in the streaming processing model to obtain a voice feature vector; preprocessing the historical conversation text to obtain a subword sequence vector; inputting the subword sequence vector into the streaming processing model to obtain a text feature vector; performing feature fusion on the voice feature vector and the text feature vector, and outputting the streaming voice fusion feature vector.

[0008] Furthermore, the step of compressing and segmenting the voice stream through the streaming processing model includes: compressing the voice stream based on the one-dimensional convolutional layer of the streaming processing model to obtain frame-level voice features; calculating the short-time energy value and zero-crossing rate of the frame-level voice features, wherein the short-time energy value is used to measure the local energy intensity of the voice signal, and the zero-crossing rate is a measure of the oscillation property of the voice signal; detecting frame-level active voice features based on the short-time energy value and the zero-crossing rate, and merging continuous frame-level voice features to obtain initial voice feature segments; for each of the initial voice feature segments, predicting the silence position in the initial voice feature segment, and performing secondary segmentation on the initial voice feature segment based on the silence position to obtain the voice feature segment.

[0009] Furthermore, the step of extracting features from N of the speech feature segments based on the state space model in the streaming processing model to obtain a speech feature vector includes: inputting the N speech feature segments into the state space model, and determining the time step parameter of the state space model when processing the speech feature segment based on the energy level of each speech feature segment, wherein the energy level represents the total energy carried by the speech signal corresponding to the speech feature segment; when processing each of the speech feature segments, dynamically configuring the time step parameter for the state space model and configuring the initial hidden state for the state space model, wherein after processing each of the speech feature segments, retaining the hidden state of the state space model as the initial hidden state for processing the next speech feature segment, and the hidden state contains the temporal features of the speech signal; based on the initial hidden state, extracting the temporal dependency features of the speech feature segment through the state space model to obtain the speech feature vector.

[0010] Furthermore, the step of preprocessing the historical conversation text to obtain a subword sequence vector includes: using a word segmenter to convert the historical conversation text into a subword sequence; and performing embedding processing on the subword sequence to obtain the subword sequence vector.

[0011] Furthermore, the step of inputting the subword sequence vector into the streaming processing model to obtain the text feature vector includes: inputting the subword sequence vector into the state space model in the streaming processing model; for each subword sequence vector, configuring the initial hidden state of the state space model, wherein the state space model updates the hidden state based on the initial hidden state using a state transfer matrix and a projection matrix, and after updating the hidden state after processing the subword sequence vector, the updated hidden state is used as the initial hidden state for processing the next subword sequence vector, the state transfer matrix is ​​dynamically adjusted, and the projection matrix is ​​obtained after mapping the subword sequence vector to the hidden state space; performing a linear transformation on the hidden state updated based on the last subword sequence vector to obtain the text feature vector.

[0012] Furthermore, the steps of performing feature fusion on the speech feature vector and the text feature vector and outputting the streaming speech fusion feature vector include: converting the speech feature vector and the text feature vector into feature vectors of the same latitude to obtain the converted speech feature vector and the text feature vector; configuring gating weights based on a gated attention mechanism; and performing weighted summation on the converted speech feature vector and the text feature vector based on the gating weights to obtain a streaming speech fusion feature vector.

[0013] According to another aspect of an embodiment of the present invention, a streaming processing device for voice conversations is also provided, including: a receiving unit, configured to receive a continuous voice stream and obtain a historical conversation text corresponding to the voice stream, wherein the voice stream is real-time voice data based on financial services; a first output unit, configured to input the voice stream and the historical conversation text into a streaming processing model, and output a streaming voice fusion feature vector, wherein the streaming processing model performs feature extraction on the voice stream based on a state space model and a dynamic selection mechanism, and the dynamic selection mechanism is configured to dynamically select model parameters of the state space model; a second output unit, configured to input the streaming voice fusion feature vector into a language processing model, and output a response text, wherein the language processing model is a pre-built model for making response decisions based on text data; and a generation unit, configured to generate response audio based on the response text, and return the response audio to a user terminal.

[0014] Furthermore, the first output unit includes: a first processing module, used to input the voice stream into the streaming processing model, compress and segment the voice stream through the streaming processing model to obtain N voice feature segments, where N is a positive integer; a first extraction module, used to extract features from the N voice feature segments based on the state space model in the streaming processing model to obtain a voice feature vector; a second processing module, used to preprocess the historical conversation text to obtain a subword sequence vector; a first input module, used to input the subword sequence vector into the streaming processing model to obtain a text feature vector; a first fusion module, used to perform feature fusion on the voice feature vector and the text feature vector, and output the streaming voice fusion feature vector.

[0015] Furthermore, the first processing module includes: a first compression submodule, used to compress the voice stream based on the one-dimensional convolutional layer of the streaming processing model to obtain frame-level voice features; a first calculation submodule, used to calculate the short-time energy value and zero-crossing rate of the frame-level voice features, wherein the short-time energy value is used to measure the local energy intensity of the voice signal, and the zero-crossing rate is a measure of the oscillation property of the voice signal; a first merging submodule, used to detect frame-level active voice features based on the short-time energy value and the zero-crossing rate, and merge continuous frame-level voice features to obtain initial voice feature segments; a first segmentation submodule, used to predict the silence position in the initial voice feature segment for each of the initial voice feature segments, and perform secondary segmentation on the initial voice feature segment based on the silence position to obtain the voice feature segment.

[0016] Furthermore, the first extraction module includes: a first determination submodule, used to input N of the speech feature segments into the state space model, and determine the time step parameter of the state space model when processing the speech feature segment based on the energy level of each speech feature segment, wherein the energy level represents the total energy carried by the speech signal corresponding to the speech feature segment; a first configuration submodule, used to dynamically configure the time step parameter for the state space model when processing each of the speech feature segments, and configure the initial hidden state for the state space model, wherein after processing each of the speech feature segments, the hidden state of the state space model is retained as the initial hidden state for processing the next speech feature segment, and the hidden state contains the temporal features of the speech signal; a first extraction submodule, used to extract the temporal dependency features of the speech feature segment through the state space model based on the initial hidden state to obtain the speech feature vector.

[0017] Furthermore, the second processing module includes: a first conversion submodule, used to use a word segmenter to convert the historical conversation text into a subword sequence; and a first processing submodule, used to perform embedding processing on the subword sequence to obtain the subword sequence vector.

[0018] Furthermore, the first input module includes: a first input submodule, used to input the subword sequence vector into the state space model in the streaming processing model; a second configuration submodule, used to configure the initial hidden state of the state space model for each subword sequence vector, wherein the state space model updates the hidden state based on the initial hidden state using a state transfer matrix and a projection matrix, and after updating the hidden state for processing the subword sequence vector, the updated hidden state is used as the initial hidden state for processing the next subword sequence vector, the state transfer matrix is ​​dynamically adjusted, and the projection matrix is ​​obtained after mapping the subword sequence vector to the hidden state space; a first acquisition submodule, used to perform a linear transformation on the hidden state updated based on the last subword sequence vector to obtain the text feature vector.

[0019] Furthermore, the first fusion module includes: a first conversion submodule, used to convert the speech feature vector and the text feature vector into feature vectors of the same latitude, to obtain the converted speech feature vector and the text feature vector; a third configuration submodule, used to configure the gating weight based on the gated attention mechanism; a second acquisition submodule, used to perform weighted summation of the converted speech feature vector and the text feature vector based on the gating weight to obtain a streaming speech fusion feature vector.

[0020] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is further provided, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute any of the above-mentioned streaming processing methods for voice conversations.

[0021] According to another aspect of an embodiment of the present invention, an electronic device is also provided, comprising one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement any one of the above-mentioned streaming processing methods for voice conversations.

[0022] In this application, the following steps are taken: receiving a continuous voice stream and obtaining the historical conversation text corresponding to the voice stream, wherein the voice stream is real-time voice data based on financial business; secondly, inputting the voice stream and the historical conversation text into a streaming processing model, and outputting a streaming voice fusion feature vector; wherein the streaming processing model extracts features from the voice stream based on a state space model and a dynamic selection mechanism, and the dynamic selection mechanism is used to dynamically select model parameters of the state space model; then, inputting the streaming voice fusion feature vector into a language processing model, and outputting a response text; wherein the language processing model is a pre-built model for making response decisions based on text data; finally, generating a response audio based on the response text, and returning the response audio to the user end.

[0023] In this application, real-time voice streams and corresponding historical conversation texts are received, and a pre-built streaming processing model is used to analyze the voice streams and text data. The model is based on a state-space model and a dynamic selectivity mechanism, and can efficiently extract voice features and text features under linear time complexity. There is no need to wait for complete voice data, and the voice stream can be processed in real time to achieve the purpose of quickly responding to voice conversations. This achieves the technical effect of reducing the delay time for waiting for data transmission and improving the efficiency of voice data processing, thereby solving the technical problem of low voice conversation processing efficiency under the cascade architecture in related technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0025] Figure 1 A hardware structure block diagram of a computer terminal (or mobile device) for implementing a streaming processing method for voice conversations is shown;

[0026] Figure 2is a flow chart of an optional method for streaming voice conversations according to an embodiment of the present invention;

[0027] Figure 3 is a schematic diagram of an optional flow of streaming processing of a voice conversation according to an embodiment of the present invention;

[0028] Figure 4 is a schematic diagram of an optional streaming processing device for voice conversations according to an embodiment of the present invention;

[0029] Figure 5 The present invention is a block diagram of the hardware structure of an electronic device (or mobile device) that optionally performs a streaming processing method for voice conversations according to an embodiment of the present invention. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0031] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0032] It should be noted that the streaming processing method and device for voice conversations in this application can be used in the field of audio and video technology when real-time voice streams are processed and responded to based on artificial intelligence, and can also be used in any field other than the field of audio and video technology when real-time voice streams are processed and responded to based on artificial intelligence. The application does not limit the application field of the streaming processing method and device for voice conversations in this application.

[0033] It should be noted that the collected information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in this application are information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation portals for users to choose to authorize or refuse. For example, an interface is set up between this system and relevant users or institutions to provide users with corresponding operation portals for users to choose to agree or refuse the automated decision-making results; if the user chooses to refuse, the expert decision-making process will be entered.

[0034] The following embodiments of the present invention can be applied to various streaming processing systems / applications / devices for voice conversations. The present invention processes continuous voice streams through a streaming processing model, supports streaming processing, and does not need to wait for complete voice data. It can effectively reduce data transmission delays, improve the efficiency of data voice conversation response, and enhance user experience. By utilizing the selective state space model of the streaming processing model, the key acoustic features and semantic dependencies in the voice stream are dynamically captured, and redundant information interference is suppressed. At the same time, combined with historical conversation text data, it can avoid the problem of context forgetting, support semantic coherence analysis, improve the accuracy of output results, and better meet the conversation needs of the user end.

[0035] The present invention will be described in detail below with reference to various embodiments.

[0036] Example 1

[0037] According to an embodiment of the present invention, an embodiment of a method for streaming processing of voice conversations is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0038] The method embodiment provided in the first embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 The hardware structure block diagram of a computer terminal (or mobile device) for implementing a streaming processing method for voice conversation is shown. Figure 1As shown, the computer terminal 10 (or mobile device) may include one or more (illustrated as 102a, 102b, ..., 102n in the figure) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0039] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry". The data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuitry may be a single independent processing module, or may be incorporated in whole or in part into any of the other components of the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuitry serves as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0040] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the method for streaming voice conversations in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, implementing the above-mentioned method for streaming voice conversations. The memory 104 may include a high-speed random access memory and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0041] The transmission device 106 is configured to receive or transmit data via a network. A specific example of the aforementioned network may include a wireless network provided by the communications provider of the computer terminal 10. In one embodiment, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission device 106 may be a radio frequency (RF) module, which is configured to communicate with the Internet wirelessly.

[0042] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0043] Under the above operating environment, this application provides Figure 2 The streaming processing method of voice dialogue shown in the figure is implemented by a streaming processing system of voice dialogue.

[0044] Figure 2 is a flow chart of an optional method for streaming voice conversation according to an embodiment of the present invention, such as Figure 2 As shown, the method includes the following steps:

[0045] Step S201: Receive a continuous voice stream and obtain a historical conversation text corresponding to the voice stream.

[0046] In the above steps, the voice data transmitted from the user end is first received. The voice data is a continuous voice stream. There is no need to wait for the complete voice data, and the voice can be streamed immediately. These voice streams come from the real-time input of the user end. The voice streams are real-time voice data based on financial services. In the financial business scenario, the voice streams may contain inquiries about product information, account operation requests or customer service feedback.

[0047] At the same time, the voice conversation streaming system retrieves historical conversation text associated with the current voice stream from a database based on user information and historical conversation records. This text records all conversations between the user and the system over the historical period, including but not limited to user questions, system responses, and any relevant account information or operation records. Historical conversation text is crucial for understanding the user's current needs because it provides essential context, helping the system better understand the conversation flow and provide personalized and contextual responses.

[0048] In these steps, the system seamlessly processes continuous speech streams and generates timely responses without waiting for complete input, reducing interaction latency and improving the user experience. Tight integration with historical conversational text enables the system to maintain conversational coherence, preventing loss of key information even in multi-turn, long-term conversations, thereby enhancing the depth and breadth of conversational understanding.

[0049] Step S202: Input the speech stream and the historical conversation text into the streaming processing model, and output the streaming speech fusion feature vector.

[0050] In step S202, the received continuous speech stream and the acquired historical conversation text are combined as input data and directed to the streaming processing model. The streaming processing model simultaneously analyzes the speech and text data, identifying speech and text features to generate a streaming speech fusion feature vector. The streaming processing model extracts features from the speech stream based on a state-space model and a dynamic selection mechanism. The dynamic selection mechanism is used to dynamically select the model parameters of the state-space model. The streaming processing model can process real-time and uninterrupted speech data. Unlike traditional processing methods, it can begin processing data as it arrives, without waiting for the complete speech data to be transmitted. The core of the streaming processing model is the state-space model. The state-space model effectively models the sequence dependencies of speech signals. Through recursive computation and linear complexity, it processes continuous input data in linear time, avoiding computational resource bottlenecks that may occur when processing long speech sequences. Furthermore, the state-space model can control the rate and attenuation of information by adjusting the parameters of the state transition matrix, input matrix, and output matrix. This is crucial for maintaining contextual coherence during multi-turn conversations.

[0051] When processing speech streams, the streaming model dynamically adjusts the model parameter Δ based on a dynamic selection mechanism, achieving refined control over the input. The Δ parameter adaptively adjusts the speed of state updates based on the characteristics of the current input (such as the energy level and signal-to-noise ratio of the speech segment), ensuring that even in noisy environments, the model remains focused on valid speech information while ignoring background noise and other irrelevant signals. This mechanism improves the model's robustness and adaptability to environmental changes, and performs particularly well under complex and uncertain input conditions.

[0052] After processing by the streaming model, information from the speech stream and historical conversation text is integrated to form a streaming speech fusion feature vector. This feature vector not only contains the acoustic features of the current speech input but also incorporates the textual context of historical conversations, providing rich contextual information for subsequent language processing tasks. The generation of feature vectors is a continuous process. Whenever a new speech stream arrives, the model updates the vector in real time, ensuring timely and consistent dialogue understanding and response generation.

[0053] Furthermore, the steps of inputting the speech stream and the historical conversation text into the streaming processing model and outputting the streaming speech fusion feature vector include: inputting the speech stream into the streaming processing model, compressing and segmenting the speech stream through the streaming processing model to obtain N speech feature segments, where N is a positive integer; performing feature extraction on the N speech feature segments based on the state space model in the streaming processing model to obtain a speech feature vector; preprocessing the historical conversation text to obtain a subword sequence vector; inputting the subword sequence vector into the streaming processing model to obtain a text feature vector; performing feature fusion on the speech feature vector and the text feature vector, and outputting the streaming speech fusion feature vector.

[0054] Specifically, after the speech stream and historical conversation text are input into the streaming processing model, they are first compressed and segmented by the streaming speech model to obtain multiple speech feature segments. Feature extraction is then performed on each speech feature segment using the state-space model within the streaming processing model. The state-space model efficiently processes speech sequences with linear complexity, maintaining fast and robust feature extraction even for long periods of speech input. Each segment is converted into a compact speech feature vector that retains key information about the speech signal, such as intonation, speech rate, and pronunciation characteristics, while eliminating redundancy and background noise.

[0055] In another optional embodiment, after obtaining multiple speech feature segments, N computing tasks are constructed based on the N speech feature segments and sent to the state space model for parallel processing to improve the processing efficiency of the speech stream.

[0056] At the same time, historical conversation texts undergo a series of preprocessing steps, including word segmentation and embedding, to be converted into machine-readable subword sequence vectors. These subword sequence vectors are then fed into a streaming model for feature extraction, generating text feature vectors that reflect the content of the historical conversation texts. This process helps the system understand the background and context of the conversation, as well as the context of the conversation.

[0057] Furthermore, the steps of compressing and segmenting the speech stream through the streaming processing model include: compressing the speech stream based on the one-dimensional convolutional layer of the streaming processing model to obtain frame-level speech features; calculating the short-time energy value and zero-crossing rate of the frame-level speech features, wherein the short-time energy value is used to measure the local energy intensity of the speech signal, and the zero-crossing rate is an indicator of the oscillation property of the speech signal; detecting the frame-level active speech features based on the short-time energy value and the zero-crossing rate, and merging the continuous frame-level speech features to obtain initial speech feature segments; for each initial speech feature segment, predicting the silence position in the initial speech feature segment, and performing secondary segmentation on the initial speech feature segment based on the silence position to obtain speech feature segments.

[0058] Specifically, compressing and segmenting the speech stream can reduce redundant computations and improve processing speed, thereby ensuring that the system can respond to users in real time. During compression and segmentation, the audio signal is fed from its original waveform form into the one-dimensional convolutional layer of the streaming processing model. The one-dimensional convolutional layer converts the continuous, high-resolution audio signal into a series of frame-level speech features, with each frame representing the signal characteristics over a short period of time. For each input sequence, the one-dimensional convolutional layer produces multiple feature maps, each corresponding to a specific convolution kernel. These feature maps can be understood as "filtering" the input signal from different angles or feature spaces, with each channel capturing local patterns or characteristics of the input signal. The one-dimensional convolutional layer captures the time and frequency dependencies of the audio signal. The convolution kernel in the time dimension extracts temporal features of speech, such as syllable duration and speech rhythm; the features in the frequency dimension (after the input is converted to a spectral representation) reflect the pitch and timbre of the sound. Thus, the one-dimensional convolutional layer compresses the original speech stream into frame-by-frame speech feature segments containing key features.

[0059] Secondly, for each frame's frame-level speech features, the short-time energy value and zero-crossing rate are calculated. The short-time energy value is a measure of the local strength of the speech signal. It is achieved by calculating the average energy of the sample points in the frame-level speech features and reflects the loudness of the sound. The zero-crossing rate is a measure of the oscillation nature of the speech signal. It is achieved by calculating the ratio of the number of zero-crossing events in the frame-level speech features. For the signal within each frame, the sign change of adjacent sampling points is calculated. In other words, if the value of the current sampling point multiplied by the value of the previous sampling point is a negative number, it is considered that a zero-crossing event has occurred. The zero-crossing rate can help identify silent or noisy segments in speech. The combination of these two indicators enables the system to accurately distinguish speech from non-speech components in the signal.

[0060] Using short-term energy values ​​and zero-crossing rates, the system can detect which frames are true speech segments (active speech features) and ignore or reduce processing of silent segments. For consecutive active speech features, the system merges them into a single initial speech feature segment. This process helps reduce computational redundancy, improves the model's responsiveness to speech content, and provides more compact and focused speech data for subsequent in-depth processing.

[0061] For each initial speech feature segment, the streaming model further predicts the locations of silence within the segment. This prediction is based on an in-depth analysis of the audio signal, allowing a trained lightweight neural network model to identify pauses between phonemes. Based on these silence locations, the initial speech feature segment is further segmented to produce more refined speech feature segments. This step helps ensure that each segment contains complete phonemes or words, avoiding the semantic fragmentation that can occur with traditional fixed segmentation strategies and enhancing the model's ability to capture semantic units.

[0062] In the above steps, one-dimensional convolution compression and the detection and merging of active speech features reduce redundant computations, improve processing speed, and ensure the system's real-time response to users. Furthermore, by predicting silence locations and performing secondary segmentation, semantic fragmentation is avoided and the model's ability to recognize complete words and phoneme units is enhanced.

[0063] Furthermore, the steps of extracting features from N speech feature segments based on the state space model in the streaming processing model to obtain a speech feature vector include: inputting N speech feature segments into the state space model, and determining the time step parameter of the state space model when processing the speech feature segment based on the energy level of each speech feature segment, wherein the energy level represents the total energy carried by the speech signal corresponding to the speech feature segment; when processing each speech feature segment, dynamically configuring the time step parameter for the state space model, and configuring the initial hidden state for the state space model, wherein after processing each speech feature segment, retaining the hidden state of the state space model as the initial hidden state for processing the next speech feature segment, and the hidden state contains the temporal features of the speech signal; based on the initial hidden state, extracting the temporal dependency features of the speech feature segment through the state space model to obtain a speech feature vector.

[0064] Specifically, during feature extraction, for each speech feature segment, the time step parameter of the state-space model is first determined based on the segment's energy level. The energy level is a quantitative indicator of the energy carried by the speech signal corresponding to the speech feature segment, reflecting the signal's strength. By associating the energy level with the time step parameter, the system can flexibly adjust the model's computational pace based on the activity of different segments. For example, for higher-energy speech segments, the time step parameter is smaller, and the system will process data at more intensive time points to capture more details. For lower-energy segments, a larger time step parameter is used to reduce unnecessary computation while maintaining reasonable sensitivity to background noise.

[0065] Next, when each speech feature segment enters the state-space model, the system configures an initial hidden state for it. This initial hidden state is based on the hidden state retained after processing the previous segment. This ensures that the model inherits the previous temporal features when processing consecutive segments, achieving continuous information transmission and memory. This mechanism is crucial for capturing coherence in conversations, especially in complex business scenarios that require understanding context and language. It enables the model to better track and utilize conversation history.

[0066] Based on the initial hidden state described above, the state-space model begins processing the current speech feature segment, recursively calculating the hidden state to extract the temporal dependency features within the segment. Here, the state-space model utilizes its unique state transition matrix A, input matrix B, and output matrix C, as well as a dynamically generated time step parameter Δ, to capture the temporal relationships within and between speech feature segments, forming a comprehensive speech feature vector. This vector not only condenses the essential properties of the speech signal but also contains rich contextual information, providing strong support for subsequent cross-modal fusion and intent analysis.

[0067] Through the above steps, the time step parameters are dynamically configured based on the segment energy level, enabling the model to flexibly respond to speech signals of varying intensities, improving computational efficiency while ensuring information integrity. By retaining and transferring hidden states, the system is able to establish a coherent memory of the conversation. Even when processing multiple rounds of dialogue or long speech sequences, it maintains sensitivity to historical information, improving the accuracy of speech understanding.

[0068] Furthermore, the steps of preprocessing the historical conversation text to obtain the subword sequence vector include: using a word segmenter to convert the historical conversation text into a subword sequence; and embedding the subword sequence to obtain the subword sequence vector.

[0069] Specifically, when preprocessing historical conversation text, an unsupervised word segmentation algorithm can effectively segment continuous text into a series of subword units. These subword units may cross word boundaries, forming a more fine-grained representation. For example, "unavailable" is not simply segmented into "un" and "available", but may produce a subword sequence such as "un", "avail", and "able". This segmentation method can better preserve the morphological information of the words. The word segmenter converts the original conversation text into a subword sequence. The subword sequence can more delicately express the structure and meaning of the text, providing richer information for subsequent embedding processing.

[0070] After the subword sequence is generated, it is embedded. Embedding is the process of mapping the subwords after segmentation into a multidimensional vector space to encode the subword sequence. These subword sequence vectors carry the semantic information and potential relationships of the subwords.

[0071] Furthermore, the subword sequence vector is input into the streaming processing model, and the step of obtaining the text feature vector includes: inputting the subword sequence vector into the state space model in the streaming processing model; for each subword sequence vector, configuring the initial hidden state of the state space model, wherein the state space model updates the hidden state based on the initial hidden state using the state transfer matrix and the projection matrix; after updating the hidden state after processing the subword sequence vector, the updated hidden state is used as the initial hidden state for processing the next subword sequence vector, the state transfer matrix is ​​dynamically adjusted, and the projection matrix is ​​obtained after mapping the subword sequence vector to the hidden state space; performing a linear transformation on the hidden state updated based on the last subword sequence vector to obtain the text feature vector.

[0072] Specifically, after the subword sequence vectors are input into the streaming model, they are fed into the state-space model, which effectively captures and represents the temporal dependencies between subword sequence vectors. For each subword sequence vector, the state-space model first sets an initial hidden state based on the final hidden state after processing the previous subword sequence vector, ensuring continuous information transfer. The model then updates the hidden state using a dynamically adjusted state transition matrix A and projection matrices B and C. The state transition matrix A controls the attenuation or enhancement of information over time, while the projection matrices B and C map the subword sequence vectors into the hidden state space to update the hidden state. Once the state-space model completes the iterative hidden state update based on all subword sequence vectors, it linearly transforms the last processed hidden state using an additional matrix W to obtain the final text feature vector. The output text feature vector captures all information about the historical conversation text, including lexical meaning, contextual relationships, and the dynamic evolution of the conversation history.

[0073] Furthermore, the steps of performing feature fusion on the speech feature vector and the text feature vector and outputting a streaming speech fusion feature vector include: converting the speech feature vector and the text feature vector into feature vectors of the same latitude to obtain the converted speech feature vector and text feature vector; configuring gating weights based on a gated attention mechanism; and performing weighted summation on the converted speech feature vector and text feature vector based on the gating weights to obtain a streaming speech fusion feature vector.

[0074] Specifically, based on the speech and text feature vectors output by the state-space model, a gated attention mechanism is used to perform feature fusion. First, the speech and text feature vectors, extracted separately by the state-space model, are transformed to ensure that they have the same feature dimension. This process can be achieved by adding or subtracting elements from the feature vectors or through linear transformations such as fully connected layers. Second, the gated attention mechanism is used to dynamically determine the importance of speech and text features during the fusion process. The gate weight α is calculated by a small neural network module that receives the transformed feature vector as input. Finally, the gate weight α is used to perform a weighted sum to generate the streaming speech fusion feature vector. This operation ensures that the importance of speech and text information is dynamically adjusted according to the specific context, so that the fused feature vector better reflects the true intent and context of the conversation.

[0075] Step S203: input the streaming speech fusion feature vector into the language processing model and output a response text.

[0076] After obtaining the streaming speech fusion feature vector, it is input into the language processing model, which understands the user's intent. The language processing model is a pre-built model used to make response decisions. The language processing model receives the streaming speech fusion feature vector, analyzes the context, intent, and necessary business information contained in the feature vector, and then generates an appropriate response text. This part may include a multi-head attention mechanism to capture long-range dependencies in the feature vector, and fully connected layers for classification and regression tasks to determine the most appropriate response strategy. Ultimately, this information is expressed in natural language, forming a fluent and accurate response.

[0077] Step S204: Generate response audio based on the response text, and return the response audio to the user terminal.

[0078] In the above step S204, the response text is used as input data of the conversion engine, and the conversion engine maps the text data to acoustic features, and then converts the acoustic features into audio waveforms to obtain response audio, thereby forming a voice dialogue returned to the user end.

[0079] Through the above steps, a continuous voice stream is received and the historical conversation text corresponding to the voice stream is obtained, wherein the voice stream is real-time voice data based on financial business. Secondly, the voice stream and the historical conversation text are input into the streaming processing model, and a streaming voice fusion feature vector is output. The streaming processing model extracts features from the voice stream based on a state space model and a dynamic selection mechanism. The dynamic selection mechanism is used to dynamically select model parameters of the state space model. Then, the streaming voice fusion feature vector is input into the language processing model, and a response text is output. The language processing model is a pre-built model for making response decisions based on text data. Finally, a response audio is generated based on the response text, and the response audio is returned to the user end.

[0080] In this embodiment, a real-time voice stream and the corresponding historical conversation text are received, and a pre-built streaming processing model is used to analyze the voice stream and text data. The model is based on a state-space model and a dynamic selectivity mechanism, and can efficiently extract voice features and text features under linear time complexity. There is no need to wait for complete voice data, and the voice stream can be processed in real time to achieve the purpose of quickly responding to voice conversations. This achieves the technical effect of reducing the delay time for waiting for data transmission and improving the efficiency of voice data processing, thereby solving the technical problem of low voice conversation processing efficiency under the cascade architecture in related technologies.

[0081] The following describes in detail another optional specific implementation.

[0082] Figure 3 is a schematic diagram of an optional flow of voice conversation streaming according to an embodiment of the present invention, such as Figure 3 As shown in the figure, the streaming process of voice dialogue includes:

[0083] Step 1: Receive the voice stream and obtain the historical conversation text;

[0084] Step 2: Input the voice stream into the streaming processing model, compress and segment the voice stream, and perform feature processing based on the dynamic selectivity mechanism;

[0085] Step 3: Perform word segmentation and embedding processing on the historical conversation text;

[0086] Step 4: Input the embedded subword sequence vector into the streaming processing model to extract and encode the text features to obtain the text feature vector;

[0087] Step 5: Feature fusion based on the gated attention mechanism;

[0088] Step 6: Input the fused features into the language processing model, output the response text and generate the response audio.

[0089] In this embodiment of the present invention, a streaming processing model processes continuous voice streams, supporting streaming processing without waiting for complete voice data. This effectively reduces data transmission latency, improves the efficiency of data-to-voice conversation responses, and enhances the user experience. Furthermore, the streaming processing model utilizes a selective state-space model to dynamically capture key acoustic features and semantic dependencies in the voice stream, suppressing redundant information interference. Furthermore, by integrating historical conversation text data, this model avoids contextual forgetting, supports semantic coherence analysis, and improves the accuracy of output results, better meeting user-side conversation needs.

[0090] The following describes it in detail with reference to another embodiment.

[0091] Example 2

[0092] A streaming processing device for voice conversations provided in this embodiment includes multiple implementation units, each of which corresponds to each implementation step in the above-mentioned embodiment 1. Its specific implementation methods and beneficial effects can be referred to the above-mentioned method embodiments and will not be repeated here.

[0093] Figure 4 is a schematic diagram of an optional streaming processing device for voice dialogue according to an embodiment of the present invention, such as Figure 4 As shown, the streaming processing device for voice dialogue may include: a receiving unit 41, a first output unit 42, a second output unit 43, and a generating unit 44, wherein:

[0094] A receiving unit 41 is configured to receive a continuous voice stream and obtain a historical conversation text corresponding to the voice stream, wherein the voice stream is real-time voice data based on financial services;

[0095] a first output unit 42 for inputting the speech stream and the historical conversation text into a streaming processing model and outputting a streaming speech fusion feature vector, wherein the streaming processing model extracts features from the speech stream based on a state-space model and a dynamic selection mechanism, wherein the dynamic selection mechanism is used to dynamically select model parameters of the state-space model;

[0096] A second output unit 43 is configured to input the streaming speech fusion feature vector into a language processing model and output a response text, wherein the language processing model is a pre-built model for making response decisions based on text data;

[0097] The generating unit 44 is configured to generate a response audio based on the response text and return the response audio to the user terminal.

[0098] The above-mentioned streaming processing device for voice dialogue receives a continuous voice stream through a receiving unit 41 and obtains a historical dialogue text corresponding to the voice stream, wherein the voice stream is real-time voice data based on financial business; inputs the voice stream and the historical dialogue text into a streaming processing model through a first output unit 42, and outputs a streaming voice fusion feature vector, wherein the streaming processing model extracts features from the voice stream based on a state space model and a dynamic selection mechanism, and the dynamic selection mechanism is used to dynamically select model parameters of the state space model; inputs the streaming voice fusion feature vector into a language processing model through a second output unit 43, and outputs a response text, wherein the language processing model is a pre-built model for making response decisions based on text data; generates a response audio based on the response text through a generation unit 44, and returns the response audio to the user end.

[0099] In this embodiment, a real-time voice stream and the corresponding historical conversation text are received, and a pre-built streaming processing model is used to analyze the voice stream and text data. The model is based on a state-space model and a dynamic selectivity mechanism, and can efficiently extract voice features and text features under linear time complexity. There is no need to wait for complete voice data, and the voice stream can be processed in real time to achieve the purpose of quickly responding to voice conversations. This achieves the technical effect of reducing the delay time for waiting for data transmission and improving the efficiency of voice data processing, thereby solving the technical problem of low voice conversation processing efficiency under the cascade architecture in related technologies.

[0100] Furthermore, the first output unit includes: a first processing module, used to input the voice stream into the streaming processing model, compress and segment the voice stream through the streaming processing model, and obtain N voice feature segments, where N is a positive integer; a first extraction module, used to extract features from the N voice feature segments based on the state space model in the streaming processing model to obtain a voice feature vector; a second processing module, used to preprocess the historical conversation text to obtain a subword sequence vector; a first input module, used to input the subword sequence vector into the streaming processing model to obtain a text feature vector; a first fusion module, used to perform feature fusion on the voice feature vector and the text feature vector, and output a streaming voice fusion feature vector.

[0101] Furthermore, the first processing module includes: a first compression submodule, which is used to compress the speech stream based on the one-dimensional convolution layer of the streaming processing model to obtain frame-level speech features; a first calculation submodule, which is used to calculate the short-time energy value and zero-crossing rate of the frame-level speech features, wherein the short-time energy value is used to measure the local energy intensity of the speech signal, and the zero-crossing rate is a measure of the oscillation property of the speech signal; a first merging submodule, which is used to detect frame-level active speech features based on the short-time energy value and the zero-crossing rate, and merge continuous frame-level speech features to obtain initial speech feature segments; a first segmentation submodule, which is used to predict the silence position in the initial speech feature segment for each initial speech feature segment, and perform secondary segmentation on the initial speech feature segment based on the silence position to obtain speech feature segments.

[0102] Furthermore, the first extraction module includes: a first determination submodule, which is used to input N speech feature segments into the state space model, and determine the time step parameter of the state space model when processing the speech feature segment based on the energy level of each speech feature segment, wherein the energy level represents the total energy carried by the speech signal corresponding to the speech feature segment; a first configuration submodule, which is used to dynamically configure the time step parameter for the state space model when processing each speech feature segment, and configure the initial hidden state for the state space model, wherein after processing each speech feature segment, the hidden state of the state space model is retained as the initial hidden state for processing the next speech feature segment, and the hidden state contains the temporal characteristics of the speech signal; the first extraction submodule, which is used to extract the temporal dependence characteristics of the speech feature segment through the state space model based on the initial hidden state to obtain a speech feature vector.

[0103] Furthermore, the second processing module includes: a first conversion submodule, used to use a word segmenter to convert the historical conversation text into a subword sequence; and a first processing submodule, used to perform embedding processing on the subword sequence to obtain a subword sequence vector.

[0104] Furthermore, the first input module includes: a first input submodule, used to input the subword sequence vector into the state space model in the streaming processing model; a second configuration submodule, used to configure the initial hidden state of the state space model for each subword sequence vector, wherein the state space model updates the hidden state based on the initial hidden state using the state transfer matrix and the projection matrix. After updating the hidden state for processing the subword sequence vector, the updated hidden state is used as the initial hidden state for processing the next subword sequence vector. The state transfer matrix is ​​dynamically adjusted, and the projection matrix is ​​obtained after mapping the subword sequence vector to the hidden state space; a first acquisition submodule, used to perform a linear transformation on the hidden state updated based on the last subword sequence vector to obtain a text feature vector.

[0105] Furthermore, the first fusion module includes: a first conversion submodule, used to convert the speech feature vector and the text feature vector into feature vectors of the same dimension, to obtain the converted speech feature vector and text feature vector; a third configuration submodule, used to configure the gating weight based on the gated attention mechanism; a second acquisition submodule, used to perform weighted summation of the converted speech feature vector and text feature vector based on the gating weight to obtain a streaming speech fusion feature vector.

[0106] It should be noted that the receiving unit 41, the first output unit 42, the second output unit 43, and the generating unit 44 correspond to steps S201 to S204 in the first embodiment. The examples and application scenarios implemented by the above units and the corresponding steps are the same, but are not limited to the contents disclosed in the first embodiment. It should be noted that the above modules or units can be hardware components or software components stored in a memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n). The above modules or units can also be part of a device and can be run in the computer terminal 10 provided in the first embodiment.

[0107] The present invention is described below in conjunction with another optional embodiment.

[0108] Example 3

[0109] An embodiment of the present invention may further provide an electronic device, Figure 5 is a hardware structure block diagram of an electronic device (or mobile device) that optionally performs a streaming processing method for voice conversations according to an embodiment of the present invention, such as Figure 5 As shown, the electronic device may include: one or more ( Figure 5 Only one is shown) processor 502, memory 504, storage controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.

[0110] Among them, the memory can be used to store software programs and modules, such as program instructions / modules corresponding to the methods and devices in the embodiments of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, implementing the above-mentioned method. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0111] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: receive a continuous voice stream and obtain the historical conversation text corresponding to the voice stream, wherein the voice stream is real-time voice data based on financial services; input the voice stream and the historical conversation text into the streaming processing model, and output a streaming voice fusion feature vector, wherein the streaming processing model extracts features from the voice stream based on a state space model and a dynamic selection mechanism, and the dynamic selection mechanism is used to dynamically select model parameters of the state space model; input the streaming voice fusion feature vector into the language processing model, and output a response text, wherein the language processing model is a pre-built model for making response decisions based on text data; generate response audio based on the response text, and return the response audio to the user end.

[0112] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: input the voice stream and the historical conversation text into the streaming processing model, and the step of outputting the streaming voice fusion feature vector includes: inputting the voice stream into the streaming processing model, compressing and segmenting the voice stream through the streaming processing model to obtain N voice feature segments, where N is a positive integer; extracting features from the N voice feature segments based on the state space model in the streaming processing model to obtain a voice feature vector; preprocessing the historical conversation text to obtain a subword sequence vector; inputting the subword sequence vector into the streaming processing model to obtain a text feature vector; performing feature fusion on the voice feature vector and the text feature vector, and outputting the streaming voice fusion feature vector.

[0113] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: the steps of compressing and segmenting the voice stream through the streaming processing model include: compressing the voice stream based on the one-dimensional convolution layer of the streaming processing model to obtain frame-level voice features; calculating the short-time energy value and zero-crossing rate of the frame-level voice features, wherein the short-time energy value is used to measure the local energy intensity of the voice signal, and the zero-crossing rate is a measure of the oscillation property of the voice signal; detecting the frame-level active voice features based on the short-time energy value and the zero-crossing rate, and merging the continuous frame-level voice features to obtain the initial voice feature segment; for each initial voice feature segment, predicting the silence position in the initial voice feature segment, and performing secondary segmentation on the initial voice feature segment based on the silence position to obtain the voice feature segment.

[0114] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: extracting features from N speech feature segments based on the state space model in the streaming processing model, and obtaining a speech feature vector includes the following steps: inputting N speech feature segments into the state space model, and determining the time step parameter of the state space model when processing the speech feature segment based on the energy level of each speech feature segment, wherein the energy level represents the total energy carried by the speech signal corresponding to the speech feature segment; when processing each speech feature segment, dynamically configuring the time step parameter for the state space model, and configuring the initial hidden state for the state space model, wherein after processing each speech feature segment, retaining the hidden state of the state space model as the initial hidden state for processing the next speech feature segment, and the hidden state contains the temporal features of the speech signal; based on the initial hidden state, extracting the temporal dependency features of the speech feature segment through the state space model to obtain a speech feature vector.

[0115] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: preprocessing the historical conversation text, and the step of obtaining the subword sequence vector includes: using a word segmenter to convert the historical conversation text into a subword sequence; embedding the subword sequence to obtain the subword sequence vector.

[0116] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: input the subword sequence vector into the streaming processing model, and the step of obtaining the text feature vector includes: inputting the subword sequence vector into the state space model in the streaming processing model; for each subword sequence vector, configuring the initial hidden state of the state space model, wherein the state space model is based on the initial hidden state and uses the state transfer matrix and the projection matrix to update the hidden state. After the hidden state is updated after processing the subword sequence vector, the updated hidden state is used as the initial hidden state for processing the next subword sequence vector. The state transfer matrix is ​​dynamically adjusted, and the projection matrix is ​​obtained after mapping the subword sequence vector to the hidden state space; the hidden state updated based on the last subword sequence vector is linearly transformed to obtain the text feature vector.

[0117] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: perform feature fusion on the speech feature vector and the text feature vector, and the step of outputting the streaming speech fusion feature vector includes: converting the speech feature vector and the text feature vector into feature vectors of the same dimension to obtain the converted speech feature vector and text feature vector; configuring the gating weight based on the gated attention mechanism; and performing weighted summation on the converted speech feature vector and text feature vector based on the gating weight to obtain the streaming speech fusion feature vector.

[0118] An embodiment of the present invention provides a streaming processing solution for voice conversations. It receives a real-time voice stream and the corresponding historical conversation text, and analyzes the voice stream and text data using a pre-built streaming processing model. This model, based on a state-space model and a dynamic selectivity mechanism, efficiently extracts voice and text features with linear time complexity. Without waiting for complete voice data, the voice stream can be processed in real time, achieving the goal of rapidly responding to voice conversations. This reduces the latency associated with waiting for data transmission and improves voice data processing efficiency, thereby resolving the technical issue of low voice conversation processing efficiency in cascaded architectures in related technologies.

[0119] It can be understood by those skilled in the art that Figure 5 The structure shown is for illustration only, and the electronic device may also be a terminal device such as a smart phone, a tablet computer, a PDA, a mobile Internet device (MID), or a PAD. Figure 5 It does not limit the structure of the above electronic device. For example, the electronic device may also include Figure 5 More or fewer components (such as network interfaces, display devices, etc.) shown in, or with Figure 5 Different configurations shown.

[0120] A person skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0121] The present invention is described below in conjunction with another optional embodiment.

[0122] Example 4

[0123] The embodiment of the present invention further provides a computer-readable storage medium. Optionally, in the embodiment of the present invention, the computer-readable storage medium can be used to store the program code executed by the method for stream processing of voice conversations provided in the first embodiment.

[0124] Optionally, in an embodiment of the present invention, the above-mentioned storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0125] An embodiment of the present invention also provides a computer program product, which, when executed on a data processing device, is suitable for executing the steps of a streaming processing method for voice conversations: receiving a continuous voice stream and obtaining a historical conversation text corresponding to the voice stream, wherein the voice stream is real-time voice data based on financial services; inputting the voice stream and the historical conversation text into a streaming processing model, and outputting a streaming voice fusion feature vector, wherein the streaming processing model extracts features from the voice stream based on a state space model and a dynamic selection mechanism, and the dynamic selection mechanism is used to dynamically select model parameters of the state space model; inputting the streaming voice fusion feature vector into a language processing model, and outputting a response text, wherein the language processing model is a pre-built model for making response decisions based on text data; generating a response audio based on the response text, and returning the response audio to the user end.

[0126] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0127] In the above embodiments of the present invention, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0128] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of the units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0129] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0130] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0131] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk, etc. Various media that can store program codes.

[0132] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A method for streaming voice conversations, characterized in that: include: Receive a continuous voice stream and obtain historical conversation text corresponding to the voice stream, wherein the voice stream is real-time voice data based on financial services; Inputting the speech stream and the historical conversation text into a streaming processing model and outputting a streaming speech fusion feature vector, wherein the streaming processing model extracts features from the speech stream based on a state-space model and a dynamic selection mechanism, wherein the dynamic selection mechanism is used to dynamically select model parameters of the state-space model; Inputting the streaming speech fusion feature vector into a language processing model and outputting a response text, wherein the language processing model is a pre-built model for making response decisions based on text data; Generate a response audio based on the response text, and return the response audio to the user end.

2. The method according to claim 1, characterized in that The steps of inputting the speech stream and the historical conversation text into a streaming processing model and outputting a streaming speech fusion feature vector include: Inputting the voice stream into a streaming processing model, compressing and segmenting the voice stream by the streaming processing model to obtain N voice feature segments, where N is a positive integer; Performing feature extraction on the N speech feature segments based on the state space model in the streaming processing model to obtain a speech feature vector; Preprocessing the historical conversation text to obtain subword sequence vectors; Inputting the subword sequence vector into the streaming processing model to obtain a text feature vector; Perform feature fusion on the speech feature vector and the text feature vector, and output the streaming speech fusion feature vector.

3. The method according to claim 2, characterized in that The steps of compressing and segmenting the voice stream by the streaming processing model include: Compressing the speech stream based on the one-dimensional convolutional layer of the streaming processing model to obtain frame-level speech features; Calculating a short-time energy value and a zero-crossing rate for the frame-level speech feature, wherein the short-time energy value is used to measure the local energy intensity of the speech signal, and the zero-crossing rate is a measure of the oscillation property of the speech signal; Detecting frame-level active speech features based on the short-time energy value and the zero-crossing rate, and merging consecutive frame-level speech features to obtain an initial speech feature segment; For each of the initial speech feature segments, a silence position in the initial speech feature segment is predicted, and the initial speech feature segment is segmented twice based on the silence position to obtain the speech feature segment.

4. The method according to claim 2, characterized in that The step of extracting features from the N speech feature segments based on the state space model in the streaming processing model to obtain a speech feature vector includes: Inputting N speech feature segments into the state-space model, and determining a time step parameter of the state-space model when processing the speech feature segment based on an energy level of each speech feature segment, wherein the energy level represents the total energy carried by the speech signal corresponding to the speech feature segment; When processing each of the speech feature segments, dynamically configuring a time step parameter for the state-space model and configuring an initial hidden state for the state-space model, wherein after processing each of the speech feature segments, retaining the hidden state of the state-space model as the initial hidden state for processing the next speech feature segment, the hidden state including the temporal characteristics of the speech signal; Based on the initial hidden state, the time-dependent features of the speech feature segment are extracted through the state space model to obtain the speech feature vector.

5. The method according to claim 2, characterized in that The steps of preprocessing the historical conversation text to obtain subword sequence vectors include: Using a word segmenter to convert the historical conversation text into a subword sequence; Embedding processing is performed on the subword sequence to obtain the subword sequence vector.

6. The method according to claim 2, characterized in that The step of inputting the subword sequence vector into the streaming processing model to obtain a text feature vector includes: Inputting the subword sequence vector into the state space model in the streaming model; For each of the subword sequence vectors, configuring an initial hidden state of the state space model, wherein the state space model updates the hidden state based on the initial hidden state using a state transfer matrix and a projection matrix. After updating the hidden state for processing a subword sequence vector, the updated hidden state serves as the initial hidden state for processing the next subword sequence vector. The state transfer matrix is ​​dynamically adjusted, and the projection matrix is ​​obtained by mapping the subword sequence vector to the hidden state space. The hidden state updated based on the last subword sequence vector is linearly transformed to obtain the text feature vector.

7. The method according to claim 2, characterized in that The step of performing feature fusion on the speech feature vector and the text feature vector and outputting the streaming speech fusion feature vector comprises: Converting the speech feature vector and the text feature vector into feature vectors of the same latitude to obtain the converted speech feature vector and the text feature vector; Configure gate weights based on the gated attention mechanism; The converted speech feature vector and the text feature vector are weightedly summed based on the gating weight to obtain a streaming speech fusion feature vector.

8. A streaming processing device for voice dialogue, characterized in that: include: A receiving unit, configured to receive a continuous voice stream and obtain a historical conversation text corresponding to the voice stream, wherein the voice stream is real-time voice data based on financial services; a first output unit, configured to input the speech stream and the historical conversation text into a streaming processing model, and output a streaming speech fusion feature vector, wherein the streaming processing model extracts features from the speech stream based on a state-space model and a dynamic selection mechanism, wherein the dynamic selection mechanism is configured to dynamically select model parameters of the state-space model; a second output unit, configured to input the streaming speech fusion feature vector into a language processing model and output a response text, wherein the language processing model is a pre-built model for making response decisions based on text data; A generating unit is used to generate a response audio based on the response text and return the response audio to the user terminal.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the method for streaming processing of voice conversations according to any one of claims 1 to 7.

10. An electronic device, characterized in that: The invention comprises one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the streaming processing method for voice conversations as described in any one of claims 1 to 7.

Images