A one-stop construction method of a full-process voice interaction intelligent agent of a private deployment

Abstract

This invention proposes a one-stop construction method for a fully automated voice interaction agent deployed in a private environment. The method includes: receiving an audio frame sequence, performing endpoint detection and smoothing, acoustic decoding, and punctuation completion to generate structured text; parallel execution of vector and keyword recall, followed by fusion scoring and ranking to generate knowledge-enhanced objects; using an evidence consistency modulation mechanism to correct candidate word probabilities and generate text responses; performing rhythm and energy synchronization modulation on acoustic features based on the text structure to generate waveform streams; and calculating complexity metrics and releasing resources after the session ends. This invention achieves fully automated voice interaction without data leaving the domain, improves response accuracy through multi-path recall fusion and evidence consistency modulation, optimizes speech naturalness through prosodic modulation, and ensures the efficiency and security of private deployment through unified resource management.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and natural language processing technology, and in particular relates to a one-stop construction method for a privately deployed, end-to-end voice interaction intelligent agent. Background Technology

[0002] Current mainstream solutions mostly adopt a combination of capabilities provided by cloud or third-party platforms, splicing speech recognition, knowledge retrieval, semantic generation, and speech synthesis through interfaces to achieve voice question-and-answer or dialogue services. While these solutions seem convenient in the initial deployment stage, they often face a series of engineering contradictions in the actual operation of high-security industries: voice content may inherently carry sensitive information such as identity, accounts, medical conditions, and matters to be handled, and uploading voice or text to the cloud will introduce compliance risks and the possibility of leakage; enterprise knowledge bases contain internal materials such as policy terms, product rules, process specifications, and historical announcements, and if they interact with external models or external retrieval services, they are prone to knowledge leakage or loss of control; at the architectural level, components such as speech recognition, retrieval, generation, and synthesis are usually carried by different systems or services, and data frequently changes between multiple components, making it difficult to manage the state uniformly, resulting in problems such as latency jitter, resource competition in concurrent scenarios, and difficulty in anomaly recovery; in addition, even if local deployment is adopted, it often remains at the level of "single module availability", lacking a closed-loop system from voice input to voice output and a consistent data object transfer method, which makes the system only privatize in some parts and unable to form a sustainable full-process operation and maintenance capability. The aforementioned issues mean that when existing technologies are implemented in high-security industries, additional engineering patches and manual maintenance strategies are often required. This makes it difficult to ensure that data remains within the controlled scope of the enterprise's intranet and to guarantee the stability and consistency of multi-module collaborative operation, thus restricting the large-scale application of voice intelligence agents in critical business. Summary of the Invention

[0003] The purpose of this invention is to design a one-stop construction method for a privately deployed, end-to-end voice interaction intelligent agent, which can realize a closed-loop private process and consistent data object transmission, reduce system integration complexity, and improve stability and controllability.

[0004] To achieve the above objectives, this invention provides a one-stop method for constructing a privately deployed, end-to-end voice interaction intelligent agent, the method comprising: S1. Load the speech recognition module, enterprise knowledge base module, local fine-tuning large language model module and speech synthesis module on the enterprise internal server. Integrate the modules through a unified control instance to form a running instance. The unified control instance initializes a unified data structure to carry session identifiers, speech data, text data and knowledge context fields. S2. Receive the audio frame sequence sent by the client and write it into the voice field cache of the unified data structure. Use the voice recognition module to perform endpoint detection smoothing, acoustic decoding and punctuation completion on the audio frame sequence to generate a structured text object containing session identifier and text string and write it into the unified data structure. S3. Based on the text string in the structured text object, vector recall and keyword recall are executed in parallel in the enterprise knowledge base module. The two recall results are sorted through a fusion scoring framework to generate a knowledge enhancement object containing the sorted paragraph set and written into the unified data structure. S4. Read the structured text object and the knowledge enhancement object, construct the input sequence into the local fine-tuning large language model module, use the evidence consistency modulation mechanism to correct the probability of candidate words, generate a text response string and combine it into a response object and write it into the unified data structure. S5. Read the response object, use the speech synthesis module to convert the text response string into an acoustic feature frame sequence, perform rhythm and energy synchronization modulation on the acoustic feature frame sequence based on the text structure, generate a waveform sample stream and send it to the client; S6. After detecting that the audio transmission is complete and receiving confirmation from the client that playback is complete, calculate the session complexity index, release the cache resources and network connection handle associated with the session identifier, and delete the session state record.

[0005] Furthermore, the specific process of the endpoint detection smoothing process is as follows: obtain the frame-by-frame speech activity probability output by the endpoint detection network, use a first-order discrete low-pass filter to smooth the frame-by-frame speech activity probability to obtain a smoothing probability, mark the entry into a speech segment when the smoothing probability is continuously higher than the activity threshold, mark the end of the speech segment when the smoothing probability is continuously lower than the silence threshold, and extract the audio frame sequence corresponding to the speech segment and input it into the acoustic model for decoding.

[0006] Furthermore, the calculation method of the fusion scoring framework includes: Calculate the cosine similarity between the query vector and the candidate paragraph vector, use the cosine similarity calculation result as the initial ranking basis, and extract the top K candidate paragraphs to form the TopK candidate set; The TopK candidate set is input into the reordering model, and the multi-layer self-attention network in the reordering model is used to cross-encode the concatenated sequence of the query text and candidate paragraphs. The relevance score of the candidate paragraphs is calculated through the linear output layer. The TopK candidate set is reordered according to the relevance score, and the top-ranked entries are selected as the final knowledge enhancement objects.

[0007] Furthermore, the specific process of the evidence consistency modulation mechanism is as follows: statistically analyze the occurrence frequency of candidate words and the total number of evidence words in the paragraph text after word segmentation in the knowledge enhancement object, calculate the occurrence ratio of the candidate words in the evidence set, and linearly fuse the original candidate word probability output by the local fine-tuning large language model module with the occurrence ratio according to the fusion coefficient to obtain the corrected candidate word probability for decoding selection.

[0008] Furthermore, the process of rhythm and energy synchronization modulation of the acoustic feature frame sequence based on text structure includes: The text response string is read, segmented and encoded by a word segmenter, and the sub-word identifier sequence is mapped into a vector sequence through an embedding layer. The vector sequence is input into a multi-layer self-attention text encoding network to obtain a text representation sequence containing contextual semantics, and prosodic related representations are predicted based on the text representation sequence. Based on prosodic relevance representation, an acoustic feature frame sequence is generated, and the acoustic feature frame sequence is modulated: the text response string is marked with punctuation positions and numerical segments, the marks are mapped to the time steps of the acoustic feature frame sequence to generate a text-driven rhythm weight sequence, and the original acoustic feature frame vector is multiplicatively modulated using the modulation intensity coefficient and the text-driven rhythm weight sequence to obtain the modulated acoustic feature frame vector.

[0009] Furthermore, the specific method for mapping the markers to the time steps of the acoustic feature frame sequence is as follows: Frame-by-frame weight values ​​are generated based on punctuation mark and digital segment mark. The frame-by-frame weight values ​​are written into the session buffer to form a rhythm weight sequence through alignment mapping from text units to frame intervals. A phoneme-to-frame repetitive expansion method is adopted to determine the frame interval corresponding to each phoneme. The frame interval is assigned a value to form a frame-by-frame text-driven rhythm weight sequence, wherein the frame interval corresponding to the key numerical segment is assigned a first weight value, and the frame interval corresponding to the ordinary narrative interval is assigned a second weight value.

[0010] Furthermore, the calculation basis of the session complexity index is as follows: the number of characters in the response text is obtained as the first parameter, the number of knowledge paragraph entries involved in the generation is obtained as the second parameter, and the product of the natural logarithm of the first parameter and the second parameter is used as the session complexity index to characterize the degree of session resource consumption.

[0011] Furthermore, the specific order of releasing the cache resources associated with the session identifier is as follows: first, close the audio output stream bound to the session identifier, then release the voice frame buffer, acoustic feature buffer, generation buffer, and knowledge segment temporary splicing area in sequence, and finally close the network connection handle and delete the session state record.

[0012] Furthermore, the initialization process of the enterprise knowledge base module includes: reading text data from an internal document directory or database, segmenting it into natural paragraphs or fixed lengths, encoding each text segment into a fixed-dimensional vector using an embedding network, and establishing a memory vector index structure. The embedding network includes a word segmentation embedding layer, an average pooling layer, and two fully connected layers.

[0013] The beneficial technical effects of the present invention are at least as follows: To address the aforementioned issues, this invention provides a one-stop construction method for a privately deployed, end-to-end voice interaction intelligent agent. It unifies speech recognition, enterprise knowledge matching, semantic generation, and speech synthesis into a runnable whole on an internal enterprise server, and uses a unified control instance to carry out session-level data flow. This allows the processing from speech to text, text to knowledge, knowledge to response, and response to speech to proceed continuously under the same session identifier, forming a closed-loop end-to-end interaction link. On the voice input side, by smoothing the endpoint detection output and coupling it with the segmentation recognition process, the real-time voice reception maintains stable segmentation boundaries under network jitter and short pause scenarios, thereby obtaining structured text objects that can be directly used for subsequent processing. On the knowledge matching side, vector semantic similarity and keyword evidence are integrated in a unified scoring framework, and bias penalties are introduced to enhance the ability to distinguish corporate rule-type texts, making the search results more consistent with the precise expression of institutional clauses and business announcements. On the semantic generation side, corporate evidence is modulated to modulate the generation probability, so that the generation process achieves a controllable balance between language fluency and evidence consistency, improving the consistency between the answer and the document facts. On the voice output side, modulation related to text structure is introduced at the acoustic feature layer, making the broadcast of key values ​​and conclusive information clearer, meeting the requirements of information understandability in financial, medical, and government scenarios. At the end of the session, resource recycling and statistical recording through session identifiers ensure that the system maintains stable operation and maintainability under high concurrency conditions. Through the above methods, this invention achieves a closed-loop, fully private process and consistent data object transfer from the perspective of engineering architecture and step connection, reducing system integration complexity, improving stability and controllability, and meeting the consistent requirements of high-security industries for data security, compliance boundaries, and business scope without relying on external services. Attached Figure Description

[0014] The present invention will be further described with reference to the accompanying drawings, but the embodiments in the drawings do not constitute any limitation on the present invention. For those skilled in the art, other drawings can be obtained based on the following drawings without creative effort.

[0015] Figure 1 This is a flowchart of a one-stop construction method for a privately deployed, end-to-end voice-interactive intelligent agent according to the present invention. Detailed Implementation

[0016] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0017] In one or more embodiments, such as Figure 1 As shown, a one-stop construction method for a fully automated voice interaction agent with private deployment is disclosed. The method includes the following: S1. Load the speech recognition module, enterprise knowledge base module, local fine-tuning large language model module and speech synthesis module on the enterprise internal server. Integrate the modules through a unified control instance to form a running instance. The unified control instance initializes a unified data structure to carry session identifiers, speech data, text data and knowledge context fields. Specifically, step one involves the unified construction and runtime initialization of the private deployment environment. This step loads and uniformly binds the speech recognition module, enterprise knowledge base module, locally fine-tuned large language model module, and speech synthesis module on the enterprise's internal server, enabling each module to form a complete running instance under the same process control framework. The enterprise server can be a rack-mounted server or a virtualized computing node, using a general-purpose server operating system environment. The speech recognition model weight file, the enterprise knowledge base original document set, the large language model weight file, the speech synthesis model weight file, and the deployment configuration file are all pre-stored in the enterprise's internal file system or database. Upon system startup, it first reads the model path, thread parameters, and service port information from the configuration file, and then sequentially completes the loading of model weights, index construction, and runtime instance initialization, ensuring that the system is immediately ready to receive voice requests after startup.

[0018] The speech recognition module includes an endpoint detection network, an acoustic model, and a punctuation prediction network. The endpoint detection network is a three-layer one-dimensional convolutional structure, where each convolutional kernel slides within a fixed time window to extract features, outputting frame-by-frame speech activity probabilities. The acoustic model employs a multi-layer bidirectional recurrent structure, mapping continuous speech feature sequences to character-level probability distributions. The punctuation prediction network is a bidirectional recurrent structure with a fully connected layer, predicting punctuation categories from recognized text sequences. Model weights are loaded into the server's memory space via memory mapping, and model instance handles are established. The enterprise knowledge base module completes vector index construction during system startup. The system reads text data from an internal document directory or database, segments it into natural paragraphs or fixed-length segments, and encodes each segment into a fixed-dimensional vector using an embedding network. The embedding network uses a multi-layer feedforward structure, taking the word vector sequence after segmentation as input, and outputting a vector representation after average pooling and a fully connected layer. After all vectors are generated, a memory-based vector index structure is established for subsequent retrieval. For example, when the knowledge base contains 50,000 document records and each document is divided into an average of 4 segments, it will form about 200,000 vector records. The index structure is built in one go during the system startup phase and kept in memory.

[0019] During the local fine-tuning of the large language model loading, the weights of the multi-layer self-attention structure model are read into memory. Each layer contains multi-head attention units and feedforward network units. The input is the segmented and encoded text sequence, and the output is the probability distribution of the next word. The speech synthesis module loads the weight files of the acoustic prediction network and the vocoder network. The acoustic prediction network maps the text to a sequence of acoustic features, and the vocoder network converts the acoustic features into a sequence of speech waveforms. After all the above modules are loaded, they are integrated through a unified control instance, and their structure is represented as follows: ; in, It serves as a unified control instance in the system's runtime state, acting as the sole module management object within the system. This is an example of a speech recognition module, which includes an endpoint detection network, an acoustic model, and a punctuation prediction network. This is an example of an enterprise knowledge base module, containing an embedded network and a vector index structure; For local fine-tuning of large language model instances; This is an instance of the speech synthesis module. All of these module instances are actual model objects in memory and are invoked through a unified control instance. The unified control instance also initializes a unified data structure to hold the session identifier, speech data, text data, and knowledge context fields used in subsequent steps. After completing the above initialization, the system enters a ready-to-run state, and all data generated in subsequent steps resides within the unified control instance. Internal circulation forms the basis for a complete private voice interaction operation.

[0020] S2. Receive the audio frame sequence sent by the client and write it into the voice field cache of the unified data structure. Use the voice recognition module to perform endpoint detection smoothing, acoustic decoding and punctuation completion on the audio frame sequence to generate a structured text object containing session identifier and text string and write it into the unified data structure. Specifically, step two is in the unified control instance. Based on the runtime state, it completes voice reception, endpoint segmentation, recognition and decoding, and punctuation completion, and organizes the results into a structured text object and writes it. Internal data structure. On the client side, continuous voice signals are collected by the microphone, formed into a sequence of audio frames of fixed length via the local audio interface, and sent frame by frame through a WebSocket long connection; on the server side, after receiving each frame, it writes it into the data in the order of arrival. The voice field is cached and invoked immediately. Internal speech recognition module Incremental processing is performed on newly added frames to generate progressively updated text output without waiting for the entire audio segment to end. To reduce false endpoint triggering caused by network jitter, environmental noise, or short pauses, the frame-by-frame speech activity probability output by the endpoint detection network is first smoothed by a first-order discrete low-pass filter before entering the threshold determination and speech segmentation logic. Once the speech segment is determined, continuous decoding of the acoustic model is triggered.

[0021] The smoothing algorithm uses an update derived from first-order IIR low-pass filtering in digital signal processing (which can also be considered a classic form of exponential moving average). Its discrete recursive formula can be obtained by discretizing a continuous first-order low-pass system: Let the endpoint detection network apply the first... The probability of voice activity in the frame output is Let the smoothed probability be denoted as Then the recursion is: ; in, For endpoint detection networks to the first The probability of speech activity in the frame output comes from The forward computation result of the three-layer one-dimensional convolutional network is taken as a proportional value. The smoothing probability of the previous frame is stored in... The data is stored in the session data structure and continuously updated throughout the session; The smoothed probability of the current frame is used as the direct input for endpoint determination and written back to the session data structure; The smoothing factor comes from a fixed parameter in the deployment configuration file and is loaded when the session starts. The value is a proportional quantity. Mathematically, this recursive formula belongs to a convex combination. and Both are probability quantities, and linear combinations retain their probability nature, facilitating direct comparison with thresholds for endpoint segmentation. Endpoint determination employs an engineering implementation combining threshold comparison and consecutive frame counting: when... When several consecutive frames exceed the activity threshold, a segment is marked for inclusion in the speech segment; when When several consecutive frames fall below the silence threshold, the speech segment is marked as ending, and the corresponding frame sequence is retrieved from the speech field buffer as a speech segment input to the acoustic model, using the segment boundary as the input. The acoustic model uses a multi-layer bidirectional recurrent structure to output a frame-by-frame character probability distribution. In the decoding stage, a common engineering process is used to select the character with the highest probability at each time step and merge repeat and blank markers to generate a character sequence. The punctuation prediction network uses a bidirectional recurrent structure and fully connected layers to predict the punctuation category of each character in the character sequence and inserts the punctuation marks into the character sequence to form a readable text string.

[0022] Combining the above process, structured text objects in Internally, it is written in a combination of session identifier and text field, represented as: ; in, It is a structured text object, stored in The data is incorporated into the conversation data structure and used as input for subsequent knowledge retrieval. As a session identifier, by The context object is generated and bound to the WebSocket connection when the connection is established; The text string after endpoint segmentation, acoustic decoding, and punctuation completion comes from... Output and write The text field is cached. The two-field design of this object naturally binds the text to the session context, allowing subsequent steps to directly read the corresponding text within the same session. With its associated session.

[0023] S3. Based on the text string in the structured text object, vector recall and keyword recall are executed in parallel in the enterprise knowledge base module. The two recall results are sorted through a fusion scoring framework to generate a knowledge enhancement object containing the sorted paragraph set and written into the unified data structure. Specifically, step three is in the unified control instance. Enterprise knowledge matching is completed within the runtime state, and the knowledge is directly read and written in step two. Structured text objects The system first uses two of these fields for retrieval and filtering. exist In the session context, locate the knowledge domain configuration and access scope corresponding to the session, and map them to the filtering conditions of a subset of searchable documents within the knowledge base; then use Enter the enterprise knowledge base module as query text. By embedding the network Encode the query into a query vector and calculate its similarity to the paragraph vectors in the knowledge base. Select the top paragraphs with the highest similarity. The candidate paragraphs are used as a set of candidate answers. This set is then input into a re-ranking model for re-ranking, and the output is a set of ranked knowledge base paragraphs. Binding writeback The session data structure is provided for direct use in the next generation stage.

[0024] Candidate recall phase dependency The two types of index structures were constructed in step one. Vector recall generates query vectors by an embedding network and retrieves nearest neighbors in the vector index: the embedding network adopts a structure of "word segmentation embedding layer + average pooling + two fully connected layers", and the word segmentation results are processed by the server-side word segmenter. The word vectors are generated by looking up a table in the embedding layer, and the pooled sentence vectors are output as query vectors through two fully connected layers. During the initial startup phase, the knowledge base segments each document into paragraph text and encodes it as a vector. And establish a vector index so that the input It can quickly return to paragraphs with similar meanings.

[0025] The initial similarity of candidate paragraphs is defined using the classic cosine similarity in the vector space model: ; in, For query text through The query vector obtained by forward computation of the embedded network is temporarily stored in the current session context; For the knowledge base The vector representation of each paragraph comes from the vector index constructed in step one and corresponds one-to-one with the paragraph text; The inner product is obtained by summing the products of the elements in the corresponding dimensions; and It is the vector norm, obtained by taking the square root of the sum of the squares of each dimension; The similarity results are used for the initial ranking of candidate paragraphs. The system is based on... Sort by highest to lowest and truncate the top results. Each paragraph forms a TopK candidate set.

[0026] The reordering phase is composed of The rerank model is completed within the system. The rerank model uses a cross-coding structure: it converts the query text... Each candidate paragraph text is concatenated with a fixed template into a single sequence and input into a multi-layer self-attention network. Each layer contains multi-head attention units and feedforward network units. Finally, a linear output layer is used to obtain the relevance score of the candidate paragraph. The system then re-sorts the TopK candidate set according to this relevance score. To facilitate engineering implementation, the system performs rerank forward computation on each of the TopK candidates to obtain a score array of the same length. The candidate paragraphs are then sorted from high to low score, and the first few paragraphs are selected as the final knowledge matching result.

[0027] For example, when When querying "What is the loan interest rate this month?", embed network output. Vector indexing by Return the top with the highest similarity The model identifies several candidate paragraphs; then, it calculates the relevance score for each of these candidate paragraphs and re-ranks them, placing paragraphs containing key information such as "this month's definition," "applicable product scope," and "effective date" at the top. The final output set of paragraphs directly serves as the knowledge enhancement context for the next generation stage.

[0028] The steps output as knowledge enhancement objects , its in In the session data structure and The bound storage contains a set of paragraphs reordered using the rerank model; each paragraph entry carries the paragraph text and a source identifier, which is generated when the knowledge base is added and stored in the index along with the paragraph record. The next step is in the same... Reading from above of With the generation of this step The paragraph set allows you to input both the "identified user issues" and the "matched enterprise document evidence" into the semantic generation module simultaneously.

[0029] S4. Read the structured text object and the knowledge enhancement object, construct the input sequence into the local fine-tuning large language model module, use the evidence consistency modulation mechanism to correct the probability of candidate words, generate a text response string and combine it into a response object and write it into the unified data structure. Specifically, step four is in the unified control instance. Intelligent response generation is completed within the same session context, directly from... Read the structured text object generated in step two from the session data structure. With the knowledge enhancement objects generated in step three Both are used to generate input construction and decoding constraints. Provide a complete semantic expression of the user's question. It serves as a session key to locate the set of knowledge segments and the generation buffer corresponding to the current session. Provides enterprise document paragraph text and source identifiers sorted by relevance. System call unified control instance. Large language model within Complete text generation. This is a multi-layer self-attention network structure, comprising a word embedding layer, a multi-head attention layer, a feedforward layer, and an output projection layer. The input text is encoded into a word identifier sequence by a word segmenter, mapped to an embedding vector sequence, and then enters the multi-layer attention and feedforward stacked structure. Finally, the output projection and normalization layer generates the candidate word probability distribution for each time step. This probability distribution forms the basis for decoding in classic autoregressive language models, and the coherence of the generated content comes from the model's conditional probability modeling of the context.

[0030] Before generation, construct the model input sequence, and... and The top paragraphs in the input text are combined using a fixed template to form a single input text: first, the user's question is entered. Then, "source identifier + paragraph text" is appended sequentially by paragraph, and separators are inserted between paragraphs to enable the model to distinguish between the question area and the evidence area. The text sequence after template concatenation enters the word segmenter to obtain the word identifier sequence, and then enters... Get each decoding time step Original candidate word probability Building upon this foundation, the decoding stage introduces evidence consistency modulation, ensuring that candidate word probabilities are simultaneously influenced by both "language model priors" and "the occurrence of corporate evidence." This modulation is based on the classic method of convex combination of multi-source information in probability theory, which involves linearly fusing two probabilities from the same domain according to their weights to obtain a new probability; one source being the language model's output probability. Another source is the support of the evidence set for the candidate words. . By enhancing knowledge objects Word-level statistics were performed on the paragraph text to obtain: Multiple word sets and candidate words were obtained by segmenting all paragraph texts using a word segmenter. The number of times it appears is recorded as The total number of evidence statements is recorded as ,but Based on this, the corrected probability is obtained in convex combination form. And used for candidate word sorting and selection at this time step: ; in, for At time step Output candidate words The original probability comes from the model's forward computation and the output of the normalization layer; The proportion of candidate words appearing in the evidence set, by The paragraph text was obtained by word segmentation and statistical analysis. and All statistics are collected and cached once within the session. The session data structure; The fusion coefficient comes from the deployment configuration file and is loaded at system startup. ; This represents the corrected candidate word probability. Both sides of the equation are probability quantities, with the right side being a linear combination of the two probability quantities, and the coefficients... and Since it is non-negative and its sum is 1, therefore The probability remains within a reasonable range, satisfying the common-sense requirements of probability fusion. Logically, the system first calculates... And cached in the session context, and then from at each time step take out And calculate according to the above formula. Finally Sort and select output words, and generate a sequence that expands progressively until the stopping condition is met.

[0031] To demonstrate the operability of this modulation in enterprise scenarios, a calculation process with a set of parameters is presented. Let the knowledge enhancement object be... The total word count of the evidence paragraph after word segmentation is Among them, candidate words Number of times it appears in the evidence Candidate words Number of occurrences ,but , At a certain time step , Output the original probability , Take the fusion coefficient ,but , At this point, the corrected probability satisfies... The sorting will prioritize... In scenarios such as financial interest rate inquiries, government service application material lists, and medical process guidelines, frequently occurring professional terms or clause keywords in evidence paragraphs typically correspond to business facts or rules. After this modulation, they are more likely to appear in the generated sequence, thus making the output more consistent with the wording and content of enterprise documents. During debugging, this can be achieved by recording each time step. Triples are used to observe the modulation effect. For example, during a generation process, several time steps are sampled to statistically analyze the changes in the "selection ratio of high-frequency evidence words" to assess the fit of the generated evidence after modulation and record the relevance to the conversation identifier. It is also stored in the internal debugging log for backtracking.

[0032] After decoding is complete, the system concatenates the generated word sequence into the final text response string. and with session identifier Combined into a response object Write The conversation data structure is provided for direct reading by the speech synthesis module in the next step. This writing process maintains consistency with the conversation keys used in steps two and three, ensuring... , and In the same This forms a continuous data link, completing the transformation from "identified text + corporate evidence" to "playable response text".

[0033] S5. Read the response object, use the speech synthesis module to convert the text response string into an acoustic feature frame sequence, perform rhythm and energy synchronization modulation on the acoustic feature frame sequence based on the text structure, generate a waveform sample stream and send it to the client; Specifically, step five is in the unified control instance. The speech output is generated within the same conversation context, and the text written in step four is read directly. response object Both fields are used for speech synthesis and output channel binding. Used in Internally, it locates the audio output buffer and WebSocket connection handle corresponding to the session, ensuring that the generated audio data stream is written to the correct session channel; This is the text string to be played, derived from the result of step four. System call. Internal speech synthesis module It performs end-to-end synthesis from text to speech waveform and pushes the waveform segments to the client for playback using a streaming method, realizing the actual output action of the closed loop of voice interaction.

[0034] Speech synthesis module An end-to-end neural network architecture incorporating large-model technology is adopted, which internally includes a text encoding network, a semantic / prosodic representation generation network, and a neural vocoder network. After generating acoustic representations in a streaming manner, it immediately drives the vocoder output waveform. The server side first... Perform word segmentation and sub-word encoding. Sub-word encoding is performed by... The corresponding word segmenter outputs a sequence of word identifiers. This sequence is then mapped to a vector sequence via an embedding layer and fed into a multi-layer self-attention text encoding network. Each layer contains multi-head attention units and feedforward network units to obtain a text representation sequence containing contextual semantics. Subsequently, a semantic / prosodic representation generation network predicts prosodic representations for controlling pauses, stresses, and rhythm based on the text representation sequence, and simultaneously generates an acoustic representation frame sequence for the vocoder to decode. This acoustic representation frame sequence is generated block by block at a fixed frame size and written to the session buffer. The neural vocoder network is a multi-layer one-dimensional convolutional and upsampling structure that converts the acoustic representation frame sequence block by block into a time-domain waveform sample sequence, enabling streaming synthesis with simultaneous generation and playback.

[0035] To ensure clarity of critical business information (such as numerical values, dates, clause numbers, and concluding statements) in speech output during specific scenarios, the system introduces a prosodic control term aligned with the text structure during the acoustic representation generation stage. This control term is inspired by the common practice in speech synthesis engineering of controllably modulating the prosodic / energy envelope, applying amplitude or prosodic weights consistent with the text structure to the acoustic representation, thereby highlighting key segments and reinforcing pause boundaries in the auditory perception. The system then... The position of the marker and the digital segment generate a frame-by-frame weight sequence. and at the acoustic characterization frame output, the original frame Multiplicative modulation is obtained Then As input to the vocoder network: ; in, for At time step The output raw acoustic representation frame vector comes from the forward computation results of the semantic / prosodic representation generation network; For the system to The frame-by-frame weight values ​​generated after marking punctuation positions and digital segments are written to the session buffer through the alignment mapping from "text unit to frame interval". The modulation intensity coefficients are derived from the deployment configuration file and loaded during system startup. ; This is the modulated acoustic representation frame vector, used to drive the neural vocoder to generate waveform samples.

[0036] For example, when a conversation Containing the key numerical segment "3.5 percent", after alignment, it is taken within the corresponding acoustic frame interval. Ordinary narrative intervals are taken Modulation intensity is taken The modulation coefficient of the key interval is then... The modulation coefficient in the normal range is If the original representation frame vector of a certain keyframe If the component in a certain dimension is 0.30, then after modulation, that component becomes... If the value of this dimension component in a given ordinary frame is 0.30, then the modulated result is... This frame-by-frame modulation results in key segments being more easily perceived and pauses being clearer in the vocoder output, thus meeting the engineering requirements for broadcasting key fields.

[0037] The neural vocoder network receives the modulated acoustic representation frame sequence and generates a waveform sample stream, which is then written into segments of fixed length. Internal and The bound audio output buffer is sent sequentially to the client via WebSocket. The client decodes and plays the audio in the order it is received, forming a voice output that is generated and played simultaneously. This step ultimately forms a voice data stream object bound to a session identifier. Write The session data structure, in which References or stream handles to waveform sample sequences generated and sent by the vocoder, for subsequent session termination and resource release processes to track sending and playback completion status.

[0038] S6. After detecting that the audio transmission is complete and receiving confirmation from the client that playback is complete, calculate the session complexity index, release the cache resources and network connection handle associated with the session identifier, and delete the session state record.

[0039] Specifically, step six is ​​in the unified control instance. The session termination and data recycling processes are completed within the same session context, and the voice data stream object generated in step five is read directly. and in conjunction with the session Existing internal , and Perform full lifecycle management. A unique identifier for the current session, used in Locate all data structures and network resources corresponding to the session in the session mapping table; This refers to the waveform sample stream or stream handle that has been sent to the client, used to determine the completion status of voice playback. After the bound audio is sent and the client confirms that playback is complete, the session recycling phase begins.

[0040] In the retrieval phase, to perform system load analysis and capacity planning without preserving the original speech and complete text content, this step introduces a session complexity metric based on information content approximation. When the number of results for an event is... At that time, its information content can be represented by a logarithmic function. Considering the scenario of this system, the length of the generated text and the number of cited knowledge paragraphs are regarded as the two main factors affecting session resource consumption, among which the length of the generated text... From the response text The number of knowledge paragraphs was obtained from the character count statistics. Knowledge-enhanced objects The number of paragraph entries involved in the splicing is obtained. To reflect the increasing but gradually decreasing impact of the increasing number of paragraphs on complexity, a logarithmic function is introduced to construct a complexity index: ; in, This is a metric for session complexity. For response text The number of characters is calculated by the server using a string length function at the end of the session; The number of knowledge paragraph entries generated is recorded and saved during step four when constructing the input sequence. In the session context; The natural logarithm function is used for compression. The growth rate of the number of paragraphs causes the complexity to increase logarithmically. The right side of the equation is the product of the character count and the logarithmic value, both of which are dimensionless statistics. The product is also a dimensionless value, consistent with the complexity index on the left side.

[0041] The derivation logic is: if only using As a measure of complexity, it cannot distinguish between "long texts with a single piece of evidence" and "medium-length texts with multiple pieces of evidence fused together"; if only using This would fail to reflect the scale of the text output. By introducing... ,exist The time complexity is 0, when The complexity increases with the logarithm, thus achieving a smooth amplification of the impact on the amount of evidence. This structure is equivalent to treating the number of paragraphs as an influencing factor in the resource model, with its marginal contribution decreasing, which aligns with the actual situation of system resource consumption.

[0042] A set of computational examples is given to illustrate its operability. Suppose the response text of a certain session contains 12 characters. The number of knowledge segments generated is ,but If the natural logarithm is used for calculation, ,but In another session if and ,but The comparison shows that the first session has higher text length and more evidence, and its complexity index is significantly higher, making it suitable for subsequent statistical analysis. The system will then... and Write to internal statistics logs, but do not retain them. Compared to the original audio content.

[0043] After completing the complexity calculation, the system... Release resources item by item in the session mapping table: release the speech frame buffer, acoustic feature buffer, generation buffer, and temporary knowledge segment splicing area; close the network connection handle corresponding to the session; and delete the session state record. The resource release order follows the principle of "close the output stream first, then release the memory buffer" to avoid prematurely destroying the buffer before transmission is complete. All release operations are performed through... An internal unified interface is used to ensure the consistency of the data structure.

[0044] The final output of this step is the updated unified control instance. Among them, with The associated runtime data has been recovered, with only statistical indicator records retained. By introducing a logarithmically growing information content model to quantify session complexity and uniformly releasing resources at the end of the session, this step achieves a statistically accurate, traceable, and resource-controllable session lifecycle management mechanism in a private deployment scenario.

[0045] This invention also provides a one-stop construction device for a privately deployed, end-to-end voice-interactive intelligent agent, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the steps described in the above embodiment of a one-stop construction method for a privately deployed, end-to-end voice-interactive intelligent agent. Figure 1 The steps S1 to S4 described above; or, when the processor executes the computer program, it implements the functions of each module in the above system embodiments.

[0046] For example, the computer program may be divided into one or more modules, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules may be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in a privately deployed, end-to-end voice-interactive intelligent agent one-stop construction device.

[0047] The aforementioned one-stop construction device for a privately deployed, end-to-end voice-interactive intelligent agent can be a computing device such as a desktop computer, laptop, handheld computer, or cloud server. This device may include, but is not limited to, processors and memory. Those skilled in the art will understand that the device may also include input / output devices, network access devices, buses, etc.

[0048] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. This processor is the control center of the proprietary, end-to-end voice-interactive intelligent agent one-stop construction device, connecting all parts of the device via various interfaces and lines.

[0049] The memory can be used to store the computer programs and / or modules. The processor, by running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory, realizes various functions of the one-stop construction device for a privately deployed, end-to-end voice-interactive intelligent agent. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application required for a function, etc.; the data storage area may store data created based on the operation of the air conditioner controller, etc. In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital card (SD card), flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage devices.

[0050] The modules integrated into the one-stop construction device for a privately deployed, end-to-end voice-interactive intelligent agent, if implemented as software functional units and sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0051] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0052] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A one-stop construction method of a full-process voice interaction intelligent agent for a private deployment, characterized in that, The method includes: S1. Load the speech recognition module, enterprise knowledge base module, local fine-tuning large language model module and speech synthesis module on the enterprise internal server. Integrate the modules through a unified control instance to form a running instance. The unified control instance initializes a unified data structure to carry session identifiers, speech data, text data and knowledge context fields. S2. Receive the audio frame sequence sent by the client and write it into the voice field cache of the unified data structure. Use the voice recognition module to perform endpoint detection smoothing, acoustic decoding and punctuation completion on the audio frame sequence to generate a structured text object containing session identifier and text string and write it into the unified data structure. S3. Based on the text string in the structured text object, vector recall and keyword recall are executed in parallel in the enterprise knowledge base module. The two recall results are sorted through a fusion scoring framework to generate a knowledge enhancement object containing the sorted paragraph set and written into the unified data structure. S4. Read the structured text object and the knowledge enhancement object, construct the input sequence into the local fine-tuning large language model module, use the evidence consistency modulation mechanism to correct the probability of candidate words, generate a text response string and combine it into a response object and write it into the unified data structure. S5. Read the response object, use the speech synthesis module to convert the text response string into an acoustic feature frame sequence, perform rhythm and energy synchronization modulation on the acoustic feature frame sequence based on the text structure, generate a waveform sample stream and send it to the client; S6. After detecting that the audio transmission is complete and receiving confirmation from the client that playback is complete, calculate the session complexity index, release the cache resources and network connection handle associated with the session identifier, and delete the session state record. 2.The full-process voice interaction intelligent agent one-stop construction method according to claim 1, characterized in that, The specific process of the endpoint detection smoothing process is as follows: obtain the frame-by-frame speech activity probability output by the endpoint detection network, use a first-order discrete low-pass filter to smooth the frame-by-frame speech activity probability to obtain a smoothing probability, mark the entry into the speech segment when the smoothing probability is continuously higher than the activity threshold, mark the end of the speech segment when the smoothing probability is continuously lower than the silence threshold, and extract the audio frame sequence corresponding to the speech segment and input it into the acoustic model for decoding. 3.The full-process voice interaction agent one-stop construction method of claim 1, wherein, The calculation method of the fusion scoring framework includes: calculating the cosine similarity between the query vector and the candidate paragraph vector, using the cosine similarity calculation result as the initial ranking basis, and extracting the top K candidate paragraphs to form a TopK candidate set; The TopK candidate set is input into the reordering model, and the multi-layer self-attention network in the reordering model is used to cross-encode the concatenated sequence of the query text and candidate paragraphs. The relevance score of the candidate paragraphs is calculated through the linear output layer. The TopK candidate set is reordered according to the relevance score, and the top-ranked entries are selected as the final knowledge enhancement objects.

4. The end-to-end voice interaction agent one-stop construction method of claim 1, wherein, The specific process of the evidence consistency modulation mechanism is as follows: statistically analyze the occurrence frequency of candidate words and the total number of evidence words in the paragraph text after word segmentation in the knowledge enhancement object, calculate the occurrence ratio of the candidate words in the evidence set, and linearly fuse the original candidate word probability output by the local fine-tuning large language model module with the occurrence ratio according to the fusion coefficient to obtain the corrected candidate word probability for decoding selection.

5. The end-to-end voice interaction agent one-stop construction method of claim 1, wherein, The process of rhythm and energy synchronization modulation of acoustic feature frame sequence based on text structure includes: reading the text response string, performing word segmentation and sub-word encoding through a word segmenter, and mapping the sub-word identifier sequence into a vector sequence through an embedding layer; The vector sequence is input into a multi-layer self-attention text encoding network to obtain a text representation sequence containing contextual semantics, and prosodic related representations are predicted based on the text representation sequence. Based on prosodic relevance representation, an acoustic feature frame sequence is generated, and the acoustic feature frame sequence is modulated: the text response string is marked with punctuation positions and numerical segments, the marks are mapped to the time steps of the acoustic feature frame sequence to generate a text-driven rhythm weight sequence, and the original acoustic feature frame vector is multiplicatively modulated using the modulation intensity coefficient and the text-driven rhythm weight sequence to obtain the modulated acoustic feature frame vector. 6.The full-process voice interaction intelligent agent one-stop construction method according to claim 5, characterized in that, The specific method for mapping the markers to the time step of the acoustic feature frame sequence is as follows: generate frame-by-frame weight values ​​based on punctuation position markers and digital segment markers, and write the frame-by-frame weight values ​​into the session buffer to form a rhythm weight sequence through the alignment mapping from text units to frame intervals. A phoneme-to-frame repetitive expansion method is adopted to determine the frame interval corresponding to each phoneme. The frame interval is assigned a value to form a frame-by-frame text-driven rhythm weight sequence, wherein the frame interval corresponding to the key numerical segment is assigned a first weight value, and the frame interval corresponding to the ordinary narrative interval is assigned a second weight value.

7. The end-to-end voice interaction agent one-stop construction method of claim 1, wherein, The calculation basis of the session complexity index is as follows: the number of characters in the response text is obtained as the first parameter, the number of knowledge paragraph entries generated is obtained as the second parameter, and the product of the natural logarithm of the first parameter and the second parameter is used as the session complexity index to characterize the degree of session resource consumption. 8.The full-process voice interaction intelligent agent one-stop construction method according to claim 1, characterized in that, The specific order for releasing the cached resources associated with the session identifier is as follows: first, close the audio output stream bound to the session identifier; then, release the speech frame buffer, acoustic feature buffer, generation buffer, and knowledge segment temporary splicing area in sequence; and finally, close the network connection handle and delete the session state record. 9.The full-process voice interaction intelligent agent one-stop construction method of claim 1, wherein, The initialization process of the enterprise knowledge base module includes: reading text data from an internal document directory or database, segmenting it into natural paragraphs or fixed lengths, encoding each text segment into a fixed-dimensional vector using an embedding network, and establishing a memory vector index structure. The embedding network includes a word segmentation embedding layer, an average pooling layer, and two fully connected layers.

Images