A processing method and related equipment for an intelligent telephone assistant

Abstract

This invention discloses a processing method and related equipment for an intelligent telephone assistant. The method includes: responding to a call initiated by a target object through a communication device, acquiring speech-to-text transcription to obtain a complete transcribed text, and outputting a classification result using a large language model; when the classification result is a general question-and-answer type, retrieving a target document fragment from a pre-built knowledge base based on the complete transcribed text, and generating an answer text using the large language model based on the complete transcribed text and the target document fragment; when the classification result is an intent type, calling the application programming interface corresponding to the intent type to execute the target operation, and then generating a response text based on the result of the target operation; synthesizing target speech using a speech synthesis model based on the answer text or the response text, and playing the target speech to the target object through a voice channel. This invention can significantly simplify the interaction process and effectively enhance the intelligence and real-time experience of the system, and can be widely applied in the field of audio processing technology.

Description

Technical Field

[0001] This invention relates to the field of audio processing technology, and in particular to a processing method and related equipment for an intelligent telephone assistant. Background Technology

[0002] Traditional telephone self-service systems use interactive voice response (IVR) technology. These systems typically involve pre-recording multi-level voice menus, and in practice, after dialing a number, users need to select the corresponding function nodes one by one by pressing buttons or using simple voice commands according to the voice prompts.

[0003] However, this interaction method based on a preset menu tree requires users to follow a fixed navigation path throughout, which cannot handle diverse requests expressed in natural language, resulting in cumbersome operation and rigid response. Summary of the Invention

[0004] The main objective of this invention is to provide a processing method, apparatus, electronic device, storage medium, and program product for a smart telephone assistant, aiming to solve at least one problem of the prior art.

[0005] To achieve the above objectives, one aspect of the present invention provides a processing method for an intelligent telephone assistant, the method comprising: In response to a call initiated by the target via a communication device, a voice channel is established; Collect the voice transmitted by the target object through the voice channel, and then obtain the complete transcribed text through speech transcription; Based on the complete transcribed text, a large language model is used to output classification results; the classification results include general question-and-answer class or intent class, and the intent class is associated with intent type. When the classification result is a general question and answer category, the target document fragment is retrieved from the pre-built knowledge base based on the complete transcribed text, and the answer text is generated using a large language model based on the complete transcribed text and the target document fragment. When the classification result is an intent type, the application programming interface corresponding to the intent type is called to execute the target operation, and then the response text is generated based on the result of the target operation; The target speech is synthesized from the answer text or response text using a speech synthesis model, and then played to the target audience through a speech channel.

[0006] In some embodiments, the process of acquiring the speech transmitted by the target object through a voice channel and then obtaining the complete transcribed text through speech-to-text includes the following steps: Continuously acquire audio packets from the communication device via the voice channel at a fixed frame length; The audio packet is fed into the speech activity detection model to obtain the speech activity detection result at each time point; the speech activity detection result includes speech or non-speech. When the speech activity detection result changes from non-speech to speech, the speech start time is determined by the time point when the speech start time is determined. Audio packets are continuously cached from the speech start time until the speech activity detection result changes to non-speech, and then audio data is generated. The audio data is fed into a speech recognition model for streaming recognition to obtain the complete transcribed text.

[0007] In some embodiments, the method further includes the following steps: If the duration of the non-speech function test result from the start of the test reaches the first time threshold; Play preset voice prompts to the target audience via the voice channel; If the voice activity detection result remains non-voice, a preset voice prompt will be played once every first time threshold duration. If the preset number of times the voice prompt is played reaches the preset threshold, the voice channel will be turned off.

[0008] In some embodiments, based on the complete transcribed text, a large language model is used to output classification results, including the following steps: The complete transcribed text is combined with the preset prompt word template and then input into the large language model to obtain the structured classification results output by the large language model. The prompt word template includes a first instruction that directs the large language model to perform general question-and-answer or intent recognition, and a second instruction that uses preset fields to represent different intent types.

[0009] In some embodiments, a target document fragment is retrieved from a pre-built knowledge base based on the complete transcribed text, and an answer text is generated using a large language model based on the complete transcribed text and the target document fragment, including the following steps: The complete transcribed text is transformed into question vectors using an embedding model. Similarity retrieval is used to retrieve target document fragments that match the question vector from a pre-built knowledge base constructed through document vectorization. The complete transcribed text is concatenated with the retrieved target document fragments and then input into the large language model so that the large language model can process and generate the answer text. If no target document fragment matching the question vector is found in the knowledge base, the preset prompt reply text will be used as the answer text.

[0010] In some embodiments, the intent type includes call transfer, phone number lookup, and notification. The process involves calling the application programming interface (API) corresponding to the intent type to perform the target operation, and then generating response text based on the result of the target operation, including the following steps: If the intent type is transfer, call the call transfer interface and pass in the target number to perform the target operation of transfer dialing; If the intent type is to query a phone number, call the address book query interface and pass in the name to perform the target operation of querying the number; If the intent type is notification, call the notification interface and pass in the notification content and object to perform the target operation of the content notification; Among them, the target number, name, or notification content and recipient are extracted from the complete transcribed text; Obtain the execution status of the target operation and generate response text based on the execution status mapping; The response text includes confirmation of successful operation or a description of the reason for operation failure.

[0011] In some embodiments, after the step of playing the target speech to the target object via the voice channel, the method further includes the following steps: Return to the step of collecting the voice transmitted by the target object through the voice channel to achieve multi-round voice interaction.

[0012] To achieve the above objectives, another aspect of the present invention provides a processing device for a smart phone assistant, the device comprising: The first module is used to establish a voice channel in response to a call initiated by the target object through a communication device; The second module is used to collect the voice transmitted by the target object through the voice channel, and then obtain the complete transcribed text through voice transcription. The third module is used to output classification results based on the complete transcribed text using a large language model. The classification results include general question-and-answer categories or intent categories, with intent categories associated with intent types. The fourth module is used when the classification result is a general question and answer type. It retrieves the target document fragment from the pre-built knowledge base based on the complete transcribed text, and uses a large language model to generate the answer text based on the complete transcribed text and the target document fragment. The fifth module is used to call the application programming interface corresponding to the intent type to perform the target operation when the classification result is an intent type, and then generate response text based on the result of the target operation; The sixth module is used to synthesize target speech based on the answer text or response text using a speech synthesis model, and then play the target speech to the target audience through the speech channel.

[0013] To achieve the above objectives, another aspect of the present invention provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the aforementioned method.

[0014] To achieve the above objectives, another aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned method.

[0015] To achieve the above objectives, another aspect of the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned method.

[0016] The embodiments of the present invention include at least the following beneficial effects: The present invention provides a processing method, device, electronic device, storage medium, and program product for an intelligent telephone assistant. This solution establishes a voice channel in response to a call initiated by a target object through a communication device; collects the voice transmitted by the target object through the voice channel, and then obtains a complete transcribed text through speech-to-text transcription; based on the complete transcribed text, outputs a classification result using a large language model; the classification result includes a general question-and-answer category or an intent category, with the intent category associated with an intent type; when the classification result is a general question-and-answer category, a target document fragment is retrieved from a pre-built knowledge base based on the complete transcribed text, and an answer text is generated using a large language model based on the complete transcribed text and the target document fragment; when the classification result is an intent category, an application programming interface corresponding to the intent type is called to execute the target operation, and then a response text is generated based on the result of the target operation; the target speech is synthesized using a speech synthesis model based on the answer text or the response text, and the target speech is played to the target object through the voice channel. This invention integrates voice interaction, natural language understanding, knowledge retrieval, and operation execution into a complete intelligent telephone assistant processing flow. Specifically, it utilizes a large language model to uniformly classify the complete transcribed text into questions and answers and intents, distinguishing between ordinary knowledge inquiries and intents requiring specific communication operations, thus avoiding the limitations of rule matching. When the request is a simple question and answer, the target document fragment is retrieved from the knowledge base and combined with the question in the original voice text to generate the answer text, effectively ensuring the accuracy and knowledge coverage of the answer. When the request is an intent-based request, the corresponding application programming interface is directly called to execute the target operation and generate the response text, enabling seamless information retrieval and call control, thereby significantly improving operational efficiency. Finally, the answer text or response text is converted into the target voice for playback through speech synthesis to form a complete voice loop. Throughout the entire process, the user only needs to provide voice instructions without any manual intervention, significantly simplifying the interaction process and effectively enhancing the system's intelligence and real-time experience. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of an implementation environment for the intelligent telephone assistant processing method provided in an embodiment of the present invention; Figure 2 This is a flowchart illustrating the processing method of the intelligent telephone assistant provided in an embodiment of the present invention; Figure 3 This is a schematic diagram illustrating the overall process of the intelligent telephone assistant processing method provided in this embodiment of the invention; Figure 4 This is a schematic diagram of the processing device for the intelligent telephone assistant provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of this invention; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this invention as detailed in the appended claims.

[0019] It is understood that the terms "first," "second," etc., used in this invention may be used to describe various concepts, but unless specifically stated otherwise, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of embodiments of this invention, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words "if" or "when" as used herein may be interpreted as "when," "in response to determination," or "in the event of a determination."

[0020] The terms “at least one,” “multiple,” “each,” “any,” etc., used in this invention, “at least one” includes one, two, or more than two; “multiple” includes two or more than two; “each” refers to each of the corresponding multiple; and “any” refers to any one of the multiple.

[0021] Unless otherwise defined, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used in this invention is for descriptive purposes only and is not intended to limit the invention.

[0022] Among related technologies, existing telephone assistant systems generally suffer from insufficient intelligence, poor naturalness of interaction, and low functional integration, making it difficult to meet current users' needs for efficient, one-stop intelligent telephone services.

[0023] In view of this, this invention provides a processing method and related equipment for an intelligent telephone assistant. This solution establishes a voice channel in response to a call initiated by a target object through a communication device; collects the voice transmitted by the target object through the voice channel, and then obtains a complete transcribed text through speech-to-text transcription; based on the complete transcribed text, outputs a classification result using a large language model; the classification result includes a general question-and-answer category or an intent category, with the intent category associated with an intent type; when the classification result is a general question-and-answer category, a target document fragment is retrieved from a pre-built knowledge base based on the complete transcribed text, and an answer text is generated using a large language model based on the complete transcribed text and the target document fragment; when the classification result is an intent category, the application programming interface corresponding to the intent type is called to execute the target operation, and then a response text is generated based on the result of the target operation; the target speech is synthesized using a speech synthesis model based on the answer text or the response text, and the target speech is played to the target object through the voice channel. This invention integrates voice interaction, natural language understanding, knowledge retrieval, and operation execution into a complete intelligent telephone assistant processing flow. Specifically, it utilizes a large language model to uniformly classify the complete transcribed text into questions and answers and intents, distinguishing between ordinary knowledge inquiries and intents requiring specific communication operations, thus avoiding the limitations of rule matching. When the request is a simple question and answer, the target document fragment is retrieved from the knowledge base and combined with the question in the original voice text to generate the answer text, effectively ensuring the accuracy and knowledge coverage of the answer. When the request is an intent-based request, the corresponding application programming interface is directly called to execute the target operation and generate the response text, enabling seamless information retrieval and call control, thereby significantly improving operational efficiency. Finally, the answer text or response text is converted into the target voice for playback through speech synthesis to form a complete voice loop. Throughout the entire process, the user only needs to provide voice instructions without any manual intervention, significantly simplifying the interaction process and effectively enhancing the system's intelligence and real-time experience.

[0024] It is understood that the intelligent phone assistant processing method provided by this invention can be applied to any computer device with data processing and computing capabilities, and this computer device can be various terminals or servers. When the computer device in the embodiment is a server, the server is an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Optionally, the terminal can be a smartphone, tablet computer, laptop computer, or desktop computer, but it is not limited to these.

[0025] like Figure 1The diagram shown is a schematic representation of an implementation environment provided by an embodiment of the present invention. (Refer to...) Figure 1 The implementation environment includes at least one terminal 102 and a server 101. The terminal 102 and the server 101 can be connected via a network, either wirelessly or via a wired connection, to complete data transmission and exchange.

[0026] Server 101 can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0027] Additionally, server 101 can also be a node server in a blockchain network. Blockchain is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms.

[0028] Terminal 102 can be a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. Terminal 102 and server 101 can be directly or indirectly connected via wired or wireless communication, and this embodiment of the invention does not impose any limitations.

[0029] For example, based on Figure 1 The implementation environment shown in this embodiment of the invention provides a processing method for a smart phone assistant. The following description uses the application of this smart phone assistant processing method in server 101 as an example. It can be understood that the processing method of the smart phone assistant can also be applied in terminal 102.

[0030] Reference Figure 2 , Figure 2 This is an optional flowchart of the processing method of the intelligent phone assistant provided in the embodiment of the present invention. The executing entity of the processing method of the intelligent phone assistant can be any of the aforementioned computer devices (including servers or terminals). Figure 2 The method may include, but is not limited to, steps S100 to S600.

[0031] Step S100: In response to a call initiated by the target object through a communication device, a voice channel is established; Among them, when establishing a voice channel, a preset welcome message can be played simultaneously to the target audience; For example, in some specific implementations, when a user dials the phone number corresponding to the smart phone assistant, a voice channel is first established between the smart phone assistant and the user's communication device, and then a pre-set welcome message can be played through the communication device.

[0032] Step S200: Collect the speech transmitted by the target object through the voice channel, and then obtain the complete transcribed text through speech transcription; It should be noted that, in some embodiments, step S200 may include the following steps: continuously acquiring audio packets from a communication device via a voice channel with a fixed frame length; sending the audio packets into a voice activity detection model to obtain the voice activity detection result corresponding to each time point; wherein, the voice activity detection result includes voice or non-voice; determining the voice start time in response to the time point when the voice activity detection result changes from non-voice to voice, continuously caching audio packets from the voice start time until the voice activity detection result changes from non-voice to non-voice, generating audio data; sending the audio data into a speech recognition model for streaming recognition to obtain complete transcribed text.

[0033] For example, in some specific implementations, determining when a user starts speaking is specifically implemented as follows: when a user picks up the phone or dials a number, the communication device starts sending a set threshold (e.g., a 20-millisecond voice packet) to the ASR (Automatic Speech Recognition) service. The ASR is preceded by a VAD (Voice Activity Detection) model, which detects whether someone is speaking. When speech is detected, the voice is stored. When the user is not speaking and the set silence threshold is exceeded, the currently stored speech is sent to the ASR for streaming recognition to obtain the complete transcribed text.

[0034] It should be noted that in some embodiments, the method may further include the following steps: if the duration of the non-speech function of the speech activity detection result from the start of detection reaches a first time threshold; play a preset voice prompt to the target object through the voice channel; if the speech activity detection result still remains non-speech, play the preset voice prompt once every duration corresponding to the first time threshold; if the number of times the preset voice prompt is played reaches a preset number threshold, close the voice channel.

[0035] For example, in some specific implementations, if the user remains silent for an extended period, the communication device can automatically schedule TTS (Text-To-Speech) to play a voice message, such as "Are you still there?". The entire conversation can be automatically ended by the user, or the communication device can automatically end the conversation if the user remains silent for an extended period, even after multiple voice prompts.

[0036] Step S300: Based on the complete transcribed text, output the classification result using a large language model; The classification results include general question-and-answer categories or intent categories, with intent categories associated with intent types; It should be noted that in some embodiments, step S300 may include the following steps: combining the complete transcribed text with a preset prompt word template and inputting it into a large language model to obtain a structured classification result output by the large language model; wherein, the prompt word template contains a first instruction instructing the large language model to perform ordinary question and answer or intent recognition and a second instruction using preset fields to represent different intent types.

[0037] For example, in some specific implementations, the large model distinguishes between "ordinary questions and answers" and "intents" by writing prompt words (for example, by associating various preset intent types through keyword mapping, while other content is treated as ordinary questions and answers) to drive the large language model for classification. For example, ordinary question and answer recognition: "Help me check the iteration function of XX software", output: "[Ordinary question and answer: query the iteration function of XX software]". For example, intent recognition: "Transfer to Manager Zhang's phone", output: "[Intent: transfer; Name: Manager Zhang]"; "Check Mr. Wang's phone number", output: "[Intent: check phone number; Name: Mr. Wang]".

[0038] Step S400: When the classification result is a general question and answer type, the target document fragment is retrieved from the pre-built knowledge base based on the complete transcribed text, and the answer text is generated using a large language model based on the complete transcribed text and the target document fragment. It should be noted that in some embodiments, the process of retrieving target document fragments from a pre-built knowledge base based on the complete transcribed text, and generating answer text using a large language model based on the complete transcribed text and the target document fragments, may include the following steps: converting the complete transcribed text into a question vector using an embedding model; retrieving a target document fragment matching the question vector from a pre-built knowledge base constructed through document vectorization using similarity retrieval; concatenating the complete transcribed text with the retrieved target document fragments and inputting the result into the large language model so that the large language model can process and generate answer text; if no target document fragment matching the question vector is retrieved from the knowledge base, a preset prompt reply text is used as the answer text.

[0039] For example, in some specific implementations, the knowledge base retrieval is implemented as follows: the knowledge base can be built using Dify (an open-source large language model application development platform), employing an embedding (vectorization) model, and uploading documents to store them in a database. The embedding model vectorizes the documents and retrieves them from the database. The large model combines the input question with the results retrieved from the documents to automatically output the answer. If the answer is not found in the knowledge base, prompts can be set to indicate that the question is unrelated to a specific question and only answers related questions.

[0040] Step S500: When the classification result is an intent class, the application programming interface corresponding to the intent type is called to perform the target operation, and then the response text is generated based on the result of the target operation. It should be noted that intent types include call transfer, phone number lookup, and notification. In some embodiments, calling the application programming interface corresponding to the intent type to execute the target operation, and then generating response text based on the result of the target operation, may include the following steps: if the intent type is call transfer, call the call transfer interface and pass in the target number to execute the target operation of call transfer dialing; if the intent type is phone number lookup, call the address book lookup interface and pass in the name to execute the target operation of number lookup; if the intent type is notification, call the notification interface and pass in the notification content and object to execute the target operation of content notification; wherein, the target number, name, or notification content and object are extracted from the complete transcribed text; obtain the execution status of the target operation, and generate response text based on the execution status mapping; wherein, the response text includes confirmation information of successful operation or a description of the reason for operation failure.

[0041] For example, in some specific implementations, an agent can call an API (Application Programming Interface) to trigger corresponding intents, thereby implementing scheduling operations corresponding to different intent types. The agent's functionality is as follows: it integrates multiple intents; after writing a prompt, it is recognized as a specific intent such as call transfer or notification, and the corresponding function is automatically triggered. The agent is responsible for issuing a specific intent, such as: "Look up phone number, Name: Mr. Wang". The backend queries Mr. Wang's phone number, which can then be transferred. Alternatively, the complete phone number can be written for transfer. If the transfer fails, a text-to-speech (TTS) message is sent explaining why the transfer could not be completed. For example: "Notifying Mr. Wang of a meeting at 4 PM"; triggering the notification intent will send a message to TTS: "Dear Mr. Wang, you are notified of a meeting at 4 PM."

[0042] Step S600: Synthesize the target speech based on the answer text or response text using a speech synthesis model, and play the target speech to the target object through the speech channel; For example, in some specific implementations, the answer text or response text is ultimately translated into speech and played back to the user via TTS.

[0043] It should be noted that in some embodiments, the method may further include the following steps: returning to the step of collecting the voice transmitted by the target object through the voice channel to achieve multi-round voice interaction.

[0044] For example, in some specific implementations, the user's voice that is collected again will be used as a new audio stream source to repeat the complete processing loop of speech recognition, intent classification, and question-and-answer or intent execution.

[0045] To explain in detail the principle of the technical solution of the present invention, the overall process of the present invention will be described below with reference to some specific embodiments. It is easy to understand that the following is an explanation of the technical principle of the present invention and should not be regarded as a limitation of the present invention.

[0046] First, it should be noted that existing methods of accessing information, making calls, or sending notifications via telephones, mobile phones, and other communication devices require manual dialing or consulting software, which is obviously inefficient and cannot meet the needs of today's smart phones.

[0047] With the continuous development of the economy and computer technology, artificial intelligence (AI) technology has been widely developed. The combination of AI technology and computers has received widespread attention. With the emergence of large-scale models, many scholars have designed different network architectures to solve intelligent question answering and knowledge retrieval problems in different fields, all aiming to achieve faster and better automated question answering and knowledge retrieval, thereby replacing manual labor and improving diagnostic efficiency and accuracy.

[0048] Therefore, in order to improve the efficiency of mobile phone use, this invention proposes an intelligent phone assistant with functions such as automatic dialing, call transfer, notification, inquiry, and intelligent question answering. Figure 3 The diagram shows a flowchart example of the processing method for the intelligent phone assistant of the present invention. In some specific embodiments, the processing method for the intelligent phone assistant of the present invention can be implemented as follows: 1. By using communication devices such as IPPBX (Internet Protocol Private Branch Exchange), which enable internal and external phone calls, and combining them with an intelligent question and answer database, the system can automatically query, transfer, and notify human operators for mobile or landline calls. This can also achieve the goal of building an intelligent question and answer knowledge base.

[0049] 2. To achieve intelligent voice interaction using mobile phones or landlines via voice interaction, the ASR model for speech-to-text needs to be Qwen3-ASR. The TTS system for text-to-speech uses kokoro (an open-source text-to-speech (TTS) engine) to realize the intelligent voice interaction system.

[0050] 3. Build an intelligent agent, establish a knowledge base, and combine communication devices and voice models to design an intelligent telephone assistant system.

[0051] The main process of the intelligent phone assistant processing method of the present invention can be implemented as follows: When a user dials the phone number corresponding to the intelligent phone assistant, a pre-set welcome message is first played through the communication device. The user speaks through a mobile phone or landline, and the speech is sent to ASR for translation into text, which is then sent to a large model. If the large model confirms that it is a normal question and answer, it accesses the knowledge base, outputs the result through the large model, and finally translates it into speech through TTS and plays it to the user. If it is an intent, an agent is used to implement the scheduling of this intent, triggering the corresponding intent function, such as call transfer, notification, or phone number lookup, and outputting the result to the user, thus completing the functions of call transfer, notification, and phone number lookup.

[0052] In some optional implementations, determining when a user begins to speak specifically involves the following: When a user picks up the phone or dials a number, the communication device begins sending voice packets to the ASR service at a set threshold (e.g., 20 milliseconds). The ASR is preceded by a VAD (Voice Activity Detection) model to detect if someone is speaking. When speech is detected, the voice is stored. When the user remains silent, exceeding a set silence threshold, the currently stored speech is sent to the ASR. If the user remains silent for an extended period, the communication device can automatically schedule a TTS (Text-to-Speech) message, such as "Are you still there?". The entire conversation can be automatically ended by the user, or the communication device can automatically end the conversation if the user remains silent for an extended period, even after multiple verbal prompts.

[0053] In some optional implementations, the large model distinguishes between "ordinary questions and answers" and "intents" by: driving the large language model to classify by writing prompt words (for example, by associating various preset intent types through keyword mapping, while other content is treated as ordinary questions and answers). For example, ordinary question and answer recognition: "Help me check the iterative functions of XX software", output: "[Ordinary question and answer: Query the iterative functions of XX software]". For example, intent recognition: "Transfer to Manager Zhang's phone", output: "[Intent: Transfer; Name: Manager Zhang]"; "Check Mr. Wang's phone number", output: "[Intent: Check phone number; Name: Mr. Wang]".

[0054] In some optional implementations, the knowledge base retrieval is specifically implemented as follows: the knowledge base can be built using Dify, employing an Embedding model, and documents can be uploaded and stored in a database. The Embedding model performs document vectorization and retrieves data from the database. The main model combines the input question with the results retrieved from the documents to automatically output the answer. If the answer is not found in the knowledge base, prompts can be set to indicate that the question is unrelated to a specific topic and only answers related questions.

[0055] In some optional implementations, the AGENT's function is as follows: The AGENT integrates multiple intents. After a prompt is written, it is recognized as a specific transfer intent, and the corresponding function is automatically triggered. The AGENT is responsible for issuing a specific intent, such as: "Look up phone number, Name: Mr. Wang". The backend queries Mr. Wang's phone number and can transfer the call. Alternatively, a complete phone number can be written for transfer. If the transfer fails, a text-to-speech (TTS) message is sent explaining why the transfer could not be completed. For example: "Notifying Mr. Wang of a meeting at 4 PM." This triggers the notification intent and sends the message to TTS: "Dear Mr. Wang, you are notified of a meeting at 4 PM."

[0056] In some specific application scenarios, the processing method of the intelligent telephone assistant of the present invention can be applied as follows: S1: Database creation; The database needs to be organized and uploaded via Dify for setup.

[0057] S2: Setting up chat flow; Use Dify to build a chat flow and implement the overall workflow functionality. S3: Model Application; Intent recognition and general question answering are achieved by combining a large language model (LLM).

[0058] S4: Voice interaction model; Kokoro is used for text-to-speech conversion, and Qwen3 ASR is used for speech-to-text conversion. The streaming performance of Qwen3 ASR is enhanced with speech activity detection, sentence segmentation, and processing capabilities. A Voice Activity Detection (VAD) is used to detect if someone is speaking. When the accumulated speech time reaches 200 milliseconds (adjustable depending on the server and scenario), the speech is sent to the ASR for recognition. Because the streaming function continuously transcribes, the same transcribed content will be repeatedly generated. After determining if the current semantics are complete, a threshold is set, and the process is repeated several times. Once the next 200 milliseconds of speech are completely silent, reaching the set silence threshold, the transcribed result is sent out. Different dynamic silence thresholds can be set according to different speech rates.

[0059] S5: Applications of communication equipment; By connecting communication devices to the chat flow built by dify and combining it with a voice interaction model, an intelligent telephone assistant can be realized.

[0060] In summary, this invention integrates voice interaction, natural language understanding, knowledge retrieval, and operation execution into a complete intelligent telephone assistant processing flow. Specifically, it utilizes a large language model to uniformly classify the complete transcribed text into questions and answers and intents, distinguishing between ordinary knowledge inquiries and intents requiring specific communication operations, thus avoiding the limitations of rule matching. Specifically, when the request is a simple question-and-answer question, the target document fragment is retrieved from the knowledge base and combined with the question in the original voice text to generate the answer text, effectively ensuring the accuracy and knowledge coverage of the response. When the request is an intent-based request, the corresponding application programming interface is directly called to execute the target operation and generate the response text, enabling seamless information retrieval and call control, thereby significantly improving operational efficiency. Finally, speech synthesis converts the answer text or response text into the target speech for playback, forming a complete voice loop. Throughout the entire process, the user only needs to provide voice instructions without any manual intervention, significantly simplifying the interaction process and effectively enhancing the system's intelligence and real-time experience.

[0061] like Figure 4 As shown, this embodiment of the invention also provides a processing device 900 for a smart phone assistant, which can implement the above-described method. This device may include: The first module 901 is used to establish a voice channel in response to a call initiated by the target object through a communication device; The second module 902 is used to collect the voice transmitted by the target object through the voice channel, and then obtain the complete transcribed text through voice transcription. The third module, 903, is used to output classification results based on the complete transcribed text using a large language model. The classification results include general question-and-answer categories or intent categories, with intent categories associated with intent types. The fourth module, 904, is used when the classification result is a general question-and-answer category. It retrieves the target document fragment from a pre-built knowledge base based on the complete transcribed text, and uses a large language model to generate the answer text based on the complete transcribed text and the target document fragment. The fifth module 905 is used to call the application programming interface corresponding to the intent type to perform the target operation when the classification result is an intent class, and then generate response text based on the result of the target operation; Module 6, 906, is used to synthesize target speech based on answer or response text using a speech synthesis model, and then play the target speech to the target audience through a speech channel.

[0062] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0063] This invention also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described above. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0064] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0065] like Figure 5 As shown, Figure 5 The hardware structure of an electronic device 1000 according to another embodiment is illustrated. The electronic device 1000 includes: The processor 1001 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (aSIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention. The memory 1002 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RaM). The memory 1002 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1002 and is called and executed by the processor 1001. Input / output interface 1003 is used to implement information input and output; The communication interface 1004 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 1005 transmits information between various components of the device (e.g., processor 1001, memory 1002, input / output interface 1003, and communication interface 1004); The processor 1001, memory 1002, input / output interface 1003 and communication interface 1004 are connected to each other within the device via bus 1005.

[0066] The electronic device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0067] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0068] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0069] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0070] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0071] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0072] The intelligent telephone assistant processing method, apparatus, electronic device, storage medium, and program product provided in this invention establish a voice channel in response to a call initiated by a target object through a communication device; collect the voice transmitted by the target object through the voice channel, and then obtain a complete transcribed text through speech transcription; based on the complete transcribed text, output a classification result using a large language model; the classification result includes a general question-and-answer category or an intent category, with the intent category associated with an intent type; when the classification result is a general question-and-answer category, retrieve the target document fragment from a pre-built knowledge base based on the complete transcribed text, and generate answer text using a large language model based on the complete transcribed text and the target document fragment; when the classification result is an intent category, call the application programming interface corresponding to the intent type to execute the target operation, and then generate response text based on the result of the target operation; synthesize the target speech using a speech synthesis model based on the answer text or response text, and play the target speech to the target object through the voice channel. This invention integrates voice interaction, natural language understanding, knowledge retrieval, and operation execution into a complete intelligent telephone assistant processing flow. Specifically, it utilizes a large language model to uniformly classify the complete transcribed text into questions and answers and intents, distinguishing between ordinary knowledge inquiries and intents requiring specific communication operations, thus avoiding the limitations of rule matching. When the request is a simple question and answer, the target document fragment is retrieved from the knowledge base and combined with the question in the original voice text to generate the answer text, effectively ensuring the accuracy and knowledge coverage of the answer. When the request is an intent-based request, the corresponding application programming interface is directly called to execute the target operation and generate the response text, enabling seamless information retrieval and call control, thereby significantly improving operational efficiency. Finally, the answer text or response text is converted into the target voice for playback through speech synthesis to form a complete voice loop. Throughout the entire process, the user only needs to provide voice instructions without any manual intervention, significantly simplifying the interaction process and effectively enhancing the system's intelligence and real-time experience.

[0073] The embodiments described in this invention are for the purpose of more clearly illustrating the technical solutions of the embodiments of this invention, and do not constitute a limitation on the technical solutions provided by the embodiments of this invention. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this invention are also applicable to similar technical problems.

[0074] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present invention, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0075] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0076] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0077] The preferred embodiments of the present invention have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and spirit of the present invention should be within the scope of the claims of the present invention.

Claims

1. A processing method for an intelligent telephone assistant, characterized in that, The method includes the following steps: In response to a call initiated by the target via a communication device, a voice channel is established; The speech transmitted by the target object through the voice channel is collected, and then the complete transcribed text is obtained through speech transcription. Based on the complete transcribed text, a large language model is used to output classification results; the classification results include general question-and-answer class or intent class, and the intent class is associated with intent type. When the classification result is the ordinary question-and-answer category, the target document fragment is retrieved from the pre-built knowledge base based on the complete transcribed text, and the answer text is generated using a large language model based on the complete transcribed text and the target document fragment. When the classification result is the intent class, the application programming interface corresponding to the intent type is called to perform the target operation, and then a response text is generated based on the result of the target operation; The target speech is synthesized by a speech synthesis model based on the answer text or the response text, and then played to the target object through the speech channel.

2. The method according to claim 1, characterized in that, The process of collecting the speech transmitted by the target object through the speech channel and then obtaining complete transcribed text through speech-to-text includes the following steps: Audio packets are continuously acquired from the communication device through the voice channel at a fixed frame length; The audio packet is fed into the speech activity detection model to obtain the speech activity detection result at each time point; wherein, the speech activity detection result includes speech or non-speech. When the speech activity detection result changes from non-speech to speech at the time point when the speech start is determined, the audio packet is continuously cached from the speech start time until the speech activity detection result changes to non-speech, and audio data is generated. The audio data is fed into a speech recognition model for streaming recognition to obtain the complete transcribed text.

3. The method according to claim 2, characterized in that, The method further includes the following steps: If the duration of the non-speech function test result from the start of the test reaches a first time threshold; A preset voice prompt is played to the target object through the voice channel; If the voice activity detection result remains non-voice, the preset voice prompt is played once every time interval corresponding to the first time threshold. If the preset voice prompt is played a certain number of times, the voice channel is turned off.

4. The method according to claim 1, characterized in that, The process of outputting classification results based on the complete transcribed text using a large language model includes the following steps: The complete transcribed text is combined with a preset prompt word template and then input into a large language model to obtain a structured classification result output by the large language model. The prompt word template includes a first instruction that instructs the large language model to perform ordinary question and answer or intent recognition, and a second instruction that uses preset fields to represent different intent types.

5. The method according to claim 1, characterized in that, The process of retrieving the target document fragment from a pre-built knowledge base based on the complete transcribed text, and generating the answer text using a large language model based on the complete transcribed text and the target document fragment includes the following steps: The complete transcribed text is converted into a question vector using an embedding model; The target document fragment matching the question vector is retrieved from the knowledge base pre-constructed through document vectorization using similarity retrieval. The complete transcribed text is concatenated with the retrieved target document fragment and then input into the large language model so that the large language model can process and generate the answer text. If no target document fragment matching the question vector is found in the knowledge base, the preset prompt reply text will be used as the answer text.

6. The method according to claim 1, characterized in that, The intent types include call transfer, phone number lookup, and notification. The step of calling the application programming interface corresponding to the intent type to execute the target operation, and then generating response text based on the result of the target operation, includes the following steps: If the intent type is call transfer, call the call transfer interface and pass in the target number to perform the target operation of call transfer; If the intent type is to query a phone number, call the address book query interface and pass in the name to perform the target operation of querying the number; If the intent type is notification, call the notification interface and pass in the notification content and object to perform the target operation of the content notification; The target number, the name, or the notification content and recipient are extracted from the complete transcribed text. Obtain the execution status of the target operation, and generate response text based on the execution status mapping; The response text includes confirmation of successful operation or a description of the reason for operation failure.

7. The method according to claim 1, characterized in that, After the step of playing the target speech to the target object through the voice channel, the method further includes the following steps: Return to the step of collecting the voice transmitted by the target object through the voice channel to achieve multi-round voice interaction.

8. A processing device for an intelligent telephone assistant, characterized in that, The apparatus, applicable to the method according to any one of claims 1 to 7, comprises: The first module is used to establish a voice channel in response to a call initiated by the target object through a communication device; The second module is used to collect the voice transmitted by the target object through the voice channel, and then obtain the complete transcribed text through voice transcription. The third module is used to output classification results based on the complete transcribed text using a large language model; the classification results include general question-and-answer class or intent class, and the intent class is associated with intent type; The fourth module is used to retrieve the target document fragment from a pre-built knowledge base based on the complete transcribed text when the classification result is the ordinary question-and-answer category, and to generate the answer text using a large language model based on the complete transcribed text and the target document fragment; The fifth module is used to call the application programming interface corresponding to the intent type to perform the target operation when the classification result is the intent class, and then generate response text based on the result of the target operation; The sixth module is used to synthesize target speech based on the answer text or the response text using a speech synthesis model, and to play the target speech to the target object through the speech channel.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 7.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images