Implementation method of personalized artificial intelligence assistant system based on end-cloud collaborative architecture

Abstract

The invention discloses an implementation method of a personalized artificial intelligence assistant system based on an end-cloud collaborative architecture, the personalized artificial intelligence assistant system based on the end-cloud collaborative architecture is constructed, face recognition and voice recognition tasks are deployed at a local terminal, user privacy is guaranteed, response delay is reduced, and the user experience is improved. Meanwhile, high-computing tasks such as large language model reasoning and personalized speech synthesis are deployed on the cloud, the semantic comprehension and speech generation capacity is improved, the system achieves bidirectional collaboration of the terminal and the cloud through an encrypted communication channel, a user is supported to upload knowledge documents and tone samples on a webpage side, and the user experience is improved. An exclusive question and answer knowledge base and a personalized voice model are constructed, and natural language interaction experience of thousands of people and thousands of faces is achieved. The system constructed by the method has the advantages of quick response, safety, reliability, high customizability and the like, and is suitable for multi-scene intelligent voice interaction requirements of education, accompanying, inquiry and the like.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of human-computer interaction, and particularly relates to an implementation method of a personalized artificial intelligence assistant system based on an end-cloud collaborative architecture. BACKGROUND

[0002] With the rapid development of artificial intelligence, especially large language model (LLM) technology, voice assistants have gradually evolved from early keyword matching systems to intelligent systems with natural language understanding and dialogue generation capabilities. However, most current mainstream voice assistant systems use a pure cloud architecture, which uploads user voice input to a remote server and completes voice recognition, semantic understanding, and voice synthesis processes in the cloud. Although this architecture can rely on cloud computing resources to achieve high-performance model reasoning, it still has significant limitations in real-time performance, privacy, personalization, and device resource utilization.

[0003] On the one hand, the pure cloud processing mode heavily relies on the stability of network connection. If the user is in a poor network or edge environment, the response speed and interaction fluency of the system will be significantly reduced, making it difficult to meet the needs of low latency and high interaction frequency. On the other hand, the user's raw data such as voice and image need to be frequently transmitted to the cloud for processing, which poses a certain risk of privacy leakage, especially when it comes to sensitive operations such as face recognition and identity verification. In addition, existing systems usually use unified models and standard voice synthesis schemes, lacking the ability to personalize semantic preferences, knowledge backgrounds, and voice characteristics, making it difficult to achieve a "thousand faces" exclusive interaction experience.

[0004] In this context, the end-cloud collaborative architecture, as a new intelligent system design pattern, has gradually emerged. This architecture allocates computing tasks between local terminals and the cloud based on model complexity and scenario requirements, uses lightweight models to implement face recognition, voice recognition, and other front-end processing on the terminal, while retaining the complex reasoning, question and answer generation, and voice synthesis functions of the cloud. This not only improves system response speed, but also protects user privacy locally and customizes interaction content. Especially in edge devices with certain computing power (such as Raspberry Pi, smart speakers, mobile terminals), deploying lightweight models and forming a two-way low-latency connection with the cloud through encrypted communication protocols has become an important technical path to realize multi-modal, personalized, and highly available artificial intelligence assistant systems. However, there is still a lack of a complete end-cloud collaborative method that can balance response real-time performance and resource consumption while considering user identity authentication, personalized knowledge question and answer, and private voice synthesis functions.

[0005] Therefore, there is an urgent need for a new system architecture to form a stable and efficient collaborative mechanism between the end and the cloud to realize a safe, intelligent, and customizable artificial intelligence assistant system. SUMMARY

[0006] To solve the above technical problems, the application provides an implementation method of a personalized artificial intelligence assistant system based on an end-cloud collaborative architecture, which prepositions a lightweight large model task to a local terminal to complete, and downgrades model training, semantic reasoning and model reasoning tasks to a cloud to execute, and realizes two-way low-delay data synchronization through an encryption communication protocol.

[0007] The technical scheme adopted by the application is as follows: an implementation method of a personalized artificial intelligence assistant system based on an end-cloud collaborative architecture, and the specific steps are as follows:

[0008] S1, constructing a personalized artificial intelligence assistant system based on an end-cloud collaborative architecture;

[0009] The system comprises a user terminal, a cloud platform and a web platform.

[0010] The user terminal comprises a face recognition module, a voice recognition module and a service scheduling and communication module; the cloud platform comprises a knowledge base processing module, a text generation module and a voice synthesis module.

[0011] S2, based on the system constructed in step S1, the user terminal establishes a local face recognition database through the face recognition module, and then performs local face identity verification;

[0012] S3, based on step S2, after completing the identity verification, the user terminal starts the voice assistant service through the voice recognition module, completes voice input and recognition, obtains the text output corresponding to the audio and transmits it to the cloud platform;

[0013] S4, constructing the web platform allows the user to upload or update the exclusive knowledge base document content to the knowledge base processing module to construct the exclusive vector database, realizes the reconstruction of the RAG knowledge question and answer system, and processes the text output in step S3 through the text generation module to realize intelligent question and answer and knowledge fusion;

[0014] S5, based on step S4, the user uploads the personalized voice tone voice sample, the web platform fine-tunes the Fish-Speech model deployed in the voice synthesis module, and then inputs the text generated by the text generation module into the voice synthesis module for voice reasoning and synthesis, finally transmits the synthesized audio back to the service scheduling and communication module of the user terminal for real-time playing, and realizes natural voice interaction.

[0015] Further, the step S2 is specifically as follows:

[0016] S21, constructing a local face recognition database;

[0017] The user terminal adopts a Raspberry Pi, in which a face recognition module is connected with an OV5640 camera to collect initial face image data of a user, i.e., a current image frame of the user, and a convolutional neural network (CNN) is used to pre-process the collected image to obtain a standard face image matrix.

[0018] The preprocessing includes gray-scale processing, image alignment, face region cropping and size normalization.

[0019] Then, a preset CNN network structure is called to perform forward inference on the pre-processed image to extract a key face feature vector as a user identity template, and the extracted two-dimensional feature vector data, a timestamp and user identification information are stored in a local database of the Raspberry Pi to complete establishment of a local face recognition library.

[0020] wherein f face represents a face feature vector obtained by processing a user face image through a neural network, represents a feature vector of the local face recognition library, n represents a dimension of a feature space, and R represents a real number vector space.

[0021] S22, local face identity verification;

[0022] When the user wakes up the assistant system, a DeepFace face recognition model deployed in the local Raspberry Pi face recognition module is called, the user terminal collects a face image of the current user in real time through the OV5640 camera, and a standard image input matrix I test is obtained after preprocessing.

[0023] The preprocessing includes gray-scale processing, image alignment, face region cropping and size normalization.

[0024] Then, the standard image input matrix I test is input into the DeepFace model to obtain a feature vector of the current image through forward inference, and a reference feature vector of a registered user is called from the local face recognition library to perform similarity comparison, and whether the user identity is matched is determined through the Euclidean distance, and the expression is as follows:

[0025]

[0026] wherein D face represents the Euclidean distance between two face feature vectors, represents the feature vector of the current image, and represents the feature vector of the local face recognition library.

[0027] The assistant system sets a feature distance determination threshold δ according to actual conditions, and if Dface face > δ, then the identity verification is considered to fail, and the user terminal can prompt the user to retry or reject subsequent operations.

[0028] Further, the step S3 is specifically as follows:

[0029] First, the user voice input signal x(t) is collected through the local microphone connected in the user terminal Raspberry Pi, and after the preprocessing operation, the standard audio stream input x std (t) is formed, which meets the model input specification.

[0030] The preprocessing includes voice endpoint detection, noise suppression, speech speed normalization, and speech enhancement.

[0031] Then, the FunASR model locally deployed in the speech recognition module is called to input the standard audio stream input x std (t) to perform feature extraction and acoustic modeling, and obtain the text output T text corresponding to the audio, completing the speech-to-text conversion.

[0032] The speech-to-text conversion is realized through the connection of the time series classification CCT and the attention-based model Attention-based decoding structure.

[0033] Finally, the output text T text is uploaded to the cloud platform in real time through the encrypted WebSocket communication protocol, forming a low-latency, bidirectional reliable text communication channel.

[0034] The WebSocket communication protocol is based on WSS and uses the TLS1.3 protocol for encrypted transmission.

[0035] Further, the step S4 is specifically as follows:

[0036] S41, a user-specific vector database is constructed based on a web platform;

[0037] A web platform based on the Web framework Streamlit is built to provide a graphical interactive interface. The user uploads the knowledge document to the knowledge base processing module through the web platform to construct a specific vector database, i.e., the knowledge base processing module preprocesses the knowledge base uploaded by the user, and calls the embedded model to vectorize the knowledge text into embedded vectors v i ∈ R m and stores them in the vector database of the text generation module, constructing the semantic index required by the RAG mechanism.

[0038] ​Wherein, m represents the number of dimensions; the knowledge base processing module automatically parses the user uploaded document, and supports formats including:.pdf,.docx,.txt,.md; the embedded model includes: bge-large-zh-v1.5. The knowledge base preprocessing step is specifically as follows:

[0039] (1) Format analysis and text extraction: according to the type of uploaded document, the corresponding parser is automatically called to extract the original text content from the document, and the encoding format is unified to UTF-8.

[0040] Wherein, the parser includes: pdfplumber, python-docx, markdown-it-py.

[0041] (2) Paragraph splitting and cleaning: the extracted long text content is logically split into paragraphs or sentences, and invalid characters, extra spaces and punctuation interference symbols are removed, and complete semantic units are retained.

[0042] Wherein, the invalid characters include: header, footer, directory number.

[0043] (3) Semantic sentence segmentation and word segmentation processing: according to the characteristics of Chinese language, rule matching or NLP tools are used to reasonably segment the paragraphs, and word segmentation and part-of-speech tagging are performed as needed.

[0044] Wherein, the NLP tool includes: jieba, pkuseg.

[0045] (4) Batch vectorization conversion: each processed sentence or semantic block is input into the bge-large-zh-v1.5 embedded model as an independent semantic unit, and a fixed dimension embedded vector v i ∈R m is output, and is bound to the original text.

[0046] (5) Vector index construction and storage: all semantic units and their vectors are stored in the vector database, and inverted index and label index are established to support various cross query methods.

[0047] Wherein, the cross query method includes: top-k semantic matching, context retrieval, keyword filtering.

[0048] S42, based on step S41, realizing intelligent question answering and knowledge fusion;

[0049] Based on step S3, the text generation module of the cloud platform receives the user question T text , combined with the exclusive vector database constructed by the user uploaded document in step S41, the semantic understanding and question answering generation task is executed, that is, the relevant knowledge vector v i, the top-k retrieval result is spliced as context with the question to form a Prompt through the RAG mechanism, and the Prompt is input into a large language model for natural language understanding, and a generated text A is output.

[0050] The large language model includes DeepSeek-R1 and Qwen2.5.

[0051] Further, the step S5 is specifically as follows:

[0052] S51, Fish-Speech model fine-tuning;

[0053] The Fish-Speech model fine-tuning function is provided on the webpage platform, voice sample data uploaded by a user is supported, timbre adaptation and voice style transfer of a voice synthesis model are completed, the user uploads a clear voice file x audio of no less than 30 seconds on the webpage,

[0054] The voice file format includes.wav and.mp3, and the sampling rate is greater than or equal to 16 kHz. The timbre fine-tuning process is specifically as follows:

[0055] (a) Voice data verification and preprocessing: the voice synthesis module first performs data integrity detection on the uploaded original voice sample, including audio format, duration, channel number, sampling rate and background noise level; the sample that does not meet the standard will prompt the user to re-record or denoise; the qualified audio sample will be preprocessed, including: mute segment elimination, loudness normalization, resampling, speech speed leveling; the cleaned standard training audio x train (t) is obtained.

[0056] (b) Voice feature extraction: after the audio signal is converted from time domain to frequency domain signal through Fourier transform, the mel filter extracts the mel spectrum M(x) ∈ R T×F The extracted frequency spectrum feature is used for modeling of timbre coding in the fine-tuning process.

[0057] Wherein, T represents the number of frames, F represents the number of mel frequency bands, and x represents the feature information of the text.

[0058] (c) Fish-Speech model fine-tuning: using LoRA technology to fine-tune the parameters of the Fish-Speech model, that is, freezing the model backbone structure, and inserting trainable low-rank matrix parameters only in the speaker embedding layer and timbre control module of the Fish-Speech model.

[0059] S52, voice reasoning and synthesis, realizing natural voice interaction;

[0060] Based on step S51, the fine-tuned Fish-Speech model is saved as an independent model weight file and is bound to the user account. When the user performs voice interaction, the assistant system automatically calls the exclusive Fish-Speech model of the corresponding user for voice synthesis tasks to realize user personalized voice answers. The model also supports operation in the web interface, including: listening, switching or replacing. The specific operation is as follows:

[0061] The generated text A is input into the Fish-Speech model fine-tuned and deployed in the cloud platform voice synthesis module for voice inference and synthesis. First, the text is mapped to the mel spectrum through the mel filter to extract the mel spectrum graph, and the expression is as follows:

[0062]

[0063] Wherein, represents the mel spectrum graph generated by the fine-tuned Fish-Speech model, x represents the feature information of the text, and G θ (A) represents the fine-tuned Fish-Speech model, A represents the input text, and θ represents the parameters of the model.

[0064] Then, Input the fine-tuned Fish-Speech model, and finally synthesize the waveform through the mel spectrum and neural vocoder, and the expression is as follows:

[0065]

[0066] Wherein, represents the final synthesized audio, t represents time, and HiFi-GAN is a generative adversarial network for voice synthesis.

[0067] Finally, the synthesized audio is returned to the user terminal through the service scheduling and communication module for real-time playback, realizing natural voice interaction.

[0068] The beneficial effects of this invention are as follows: This invention discloses a method for implementing a personalized AI assistant system based on an edge-cloud collaborative architecture. The system deploys face recognition and speech recognition tasks on the local terminal to protect user privacy and reduce response latency, while deploying high-computation tasks such as large language model inference and personalized speech synthesis in the cloud to enhance semantic understanding and speech generation capabilities. The system achieves bidirectional collaboration between the terminal and the cloud through an encrypted communication channel, supporting users to upload knowledge documents and voice samples on a webpage to build a dedicated question-and-answer knowledge base and personalized speech model, achieving a "personalized" natural language interaction experience. The system constructed by the method of this invention is based on an edge-cloud collaborative architecture, which deploys basic perception tasks such as facial recognition and speech recognition on the local terminal for execution. It does not require uploading original images and audio, effectively reducing dependence on network bandwidth, improving the overall system response speed, and enhancing the privacy protection of user data. It has the advantages of fast response, security and reliability, and strong customizability. Moreover, the system realizes real-time collaboration between the local and cloud through an encrypted communication mechanism. Combined with the cloud's large model inference and voice synthesis capabilities, it forms a secure, efficient, and customizable intelligent voice interaction process, which is suitable for intelligent voice interaction needs in multiple scenarios such as education, companionship, and medical consultation. Attached Figure Description

[0069] Figure 1 This is a flowchart illustrating an implementation method for a personalized AI assistant system based on an edge-cloud collaborative architecture, according to the present invention.

[0070] Figure 2 This is a schematic diagram of the personalized AI assistant system architecture based on an edge-cloud collaborative architecture in an embodiment of the present invention.

[0071] Figure 3 This is a flowchart of the system workflow described in an embodiment of the present invention.

[0072] Figure 4 This is a schematic diagram illustrating the use of the RAG knowledge base in an embodiment of the present invention.

[0073] Figure 5 This is a schematic diagram of tone training according to an embodiment of the present invention.

[0074] Figure 6 This is a schematic diagram illustrating the graphics card usage during model training in an embodiment of the present invention. Detailed Implementation

[0075] The method of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0076] like Figure 1 The flowchart shown illustrates a method for implementing a personalized AI assistant system based on an edge-cloud collaborative architecture according to the present invention. The specific steps are as follows:

[0077] S1, constructing a personalized artificial intelligence assistant system based on an end-cloud collaborative architecture;

[0078] As shown in Figure 2 The system includes a user terminal, a cloud platform, and a web platform. The user terminal is responsible for image acquisition, face recognition, and voice recognition functions; the cloud platform is responsible for knowledge retrieval, large language model reasoning, and voice synthesis; the web platform is used for document uploading, timbre training, and system configuration management, and the three are connected through a secure channel for efficient linkage.

[0079] The user terminal includes a face recognition module, a voice recognition module, a service scheduling and communication module; the cloud platform includes a knowledge base processing module, a text generation module, and a voice synthesis module. The system workflow is shown in Figure 3 .

[0080] S2, based on the system constructed in step S1, the user terminal establishes a local face recognition database through the face recognition module, and then performs local face identity verification;

[0081] S3, based on step S2, after identity verification, the user terminal starts the voice assistant service through the voice recognition module, completes voice input and recognition, obtains the text output corresponding to the audio, and transmits it to the cloud platform;

[0082] S4, constructing a web platform allows users to upload or update exclusive knowledge base document content to the knowledge base processing module to build an exclusive vector database, realize the reconstruction of the RAG (Retrieval-Augmented Generation) knowledge question and answer system, and process the text output in step S3 through the text generation module to realize intelligent question and answer and knowledge fusion;

[0083] S5, based on step S4, the user uploads personalized timbre voice samples, the web platform fine-tunes the Fish-Speech model deployed in the voice synthesis module (completes the customized deployment of the exclusive timbre assistant), and then inputs the text generated by the text generation module into the voice synthesis module for voice reasoning and synthesis. Finally, the synthesized audio is transmitted back to the service scheduling and communication module of the user terminal for real-time playback, realizing natural voice interaction.

[0084] In this embodiment, the step S2 is as follows:

[0085] S21, local face recognition database construction;

[0086] The user terminal adopts a Raspberry Pi, in which a face recognition module is connected with an OV5640 camera to collect initial face image data of a user, i.e., a current image frame of the user, and a convolutional neural network (CNN) is used to pre-process the collected image to obtain a standard face image matrix.

[0087] The preprocessing includes gray-scale processing, image alignment, face region cropping and size normalization.

[0088] Then, a preset CNN network structure is called to perform forward inference on the pre-processed image to extract a key face feature vector as a user identity template, and the extracted two-dimensional feature vector data, a timestamp and user identification (identity tag) information are stored in a local database of the Raspberry Pi to complete the establishment of a local face recognition library. The local face recognition library establishment process is completely completed on the terminal side, avoiding the uploading of original face images and improving privacy security.

[0089] wherein f face represents a face feature vector obtained by processing a user face image through a neural network, represents a local face recognition library feature vector, n represents a dimension (a positive integer) of a feature space, n = 128 in this embodiment, and R represents a real number vector space used to describe high-dimensional semantic features of a face.

[0090] S22, local face identity verification;

[0091] When the user wakes up the assistant system, a DeepFace face recognition model deployed in the local face recognition module of the Raspberry Pi is called, the user terminal collects a face image of the current user in real time through the OV5640 camera, and a standard image input matrix I test is obtained after preprocessing.

[0092] The preprocessing includes gray-scale processing, image alignment, face region cropping and size normalization. The face recognition module adopts a DeepFace public model, which uses a deep convolutional neural network to realize face image alignment and feature extraction. The extracted face feature vector is used for similarity comparison with an identity template in the local database. Since the model structure and training method are known to those skilled in the art, they will not be described here.

[0093] The standard image input matrix I test is then input into the DeepFace model to obtain a feature vector f of the current image through forward inference, and a reference feature vector of a registered user is called from the local face recognition library for similarity comparison. Whether the user identity is matched is determined by the Euclidean distance, and the expression is as follows:

[0094]

[0095] wherein D face represents the Euclidean distance between two face feature vectors, represents the current image feature vector, represents the local face recognition library feature vector.

[0096] The assistant system sets a feature distance determination threshold δ according to the actual situation, and if D face < δ, it is considered that the current user identity verification is passed. If D face > δ, it is considered that the identity verification fails, and the user terminal can prompt the user to retry or reject the subsequent operation.

[0097] The local face identity verification process does not need to rely on the cloud, fully utilizes the end-side computing power, and guarantees the privacy of user identity.

[0098] In the present embodiment, the step S3 is specifically as follows:

[0099] As shown in Figure 3 , first, the user voice input signal x(t) is collected through the local microphone connected in the user terminal Raspberry Pi, and after pre-processing operation, the standard audio stream input x std (t) is formed, which meets the model input specification.

[0100] Among them, the pre-processing includes: voice activity detection (VAD), noise suppression, speech speed normalization and speech enhancement.

[0101] Then the FunASR model deployed locally in the speech recognition module is called, the standard audio stream input x std (t) is input for feature extraction and acoustic modeling, and the text output T text corresponding to the audio is obtained, and the speech-to-text conversion is completed.

[0102] Among them, the speech-to-text conversion is realized through the connection of the time series classification CCT (Connectionist Temporal Classification) and the attention-based model Attention-based decoding structure, which has high robustness.

[0103] Finally, the output text T text is uploaded to the cloud platform in real time through the encrypted WebSocket communication protocol, forming a low-latency, bidirectional reliable text communication channel. Provide input for subsequent semantic reasoning and speech generation, and this process demonstrates the core characteristics of end-side reasoning and cloud-side large model collaboration.

[0104] The WebSocket communication protocol is based on WSS (WebSocket Secure) and uses a TLS 1.3 protocol for encrypted transmission.

[0105] In the embodiment, the step S4 is specifically as follows:

[0106] S41, constructing a user-specific vector database based on a web platform;

[0107] A web platform based on a Web framework Streamlit is built to provide a graphical interactive interface to facilitate vector database management and Fish-Speech model personalized timbre training calling.

[0108] As shown in Figure 4 , the user uploads the knowledge document to the knowledge base processing module through the web page to construct a specific vector database, that is, the knowledge base processing module pre-processes the knowledge base uploaded by the user, and calls an embedded model to vectorize the knowledge text into embedded vectors v i ∈R m and stores them in the vector database of the text generation module, to construct the semantic index required by the RAG (Retrieval Augmented Generation) mechanism.

[0109] Wherein, m represents the number of dimensions, m = 1024 in the embodiment; the knowledge base processing module automatically parses the user uploaded document, and supports formats including:.pdf,.docx,.txt,.md; the embedded model includes: bge-large-zh-v1.5. The knowledge base preprocessing step is specifically as follows:

[0110] (1) Format analysis and text extraction: automatically call the corresponding parser according to the type of uploaded document to extract the original text content from the document, and unify the encoding format to UTF-8 to avoid garbled characters or character errors;

[0111] Wherein, the parser includes: pdfplumber, python-docx, markdown-it-py.

[0112] (2) Paragraph splitting and cleaning: split the extracted long text content at the logical paragraph or sentence level, and remove invalid characters, redundant spaces, and punctuation interference symbols, while retaining complete semantic units;

[0113] Wherein, the invalid characters include: header, footer, and directory number.

[0114] (3) Semantic sentence splitting and word segmentation processing: according to the characteristics of Chinese language, use rule matching or NLP tools to reasonably split the paragraphs, and perform word segmentation and part-of-speech tagging as needed, to provide clear semantic boundaries for subsequent vectorization operations;

[0115] The NLP tool includes jieba and pkuseg.

[0116] (4) Batch vectorization conversion: input each processed sentence segment or semantic block as an independent semantic unit into the bge-large-zh-v1.5 embedded model, and output an embedded vector v of a fixed dimension. i ∈R 1024 , and bind mapping with the original text;

[0117] (5) Vector index construction and storage: store all semantic units and their vectors into the Faiss or Milvus vector database, and establish an inverted index and a label index to support various cross-query methods.

[0118] The cross-query methods include top-k semantic matching, context retrieval, and keyword filtering.

[0119] S42, based on step S41, realizing intelligent question answering and knowledge fusion;

[0120] Based on step S3, the text generation module of the cloud platform receives the user question T text , and performs semantic understanding and question answering generation tasks in combination with the exclusive vector database constructed by the user uploaded document in step S41, that is, retrieves relevant knowledge vectors v i from the user exclusive vector database, forms a Prompt by splicing the top-k retrieval result as the context and the question through the RAG (Retrieval-Augmented Generation) mechanism, and inputs the Prompt into a large language model for natural language understanding to output the generated text A.

[0121] The large language model includes DeepSeek-R1 and Qwen2.5.

[0122] In this embodiment, step S5 is specifically as follows:

[0123] In addition to the vector database management function, the web platform also provides Fish-Speech model (personalized timbre model) fine-tuning function, supports user uploading of voice sample data, to complete the timbre adaptation and voice style transfer of the speech synthesis model, so as to realize the personification of the voice interaction experience of "each person has a voice". As shown in Figure 5 , the user uploads a clear voice file x audio of no less than 30 seconds on the web page, triggers the cloud platform to fine-tune the timbre, and obtains the exclusive Fish-Speech model corresponding to the user.

[0124] The voice file format includes.wav,.mp3, and the sampling rate is greater than or equal to 16 kHz. The Fish-Speech model is a text-to-speech tool opened by Fish Audio, and the model structure includes a text encoder, a speaker embedding layer, a timbre control module, an acoustic model, and a vocoder.

[0125] The timbre fine-tuning process is as follows:

[0126] (a) Voice data verification and preprocessing: The voice synthesis module first performs data integrity detection on the uploaded original voice sample, including audio format, duration, channel number, sampling rate, and background noise level. Samples that do not meet the standard will prompt the user to re-record or noise reduction processing. Qualified audio samples will be preprocessed, including: silent segment elimination, loudness normalization, resampling, speech speed leveling, to obtain the cleaned standard training audio x train (t);

[0127] (b) Voice feature extraction: After Fourier transform converts the audio signal from time domain to frequency domain signal, the Mel filter extracts the Mel spectrogram M(x) ∈ R T×F , the extracted spectral features are used for modeling of timbre coding in the fine-tuning process;

[0128] where T represents the number of frames, F represents the number of Mel frequency bands, and x represents the feature information of the text.

[0129] (c) Fine-tuning of Fish-Speech model: The LoRA (Low-Rank Adaptation) technology is used to fine-tune the parameters of the basic voice synthesis model Fish-Speech. The specific strategy is to freeze the model backbone structure, and only insert trainable low-rank matrix parameters in the speaker embedding layer and timbre control module of Fish-Speech. As shown in Figure 6 , the system training stage occupies less GPU resources, and single card can complete fast training, with good deployment and engineering adaptability. Through this method, only the parameters related to timbre are optimized, so that the synthesized voice can fully reflect the timbre characteristics of the audio.

[0130] S52, voice inference and synthesis, realize natural voice interaction;

[0131] Based on step S51, the fine-tuned Fish-Speech model is saved as an independent model weight file and bound to the user account. When the user interacts via voice, the assistant system automatically calls the user's exclusive timbre feature model for the speech synthesis task, enabling personalized voice responses. This model also supports operations on the web interface, including: previewing, switching, or replacing. Details are as follows:

[0132] The generated text A is input into the finely tuned Fish-Speech model deployed in the speech synthesis module of the cloud platform for speech inference and synthesis. First, the text is converted to a Mel spectrogram using a Mel filter, and the Mel spectrogram is extracted, as shown in the following expression:

[0133]

[0134] in, This represents the Mel spectrogram generated by the fine-tuned Fish-Speech model, where x represents the feature information of the text, and G... θ (A) represents the fine-tuned Fish-Speech model, where A represents the input text and θ represents the model parameters.

[0135] Then The finely tuned Fish-Speech model is input, and the final waveform is synthesized via neural vocoder synthesis using Mel spectrum analysis. The expression is as follows:

[0136]

[0137] in, The final synthesized audio is represented by t, where t represents time. HiFi-GAN is a generative adversarial network used for speech synthesis.

[0138] Finally, the synthesized audio will be The data is transmitted back to the user terminal via the service scheduling and communication module for real-time playback, enabling natural voice interaction.

[0139] The method of this invention is based on an edge-cloud collaborative architecture, which utilizes the capabilities of large models to generate personalized responses and forms a natural interaction chain of "voice input - semantic understanding - voice output" through a two-way closed loop between the edge and cloud.

[0140] To sum up, the method of the application constructs a system based on an end-cloud collaborative architecture, deploys basic perception tasks such as face recognition and voice recognition in a local terminal for execution, does not need to upload original images and audio, effectively reduces the dependence on network bandwidth, improves the overall response speed of the system, and enhances the privacy protection capability of user data, has the advantages of fast response, safety and reliability, strong customizability, and the like, and the system realizes real-time collaboration between the local and the cloud through an encryption communication mechanism, combines the cloud large model inference and tone synthesis capability, forms a safe, efficient and customizable intelligent voice interaction process, and is suitable for intelligent voice interaction requirements in multiple scenes such as education, accompanying, diagnosis and the like.

[0141] Those skilled in the art will realize that the embodiments described herein are for the purpose of illustration and should not be construed as limiting the scope of the present application. The present application can be modified and varied in various ways, and any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. An implementation method of a personalized artificial intelligence assistant system based on an end-cloud collaborative architecture, the specific steps being as follows: S1, constructing a personalized artificial intelligence assistant system based on an end-cloud collaborative architecture; The system comprises: a user terminal, a cloud platform, and a web platform; wherein the user terminal comprises a face recognition module, a voice recognition module, and a service scheduling and communication module; the cloud platform comprises a knowledge base processing module, a text generation module, and a speech synthesis module; S2, based on the system constructed in step S1, the user terminal establishes a local face recognition library through the face recognition module, and then performs local face identity verification; S3, based on step S2, after identity verification, the user terminal starts the voice assistant service through the voice recognition module, completes voice input and recognition, obtains the text output corresponding to the audio, and transmits it to the cloud platform; S4, constructing the web platform allows the user to upload or update the exclusive knowledge base document content to the knowledge base processing module to build an exclusive vector database, realize the reconstruction of the RAG knowledge question and answer system, and process the text output in step S3 through the text generation module to realize intelligent question and answer and knowledge fusion; S5, based on step S4, the user uploads personalized voice samples, the web platform fine-tunes the Fish-Speech model deployed in the speech synthesis module, and then inputs the text generated by the text generation module into the speech synthesis module for speech reasoning and synthesis, finally transmits the synthesized audio back to the service scheduling and communication module of the user terminal for real-time playing, realizing natural voice interaction.

2. The implementation method of the personalized artificial intelligence assistant system based on the end-cloud collaborative architecture according to claim 1, characterized in that, The step S2 is specifically as follows: S21, local face recognition library construction; The user terminal uses Raspberry Pi, in which the face recognition module is connected to the OV5640 camera, the initial face image data of the user is collected, that is, the current image frame of the user, and the convolutional neural network (CNN) is used to preprocess the collected image to obtain a standard face image matrix; Wherein, the preprocessing includes: grayscale, image alignment, face region cropping and size normalization; Then call the preset CNN network structure to forward inference the preprocessed image, and extract the key face feature vector As a user identity template, and the extracted two-dimensional feature vector data and time stamp and user identification information are stored in the local database of Raspberry Pi, and the local face recognition library is established. wherein f face represents a face feature vector obtained after the user's face image is processed by the neural network, represents a local face recognition library feature vector, n represents the dimension of the feature space, and R represents a real number vector space. S22, local face identity verification; When the user wakes up the assistant system, the DeepFace face recognition model deployed locally in the Raspberry Pi face recognition module is called, the user terminal acquires the face image of the current user in real time through the OV5640 camera, and a standard image input matrix I that meets the recognition standard is obtained after preprocessing test ; Wherein, the preprocessing includes: grayscale, image alignment, face region cropping and size normalization; The standard image is then input into a matrix I test The DeepFace model is input, and the feature vector of the current image is obtained through forward inference The reference feature vector of the registered user is called from the local face recognition library for similarity comparison, and the Euclidean distance is used to determine whether the user identity matches, expressed as follows: where D face denotes the Euclidean distance between two face feature vectors, denotes the current image feature vector, denotes the local face recognition library feature vector; The assistant system sets a characteristic distance determination threshold δ according to actual conditions, if D face < δ, it is considered that the current user identity verification is passed; if D face > δ, it is considered that the identity verification fails, and the user terminal can prompt the user to retry or reject subsequent operations.

3. The implementation method of the personalized artificial intelligence assistant system based on the end-cloud collaborative architecture according to claim 1, characterized in that, The step S3 is specifically as follows: Firstly, the user voice input signal x(t) is collected by the local microphone connected in the user terminal Raspberry Pi, and after the preprocessing operation, the standard audio stream input x std (t) is formed, which meets the model input specification; Wherein, the preprocessing includes: voice endpoint detection, noise suppression, speech speed normalization and speech enhancement; Then a FunASR model locally deployed in the speech recognition module is called to input the standard audio stream input x std (t) feature extraction and acoustic modeling are performed to obtain the text output T corresponding to the audio text , and the speech-to-text conversion is completed; Wherein, the conversion from voice to text is realized through connection of the time sequence classification (CCT) and the attention-based decoding structure; Finally the output text T text Real-time upload to the cloud platform through encrypted WebSocket communication protocol, forming a low-latency, two-way reliable text communication channel; The WebSocket communication protocol is based on WSS and uses TLS1.3 protocol for encrypted transmission.

4. The implementation method of the personalized artificial intelligence assistant system based on the end-cloud collaborative architecture according to claim 1, characterized in that, The step S4 is specifically as follows: S41, constructing a user exclusive vector database based on the web platform; A web-based platform based on the Web framework Streamlit is built to provide a graphical interactive interface. Users upload knowledge documents to the knowledge base processing module through the web to build a dedicated vector database, i.e., the knowledge base processing module preprocesses the knowledge base uploaded by the user, and calls the embedded model to vectorize the knowledge text into embedded vectors v i ∈R m The vector database stored in the text generation module constructs the semantic index required by the RAG mechanism; Wherein, m represents the dimension number; the knowledge base processing module automatically parses the user uploaded documents, supporting formats:.pdf,.docx,.txt,.md; the embedded model includes: bge-large-zh-v1.5; the knowledge base preprocessing step is specifically as follows: (1) Format analysis and text extraction: automatically call the corresponding parser according to the type of uploaded document, extract the original text content from the document, and unify the encoding format to UTF-8; Among them, the parser includes: pdfplumber, python-docx, markdown-it-py; (2) Paragraph splitting and cleaning: split the extracted long text content into logical paragraphs or sentence level, and remove invalid characters, extra spaces and punctuation interference symbols, and keep the complete semantic units; Among them, the invalid characters include: header, footer, directory number; (3) Semantic sentence segmentation and word segmentation processing: according to the characteristics of Chinese language, use rule matching or NLP tools to reasonably segment the paragraph, and perform word segmentation and part-of-speech tagging as needed; Among them, the NLP tools include: jieba, pkuseg; (4) Batch vectorization conversion: input each processed sentence segment or semantic block as an independent semantic unit into the bge-large-zh-v1.5 embedded model, and output an embedded vector v of fixed dimension i ∈R m , and bind map with the original text; (5) Vector index construction and storage: store all semantic units and their vectors into the vector database, and establish inverted index and label index to support various cross query methods; Among them, the cross query methods include: top-k semantic matching, context retrieval, keyword filtering; S42, based on step S41, realize intelligent question answering and knowledge fusion; Based on step S3, the text generation module of the cloud platform receives the user question T text , in combination with step S41, the user uploads the document to build a dedicated vector database to perform semantic understanding and question answering generation tasks, that is, to retrieve relevant knowledge vectors v i from the user's dedicated vector database, through the RAG mechanism, the top-k retrieval result is spliced as the context and the question to form Prompt, and the Prompt is input into the large language model for natural language understanding, and the output generates text A; Among them, the large language model includes: DeepSeek-R1, Qwen2.

5.

5. The implementation method of the personalized artificial intelligence assistant system based on the end-cloud collaborative architecture according to claim 1, characterized in that, The step S5 is specifically as follows: S51, Fish-Speech model fine-tuning; The webpage platform provides Fish-Speech model fine-tuning function, supports user uploading voice sample data, completes voice synthesis model timbre adaptation and voice style transfer, and the user uploads a clear voice file x of not less than 30 seconds on the webpage audio , triggers timbre fine-tuning, and obtains a special Fish-Speech model corresponding to the user; Among them, the speech file format includes:.wav,.mp3, and the sampling rate is greater than or equal to 16kHz; the timbre fine-tuning process is specifically as follows: (a) Speech data verification and preprocessing: The speech synthesis module first performs data integrity detection on the uploaded original speech samples, including: audio format, duration, channel number, sampling rate, and background noise level; samples that do not meet the standard will prompt the user to re-record or noise reduction processing; qualified audio samples will be preprocessed, including: silence segment elimination, loudness normalization, resampling, speech speed leveling; get the cleaned standard training audio x train (t); (b) Speech feature extraction: After the audio signal is converted from time domain to frequency domain signal by Fourier transform, the mel filter extracts the mel spectrogram M(x) e R T×F , and the extracted spectral features are used for modeling timbre coding in the fine-tuning process; Among them, T represents the number of frames, F represents the number of mel frequency bands, and x represents the feature information of the text; (c) Fish-Speech model fine-tuning: use LoRA technology to fine-tune the parameters of Fish-Speech model, that is, freeze the model backbone structure, and only insert trainable low-rank matrix parameters in the speaker embedding layer and timbre control module of Fish-Speech model; S52, voice reasoning and synthesis, realize natural voice interaction; Based on step S51, save the fine-tuned Fish-Speech model as an independent model weight file, and bind it with the user account. When the user performs voice interaction, the assistant system automatically calls the exclusive Fish-Speech model of the corresponding user for voice synthesis task, realizing user personalized voice answer; and the model also supports operation in the web interface, including: listening, switching or replacing; specifically as follows: Input the generated text A into the Fish-Speech model fine-tuned in the cloud platform speech synthesis module for voice reasoning and synthesis. First, convert the text to mel spectrum mapping through the mel filter, extract the mel spectrum graph, and the expression is as follows: wherein, represents the mel-spectrogram generated by the fine-tuned Fish-Speech model, x represents the feature information of the text, G θ (A) represents the fine-tuned Fish-Speech model, A represents the input text, and θ represents the parameters of the model. Then the The fine-tuned Fish-Speech model is inputted to synthesize the final waveform through the neural vocoding of the mel-spectrogram, and the expression is as follows: wherein, represents the final synthesized audio, t represents time, and HiFi-GAN is a generative adversarial network for speech synthesis. Finally, the synthesized audio will be The service scheduling and communication module transmits the data back to the user terminal for real-time playback, enabling natural voice interaction.

Images