Multi-terminal intelligent voice coordination interaction method and system

Abstract

The invention discloses a multi-terminal intelligent voice coordination interaction method and system, and the method comprises the steps: collecting voice data, carrying out the preprocessing of the voice data, and carrying out the voice recognition based on a voice recognition model of an attention mechanism; performing intention recognition sample processing on a voice recognition result, and performing intention recognition based on an intention recognition model; generating voice response content according to the recognized intention, and performing voice response through a virtual image of a self-defined voice assistant; and performing multi-terminal adaptation and cross-device cooperative processing based on a multi-terminal adaptation mechanism and a resource allocation mechanism, and synchronizing and feeding back the voice interaction content and the voice response of the virtual image. According to the invention, the overall capabilities of voice recognition, semantic comprehension, personalized service and multi-terminal cooperation are improved, and efficient, natural and intelligent voice interaction experience is provided for users.

Description

Technical Field

[0001] The present invention relates to the field of voice processing technology, and in particular to a multi-terminal intelligent voice coordination interaction method and system. Background Art

[0002] With the rapid development of artificial intelligence and voice interaction technologies, the use of intelligent voice systems has become a key strategic direction for the intelligent transformation of various industries. Whether in smart TVs, smart cars, or smart speakers, voice interaction is a key interactive method for exchanging information between users and devices. However, existing intelligent voice systems still face many technical challenges and limitations in voice recognition capabilities, personalized customization, and multi-terminal collaboration, which prevents them from fully meeting users' needs for efficient, flexible, and intelligent voice interaction.

[0003] 1. Insufficient speech recognition and semantic understanding capabilities: Although speech recognition and natural language understanding technologies have made certain progress, the speech recognition accuracy and semantic understanding capabilities of existing systems are still limited in complex real-world environments and multi-round dialogue scenarios, and are often unable to provide users with an efficient and satisfactory interactive experience.

[0004] 2. Insufficient personalization and customization: Existing intelligent voice systems often focus on a single standardized voice service and lack the ability to be customized according to user needs, scenarios, device types, etc., and cannot fully meet users' needs to create personalized and customized products.

[0005] 3. Poor multi-terminal collaboration: Existing intelligent voice products on the market have poor collaboration and interoperability with each other. User information and interaction status cannot be efficiently shared between devices, resulting in a broken user experience on different terminals and a lack of coherence and consistency.

[0006] 4. Lack of flexible terminal adaptation and resource allocation mechanism: In the existing system's multi-device, cross-platform application environment, the system's resource allocation, configuration, and management are difficult to be flexible and efficient, and cannot be optimized in a timely manner according to the device's hardware characteristics and user needs.

[0007] In summary, existing intelligent voice systems still have problems such as low voice recognition accuracy, poor multi-terminal collaboration, and insufficient personalization and customization; therefore, existing technologies need to be improved. Summary of the Invention

[0008] The technical problem to be solved by the present invention is that, in response to the defects of the existing technology, the present invention provides a multi-terminal intelligent voice coordination interaction method and system to solve the problems of low voice recognition accuracy, poor multi-terminal coordination, and insufficient personalization and customization in the existing intelligent voice system.

[0009] The technical solutions adopted by the present invention to solve the technical problems are as follows: In a first aspect, the present invention provides a multi-terminal intelligent voice coordination interaction method, comprising: Collecting speech data, preprocessing the speech data, and performing speech recognition based on a speech recognition model with an attention mechanism; Perform intent recognition sample processing on the speech recognition results and perform intent recognition based on the intent recognition model; Generate voice response content based on the recognized intent and respond through a customized voice assistant avatar; Based on the multi-terminal adaptation mechanism and resource allocation mechanism, multi-terminal adaptation and cross-device collaborative processing are performed to synchronize and feedback the voice interaction content and the voice response of the virtual image.

[0010] In one implementation, collecting speech data, preprocessing the speech data, and performing speech recognition based on a speech recognition model using an attention mechanism include: The system collects voice data from real user conversations through microphones in various terminal devices, generates diversified voice samples based on the collected voice data, and obtains supplementary data from public voice datasets to expand the sample size. Performing spectrum subtraction, adaptive filtering, audio segmentation, feature extraction, and data enhancement on the expanded speech data to obtain preprocessed speech data; The speech recognition model based on the attention mechanism performs speech recognition on the preprocessed speech data to obtain a text result of the speech recognition.

[0011] In one implementation, the speech recognition model based on the attention mechanism performs speech recognition on the preprocessed speech data to obtain a text result of the speech recognition, which includes: Dividing the preprocessed speech data into a training set, a validation set, and a test set according to a preset ratio; Use cross entropy as the loss function and continuously optimize model parameters through batch training iterations; Adjusting the hyperparameters of the speech recognition model using a Bayesian optimization method, and performing transfer learning using a pre-trained speech recognition model; The word error rate and the phoneme error rate of the speech recognition of the test set are calculated, and the speech recognition model is evaluated according to the word error rate and the phoneme error rate to obtain the trained speech recognition model.

[0012] In one implementation, performing intent recognition sample processing on the speech recognition result and performing intent recognition based on the intent recognition model includes: Perform word segmentation, part-of-speech tagging, and entity recognition on the text results obtained from speech recognition; Based on a large language model, combined with knowledge bases in various fields and predefined rules, a diverse set of user dialogue data is generated to obtain a complete set of intent recognition samples. Based on the classification model, performing intent classification on the improved intent recognition sample set; Based on the sequence labeling model, the slots of the classified intent recognition sample set are filled to obtain the processed intent recognition samples; The processed intent recognition sample is input into the intent recognition model to perform intent recognition to obtain an intent recognition result.

[0013] In one implementation, generating voice response content based on the recognized intent and providing a voice response through a customized voice assistant avatar includes: Generate interactive response content based on the customized question-answer pair using a generative language model, and convert the text corresponding to the interactive response content into speech output using speech synthesis technology; The converted speech is responded to in a voice by a customized virtual human image, and the voice response form of the virtual human image is optimized according to the speech interaction optimization strategy.

[0014] In one implementation, optimizing the voice response form of the avatar according to the voice interaction optimization strategy includes: Optimize personalized response content by analyzing historical interaction data; Use real-time voice emotion analysis technology to determine current emotions and optimize emotional response content; By analyzing voice input and behavior data in real time, the voice interaction strategy is dynamically adjusted, and the voice response strategy is automatically adjusted based on user feedback and historical data.

[0015] In one implementation, multi-terminal adaptation and cross-device collaborative processing are performed based on a multi-terminal adaptation mechanism and a resource allocation mechanism, and voice interaction content and the avatar's voice response are synchronized and fed back, including: Manage and synchronize voice interaction data and device status information of each terminal through the intelligent voice platform; Detecting compatibility between devices based on the multi-terminal adaptation mechanism and optimizing the quality of voice input and output according to a preset adaptive module; Based on the resource allocation mechanism, cross-device collaborative processing is performed to synchronize and provide feedback on the voice interaction content and the voice response of the virtual image.

[0016] In a second aspect, the present invention provides a multi-terminal intelligent voice coordination and interaction system, comprising: A speech recognition module is used to collect speech data, pre-process the speech data, and perform speech recognition based on a speech recognition model with an attention mechanism; The intent recognition module is used to process the speech recognition results into intent recognition samples and perform intent recognition based on the intent recognition model; The voice response module is used to generate voice response content based on the recognized intent and provide voice response through a customized voice assistant avatar; The multi-terminal collaboration module is used to perform multi-terminal adaptation and cross-device collaborative processing based on the multi-terminal adaptation mechanism and resource allocation mechanism, and synchronize and feedback the voice interaction content and the voice response of the virtual image.

[0017] In the third aspect, the present invention provides a terminal comprising: a processor and a memory, wherein the memory stores a multi-terminal intelligent voice coordination interaction program, and when the multi-terminal intelligent voice coordination interaction program is executed by the processor, it is used to implement the operation of the multi-terminal intelligent voice coordination interaction method as described in the first aspect.

[0018] In a fourth aspect, the present invention also provides a medium, which is a computer-readable storage medium, and the medium stores a multi-terminal intelligent voice coordination interaction program, and when the multi-terminal intelligent voice coordination interaction program is executed by a processor, it is used to implement the operation of the multi-terminal intelligent voice coordination interaction method as described in the first aspect.

[0019] The present invention adopts the above technical solution to achieve the following effects: 1) This invention uses advanced speech recognition models and intent recognition models for recognition, improving the accuracy of speech recognition and semantic understanding in noisy environments, complex contexts, and multi-round conversations, thereby further improving system accuracy and response speed.

[0020] 2) The present invention can customize the timbre, intonation, appearance and other characteristics of the virtual image according to needs, enhancing the user's voice interaction experience. It can also dynamically adjust the response strategy based on user behavior habits, user profile and contextual dialogue environment to provide users with highly personalized voice services.

[0021] 3) The present invention realizes seamless collaboration between multiple terminal devices, ensuring that users obtain a consistent voice interaction experience on different devices, while supporting instant information sharing and interaction status synchronization between smart devices.

[0022] 4) The present invention can flexibly adjust resource allocation and optimization strategies according to the hardware characteristics of different terminal devices and user needs, thereby improving device adaptability and system operating efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying any creative work.

[0024] Figure 1 This is a flow chart of the multi-terminal intelligent voice coordination interaction method in the present invention.

[0025] Figure 2 It is a functional principle diagram of a terminal in one implementation of the present invention.

[0026] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solutions and advantages of the present invention more clear and distinct, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0028] Exemplary Methods Existing intelligent voice systems still face many technical challenges and limitations in terms of voice recognition capabilities, personalized customization, and multi-terminal collaboration, which makes it impossible to fully meet users' needs for efficient, flexible and intelligent voice interaction.

[0029] 1. Insufficient speech recognition and semantic understanding capabilities: Although speech recognition and natural language understanding technologies have made certain progress, the speech recognition accuracy and semantic understanding capabilities of existing systems are still limited in complex real-world environments and multi-round dialogue scenarios, and are often unable to provide users with an efficient and satisfactory interactive experience.

[0030] 2. Insufficient personalization and customization: Existing intelligent voice systems often focus on a single standardized voice service and lack the ability to be customized according to user needs, scenarios, device types, etc., and cannot fully meet users' needs to create personalized and customized products.

[0031] 3. Poor multi-terminal collaboration: Existing intelligent voice products on the market have poor collaboration and interoperability with each other. User information and interaction status cannot be efficiently shared between devices, resulting in a broken user experience on different terminals and a lack of coherence and consistency.

[0032] 4. Lack of flexible terminal adaptation and resource allocation mechanism: In the existing system's multi-device, cross-platform application environment, the system's resource allocation, configuration, and management are difficult to be flexible and efficient, and cannot be optimized in a timely manner according to the device's hardware characteristics and user needs.

[0033] In response to the above technical problems, an embodiment of the present invention provides a multi-terminal intelligent voice coordinated interaction method, which includes: collecting voice data, preprocessing the voice data, and performing voice recognition based on a voice recognition model of an attention mechanism; performing intent recognition sample processing on the results of voice recognition, and performing intent recognition based on the intent recognition model; generating voice response content according to the recognized intent, and performing voice response through a customized voice assistant's virtual image; performing multi-terminal adaptation and cross-device collaborative processing based on a multi-terminal adaptation mechanism and a resource allocation mechanism, and synchronizing and feedbacking the voice interaction content and the voice response of the virtual image. The embodiment of the present invention improves the overall capabilities of voice recognition, semantic understanding, personalized services and multi-terminal collaboration, and provides users with an efficient, natural and intelligent voice interaction experience.

[0034] like Figure 1 As shown, an embodiment of the present invention provides a multi-terminal intelligent voice coordination interaction method, including the following steps: Step S100: collect voice data, pre-process the voice data, and perform voice recognition based on a speech recognition model of an attention mechanism.

[0035] In this embodiment, a multi-terminal intelligent voice coordination interaction method based on artificial intelligence and voice interaction technology is constructed to improve the overall capabilities of voice recognition, semantic understanding, personalized services and multi-terminal collaboration, and provide users with an efficient, natural and intelligent voice interaction experience; the specific technical solutions of the method include the following parts: voice recognition module optimization, improved semantic understanding accuracy, personalized voice services, multi-terminal collaboration and cross-platform adaptation, and flexible terminal adaptation and resource allocation mechanisms.

[0036] In order to realize the multi-terminal intelligent voice coordination interaction method, this embodiment also provides a multi-terminal intelligent voice coordination interaction framework based on artificial intelligence and voice interaction technology. The framework is mainly divided into three levels: terminal layer, platform core capability layer and underlying technical support layer, covering the full process design from multi-terminal adaptation, voice recognition, intent analysis to model fine-tuning and platform management. Through clear interfaces and communication mechanisms between modules, it ensures efficient and stable operation of the entire process from voice input to response output, while supporting flexible functional expansion and continuous optimization.

[0037] The terminal layer architecture describes the basic process of users interacting with the platform through voice through different terminal devices (for example, TVs, speakers, and car-mounted devices), laying the foundation for the collection and processing of voice data.

[0038] The platform's core capability layer describes in detail the core functional modules of the intelligent voice platform, including: product management, skill management, model management, terminal access management, virtual human management, data dashboard, platform management, and integration management.

[0039] The underlying technical support layer provides the technical capabilities and system services that underpin these core functions. These include ASR (automatic speech recognition) capabilities, intent recognition, TTS synthesis (a technology that converts text to natural speech), multimodal media resource services, and recommendation profiling services. The platform also specifies the various data types managed by the platform, such as skill data, user data, and business data. These data provide crucial support for the iterative optimization and intelligent upgrade of platform functions.

[0040] The communication relationship between layers and modules is as follows: 1. Terminal layer and platform core capability layer: The terminal device transmits the user's voice request to the ASR module of the platform's core capability layer through the voice interaction module for recognition, and then enters the intent recognition module for analysis, and finally responds by the skill management module or model management module.

[0041] 2. Core capability layer and technical support layer: The platform's core modules leverage the ASR recognition, intent recognition, and TTS generation services provided by the underlying technical support layer to achieve a complete link between speech recognition, intent understanding, and speech synthesis. Furthermore, each platform core module leverages the underlying multimodal media resource services and data management modules to allocate and optimize resources.

[0042] Based on the aforementioned multi-terminal intelligent voice coordination and interaction framework, this embodiment optimizes the voice recognition module by combining deep learning methods such as convolutional neural networks (CNNs) and long short-term memory networks (LSTMs) to improve existing voice recognition algorithms. Furthermore, advanced noise suppression algorithms, such as adaptive filtering and deep neural network (DNN) models, are introduced to eliminate background noise interference on voice recognition. A noise suppression method based on the time and frequency domains filters noise in real time during signal acquisition, improving signal quality. Signal processing techniques enhance voice features during voice signal transmission, reducing the impact of ambient noise. This results in clearer voice reception and recognition by the system, improving voice input accuracy.

[0043] Specifically, in one implementation of this embodiment, step S100 includes the following steps: Step S101: Collect voice data from real user conversation scenarios through microphones in various terminal devices, generate diversified voice samples based on the collected voice data, and obtain supplementary data from public voice datasets to expand the sample size; Step S102, performing spectrum subtraction, adaptive filtering, audio segmentation, feature extraction, and data enhancement processing on the expanded speech data to obtain preprocessed speech data; Step S103: Perform speech recognition on the preprocessed speech data based on the speech recognition model of the attention mechanism to obtain a text result of the speech recognition.

[0044] In this embodiment, the process of optimizing the speech recognition module mainly includes the following steps: speech data collection, speech data preprocessing, speech model architecture design, speech model training, and speech model evaluation.

[0045] During the voice data collection process, we first collected voice data from real user conversations using microphones in various terminal devices to achieve multi-scenario voice collection. This included everyday conversations, phone calls, and voice commands in noisy environments, ensuring the model could accurately recognize user voices in various scenarios. We then collected voice samples from users of diverse genders, ages, regions, and accents to enrich the data, achieve diverse voice samples, and improve the model's universality. Finally, we obtained supplementary data from public voice datasets such as LibriSpeech and VoxCeleb to expand the sample size.

[0046] During speech data preprocessing, spectral subtraction is used to filter out background noise in the speech data, thereby increasing the signal-to-noise ratio of the speech signal to a higher level. Adaptive filtering technology is used to adjust filter parameters to adapt to changes in ambient noise in real time. Afterwards, long recordings are cut into short audio slices based on semantics or pause points to facilitate subsequent annotation and processing of the audio data. Key features such as the Mel-frequency cepstral coefficients (MFCC) and Log-Mel spectrogram of the speech signal are extracted to provide feature input for subsequent speech recognition or semantic understanding. Finally, diverse speech samples are generated by adding background noise, changing speech speed, and changing pitch to improve the robustness of the model.

[0047] In the process of designing the speech model architecture, an end-to-end speech recognition model based on the attention mechanism is used to replace the traditional step-by-step speech processing method. This enables direct conversion from audio waveform to text output and reduces error accumulation in the intermediate feature extraction step.

[0048] Specifically, in one implementation of this embodiment, the following steps are included before step S103: Step S103a, dividing the pre-processed speech data into a training set, a validation set, and a test set according to a preset ratio; Step S103b, using cross entropy as the loss function, and continuously optimizing the model parameters through batch training iterations; Step S103c, adjusting the hyperparameters of the speech recognition model using a Bayesian optimization method, and performing transfer learning using the pre-trained speech recognition model; Step S103d, calculating the word error rate and the phoneme error rate of the speech recognition of the test set, and evaluating the speech recognition model according to the word error rate and the phoneme error rate to obtain the trained speech recognition model.

[0049] In this embodiment, during speech model training, all speech data is first divided into a training set, a validation set, and a test set in a ratio of 7:2:1. Then, using cross entropy as the loss function, the model parameters are continuously optimized through batch training iterations to ensure that the model can fully learn the characteristics of the sample data. Furthermore, Bayesian optimization methods are used to adjust the model's hyperparameters, such as the learning rate and the number of hidden layers, to improve model performance. During training, transfer learning can be performed using the pre-trained speech recognition model Wav2Vec 2.0, accelerating model training and improving results.

[0050] During speech model evaluation, we first evaluate whether the model performance meets expectations by calculating key indicators such as the word error rate (WER) and phoneme error rate (PER) of the test set speech recognition. Then, combining professional knowledge from different fields, we further check the semantic accuracy of the speech recognition results to ensure that the recognized text meets the needs of the business scenario.

[0051] In this embodiment, based on the above two steps of speech model training and speech model evaluation, a trained speech recognition model based on the attention mechanism can be obtained; then, the speech data to be recognized is input into this speech recognition model, and the preprocessed speech data is used to perform speech recognition, and the text result of the speech recognition can be output.

[0052] This embodiment achieves full-link optimization from user voice input to semantic understanding through the deep integration of speech recognition and natural language processing technologies. The use of an adaptive acoustic model and a dynamic language model update mechanism enables the system to dynamically adjust to the user's voice characteristics and scenario requirements, significantly improving the accuracy of recognition and understanding. This addresses the limitations of existing voice assistants, which are unable to provide high-precision understanding for diverse users and complex speech scenarios, especially the difficulty of recognition in dialects and noisy environments.

[0053] In specific application scenarios, deep learning algorithms can be used to optimize acoustic models and combined with context-aware NLP to achieve efficient integration of speech and language processing across the entire chain.

[0054] like Figure 1 As shown, an embodiment of the present invention provides a multi-terminal intelligent voice coordination interaction method, including the following steps: Step S200: Perform intent recognition sample processing on the result of speech recognition, and perform intent recognition based on the intent recognition model.

[0055] Based on the above multi-terminal intelligent voice coordination interaction framework, this embodiment improves the accuracy of semantic understanding by collecting and annotating real user conversation data in different fields, building a private user corpus knowledge base, and using this data to train and fine-tune the intent model to improve the accuracy of semantic understanding; and through a small number of given example samples, combined with the capabilities of the large model, domain semantic knowledge and predefined rules, intelligently generate a user conversation corpus covering different expressions, language styles, regional expressions and long-tail use cases, providing richer training materials for the intent model. Finally, by continuously tracking and understanding the user's conversation history, the user's current speech is recognized in combination with contextual information, so that the user's true intention can be more accurately identified in multi-round conversation scenarios, improving the robustness and accuracy of semantic understanding.

[0056] Specifically, in one implementation of this embodiment, step S200 includes the following steps: Step S201, performing word segmentation, part-of-speech tagging, and entity recognition on the text result obtained by speech recognition; Step S202: Based on the large language model, combined with knowledge bases in various fields and predefined rules, a diverse set of user dialogue data is generated to obtain a refined intent recognition sample set; Step S203: performing intent classification on the improved intent recognition sample set based on the classification model; Step S204: Filling slots in the classified intent recognition sample set based on the sequence labeling model to obtain processed intent recognition samples; Step S205 : inputting the processed intent recognition sample into the intent recognition model to perform intent recognition and obtain an intent recognition result.

[0057] In this embodiment, the process of improving the accuracy of semantic understanding mainly includes the following steps: intent recognition sample processing, intent recognition model architecture design, intent recognition model training, and intent recognition result evaluation.

[0058] Specifically, during the intent recognition sample processing process, the text results obtained by the terminal through speech recognition are first preprocessed by word segmentation, part-of-speech tagging, entity recognition, etc.; then, the text generation capability of the large language model (LLM) is utilized, combined with knowledge bases in various fields and predefined rules, to generate diverse user dialogue materials, ensuring that the training samples can cover different user expressions, regionalized language habits and long-tail use cases, further improving the intent recognition sample set; and based on the classification model RoBERTa, the intent sample data is intelligently and automatically classified to facilitate learning and analyzing the characteristics of various intents and improve the accuracy of the intent recognition model; finally, the sequence labeling model BiLSTM-CRF is used to extract key semantic slot information from the user dialogue data. For example, in navigation intents, important information about "location" and "time" is extracted from the user's words.

[0059] During the design of the intent recognition model architecture, the pre-trained GPT language model will be combined with the long short-term memory network (LSTM) module, so that the intent model can update and optimize the recognition results in real time based on the user's historical conversation records, ensuring that the user's intentions can always be accurately understood in multiple rounds of conversations.

[0060] During intent recognition model training, a multi-task learning approach is employed to jointly train intent classification and slot filling tasks, improving the model's overall semantic understanding capabilities. Furthermore, targeted model fine-tuning is performed in specific domains (such as smart home or automotive) to achieve domain adaptation and improve the model's semantic understanding precision and accuracy in those specific areas.

[0061] During the intent recognition evaluation process, the accuracy of the current intent model is evaluated by calculating the precision, recall, and F1-score of intent classification. Furthermore, the quality of intelligent slot filling is further evaluated by calculating the values of indicators such as slot filling precision and recall.

[0062] In this embodiment, based on the above process of intent recognition model training and intent recognition result evaluation, a trained intent recognition model can be obtained, and the processed intent recognition samples are input into the intent recognition model for intent recognition to obtain the intent recognition result.

[0063] In this embodiment, the intent recognition model is optimized, which can separate and synchronously process multiple user intentions in a single voice input, and a context memory and understanding module is added, so that the intent model can update and optimize the recognition results in real time in combination with the user's historical conversation records, thereby improving the accuracy of identifying user intentions in multiple rounds of conversations; it overcomes the problem that traditional voice assistants can only process a single intent at a time and cannot remember contextual historical conversation information, realizes multi-tasking parallelism, and improves user interaction efficiency and accuracy.

[0064] In specific application scenarios, multi-intent detection can be used to identify and process multi-intent voice input, combined with sequence labeling technology and deep attention mechanisms. The pre-trained GPT language model is combined with a long short-term memory network to record and analyze contextual information from multiple rounds of conversation.

[0065] like Figure 1 As shown, an embodiment of the present invention provides a multi-terminal intelligent voice coordination interaction method, including the following steps: Step S300: Generate voice response content based on the recognized intention, and make a voice response through the customized voice assistant's virtual image.

[0066] Based on the above multi-terminal intelligent voice coordination and interaction framework, this embodiment provides flexible virtual human customization capabilities for personalized voice services, allowing users to quickly build customized voice assistants based on their personal preferences and business scenarios. Users can customize multiple elements of the virtual human's timbre, intonation, appearance, and other elements, making the voice assistant's interaction style more tailored to their needs. It also implements a personalized response strategy based on historical data. By analyzing user interests, habits, historical behavior, and other data, and combining it with a deep learning model, it dynamically adjusts the voice assistant's response content to provide each user with a personalized voice interaction experience, ensuring a more targeted and personalized service. Furthermore, it implements dynamic voice feedback and emotional interaction capabilities, supporting the recognition of user emotional changes and adaptively adjusting the voice assistant's response. For example, when a user is confused or anxious, the voice assistant adopts a softer tone and soothing language to provide assistance; when a user expresses excitement or joy, the voice assistant adopts a more enthusiastic tone to respond, enhancing the user's emotional resonance.

[0067] Specifically, in one implementation of this embodiment, step S300 includes the following steps: Step S301: Generate interactive response content using a generative language model based on a custom question-answer pair, and convert the text corresponding to the interactive response content into speech output using speech synthesis technology; Step S302 , the converted speech is responded to in a voice by a customized avatar, and the voice response form of the avatar is optimized according to a speech interaction optimization strategy.

[0068] In this embodiment, the process of personalized voice service mainly includes the following steps: voice response generation, avatar image customization, and voice interaction optimization.

[0069] During voice response generation, we first generate accurate responses based on the user's uploaded custom question-and-answer pairs, thus meeting personalized needs. We then leverage the generative language model ChatGPT to generate more natural and personalized interactive responses. Finally, we convert the response text into speech output using Text-to-Speech (TTS) technology, and further enhance the quality of the generated speech using WaveNet.

[0070] During the customization of the virtual image, the ability to customize the virtual image of the voice assistant according to personal needs can be achieved, supporting users to configure the timbre, tone, accent, and visual appearance of the voice assistant's voice, allowing users to quickly create their own unique voice assistant on the voice platform.

[0071] Specifically, in one implementation of this embodiment, step S302 includes the following steps: Step S302a, optimizing personalized response content by analyzing historical interaction data; Step S302b, using real-time voice emotion analysis technology to determine the current emotion and optimize the emotional response content; Step S302c: Dynamically adjust the voice interaction strategy by analyzing voice input and behavior data in real time, and automatically adjust the voice response strategy based on user feedback and historical data.

[0072] In this embodiment, during the process of voice interaction optimization, optimization is mainly carried out in the following aspects: 1) Personalized Response: By analyzing historical user interaction data (e.g., voice commands, behavioral data, etc.), a personalized voice service strategy is established for each user. For example, the system adjusts the voice assistant's tone, response method, and service content based on the user's voice usage habits (e.g., frequency of use, preferred scenarios, etc.) to enhance the naturalness and affinity of the interaction.

[0073] 2) Emotional Response: Real-time voice emotion analysis technology is used to determine the user's current mood (such as happiness, confusion, irritation, etc.) and adjust the voice assistant's feedback accordingly. For example, if the system detects that the user is depressed, the voice assistant's tone will automatically adjust to comfort or provide assistance.

[0074] 3) Real-time feedback mechanism: By analyzing user voice input, behavioral data, etc. in real time, the voice interaction strategy is dynamically adjusted. When it is detected that the user is confused or does not receive a satisfactory response in a conversation, the voice tone is automatically adjusted or further assistance is provided in a timely manner to enhance the smoothness of the interaction and user satisfaction.

[0075] 4) Dynamic Strategy Adjustment: Automatically adjust service strategies based on user feedback and historical data. For example, if a user repeatedly asks the same question, the system will adjust its answer strategy through active learning to more efficiently address the user's needs.

[0076] In this embodiment, the voice assistant uses emotion recognition and feedback mechanisms to sense the user's emotional state and respond emotionally. When the user's emotions fluctuate, the voice assistant can adjust its tone and intonation or provide soothing responses, thereby enhancing the emotional depth of the interaction and the user's sense of intimacy. This approach overcomes the problem of traditional voice assistants' inability to understand and respond to users' emotional states, enabling voice assistants to establish a higher-level emotional connection with users, increasing user stickiness and frequency of use.

[0077] In specific application scenarios, emotion recognition technology and natural language processing algorithms can be used to analyze the emotional components in the user's voice, dynamically adjust the voice output of the voice assistant according to changes in the user's emotions, and perform emotional resonance and personalized feedback.

[0078] like Figure 1 As shown, an embodiment of the present invention provides a multi-terminal intelligent voice coordination interaction method, including the following steps: Step S400 , performing multi-terminal adaptation and cross-device collaborative processing based on the multi-terminal adaptation mechanism and resource allocation mechanism, and synchronizing and feeding back the voice interaction content and the voice response of the virtual image.

[0079] Based on the aforementioned multi-terminal intelligent voice coordination and interaction framework, this embodiment addresses multi-terminal collaboration and cross-platform adaptation by building a centralized intelligent voice platform to unify the management of multiple terminal devices (such as smart speakers, smart TVs, and smart car systems). This ensures that data synchronization, status updates, and command distribution across all devices can be controlled through the same system, and that timely voice interaction status is shared. Furthermore, a device adaptation module is implemented to ensure compatibility across different terminal devices. Regardless of whether a user is using a smart speaker, smart TV, or car system, the system automatically optimizes the quality of voice input and output, as well as the interaction method, based on the capabilities and characteristics of different hardware devices (such as speaker quality, microphone sensitivity, and screen display), providing a consistent voice interaction experience. Furthermore, a cloud-based data synchronization mechanism ensures consistent user operations across multiple terminals. When a user switches between devices, the system responds instantly and seamlessly switches the interaction scenario between the corresponding devices, eliminating the need for the user to re-enter the system. For example, commands issued on a smart speaker can be instantly synchronized to a smart TV or car system.

[0080] This embodiment utilizes a modular design for flexible terminal adaptation and resource allocation mechanisms. This allows the system to automatically allocate resources based on the hardware requirements of different terminals (such as processing power and storage resources). This allows for flexible expansion of new devices and customized configurations. For example, computing resource requirements can be automatically reduced on low-power devices, while more complex speech analysis models can be selected on high-performance devices.

[0081] In addition, data from all terminal devices is uploaded to the intelligent voice platform in real time and synchronously, enabling the platform to dynamically adjust the allocation of voice processing tasks based on the current status of the device (such as network connection, computing load, etc.), ensuring efficient use of computing resources.

[0082] And through the centralized resource scheduling module, the computing resource status of the device (such as CPU load, memory usage, etc.) is analyzed in real time, and resources are dynamically allocated as needed to ensure that the voice interaction experience is not affected when multiple devices are running at the same time, avoiding resource conflicts or overloads.

[0083] Specifically, in one implementation of this embodiment, step S400 includes the following steps: Step S401: managing and synchronizing the voice interaction data and device status information of each terminal through the intelligent voice platform; Step S402: detecting compatibility between devices based on the multi-terminal adaptation mechanism, and optimizing the quality of voice input and output according to a preset adaptive module; Step S403: Perform cross-device collaborative processing based on the resource allocation mechanism to synchronize and provide feedback on the voice interaction content and the voice response of the virtual image.

[0084] In this embodiment, the voice interaction process of the avatar on each terminal is managed through a centralized management platform. All terminal devices (such as smart speakers, smart TVs, and smart cars) are centrally managed through a unified intelligent voice platform. This platform can manage and synchronize voice interaction data, device status, and other information from each terminal, ensuring a consistent interactive experience across devices in different scenarios.

[0085] Based on the centralized management platform, this embodiment adopts a modular design approach to the overall architecture of the intelligent voice system, creating multiple independent modules such as voice recognition, voice synthesis, and user profiling, enabling the system to be dynamically optimized according to the specific characteristics of the terminal device, thereby improving the system's flexibility and scalability. In addition, a unified access evaluation process is designed for different hardware (such as speakers, microphones, displays, etc.) to detect compatibility between devices, and an adaptive module is added to enable the system to automatically optimize the quality of voice input and output according to the hardware performance of different devices, ensuring that users can obtain the best voice interaction experience on each terminal.

[0086] Based on the centralized management platform, modular design, and multi-terminal adaptation mechanism described above, this embodiment automatically synchronizes the voice interaction status of the current device to other devices during cross-device collaboration when the user switches devices. For example, when a user interacts with a voice assistant on a smart speaker, the system will synchronize the conversation content and status to a smart TV or in-car device in real time, ensuring that the user can continue the conversation with the voice assistant anytime, anywhere, and avoiding repeated operations.

[0087] Moreover, in the process of cross-device collaboration, real-time resource allocation can also be performed. Specifically, the system can dynamically allocate resources based on the real-time resource status of the device. When multiple terminal devices are used at the same time, some voice processing tasks are allocated to the cloud by calculating the real-time load to optimize the performance of local devices and ensure a smooth user interaction experience.

[0088] Furthermore, during cross-device collaboration, this embodiment incorporates a data synchronization and feedback mechanism. Specifically, the platform synchronizes and monitors device status in real time. By incorporating machine learning technology, it intelligently locates indicators experiencing abnormalities (e.g., recognition results that deviate from user intent) and promptly provides feedback and synchronization to platform administrators, enabling them to quickly adjust device performance. If a device experiences performance bottlenecks or delays, tasks are automatically assigned to other devices, ensuring the continuity and stability of voice interaction.

[0089] This embodiment utilizes a unified platform's centralized management architecture to ensure data synchronization and collaboration across multiple devices (such as smart speakers, smart TVs, and smart car systems). Cross-platform adaptation technology ensures a seamless voice interaction experience across different devices. This addresses the issue of seamless collaboration and adaptation between devices in traditional voice assistant systems, enabling a consistent user experience across different devices and improving the convenience and fluidity of cross-device use.

[0090] In specific application scenarios, a distributed system architecture and device adaptation module can be adopted. Through cloud data synchronization technology and cross-platform API interface, different devices can share the same user data and interaction status.

[0091] This embodiment introduces a dynamic resource allocation mechanism that automatically adjusts computing resources and bandwidth allocation based on the hardware characteristics of different terminals and the current system load. By leveraging real-time data feedback, it optimizes computing resources for voice processing and recognition, ensuring the system maintains high efficiency and responsiveness. This addresses the issues of inflexible resource scheduling and wasted computing resources in traditional voice assistant systems. By dynamically adjusting resource allocation, it improves system responsiveness and processing efficiency.

[0092] In specific application scenarios, a resource scheduling algorithm based on real-time data stream analysis can be used to adjust the allocation of cloud and local computing resources according to device status and load conditions to ensure the real-time and efficient performance of speech recognition, processing, and feedback.

[0093] This embodiment establishes an abnormal scenario detection and emergency response mechanism, which can automatically detect problems and restore functionality in special situations such as noise interference or hardware failure. This addresses the shortcomings of traditional voice assistants, such as interaction failures in non-ideal environments and the lack of emergency response.

[0094] In specific application scenarios, interaction data can be monitored through anomaly detection algorithms based on probability distribution, and intelligent fault recovery can be performed in combination with backup models or offline modes.

[0095] The above innovations ensure breakthroughs and optimizations in intelligent voice recognition, personalized services, multi-terminal collaboration, resource allocation, etc. in this embodiment by integrating advanced technical means, providing an efficient, accurate and emotional voice interaction solution.

[0096] The alternatives that can be changed in this embodiment are as follows: 1) Intelligent speech recognition and understanding: Multimodal recognition technology: Solution Description: Building on existing speech recognition technology, this approach integrates visual and tactile information, using multimodal recognition technology (such as combining input from cameras, sensors, and other devices) to enhance the system's ability to understand voice commands. For example, the system can use facial expressions, gestures, or eye movements to determine the accuracy of voice commands and user intent.

[0097] How it works: By integrating voice, image, and motion recognition technologies, data from various sources is comprehensively analyzed to improve the recognition rate and accuracy of complex voice commands. A deep learning model based on a multimodal neural network is used for data processing and semantic understanding.

[0098] Advantages and application scenarios: This technology will improve the model's accuracy in identifying user intent in noisy environments and enhance the adaptability and intelligence of voice interaction.

[0099] Fault tolerance mechanism for speech recognition: Solution Description: Introducing a stronger speech recognition error-tolerance mechanism. When the user's pronunciation is unclear or errors occur in the voice command, the system can adaptively correct and understand the user's intent. This approach uses a combination of acoustic and language models, incorporating contextual information for error tolerance and speech repair.

[0100] Implementation Principle: The speech recognition model is trained using a reinforcement learning algorithm, enabling it to accurately understand user intent in diverse pronunciation environments and accents. Deep neural networks and contextual models are used to automatically identify and correct mispronounced or incomplete speech input. Advantages and Application Scenarios: This system is suitable for multilingual and multi-accent environments, particularly in cross-cultural and cross-regional smart devices, effectively reducing recognition errors caused by pronunciation differences.

[0101] 2) Multi-scenario application of virtual human images: Solution Description: The virtual voice assistant's role and style can be switched in real time based on different application scenarios (such as home, office, and car). For example, in a home setting, a gentle and friendly tone of voice can be used to make users feel comfortable and warm; in a work environment, a more rigorous and concise tone of voice can be used to highlight the professionalism of the response.

[0102] How it works: By leveraging contextual recognition technology and emotional computing models, combined with data from multiple scenarios, the system automatically switches the virtual voice assistant's tone, language style, and interaction method. Advantages and Applications: Suitable for in-car voice assistants, office assistants, and smart home devices, it enables intelligent voice assistants to provide the most appropriate interaction method in various usage environments, further enhancing the user experience.

[0103] Improvements or changes can be made based on the above description, and all these improvements and changes should fall within the protection scope of the embodiments of the present invention.

[0104] This embodiment achieves the following technical effects through the above technical solution: 1) This embodiment uses advanced speech recognition models and intent recognition models for recognition, improving the accuracy of speech recognition and semantic understanding in noisy environments, complex contexts, and multi-round conversations, thereby further improving system accuracy and response speed.

[0105] 2) This embodiment can customize the avatar's timbre, intonation, appearance, and other characteristics as needed to enhance the user's voice interaction experience. It can also dynamically adjust the response strategy based on user behavior habits, user profile, and contextual dialogue environment to provide users with highly personalized voice services.

[0106] 3) This embodiment achieves seamless collaboration between multiple terminal devices, ensuring that users have a consistent voice interaction experience on different devices, while supporting instant information sharing and interaction status synchronization between smart devices.

[0107] 4) This embodiment can flexibly adjust resource allocation and optimization strategies according to the hardware characteristics of different terminal devices and user needs, thereby improving device adaptability and system operating efficiency.

[0108] Exemplary devices Based on the above embodiments, the present invention further provides a multi-terminal intelligent voice coordination interaction system, comprising: A speech recognition module is used to collect speech data, pre-process the speech data, and perform speech recognition based on a speech recognition model with an attention mechanism; The intent recognition module is used to process the speech recognition results into intent recognition samples and perform intent recognition based on the intent recognition model; The voice response module is used to generate voice response content based on the recognized intent and provide voice response through a customized voice assistant avatar; The multi-terminal collaboration module is used to perform multi-terminal adaptation and cross-device collaborative processing based on the multi-terminal adaptation mechanism and resource allocation mechanism, and synchronize and feedback the voice interaction content and the voice response of the virtual image.

[0109] This embodiment achieves the following technical effects through the above technical solution: 1) This embodiment uses advanced speech recognition models and intent recognition models for recognition, improving the accuracy of speech recognition and semantic understanding in noisy environments, complex contexts, and multi-round conversations, thereby further improving system accuracy and response speed.

[0110] 2) This embodiment can customize the avatar's timbre, intonation, appearance, and other characteristics as needed to enhance the user's voice interaction experience. It can also dynamically adjust the response strategy based on user behavior habits, user profile, and contextual dialogue environment to provide users with highly personalized voice services.

[0111] 3) This embodiment achieves seamless collaboration between multiple terminal devices, ensuring that users have a consistent voice interaction experience on different devices, while supporting instant information sharing and interaction status synchronization between smart devices.

[0112] 4) This embodiment can flexibly adjust resource allocation and optimization strategies according to the hardware characteristics of different terminal devices and user needs, thereby improving device adaptability and system operating efficiency.

[0113] Based on the above embodiment, the present invention further provides a terminal, whose principle block diagram can be shown as follows: Figure 2 shown.

[0114] The terminal includes: a processor, memory, interface, display screen and communication module connected via a system bus; wherein the processor of the terminal is used to provide computing and control capabilities; the memory of the terminal includes a storage medium and an internal memory; the storage medium stores an operating system and computer programs; the internal memory provides an environment for the operation of the operating system and computer programs in the storage medium; the interface is used to connect to external devices; the display screen is used to display corresponding information; and the communication module is used to communicate with a cloud server or other devices.

[0115] When the computer program is executed by a processor, it is used to implement the operation of the multi-terminal intelligent voice coordinated interaction method.

[0116] It will be understood by those skilled in the art that Figure 2 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the solution of the present invention, and does not constitute a limitation on the terminal to which the solution of the present invention is applied. The specific terminal may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0117] In one embodiment, a terminal is provided, which includes: a processor and a memory, the memory storing a multi-terminal intelligent voice coordination interaction program, and the multi-terminal intelligent voice coordination interaction program is used to implement the operation of the above-mentioned multi-terminal intelligent voice coordination interaction method when executed by the processor.

[0118] In one embodiment, a storage medium is provided, wherein the storage medium stores a multi-terminal intelligent voice coordination interaction program, which is used to implement the operation of the above multi-terminal intelligent voice coordination interaction method when executed by a processor.

[0119] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile storage medium. When executed, the computer program can include the processes in the above-described method embodiments. Any reference to memory, storage, database, or other media used in the various embodiments provided herein may include both non-volatile and volatile memory.

[0120] In summary, the present invention provides a multi-terminal intelligent voice coordinated interaction method and system, including: collecting voice data, pre-processing the voice data, and performing voice recognition based on a voice recognition model of an attention mechanism; performing intent recognition sample processing on the results of voice recognition, and performing intent recognition based on the intent recognition model; generating voice response content according to the recognized intent, and performing voice response through a customized voice assistant's virtual image; performing multi-terminal adaptation and cross-device collaborative processing based on a multi-terminal adaptation mechanism and a resource allocation mechanism, and synchronizing and feeding back the voice interaction content and the voice response of the virtual image. The present invention improves the overall capabilities of voice recognition, semantic understanding, personalized services, and multi-terminal collaboration, and provides users with an efficient, natural, and intelligent voice interaction experience.

[0121] It should be understood that the application of the present invention is not limited to the above examples. For those skilled in the art, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A multi-terminal intelligent voice coordination interaction method, characterized in that: include: Collecting speech data, preprocessing the speech data, and performing speech recognition based on a speech recognition model with an attention mechanism; Perform intent recognition sample processing on the speech recognition results and perform intent recognition based on the intent recognition model; Generate voice response content based on the recognized intent and respond through a customized voice assistant avatar; Based on the multi-terminal adaptation mechanism and resource allocation mechanism, multi-terminal adaptation and cross-device collaborative processing are performed to synchronize and feedback the voice interaction content and the voice response of the virtual image.

2. The multi-terminal intelligent voice coordination interaction method according to claim 1, characterized in that: The collecting of speech data, preprocessing of the speech data, and performing speech recognition based on a speech recognition model of an attention mechanism include: The system collects voice data from real user conversations through microphones in various terminal devices, generates diversified voice samples based on the collected voice data, and obtains supplementary data from public voice datasets to expand the sample size. Performing spectrum subtraction, adaptive filtering, audio segmentation, feature extraction, and data enhancement on the expanded speech data to obtain preprocessed speech data; The speech recognition model based on the attention mechanism performs speech recognition on the preprocessed speech data to obtain a text result of the speech recognition.

3. The multi-terminal intelligent voice coordination interaction method according to claim 2, characterized in that: The speech recognition model based on the attention mechanism performs speech recognition on the preprocessed speech data to obtain a text result of the speech recognition, which includes: Dividing the preprocessed speech data into a training set, a validation set, and a test set according to a preset ratio; Use cross entropy as the loss function and continuously optimize model parameters through batch training iterations; Adjusting the hyperparameters of the speech recognition model using a Bayesian optimization method, and performing transfer learning using a pre-trained speech recognition model; The word error rate and the phoneme error rate of the speech recognition of the test set are calculated, and the speech recognition model is evaluated according to the word error rate and the phoneme error rate to obtain the trained speech recognition model.

4. The multi-terminal intelligent voice coordination interaction method according to claim 1, characterized in that: The performing intention recognition sample processing on the speech recognition result and performing intention recognition based on the intention recognition model includes: Perform word segmentation, part-of-speech tagging, and entity recognition on the text results obtained from speech recognition; Based on a large language model, combined with knowledge bases in various fields and predefined rules, a diverse set of user dialogue data is generated to obtain a complete set of intent recognition samples. Based on the classification model, performing intent classification on the improved intent recognition sample set; Based on the sequence labeling model, the slots of the classified intent recognition sample set are filled to obtain the processed intent recognition samples; The processed intent recognition sample is input into the intent recognition model to perform intent recognition to obtain an intent recognition result.

5. The multi-terminal intelligent voice coordination interaction method according to claim 1, characterized in that: Generating voice response content based on the recognized intent and providing a voice response through a customized voice assistant avatar includes: Generate interactive response content based on the customized question-answer pair using a generative language model, and convert the text corresponding to the interactive response content into speech output using speech synthesis technology; The converted speech is responded to in a voice by a customized virtual human image, and the voice response form of the virtual human image is optimized according to the speech interaction optimization strategy.

6. The multi-terminal intelligent voice coordination interaction method according to claim 5, characterized in that: Optimizing the voice response form of the virtual human image according to the voice interaction optimization strategy includes: Optimize personalized response content by analyzing historical interaction data; Use real-time voice emotion analysis technology to determine current emotions and optimize emotional response content; By analyzing voice input and behavior data in real time, the voice interaction strategy is dynamically adjusted, and the voice response strategy is automatically adjusted based on user feedback and historical data.

7. The multi-terminal intelligent voice coordination interaction method according to claim 1, characterized in that: Based on the multi-terminal adaptation mechanism and resource allocation mechanism, multi-terminal adaptation and cross-device collaborative processing are carried out to synchronize and feedback the voice interaction content and the voice response of the virtual image, including: Manage and synchronize voice interaction data and device status information of each terminal through the intelligent voice platform; Detecting compatibility between devices based on the multi-terminal adaptation mechanism and optimizing the quality of voice input and output according to a preset adaptive module; Based on the resource allocation mechanism, cross-device collaborative processing is performed to synchronize and provide feedback on the voice interaction content and the voice response of the virtual image.

8. A multi-terminal intelligent voice coordination and interaction system, characterized in that: include: A speech recognition module is used to collect speech data, pre-process the speech data, and perform speech recognition based on a speech recognition model with an attention mechanism; The intent recognition module is used to process the speech recognition results into intent recognition samples and perform intent recognition based on the intent recognition model; The voice response module is used to generate voice response content based on the recognized intent and provide voice response through a customized voice assistant avatar; The multi-terminal collaboration module is used to perform multi-terminal adaptation and cross-device collaborative processing based on the multi-terminal adaptation mechanism and resource allocation mechanism, and synchronize and feedback the voice interaction content and the voice response of the virtual image.

9. A terminal, characterized in that: include: A processor and a memory, wherein the memory stores a multi-terminal intelligent voice coordination interaction program, and when the multi-terminal intelligent voice coordination interaction program is executed by the processor, it is used to implement the operation of the multi-terminal intelligent voice coordination interaction method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a multi-terminal intelligent voice coordination interaction program, which, when executed by a processor, is used to implement the operation of the multi-terminal intelligent voice coordination interaction method according to any one of claims 1 to 7.

Images