Cooperative office method and system based on voice interaction and large model

Abstract

According to the collaborative office method and system based on voice interaction and a large model, terminal acquisition is carried out through an intelligent microphone array, key features are reserved while voice definition and purity are improved through real-time audio stream preprocessing, environmental noise is effectively restrained, and the quality of voice signals is improved; the method comprises the following steps: firstly, performing conversion by combining an end-to-end deep neural network Transform model, and greatly improving the precision of speech recognition and natural language understanding through speech understanding and analysis, so that a speech instruction of a user can be recognized more accurately, the intention and context of a dialogue can be deeply understood, and the intelligent level of the system and the user experience are improved; and a multi-modal feedback mechanism feeds back an operation result in time, so that the working efficiency and the instantaneity of user experience are further improved, and the efficient response and feedback mechanism can remarkably improve the office efficiency and reduce the waiting time of the user.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a collaborative office method and system based on voice interaction and large models. Background Art

[0002] At present, there are many collaborative office systems based on voice interaction on the market. Using collected voice information for interactive office work has become a trend, but they still have varying degrees of deficiencies in voice collection, recognition, understanding, task execution and feedback. The current technology makes voice collection unclear, which leads to inaccurate voice recognition; the existing collaborative system has a low level of intelligence and cannot be executed based on simple semantics, or the system cannot respond due to semantic deviations; secondly, the feedback is not timely, or it is impossible to provide a personalized and efficient office experience based on user behavior.

[0003] The present invention aims to significantly improve office efficiency, optimize user experience, and realize personalization and intelligence of office systems by integrating advanced voice collection and preprocessing technologies, accurate voice recognition and natural language understanding algorithms, efficient task execution and instant multimodal feedback mechanisms, and continuous user behavior analysis and model iteration capabilities. Summary of the invention

[0004] The purpose of the present invention is to solve the shortcomings of the prior art and to propose a collaborative office method and system based on voice interaction and large models.

[0005] In order to achieve the above object, the present invention adopts the following technical solution, a collaborative office method based on voice interaction and large model, comprising the following steps:

[0006] S1: Voice collection, voice collection through intelligent microphone array;

[0007] S2: Real-time audio stream processing, sending the collected audio signal to the real-time audio stream processing module, analyzing the spectral characteristics of the audio signal, improving the clarity and purity of the voice, and retaining the key features of the voice signal;

[0008] S3: End-to-end deep neural network, converting the processed speech features into text information;

[0009] S4: Natural language understanding, using natural language understanding (NLU) algorithms to accurately capture cultural information and obtain demand information;

[0010] S5: Rapid execution and feedback mechanism, providing output or response according to user intentions and needs;

[0011] S6: Continuous analysis and model iteration. By recording the voice commands and operational feedback data of users when using the system, the model parameters are updated in real time using online learning and personalized model fine-tuning technology.

[0012] Preferably, the audio stream processing includes speech enhancement (SE) and voice activity detection (VAD), mainly performing framing, windowing, Fourier transform, echo cancellation and volume adjustment.

[0013] Preferably, the end-to-end deep neural network is a Transformer model, which recognizes text directly from the audio stream.

[0014] Preferably, the natural language understanding NL algorithm performs semantic analysis on the recognized text including domain detection, intent recognition and slot filling.

[0015] Preferably, the NLU algorithm maintains the conversation history and understands the context through the RNN or LSTM model; and combines the CRF and BERT models for semantic analysis.

[0016] Preferably, the specific process of the rapid execution and feedback mechanism is:

[0017] A1. Intent recognition and processing: The text intent is obtained after being processed by the speech recognition and natural language understanding modules, and then the user's intent is processed;

[0018] A2. Entity extraction: extracting entity information related to intent through the NER model;

[0019] A3. Command formatting: formatting the extracted intent and entity information into one or more operation instructions;

[0020] A4. Instruction mapping, mapping the formatted operation instructions to specific execution commands;

[0021] A5. Verification and adjustment. Before the system executes the mapped command, the system performs simple verification and adjustment.

[0022] A6. Execute the command, and the system sends it to the execution engine;

[0023] A7. Feedback generation: the system generates corresponding feedback information and informs the user of the results through the feedback mechanism.

[0024] Preferably, the feedback mechanism mainly includes instant feedback, multi-modal feedback and personalized feedback.

[0025] Preferably, the specific steps of the continuous analysis and model iteration are:

[0026] B1. User behavior log collection. Whenever a user uses the system, the user behavior log module will record the user's voice commands, operation feedback, usage time, frequency, and any related data on interaction with the system;

[0027] B2. Data preprocessing and analysis: Regularly preprocess the collected user behavior logs, including data cleaning, feature extraction and statistical analysis;

[0028] B3. Develop personalized model fine-tuning strategies, including selecting specific model parameters for adjustment, optimizing the model's training data set, and adjusting the model's training objectives;

[0029] B4. Model fine-tuning implementation, using incremental learning technology to fine-tune the model to adjust the parameters and weights of the model;

[0030] B5. Evaluation and feedback: Test and evaluate the fine-tuning effect, including using user-specific voice samples for recognition testing, observing the performance of the system in actual use, etc. At the same time, collect user feedback;

[0031] B6. Continuous iteration and optimization. Treat the above process as a continuous iterative cycle and adjust the model fine-tuning strategy and implementation fine-tuning as needed;

[0032] Preferably, the system of the collaborative office method based on voice interaction and large models includes: a voice acquisition terminal, an audio preprocessing module, a conversion module, a speech understanding and parsing module, an execution and feedback module, an analysis and iteration module and an output module. The voice acquisition terminal transmits the collected information to the audio preprocessing module for processing, and converts it through an end-to-end deep neural network, and then the speech understanding and parsing module performs understanding and parsing. The execution and feedback module calls the corresponding system module according to the identified user intentions and needs. The analysis and iteration module iteratively optimizes the system's speech recognition, natural language understanding and other algorithms according to the analysis results to improve the accuracy and adaptability of the system.

[0033] The beneficial effects of the present invention are:

[0034] 1. Use intelligent microphone arrays to collect voice signals, and combine real-time audio stream preprocessing to perform noise reduction, voice activity detection and other preprocessing on audio data to ensure the input quality of subsequent speech recognition modules and improve the overall performance of the system.

[0035] 2. By adopting an end-to-end deep neural network Transformer model to directly recognize text from the audio stream, and combining the RNN / LSTM model to maintain the conversation history and understand the context, as well as combining models such as CRF and BERT for semantic analysis, high-precision speech recognition and natural language understanding are achieved, enhancing the intelligence level of the system.

[0036] 3. Command mapping and execution engine ensures that the system can quickly respond to user voice commands and execute corresponding tasks. At the same time, by providing multiple feedback methods such as voice, text, and graphical interface, a multi-modal feedback mechanism is implemented, which improves the immediacy and richness of the user experience.

[0037] 4. Continuous user behavior analysis and model iteration: The established user behavior analysis mechanism records the voice commands, operation feedback and other data of users when using the system, and uses online learning and personalized model fine-tuning technology to update model parameters in real time. This continuous user behavior analysis and model iteration capability enables the system to gradually adapt to the user's voice and language habits, improving recognition accuracy and personalized service levels. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a schematic diagram of the workflow of the present invention;

[0039] Figure 2 Refine the flow chart for execution and feedback module work;

[0040] Figure 3 Refine the flow chart for analysis and iterative module work;

[0041] Figure 4 It is a flow chart of the system of the present invention. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0043] In the description of the present invention, it should be understood that the terms "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside" and "outside" etc., indicating directions or positional relationships, are based on the directions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore cannot be understood as a limitation on the present invention. In the description of the present invention, "multiple" means two or more, unless otherwise clearly and specifically defined.

[0044] Embodiment 1

[0045] See also Figure 1 , a collaborative office method based on voice interaction and large models, characterized in that it includes the following steps: S1: voice collection, voice collection is performed through an intelligent microphone array; S2: real-time audio stream processing, the collected audio signal is sent to the real-time audio stream processing module, the spectral characteristics of the audio signal are analyzed, the voice clarity and purity are improved, and the key features of the voice signal are retained; S3: end-to-end deep neural network, the processed voice features are converted into text information; S4: natural language understanding, cultural information is accurately captured through the natural language understanding NLU algorithm, and demand information is obtained; S5: fast execution and feedback mechanism, output or response is provided according to the user's intention and needs; S6: continuous analysis and model iteration, by recording the voice instructions and operation feedback data when the user uses the system, the model parameters are updated in real time using online learning and personalized model fine-tuning technology.

[0046] The audio stream processing includes speech enhancement (SE) and voice activity detection (VAD). The main purpose of using these two technologies is to optimize the speech signal, mainly performing framing, windowing, Fourier transform, echo cancellation and volume adjustment to ensure the input quality of the subsequent speech recognition module and improve the overall performance of the system.

[0047] The end-to-end deep neural network is a Transformer model that recognizes text directly from an audio stream.

[0048] The natural language understanding NL algorithm performs semantic analysis on the recognized text, including domain detection, intent recognition, and slot filling.

[0049] The NLU algorithm maintains the conversation history and understands the context through the RNN or LSTM model, and combines the CRF and BERT models for semantic analysis.

[0050] The end-to-end deep neural network Transformer model recognizes text directly from the audio stream; maintains the conversation history and understands the context through the RNN or LSTM model; and combines models such as CRF (conditional random field) and BERT for semantic analysis.

[0051] See also Figure 2 , the specific process of the rapid execution and feedback mechanism is as follows:

[0052] A1. Intent recognition and processing: The text intent is obtained after being processed by the speech recognition and natural language understanding modules, and then the user's intent is processed;

[0053] A2. Entity extraction: extracting entity information related to intent through the NER model;

[0054] A3. Command formatting: formatting the extracted intent and entity information into one or more operation instructions;

[0055] A4. Instruction mapping, mapping the formatted operation instructions to specific execution commands;

[0056] A5. Verification and adjustment. Before the system executes the mapped command, the system performs simple verification and adjustment.

[0057] A6. Execute the command, and the system sends it to the execution engine;

[0058] A7. Feedback generation: the system generates corresponding feedback information and informs the user of the results through the feedback mechanism.

[0059] For example, the following specific steps:

[0060] a1. Intent Recognition

[0061] Input: text intent obtained after processing by the speech recognition and natural language understanding modules (such as "remind me to attend a meeting at 9 am tomorrow").

[0062] Processing: The system first parses the text and recognizes that the user's intent is "set a reminder." This step typically relies on an intent classification model that has been trained to recognize a predefined set of user intents.

[0063] a2. Entity Extraction

[0064] Processing: After identifying the user's intent, the system further analyzes the text and extracts entity information related to the intent, such as time ("tomorrow at 9 am"), event type ("meeting"), etc. This step is usually implemented using Named Entity Recognition (NER) technology.

[0065] a3. Command formatting

[0066] Processing: Based on the extracted intent and entity information, the system formats the information into one or more action instructions. In this example, an action instruction may be a structured data object containing time, event, and reminder action, such as {"action":"set_reminder","time":"tomorrow morning 9:00","event":"meeting"}.

[0067] a4. Instruction Mapping

[0068] Processing: The system maps the formatted operation instructions to specific execution commands. This may require a mapping table or rule set that defines how different types of operation instructions are converted into execution commands that the system can understand. For example, the above operation instruction may be mapped to call the API of the collaborative office software to perform the operation of setting a reminder.

[0069] a5. Verification and adjustment

[0070] Processing: Before executing the mapped command, the system may perform a simple verification to ensure the legitimacy and correctness of the command. If a problem is found, the system may adjust the command or return an error message to the user.

[0071] a6. Execute commands

[0072] Output: Once the command is validated, the system sends it to the execution engine. The execution engine is responsible for calling the corresponding service or interface to execute the command. In this example, the execution engine may call the API of the collaborative office software to set a meeting reminder for 9 am tomorrow.

[0073] a7. Feedback generation

[0074] Processing: After executing the command, the execution engine returns the execution result to the task execution and feedback module. Based on the execution result, the system generates corresponding feedback information and informs the user of the result through a multi-modal feedback mechanism (such as voice, text or graphical interface).

[0075] The feedback mechanism mainly includes instant feedback, multi-modal feedback and personalized feedback.

[0076] Instant feedback: Instant feedback is provided at every stage of user-system interaction, allowing users to understand the operation results and system status in a timely manner.

[0077] Information is delivered to users through various means such as voice broadcast, text prompts and graphical interfaces, improving the readability and comprehensibility of information.

[0078] Multimodal feedback: Combine multiple feedback methods such as voice, text and graphical interface to provide users with a more comprehensive and intuitive feedback experience. In different scenarios, choose the most appropriate feedback method to improve the efficiency and accuracy of feedback.

[0079] Personalized feedback: Provide personalized feedback content and methods based on user behavior patterns and preferences.

[0080] For example, for users who frequently use voice commands, the proportion of voice feedback can be increased; for users who like reading, more text prompts can be provided.

[0081] Continuous improvement: Continuously optimize the system's feedback mechanism by continuously collecting user feedback and behavior data. Adjust the content and method of feedback according to user needs and expectations to improve user satisfaction and loyalty

[0082] See also Figure 3 , the specific steps of the continuous analysis and model iteration are:

[0083] B1. User behavior log collection. Whenever a user uses the system, the user behavior log module will record the user's voice commands, operation feedback, usage time, frequency, and any related data on interaction with the system;

[0084] B2. Data preprocessing and analysis: Regularly preprocess the collected user behavior logs, including data cleaning, feature extraction and statistical analysis;

[0085] B3. Develop personalized model fine-tuning strategies, including selecting specific model parameters for adjustment, optimizing the model's training data set, and adjusting the model's training objectives;

[0086] B4. Model fine-tuning implementation, using incremental learning technology to fine-tune the model to adjust the parameters and weights of the model;

[0087] B5. Evaluation and feedback: Test and evaluate the fine-tuning effect, including using user-specific voice samples for recognition testing, observing the performance of the system in actual use, etc. At the same time, collect user feedback;

[0088] B6. Continuous iteration and optimization. Treat the above process as a continuous iterative cycle and adjust the model fine-tuning strategy and implementation fine-tuning as needed.

[0089] For example, the following specific steps:

[0090] b1. User behavior log collection

[0091] Process: Whenever a user uses the system, the user behavior log module records the user's voice commands, operation feedback, usage time, frequency, and any related data of interaction with the system. These data are stored in a secure database for subsequent analysis.

[0092] Purpose: To accumulate enough data to understand the user's voice characteristics, language habits, common commands, etc.

[0093] b2. Data preprocessing and analysis

[0094] Process: Regularly pre-process the collected user behavior logs, including data cleaning (removing noise, outliers, etc.), feature extraction (such as user pronunciation characteristics, common vocabulary, command patterns, etc.) and statistical analysis.

[0095] Purpose: Extract useful information for model fine-tuning from raw data and identify unique usage patterns of users.

[0096] b3. Personalized model fine-tuning strategy formulation

[0097] Process: Based on the results of data analysis, develop a personalized model fine-tuning strategy. This may include selecting specific model parameters for adjustment, optimizing the model's training data set (such as adding user-specific voice samples), adjusting the model's training objectives (such as focusing more on the recognition accuracy of users' frequently used commands), etc.

[0098] Purpose: To ensure that model fine-tuning can be optimized for the unique needs of users and improve personalized recognition accuracy.

[0099] b4. Model fine-tuning implementation

[0100] Process: Fine-tune the model using incremental learning techniques such as online gradient descent. This typically involves further training the existing model using user-specific data to adjust the model's parameters and weights.

[0101] Purpose: To gradually adapt the model to the user's voice and language habits, and improve recognition accuracy and response speed.

[0102] b5. Evaluation and feedback

[0103] Process: After the model is fine-tuned, a series of tests are performed to evaluate the fine-tuning effect. These tests may include using user-specific speech samples for recognition testing, observing the performance of the system in actual use, etc. At the same time, user feedback is collected to understand their feelings about the system improvements.

[0104] Purpose: To verify whether the fine-tuning effect has achieved the expected goal, and to make necessary adjustments and optimizations based on the evaluation results and user feedback.

[0105] b6.Continuous iteration and optimization

[0106] Process: Think of the above process as a continuous iterative cycle. As users continue to use the system and generate new behavioral data, the system will continue to collect and analyze this data, and adjust the model fine-tuning strategy and implementation fine-tuning as needed.

[0107] The model update parameter determination process is as follows:

[0108] Data collection and preprocessing: Data collection: First, it is necessary to collect various types of user behavior data during the use of the system, including but not limited to clicks, browsing, purchases, comments, feedback, etc. These data should cover a sufficient time range and user groups to ensure the representativeness of the analysis results.

[0109] Data preprocessing: The collected raw data often contains noise, missing values ​​or inconsistencies. Therefore, before analysis, data needs to be cleaned, missing values ​​filled, data type converted and other preprocessing work to improve the accuracy and usability of the data.

[0110] User behavior analysis: Behavioral feature extraction: Extract user behavior features from preprocessed data, such as visit frequency, dwell time, purchase conversion rate, preference category, etc. These features should be able to fully reflect the user's behavior patterns and preferences.

[0111] Behavioral pattern recognition: Use machine learning algorithms to cluster, classify, or perform association analysis on user behavior characteristics to identify different user groups and their behavior patterns.

[0112] Feedback mechanism integration: Integrate feedback provided by users through voice, text, graphical interface, etc. into behavioral analysis to more fully understand user needs and expectations.

[0113] Model selection and training: Model selection: Based on the results of user behavior analysis, select a suitable machine learning model for training. The choice of model should be based on the characteristics of the data, the complexity of the problem, and the limitations of computing resources.

[0114] Parameter initialization: Set initial parameters for the model. These parameters can be randomly generated or pre-trained based on prior knowledge or historical data.

[0115] Online learning technology: Introducing online learning algorithms, such as incremental learning and adaptive learning, enables the model to update parameters in real time based on newly collected user behavior data.

[0116] Parameter update and model evaluation: Real-time update: Online learning technology allows the model to update parameters immediately when receiving new data without retraining the entire model. This greatly improves the update speed and adaptability of the model.

[0117] Model evaluation: Regularly evaluate the performance of the model, including accuracy, recall, F1 score and other indicators. At the same time, by comparing the performance differences between the new and old models, the effect of model iteration can be quantified.

[0118] Feedback loop: Integrate the results of model evaluation and user feedback into user behavior analysis again to form a closed-loop feedback mechanism to continuously optimize the model.

[0119] Deployment and monitoring: Model deployment: Deploy the updated model to the production environment to actually serve users.

[0120] Performance monitoring: Continuously monitor the performance changes of the model to promptly discover and resolve potential problems. At the same time, collect new user behavior data to prepare for the next round of model iteration.

[0121] Purpose: To ensure that the system can continuously learn and adapt to the changing needs of users, and maintain a high level of recognition accuracy and response speed.

[0122] See also Figure 4 The system of the collaborative office method based on voice interaction and large models includes: a voice acquisition terminal, an audio preprocessing module, a conversion module, a speech understanding and parsing module, an execution and feedback module, an analysis and iteration module and an output module. The voice acquisition terminal transmits the collected information to the audio preprocessing module for processing, and converts it through an end-to-end deep neural network. Then the speech understanding and parsing module performs understanding and parsing. The execution and feedback module calls the corresponding system module according to the identified user intentions and needs. The analysis and iteration module iteratively optimizes the system's speech recognition, natural language understanding and other algorithms according to the analysis results to improve the accuracy and adaptability of the system.

[0123] The above are only preferred embodiments of the present invention and are not intended to limit the present invention in other forms. Any technician familiar with the profession may use the technical contents disclosed above to change or modify them into equivalent embodiments with equivalent changes and apply them to other fields. However, any simple modification, equivalent change and modification made to the above embodiments based on the technical essence of the present invention without departing from the technical solution of the present invention still falls within the protection scope of the technical solution of the present invention.

Claims

1. A collaborative office method based on voice interaction and large models, characterized in that: The following steps are involved: S1: Voice collection, voice collection through intelligent microphone array; S2: Real-time audio stream processing, sending the collected audio signal to the real-time audio stream processing module, analyzing the spectral characteristics of the audio signal, improving the clarity and purity of the voice, and retaining the key features of the voice signal: S3: End-to-end deep neural network, converting the processed speech features into text information; S4: Natural language understanding, using natural language understanding (NLU) algorithms to accurately capture cultural information and obtain demand information; S5: Rapid execution and feedback mechanism, providing output or response according to user intentions and needs; S6: Continuous analysis and model iteration. By recording the voice commands and operational feedback data of users when using the system, the model parameters are updated in real time using online learning and personalized model fine-tuning technology.

2. According to claim 1, a collaborative office method based on voice interaction and large models is characterized in that: The audio stream processing includes speech enhancement (SE) and voice activity detection (VAD), and mainly performs framing, windowing, Fourier transform, echo cancellation and volume adjustment.

3. The collaborative office method based on voice interaction and large models according to claim 1, characterized in that: The end-to-end deep neural network is a Transformer model that recognizes text directly from an audio stream.

4. A collaborative office method based on voice interaction and large models according to claim 1, characterized in that :The natural language understanding NL algorithm performs semantic analysis on the recognized text, including domain detection, intent recognition and slot filling.

5. The collaborative office method based on voice interaction and large models according to claim 1, characterized in that The NLU algorithm maintains the conversation history and understands the context through the RNN or LSTM model, and combines the CRF and BERT models for semantic analysis.

6. The collaborative office method based on voice interaction and large models according to claim 1, characterized in that The specific process of the rapid execution and feedback mechanism is as follows: A1. Intent recognition and processing: The text intent is obtained after being processed by the speech recognition and natural language understanding modules, and then the user's intent is processed; A2. Entity extraction: extracting entity information related to intent through the NER model; A3. Command formatting: formatting the extracted intent and entity information into one or more operation instructions; A4. Instruction mapping, mapping the formatted operation instructions to specific execution commands; A5. Verification and adjustment. Before the system executes the mapped command, the system performs simple verification and adjustment. A6. Execute the command, and the system sends it to the execution engine; A7. Feedback generation: the system generates corresponding feedback information and informs the user of the results through the feedback mechanism.

7. The collaborative office method based on voice interaction and large models according to claim 1, characterized in that :The feedback mechanism mainly includes instant feedback, multi-modal feedback and personalized feedback.

8. The collaborative office method based on voice interaction and large models according to claim 1, characterized in that The specific steps of the continuous analysis and model iteration are: B1. User behavior log collection. Whenever a user uses the system, the user behavior log module will record the user's voice commands, operation feedback, usage time, frequency, and any related data on interaction with the system; B2. Data preprocessing and analysis: Regularly preprocess the collected user behavior logs, including data cleaning, feature extraction and statistical analysis; B3. Develop personalized model fine-tuning strategies, including selecting specific model parameters for adjustment, optimizing the model's training data set, and adjusting the model's training objectives; B4. Model fine-tuning implementation, using incremental learning technology to fine-tune the model to adjust the parameters and weights of the model; B5. Evaluation and feedback: Test and evaluate the fine-tuning effect, including using user-specific voice samples for recognition testing, observing the performance of the system in actual use, etc. At the same time, collect user feedback; B6. Continuous iteration and optimization. Treat the above process as a continuous iterative cycle and adjust the model fine-tuning strategy and implementation fine-tuning as needed.

9. A collaborative office system based on voice interaction and large models, characterized by: According to the collaborative office method based on voice interaction and large models according to any one of claims 1-8, the system includes: a voice acquisition terminal, an audio preprocessing module, a conversion module, a speech understanding and parsing module, an execution and feedback module, an analysis and iteration module and an output module. The voice acquisition terminal transmits the collected information to the audio preprocessing module for processing, and converts it through an end-to-end deep neural network, and then the speech understanding and parsing module performs understanding and parsing. The execution and feedback module calls the corresponding system module according to the identified user intentions and needs. The analysis and iteration module iteratively optimizes the system's speech recognition, natural language understanding and other algorithms according to the analysis results to improve the accuracy and adaptability of the system.

Images