A voice and video agent assistance prompt method based on large model

Abstract

The present invention relates to the field of voice and video agent assistance technology, and in particular to a method for voice and video agent assistance prompting based on a large model, which includes collecting user conversation data to construct a multimodal feature matrix, generating and analyzing context vectors, generating personalized prompt content and optimizing output priority, delivering the prompt content to the agent end, and providing feedback to update the model. This application improves the context parsing capability in complex scenarios through multimodal data processing and context understanding modules, improves prompt accuracy and flexibility through personalized service generation and real-time optimization modules, and uses feedback optimization modules to achieve continuous system improvement, thereby significantly improving the response speed, prompt accuracy, and personalized service level in voice and video agent scenarios, and improving user experience.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and customer service technology, and specifically provides a voice and video agent auxiliary prompt method based on a large model. Background Art

[0002] Intelligent interaction technology refers to the use of artificial intelligence, natural language processing, machine learning, and other related technologies to enhance the interaction between people and computer systems. When intelligent interaction technology is applied to voice and video agent-assisted prompting methods, it can improve customer service efficiency and optimize user experience.

[0003] The existing technology has the following deficiencies:

[0004] Currently, most agent assistance systems still rely on static rule bases or template-driven prompt mechanisms. They lack the ability to jointly model multimodal interaction features (such as voice intonation, facial expressions, text semantics, etc.) and real-time context perception. These systems are unable to dynamically adapt to complex and changing user intentions, and it is difficult to balance prompt quality and timeliness, which affects customer service efficiency and user experience. Therefore, a voice and video agent assistance prompt method based on a large model is proposed.

[0005] The above information disclosed in this Background section is only for enhancement of understanding of the background of the present disclosure and therefore it may contain information that does not form the prior art that is already known to a person of ordinary skill in the art. Summary of the Invention

[0006] To overcome the aforementioned shortcomings of the existing technology, the present invention provides a large-scale model-based voice and video agent assisted prompting method. By introducing a multimodal data processing module, a context understanding module, and a personalized service generation module, combined with a real-time optimization mechanism, this method addresses the existing technology's issues of slow response speed, insufficient prompt accuracy, and limited personalized service capabilities in complex scenarios. This method enables more efficient assisted prompting in voice and video agent scenarios, improving the user experience.

[0007] To achieve the above object, the present invention provides the following technical solutions:

[0008] A method for assisting voice and video agent prompts based on a large model includes the following steps:

[0009] The following steps are involved:

[0010] Step S1: Collect user session data and construct a multimodal feature matrix, setting a time window to capture dynamic interaction information;

[0011] Step S2: Generate a context vector using the multimodal feature matrix and perform in-depth analysis of the current conversation in combination with the context understanding module;

[0012] Step S3: Generate personalized prompt content based on the context vector and adjust the output priority through the real-time optimization module;

[0013] Step S4: The generated prompt content is transmitted to the agent end, and the prompt generation model is updated according to the feedback information.

[0014] In a preferred embodiment, in step S1, the user session data includes voice signals, video signals and text input;

[0015] The speech signal is converted into a spectral feature map through the acoustic feature extraction module;

[0016] The video signal is used to extract key point locations and emotional features through the facial expression recognition module;

[0017] The text input is converted into a sequence of word vectors through the natural language processing module;

[0018] The voice signal, video signal and text input are stored in independent feature buffers respectively and merged into a multimodal feature matrix through a feature fusion algorithm. The feature fusion algorithm adopts a weighted average method, and the weight is determined by the feature contribution in the historical data.

[0019] In a preferred embodiment, in step S1, the time window is set by a sliding window mechanism, the length of the sliding window is a fixed value, and the step size is dynamically adjusted according to the real-time load;

[0020] When the system detects high concurrent requests, it shortens the step size of the sliding window to reduce computing resource consumption;

[0021] When the system load is low, the step size of the sliding window is extended to improve the feature capture accuracy, and the data in the sliding window is converted into a time dimension feature vector through a time series encoder.

[0022] In a preferred embodiment, in step S2, the multimodal feature matrix is input into the context understanding module, which includes a three-layer neural network structure:

[0023] The input layer receives the multimodal feature matrix and normalizes it before passing it to the attention layer. The attention layer calculates the importance weight of each feature through the self-attention mechanism.

[0024] The formula is ;

[0025] in, 、 and are the query matrix, key matrix, and value matrix of the input features respectively, is the dimension of the key matrix, and the output layer multiplies the attention weights by the input features to generate a context vector.

[0026] In a preferred embodiment, in step S2, after the context vector is generated, the current conversation is deeply analyzed by a context analysis algorithm, and the context analysis algorithm adopts a hierarchical analysis method;

[0027] First, the context vector is subject to topic classification to determine the topic category of the current session. Second, the topic category is refined in combination with the context information to generate fine-grained context labels. The fine-grained context labels are stored in the context label library and compared with historical session data through the context matching algorithm.

[0028] In a preferred embodiment, in step S3, the generation of personalized prompt content is realized by a prompt generation module. The prompt generation module includes two parts: a template generator and a dynamic generator. The template generator generates basic prompt content based on predefined templates in the context tag library. The dynamic generator generates customized prompt content based on the context vector and user portrait. The user portrait is generated by a historical interaction data analysis module, which includes the user's preferences, behavior patterns and emotional tendencies.

[0029] In a preferred embodiment, in step S3, the real-time optimization module adjusts the output order of the prompt content through a priority scheduling algorithm. The priority scheduling algorithm comprehensively considers the importance and urgency of the prompt content. The importance is determined by the weight of the context vector, and the urgency is determined by the interaction frequency within the time window. The priority calculation formula is P=α·W+β·F;

[0030] Among them, P is the priority of the prompt content, W is the weight of the context vector, F is the interaction frequency, and α and β are weight coefficients.

[0031] In a preferred embodiment, in step S4, the generated prompt content is transmitted to the agent end through the agent end interface. The agent end interface supports the transmission of multiple data formats, including text, images and voice. The display method of the prompt content is dynamically adjusted according to the type of agent end device. The system obtains the agent end operation records and user evaluations through the feedback collection module to update the prompt generation model.

[0032] In a preferred embodiment, in step S4, the feedback collection module obtains feedback information through a multi-channel data collection mechanism, including the agent's operation log, the user's satisfaction score, and the emotional characteristics in the voice and video signals;

[0033] The feedback information is stored in the feedback database and the prompt generation model is updated through the incremental learning algorithm. The incremental learning algorithm adopts the small batch gradient descent method, and only the latest feedback data is used in each update.

[0034] A voice and video agent assistance prompt system based on a large model includes a data acquisition module, a context understanding module, a prompt generation module and a feedback optimization module;

[0035] The data acquisition module is used to collect user session data and construct a multimodal feature matrix, which is then passed to the context understanding module.

[0036] The context understanding module is used to generate context vectors and perform in-depth analysis of the current conversation. The analysis results are then passed to the prompt generation module.

[0037] The prompt generation module is used to generate personalized prompt content and adjust the output priority through the real-time optimization module, and finally deliver the prompt content to the seat end;

[0038] The feedback optimization module is used to update the prompt generation model by collecting feedback information, forming a closed-loop optimization mechanism.

[0039] Technical effects and advantages of the present invention:

[0040] The present invention solves the problem of insufficient context understanding ability of existing technologies in complex scenarios by introducing a multimodal data processing module and a context understanding module; improves the flexibility and accuracy of prompt content through a personalized service generation module and a real-time optimization module; and achieves continuous improvement of the system through a feedback optimization module, meeting the actual needs of voice and video agent scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 The figure is a flow chart of a method for assisting voice and video agents based on a large model according to the present invention.

[0042] Figure 2 This is a module structure diagram of a voice and video agent auxiliary prompt system based on a large model of the present invention.

[0043] Figure 3 This is a schematic diagram of a sliding window mechanism in a large-model-based voice and video agent assistance prompting method of the present invention. DETAILED DESCRIPTION

[0044] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0045] The present invention provides a method and system for assisting voice and video agent prompts based on a large model, and its specific implementation is as follows: Figures 1 to 3As shown; the specific implementation of the present invention will be described in detail below with reference to the accompanying drawings to ensure that technicians in the relevant technical field can implement the technology according to the contents of the specification.

[0046] like Figure 1 As shown, the overall process of the present invention includes four main links: data collection, context understanding, prompt generation and feedback optimization, which are respectively completed by the data collection module, context understanding module, prompt generation module and feedback optimization module.

[0047] These modules are interconnected through data streams to form a closed-loop system, thus realizing the complete process from collecting user session data to generating prompt content and model optimization.

[0048] In step S1, the data acquisition module is responsible for collecting user session data and constructing a multimodal feature matrix.

[0049] User conversation data consists of three types: voice signals, video signals, and text input. Voice signals are converted into spectral feature maps through the acoustic feature extraction module. Video signals are extracted for key point locations and emotional features through the facial expression recognition module. Text input is converted into word vector sequences through the natural language processing module.

[0050] The above three types of features are stored in independent feature buffers respectively and merged into a multimodal feature matrix through a feature fusion algorithm.

[0051] The feature fusion algorithm uses a weighted average method, where the weights are determined by the contribution of features in the historical data. The time window is set using a sliding window mechanism.

[0052] like Figure 3 As shown, the length of the sliding window is fixed, and the step size can be dynamically adjusted according to the real-time load. When the system detects high concurrent requests, the step size of the sliding window is shortened to reduce computing resource consumption;

[0053] When the system load is low, the step size of the sliding window is extended to improve feature capture accuracy.

[0054] The data in the sliding window is converted into a time dimension feature vector through a time series encoder for subsequent context vector generation.

[0055] The data acquisition module passes the generated multimodal feature matrix to the context understanding module.

[0056] Realize collaborative modeling of multimodal data: Collect three types of conversation data: user voice signals, video signals and text input, and convert them into spectral feature maps, emotional key point features and word vector sequences through modules such as acoustic feature extraction, facial expression recognition and natural language processing, so as to construct a multimodal feature set with speech, vision and semantic dimensions, and improve the input data dimension and information density of downstream context understanding.

[0057] Enhance the robustness and dynamic adaptability of feature fusion: A weighted average fusion strategy is adopted, and the weight coefficient is dynamically adjusted based on the feature contribution of historical data statistics. This enables the multimodal feature fusion process to have a semantic association reinforcement mechanism, which can achieve the optimal combination of feature information in different user scenarios and improve the discriminative performance of the fused feature matrix.

[0058] Improve the system's real-time response and resource regulation capabilities: The fused sliding window mechanism processes multimodal data within a time segment in batches. The window length is fixed and the step size is adjustable. The system automatically shortens the step size to reduce resource consumption under high load and extends the step size to increase data granularity and analysis accuracy under low load, taking into account both system stability and feature capture integrity.

[0059] Support for time series modeling capabilities: The time series encoder performs sequence modeling on the data in the sliding window to generate a time feature vector with time dimension dependency, providing stable and time-consistent data input for subsequent context vector generation and user intent recognition, thereby enhancing the continuity and context consistency of context perception.

[0060] In step S2, the context understanding module receives the multimodal feature matrix from the data acquisition module and processes it through a three-layer neural network structure.

[0061] The input layer receives the multimodal feature matrix, normalizes it, and passes it to the attention layer; the attention layer calculates the importance weight of each feature through the self-attention mechanism;

[0062] The formula is ;

[0063] in, 、 and are the query matrix, key matrix, and value matrix of the input features respectively, The dimension of the key matrix is , and the output layer multiplies the attention weights by the input features to generate a context vector, which is used to represent the global context information of the current session.

[0064] The generated context vector is then deeply analyzed by the context parsing algorithm. The context parsing algorithm adopts a hierarchical analysis method. First, the context vector is classified into topics to determine the topic category of the current conversation. Secondly, the topic category is refined based on the context information to generate fine-grained context labels.

[0065] Fine-grained context tags are stored in a context tag library and compared with historical session data through a context matching algorithm to improve the accuracy of context parsing.

[0066] The context understanding module passes the generated context vector and fine-grained context label to the prompt generation module.

[0067] Strengthening the semantic extraction capability of multimodal features: A three-layer neural network structure is used to process the multimodal feature matrix from the data acquisition module. The input layer implements standardization operations to ensure the consistency of features in numerical scale. The middle layer introduces a self-attention mechanism to calculate the importance weight of each modal feature in the current conversation context, effectively highlighting the feature dimensions that have a key contribution to the global semantics, and improving the accuracy and robustness of the overall semantic extraction.

[0068] The self-attention mechanism is introduced to improve the accuracy of context modeling: by constructing the query matrix, key matrix, and value matrix, and performing feature weighting according to the following attention mechanism formula.

[0069] Achieve hierarchical analysis of context vectors: The context vector generated by the output layer not only contains multimodal fusion information, but also has global perception capabilities in the context dimension; the vector is hierarchically processed through the context analysis algorithm, first achieving topic-level context classification, and then combining historical context to refine it into more recognizable fine-grained context labels, effectively improving the system's semantic understanding capabilities in multi-turn and multi-topic alternation scenarios.

[0070] Enhance the dynamic adaptability and evolution of contextual tags: By building a contextual tag library and combining it with a context matching algorithm to compare and analyze historical session data, we can automatically compare and dynamically integrate the current context with the existing tag system, thereby improving the intelligence level of tag updates and adaptation, and further optimizing the response logic of subsequent prompt generation modules.

[0071] Improve the quality of intelligent interaction of the entire system: This step not only enhances the context understanding module's ability to comprehensively interpret multimodal data, but also provides the prompt generation module with deep context information with clear structure, precise semantics, and consistent context, providing a solid data foundation for achieving personalized and semantically consistent system responses.

[0072] In step S3, the prompt generation module generates personalized prompt content based on the context vector and fine-grained context labels. The prompt generation module includes two parts: a template generator and a dynamic generator.

[0073] The template generator generates basic prompt content based on predefined templates in the context tag library, while the dynamic generator combines context vectors and user portraits to generate customized prompt content.

[0074] User portraits are generated through the historical interaction data analysis module and contain information such as user preferences, behavior patterns, and emotional tendencies.

[0075] The generated prompt content is prioritized by the real-time optimization module. This module uses a priority scheduling algorithm to comprehensively consider the importance and urgency of the prompt content. Importance is determined by the weight of the context vector, while urgency is determined by the frequency of interactions within the time window.

[0076] The priority calculation formula is P = α·W + β·F;

[0077] Among them, P is the priority of the prompt content, W is the weight of the context vector, F is the interaction frequency, and α and β are weight coefficients, which can be determined through experimental tuning.

[0078] The prompt generation module passes the generated prompt content to the feedback optimization module.

[0079] Improve the personalization and targeting of prompt content: The prompt generation module integrates a template generator and a dynamic generator. The former generates efficient and standardized basic prompt content based on the structured corpus template in the context tag library, while the latter customizes content based on real-time context vectors and user portrait information, fully considering the user's historical preferences, behavioral patterns and emotional tendencies, and realizing personalized prompt generation that links context perception with user intentions, significantly improving the adaptability of system prompts and user satisfaction.

[0080] Achieve context-enhanced output driven by user portraits: Build multi-dimensional user portraits through the historical interaction data analysis module and perform correlation analysis with the current context vector to enhance the adaptability of prompt content to the current user status. Support cross-session context maintenance and content continuity modeling, and be suitable for complex application scenarios such as multi-round question and answer and multimodal interaction.

[0081] Improve the real-time performance and response efficiency of prompt scheduling: Introduce a real-time optimization module, dynamically sort the generated prompt content, and adopt a priority scheduling formula.

[0082] Taking into account both the stability of structural templates and the flexibility of semantic generation: The dual-generator architecture effectively combines the versatility of structured language templates with the flexibility of neural generation models, ensuring both the consistency and language quality of prompt content and the diversity and adaptability, thus solving the problem of severe templateization and lack of variation in prompt content in traditional systems.

[0083] Enhance the system's interactive guidance and behavioral incentive capabilities: The generated prompt content is not only output as a passive response item, but can also be actively used to guide user behavior, stimulate user participation, and correct user paths, thereby building a more proactive and guiding interactive closed loop in scenarios such as smart assistants, medical recommendations, and educational interactions.

[0084] In step S4, the feedback optimization module transmits the prompt content to the agent end through the agent end interface. The agent end interface supports the transmission of multiple data formats, including text, image and voice.

[0085] The display method of prompt content is dynamically adjusted according to the type of agent terminal device. For example, it is displayed in the form of a pop-up window on desktop devices and in the form of a notification bar on mobile devices.

[0086] At the same time, the system obtains the agent's operation records and user evaluations through the feedback collection module to update the prompt generation model.

[0087] Achieve multi-terminal adaptive distribution of prompt content: The feedback optimization module delivers prompt content to different types of agent terminals through the agent-side interface. This interface supports dynamic transmission of multiple data formats such as text, images, and voice, ensuring the compatibility and display consistency of prompt content on multiple terminals such as desktops, mobile terminals, and wearable devices, significantly improving the flexibility of system deployment and cross-platform operation capabilities.

[0088] Enhanced context-awareness of prompt presentation: The presentation format of prompt content can be dynamically adjusted based on the type of agent device. For example, desktop devices use pop-up prompts to enhance attention capture, while mobile devices use notification bar prompts to reduce interruptions. This allows the presentation format to match user habits in different interaction scenarios, improving the acceptance rate and response efficiency of prompt information.

[0089] Build a closed-loop feedback mechanism for prompt generation: The feedback collection module collects agent-side operation records and user evaluations in real time, including indicators such as click behavior, ignoring frequency, and satisfaction ratings, thereby establishing a response mapping relationship between prompt output and user behavior. This feeds back to the prompt generation model and continuously optimizes the accuracy and applicability of prompt content, driving the system's evolution from static recommendation to dynamic adaptation.

[0090] Improve the system's self-learning and dynamic evolution capabilities: Feedback information serves as an important training sample for the prompt generation model. It is embedded in the model training process through a periodic update mechanism, enabling the system to continuously adjust the content generation strategy based on real interaction data, realizing the evolution of the model from rule-driven to data-driven, and improving the intelligence and robustness of the system in long-term operation.

[0091] Support system performance evaluation and strategy iteration: The structured collection of operation records and user evaluations helps to build a system prompt effect evaluation indicator system, such as prompt response rate, prompt hit rate, user satisfaction curve, etc., to provide a quantitative basis for prompt strategy optimization, interface design improvement and model parameter fine-tuning, and support the debuggability and portability of the system in different business fields.

[0092] The feedback collection module obtains feedback information through a multi-channel data collection mechanism, including agent operation logs, user satisfaction scores, and emotional features in voice and video signals.

[0093] Feedback information is stored in a feedback database and used to update the model using an incremental learning algorithm. The incremental learning algorithm uses mini-batch gradient descent, and each update only uses the latest feedback data to reduce computational overhead.

[0094] The feedback optimization module feeds back the updated prompt generation model to the prompt generation module, forming a closed-loop optimization mechanism.

[0095] The data acquisition module, context understanding module, prompt generation module and feedback optimization module are closely connected in the form of data flow.

[0096] The data acquisition module passes the multimodal feature matrix to the context understanding module, the context understanding module passes the generated context vector and fine-grained context label to the prompt generation module, the prompt generation module passes the generated prompt content to the feedback optimization module, and the feedback optimization module feeds back the updated prompt generation model to the prompt generation module.

[0097] This collaborative relationship between modules ensures efficient operation and continuous optimization of the system.

[0098] In practical applications, the present invention can be widely used in voice and video agent scenarios, such as customer service, technical support, and telemedicine.

[0099] Taking customer service as an example, when a user interacts with an agent through voice or video, the data acquisition module collects the user's voice signals, facial expressions, and text input in real time, and generates a multimodal feature matrix through a feature fusion algorithm.

[0100] The context understanding module deeply analyzes the multimodal feature matrix to generate context vectors and fine-grained context labels.

[0101] The prompt generation module generates personalized prompt content based on the context vector and user portrait, such as recommending solutions or providing operational guidance, and adjusts the output priority through the real-time optimization module.

[0102] The feedback optimization module continuously optimizes the prompt generation model by collecting agent operation records and user evaluations, thereby improving the system's response speed and prompt accuracy.

[0103] The present invention solves the problem of insufficient context understanding ability in complex scenarios in the existing technology by introducing a multimodal data processing module and a context understanding module;

[0104] The flexibility and accuracy of prompt content are improved through the personalized service generation module and real-time optimization module;

[0105] The feedback optimization module enables continuous improvement of the system, meeting the actual needs of voice and video agent scenarios.

[0106] In order to better enable relevant personnel in this technical field to fully understand and implement the present invention, the specific implementation principle of the present invention is further supplemented below with reference to a specific application scenario.

[0107] In telemedicine scenarios, when patients interact with doctors through video seats, this system can assist doctors in providing more accurate diagnostic recommendations in real time.

[0108] First, the data acquisition module collects the patient's voice signal and video signal through the microphone and camera respectively, and simultaneously receives the text description information input by the patient.

[0109] The speech signal is converted into a spectral feature map after being processed by the acoustic feature extraction module, and the video signal is extracted through the facial expression recognition module to extract the patient's facial key point positions and emotional features, such as pain expression or anxiety state.

[0110] The text input is converted into a sequence of word vectors through the natural language processing module to capture semantic information. The three types of feature data mentioned above are stored in independent feature buffers and merged into a multimodal feature matrix through a feature fusion algorithm.

[0111] The feature fusion algorithm adopts a weighted average method, where the weights are determined by the feature contributions in the historical data, thus ensuring that the multimodal feature matrix can fully reflect the patient's current status.

[0112] Subsequently, the context understanding module receives the multimodal feature matrix from the data acquisition module and processes it through a three-layer neural network structure.

[0113] The input layer normalizes the multimodal feature matrix and passes it to the attention layer. The attention layer calculates the importance weight of each feature through the self-attention mechanism.

[0114] For example, when a patient describes their symptoms, the tone changes in the speech signal may be more representative than the text input, so its weight will be dynamically increased.

[0115] The output layer multiplies the attention weights by the input features to generate a context vector, which is used to represent the global context information of the current session.

[0116] The generated context vector is further deeply analyzed through the context parsing algorithm. First, the context vector is classified by topic, for example, determining whether the topic of the current conversation is "acute symptoms" or "chronic disease management."

[0117] Then, the topic categories are refined by combining contextual information to generate fine-grained context labels, such as “acute abdominal pain” or “hypertension follow-up”.

[0118] These fine-grained context tags are stored in a context tag library and compared with historical session data through a context matching algorithm to improve the accuracy of context parsing.

[0119] The prompt generation module generates personalized prompt content based on the context vector and fine-grained context labels.

[0120] The template generator generates basic prompt content based on predefined templates in the context tag library, such as common diagnostic suggestions for "acute abdominal pain".

[0121] The dynamic generator combines context vectors and patient portraits to generate customized prompt content.

[0122] Patient portraits are generated through the historical interactive data analysis module and include information such as the patient's medical history, medication records, and medical preferences.

[0123] For example, for a patient with a history of gastric ulcers, the system will prioritize recommending tips related to gastric examinations.

[0124] The generated prompt content adjusts the output priority through the real-time optimization module.

[0125] The real-time optimization module comprehensively considers the importance and urgency of the prompt content through a priority scheduling algorithm.

[0126] For example, when the symptoms described by the patient are more severe and the interaction frequency is higher, the priority of the prompt content will be significantly increased.

[0127] The priority calculation formula is P=α·W+β·F;

[0128] Among them, P is the priority of the prompt content, W is the weight of the context vector, F is the interaction frequency, and α and β are weight coefficients, which can be determined through experimental tuning.

[0129] The feedback optimization module transmits prompts to the doctor's device through the agent's interface. On desktop devices, prompts are displayed as pop-ups; on mobile devices, they are presented in the notification bar.

[0130] At the same time, the system obtains doctors' operation records and patients' satisfaction scores through the feedback collection module.

[0131] For example, whether the doctor adopts the system's prompts and the patient's satisfaction with the diagnostic process.

[0132] The system also analyzes patient feedback based on emotional characteristics in voice and video signals. This feedback is stored in a feedback database and updated using an incremental learning algorithm to generate prompts.

[0133] The incremental learning algorithm adopts the mini-batch gradient descent method, and only uses the latest feedback data for each update to reduce computational overhead.

[0134] For example, when the system finds that the adoption rate of a certain type of prompt content is low, it will automatically adjust the relevant weights to optimize the prompt content generation strategy.

[0135] Through the above steps, the present invention achieves efficient auxiliary prompting in telemedicine scenarios. The data acquisition module captures the patient's multimodal data in real time, the context understanding module deeply analyzes the complex context, the prompt generation module generates personalized prompt content, and the feedback optimization module continuously improves system performance through a closed-loop mechanism.

[0136] This close collaboration between modules ensures the efficient operation of the system, which not only improves the work efficiency of doctors, but also significantly improves the medical experience of patients.

[0137] The above embodiments may be implemented in whole or in part through software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments may be implemented in whole or in part in the form of a computer program product.

[0138] Those skilled in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application of the technical solution and the invention constraints. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0139] In addition, each functional module in each embodiment of the present application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0140] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

[0141] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A voice and video agent auxiliary prompt method based on a large model, characterized in that: The following steps are involved: Step S1: Collect user session data and construct a multimodal feature matrix, setting a time window to capture dynamic interaction information; Step S2: Generate a context vector using the multimodal feature matrix and perform in-depth analysis of the current conversation in combination with the context understanding module; Step S3: Generate personalized prompt content based on the context vector and adjust the output priority through the real-time optimization module; In step S3, the real-time optimization module adjusts the output order of the prompt content through a priority scheduling algorithm. The priority scheduling algorithm comprehensively considers the importance and urgency of the prompt content. The importance is determined by the weight of the context vector, and the urgency is determined by the interaction frequency within the time window. The priority calculation formula is P = α·W + β·F; Among them, P is the priority of the prompt content, W is the weight of the context vector, F is the interaction frequency, and α and β are weight coefficients; Step S4: The generated prompt content is transmitted to the agent end, and the prompt generation model is updated according to the feedback information.

2. The method for providing voice and video agent assistance prompts based on a large model according to claim 1, characterized in that: In step S1, user conversation data includes voice signals, video signals and text input; The speech signal is converted into a spectral feature map through the acoustic feature extraction module; The video signal is used to extract key point locations and emotional features through the facial expression recognition module; The text input is converted into a sequence of word vectors through the natural language processing module; The voice signal, video signal and text input are stored in independent feature buffers respectively and merged into a multimodal feature matrix through a feature fusion algorithm. The feature fusion algorithm adopts a weighted average method, and the weight is determined by the feature contribution in the historical data.

3. The method for providing voice and video agent assistance prompts based on a large model according to claim 2, characterized in that: In step S1, the time window is set using a sliding window mechanism. The length of the sliding window is fixed, and the step size is dynamically adjusted according to the real-time load. When the system detects high concurrent requests, it shortens the step size of the sliding window to reduce computing resource consumption; The current computing load factor of the system is calculated by calculating the ratio of the number of tasks processed within the current sliding window length to the maximum number of concurrently processed tasks that can be carried out. The system's currently calculated load coefficient is compared with the preset load threshold. If the system's currently calculated load coefficient is less than the preset load threshold, the step size of the sliding window is extended to improve feature capture accuracy.

4. The method for providing voice and video agent assistance prompts based on a large model according to claim 3, characterized in that: In step S2, the multimodal feature matrix is passed as input to the context understanding module, which includes a three-layer neural network structure: The input layer receives the multimodal feature matrix and normalizes it before passing it to the attention layer. The attention layer calculates the importance weight of each feature through the self-attention mechanism. The formula is ; in, 、 and are the query matrix, key matrix, and value matrix of the input features respectively, is the dimension of the key matrix, and the output layer multiplies the attention weights by the input features to generate a context vector.

5. The method for providing voice and video agent assistance prompts based on a large model according to claim 4, characterized in that: In step S2, after the context vector is generated, the current conversation is deeply analyzed using a context analysis algorithm that uses a hierarchical analysis method. First, the context vector is subject to topic classification to determine the topic category of the current session. Second, the topic category is refined in combination with the context information to generate fine-grained context labels. The fine-grained context labels are stored in the context label library and compared with historical session data through the context matching algorithm.

6. The method for providing voice and video agent assistance prompts based on a large model according to claim 5, characterized in that: In step S3, the generation of personalized prompt content is realized through the prompt generation module. The prompt generation module includes two parts: a template generator and a dynamic generator. The template generator generates basic prompt content based on the predefined templates in the context tag library. The dynamic generator combines the context vector and user portrait to generate customized prompt content. The user portrait is generated by the historical interaction data analysis module, which includes the user's preferences, behavior patterns and emotional tendencies.

7. The method for providing voice and video agent assistance prompts based on a large model according to claim 1, characterized in that: In step S4, the generated prompt content is transmitted to the agent side through the agent side interface. The agent side interface supports the transmission of multiple data formats, including text, images and voice. The display method of the prompt content is dynamically adjusted according to the type of agent side device. The system obtains the agent side operation records and user evaluations through the feedback collection module to update the prompt generation model.

8. The method for providing voice and video agent assistance prompts based on a large model according to claim 7, characterized in that: In step S4, the feedback collection module obtains feedback information through a multi-channel data collection mechanism, including the agent's operation log, the user's satisfaction score, and the emotional characteristics in the voice and video signals; The feedback information is stored in the feedback database and the prompt generation model is updated through the incremental learning algorithm. The incremental learning algorithm adopts the small batch gradient descent method, and only the latest feedback data is used in each update.

9. A large-scale model-based voice and video agent-assisted prompting system, used to implement the large-scale model-based voice and video agent-assisted prompting method according to any one of claims 1 to 8, characterized in that: It includes data collection module, context understanding module, prompt generation module and feedback optimization module; The data acquisition module is used to collect user session data and construct a multimodal feature matrix, which is then passed to the context understanding module. The context understanding module is used to generate context vectors and perform in-depth analysis of the current conversation. The analysis results are then passed to the prompt generation module. The prompt generation module is used to generate personalized prompt content and adjust the output priority through the real-time optimization module, and finally deliver the prompt content to the seat end; The feedback optimization module is used to update the prompt generation model by collecting feedback information, forming a closed-loop optimization mechanism.

Images