Multi-modal agent interaction method and system under agent framework

Abstract

The invention discloses a multi-modal agent interaction method and system under an agent framework, and belongs to the technical field of artificial intelligence, and the method is implemented by the following steps: multi-modal input: supporting input of multiple modals including texts, voices and images, and automatically identifying and converting into a format which can be processed by the system; mode switching: automatically switching an interaction mode according to an input mode of a user so as to ensure the naturalness and fluency of interaction; multi-modal fusion: carrying out fusion processing on the input information of different modals; efficient processing and response to multi-modal input are achieved based on an intelligent agent framework, and the intelligent agent framework dynamically adjusts an interaction strategy according to input of a user and historical interaction records. According to the method, the interaction modes can be automatically switched according to the input mode of the user, richer and more accurate feedback is provided through the multi-mode fusion technology, meanwhile, personalized adaptation is performed according to different application scenes, the requirements of different users are met, and the user experience and the interaction efficiency are improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and specifically to a multimodal intelligent agent interaction method and system under an intelligent agent framework. Background Art

[0002] With the rapid development of artificial intelligence (AI), human-computer interaction methods are becoming increasingly diverse. Traditional interaction methods primarily rely on a single modality, such as text input or voice commands, but this approach has limitations in certain scenarios. For example, users may need to interact using multiple modalities simultaneously when performing complex tasks. Furthermore, single-modal interaction methods fail to fully utilize humans' multimodal perception capabilities, resulting in low interaction efficiency. Therefore, developing an intelligent agent interaction system that supports multimodal input is of great practical significance. Summary of the Invention

[0003] The technical task of the present invention is to address the above shortcomings and provide a multimodal intelligent agent interaction method and system under an intelligent agent framework, which can automatically switch the interaction mode according to the user's input mode, and provide richer and more accurate feedback through multimodal fusion technology. At the same time, it can perform personalized adaptation according to different application scenarios to meet the needs of different users, significantly improving user experience and interaction efficiency.

[0004] The technical solution adopted by the present invention to solve its technical problem is:

[0005] A multimodal agent interaction method under an agent framework, the implementation of which includes the following steps:

[0006] S1. Multimodal input: supports input in multiple modes including text, voice, and images, and automatically recognizes and converts them into a format that the system can process.

[0007] S2, mode switching, automatically switches the interaction mode according to the user's input mode to ensure the naturalness and smoothness of the interaction;

[0008] S3, multimodal fusion, fuses input information from different modalities to provide richer and more accurate feedback;

[0009] S4. Efficiently process and respond to multimodal inputs based on an agent-based framework. The agent-based framework dynamically adjusts interaction strategies based on user input and historical interaction records, providing a more personalized interaction experience.

[0010] S5. Application scenario adaptation: Adapt according to different application scenarios (such as smart home, smart office, smart customer service, etc.) to provide personalized interactive experience.

[0011] This approach achieves efficient and natural human-computer interaction by integrating modules such as multimodal input, intelligent modal switching, multimodal fusion, an intelligent agent framework, and application scenario adaptation. It automatically switches interaction modes based on the user's input modality, provides richer and more accurate feedback through multimodal fusion technology, and offers personalized adaptation to meet the needs of diverse users based on different application scenarios. It is widely used in smart homes, smart offices, smart customer service, and other fields, effectively improving user experience and interaction efficiency.

[0012] Furthermore, the multimodal input,

[0013] Text input is entered through devices such as keyboards or touch screens, and the system converts it into a unified text format;

[0014] Voice input is collected through a microphone and converted into text information by a speech recognition unit. The speech recognition unit is based on deep learning algorithms, including recurrent neural networks (RNN) or convolutional neural networks (CNN), to achieve high-precision speech-to-text functions.

[0015] Image input is collected by the camera and processed by the image recognition unit to extract key information and convert it into a format that the system can recognize. The image recognition unit uses computer vision technology, including target detection and image segmentation, to extract key information from the image, including object location and color.

[0016] The specific implementation steps of multimodal input include:

[0017] Step S11: The user inputs text via the keyboard, and the system receives and formats the text;

[0018] Step S12: The user inputs voice through the microphone, and the system performs voice recognition and converts it into text;

[0019] Step S13: The user inputs an image through the camera, and the system performs image recognition and extracts key information.

[0020] Furthermore, the mode switching is performed by monitoring the user's input type in real time and using a pattern recognition algorithm to determine the current input mode; by detecting the characteristics of the input signal, including frequency, amplitude and other characteristics, to distinguish between voice input and text input; and by analyzing the pixel information of the input data to identify image input;

[0021] During the switching process, the interaction interface and feedback method are optimized according to the characteristics of the current mode to ensure the continuity and naturalness of the interaction.

[0022] The specific implementation steps of mode switching include:

[0023] Step S21: The system monitors the user's input type in real time;

[0024] Step S22: determining whether the input mode is voice, text or image;

[0025] Step S23: Automatically switch to the corresponding interaction mode according to the input mode.

[0026] Furthermore, the multimodal fusion uses a fusion algorithm to fuse the input information of different modalities, including feature-level fusion, decision-level fusion, etc.

[0027] Feature-level fusion extracts feature information from different modal inputs and maps them into the same feature space for fusion processing. For example, the acoustic features of a speech signal and the visual features of an image are fused to generate a comprehensive feature vector.

[0028] Decision-level fusion generates the final decision result based on the processing results of each modality through weighted averaging, voting and other methods;

[0029] The specific implementation steps of multimodal fusion include:

[0030] Step S31: collecting feature information of different modal inputs;

[0031] Step S32: Mapping the feature information to the same feature space for fusion processing;

[0032] Step S33: Generate comprehensive feedback information based on the fusion result.

[0033] Furthermore, the multimodal fusion dynamically adjusts the fusion strategy according to the user's historical interaction records and context information to further improve the fusion effect.

[0034] Furthermore, the intelligent agent framework adopts a modular design, including a task planning unit, a knowledge base management unit and a user interaction unit.

[0035] The task planning unit generates the optimal task execution path based on user input and historical interaction records to ensure the efficiency and accuracy of interaction;

[0036] The knowledge base management unit is responsible for storing and updating the system's knowledge base, including domain knowledge, user preferences and other information, to provide knowledge support for interaction;

[0037] The user interaction unit is responsible for real-time interaction with users, dynamically adjusting interaction strategies based on user feedback to provide more personalized services;

[0038] The specific implementation steps of the intelligent agent framework include:

[0039] Step S41: System initialization, loading the agent framework and related modules;

[0040] Step S42: The task planning unit generates a task execution path based on the user input and historical records;

[0041] Step S43: the knowledge base management unit provides knowledge support, and the user interaction unit interacts with the user in real time.

[0042] Furthermore, the application scenario adaptation implements personalized support for different scenarios by configuring different adaptation strategies, including:

[0043] In smart home scenarios, by interacting with the communication interface of home appliances, the user can control home appliances based on voice or image commands, such as lighting adjustment and temperature control.

[0044] In smart office scenarios, text input and voice input are combined to provide services including meeting record organization and document editing;

[0045] In intelligent customer service scenarios, text and voice input can be used to quickly identify user questions and provide accurate answers based on the knowledge base.

[0046] The specific implementation steps for application scenario adaptation include:

[0047] Step S51: Load the corresponding adaptation strategy according to the application scenario;

[0048] Step S52: In the smart home scenario, control home appliances according to user instructions;

[0049] Step S53: In the smart office scenario, provide meeting record organization and document editing services;

[0050] Step S54: In the intelligent customer service scenario, customer service is provided in combination with the knowledge base.

[0051] The present invention also claims protection for a multimodal agent interaction system under an agent framework, comprising:

[0052] Multimodal input module supports input in multiple modes including text, voice, and images, and automatically recognizes and converts them into a format that the system can process;

[0053] The modal switching module automatically switches the interaction mode according to the user's input mode to ensure the naturalness and smoothness of the interaction;

[0054] Multimodal fusion module, which fuses input information from different modalities to provide richer and more accurate feedback;

[0055] The intelligent agent framework module is used to achieve efficient processing and response to multimodal input. The intelligent agent framework dynamically adjusts the interaction strategy based on user input and historical interaction records to provide a more personalized interaction experience;

[0056] Application scenario adaptation module, used to adapt to different application scenarios (such as smart home, smart office, smart customer service, etc.) to provide personalized interactive experience;

[0057] The system specifically realizes multimodal intelligent agent interaction through the above method.

[0058] The present invention also claims protection for a multimodal agent interaction device under an agent framework, comprising: at least one memory and at least one processor;

[0059] The at least one memory is configured to store a machine-readable program;

[0060] The at least one processor is configured to call the machine-readable program to implement the above method.

[0061] The present invention also claims protection for a computer-readable medium having computer instructions stored thereon, which implement the above method when executed by a processor.

[0062] Compared with the prior art, the multimodal agent interaction method and system under the agent framework of the present invention has the following beneficial effects:

[0063] 1. Multimodal input support: By integrating multiple modal inputs (text, voice, image, etc.), the system can meet the interaction needs of different users in different scenarios, improving the flexibility and adaptability of the system.

[0064] 2. Automatic mode switching: The mode switching module can automatically switch the interaction mode according to the user's input mode, ensuring the naturalness and smoothness of the interaction and improving the user experience.

[0065] 3. Multimodal fusion technology: By fusing input information from different modalities, the system can provide richer and more accurate feedback, fully utilizing human multimodal perception capabilities and improving the accuracy and efficiency of interaction.

[0066] 3. Intelligent agent framework: Based on the intelligent agent framework, the system can dynamically adjust the interaction strategy according to the user's input and historical interaction records, provide a more personalized interaction experience, and improve the intelligence level of the system.

[0067] 5. Application scenario adaptation: The system is adapted according to different application scenarios, providing a personalized interactive experience, ensuring wide application in smart home, smart office, smart customer service and other fields, and improving the practicality and adaptability of the system.

[0068] To sum up, the present invention effectively solves the problems of single interaction mode and low interaction efficiency in the existing technology through technical means such as multimodal input, automatic modal switching, multimodal fusion, intelligent body framework and application scenario adaptation, and significantly improves user experience and interaction efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 1 is a diagram illustrating the architecture of a multimodal agent interaction method under an agent framework provided by an embodiment of the present invention;

[0070] Figure 2 is a schematic diagram of multimodal input provided by an embodiment of the present invention;

[0071] Figure 3 is a schematic diagram of mode switching provided by an embodiment of the present invention;

[0072] Figure 4 is a schematic diagram of multimodal fusion provided by an embodiment of the present invention;

[0073] Figure 5 Schematic diagram of the intelligent agent framework provided by an embodiment of the present invention;

[0074] Figure 6 This is a schematic diagram of application scenario adaptation provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0075] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0076] A multimodal agent interaction method under an agent framework, the implementation of which includes the following steps:

[0077] S1. Multimodal input: supports input in multiple modes including text, voice, and images, and automatically recognizes and converts them into a format that the system can process.

[0078] S2, mode switching, automatically switches the interaction mode according to the user's input mode to ensure the naturalness and smoothness of the interaction;

[0079] S3, multimodal fusion, fuses input information from different modalities to provide richer and more accurate feedback;

[0080] S4. Efficiently process and respond to multimodal inputs based on an agent-based framework. The agent-based framework dynamically adjusts interaction strategies based on user input and historical interaction records, providing a more personalized interaction experience.

[0081] S5. Application scenario adaptation: Adapt according to different application scenarios (such as smart home, smart office, smart customer service, etc.) to provide personalized interactive experience.

[0082] This method addresses the existing issues of single interaction methods and low interaction efficiency. By integrating multiple input modalities (such as text, voice, and images), combining an intelligent agent framework with advanced algorithms, it achieves efficient and natural human-computer interaction. It can automatically switch interaction modes based on the user's input modality, provide richer and more accurate feedback through multimodal fusion technology, and provide personalized adaptation to meet the needs of different users based on different application scenarios. It is widely used in smart homes, smart offices, smart customer service, and other fields, effectively improving user experience and interaction efficiency.

[0083] The specific implementation of this method is as follows:

[0084] 1. Multimodal input.

[0085] It supports input in multiple modes such as text, voice, and images, and can automatically identify and convert them into a format that the system can process.

[0086] Text input is entered through devices such as keyboards or touch screens, and the system converts it into a unified text format.

[0087] Voice input is collected through a microphone and converted into text by the speech recognition unit. The speech recognition unit is based on deep learning algorithms such as recurrent neural networks (RNNs) or convolutional neural networks (CNNs), achieving high-precision speech-to-text conversion.

[0088] Image input is captured by a camera and processed by the image recognition unit, which extracts key information and converts it into a format recognizable by the system. The image recognition unit uses computer vision techniques such as object detection and image segmentation to extract key information from the image, including object location and color.

[0089] The specific implementation steps of multimodal input are as follows:

[0090] Step S11: The user inputs text via the keyboard, and the system receives and formats the text;

[0091] Step S12: The user inputs voice through the microphone, and the system performs voice recognition and converts it into text;

[0092] Step S13: The user inputs an image through the camera, and the system performs image recognition and extracts key information.

[0093] 2. Mode switching.

[0094] Automatically switch the interaction mode according to the user's input mode to ensure the naturalness and smoothness of the interaction.

[0095] Modal switching monitors the user's input type in real time and uses pattern recognition algorithms to determine the current input mode. For example, it can distinguish between voice input and text input by detecting characteristics such as the frequency and amplitude of the input signal, and recognize image input by analyzing the pixel information of the input data.

[0096] During the switching process, the interaction interface and feedback method are optimized according to the characteristics of the current mode to ensure the continuity and naturalness of the interaction.

[0097] The specific steps for implementing mode switching are as follows:

[0098] Step S21: The system monitors the user's input type in real time;

[0099] Step S22: determining whether the input mode is voice, text or image;

[0100] Step S23: Automatically switch to the corresponding interaction mode according to the input mode.

[0101] 3. Multimodal fusion.

[0102] Fuse input information from different modalities to provide richer and more accurate feedback.

[0103] The multimodal fusion module adopts advanced fusion algorithms, such as feature-level fusion and decision-level fusion.

[0104] Feature-level fusion extracts feature information from different modal inputs and maps it into the same feature space for fusion processing. For example, the acoustic features of a speech signal and the visual features of an image are fused to generate a comprehensive feature vector.

[0105] Decision-level fusion generates the final decision based on the processing results of each modality through methods such as weighted averaging and voting. In addition, this module can dynamically adjust the fusion strategy based on the user's historical interaction records and contextual information to further improve the fusion effect.

[0106] The specific implementation steps of multimodal fusion are as follows:

[0107] Step S31: collecting feature information of different modal inputs;

[0108] Step S32: Mapping the feature information to the same feature space for fusion processing;

[0109] Step S33: Generate comprehensive feedback information based on the fusion result.

[0110] 4. Intelligent agent framework.

[0111] Based on the intelligent agent framework, efficient processing and response to multimodal inputs can be achieved.

[0112] The intelligent agent framework adopts a modular design, mainly including a task planning unit, a knowledge base management unit and a user interaction unit.

[0113] The task planning unit generates the optimal task execution path based on user input and historical interaction records to ensure the efficiency and accuracy of the interaction.

[0114] The knowledge base management unit is responsible for storing and updating the system's knowledge base, including domain knowledge, user preferences and other information, to provide knowledge support for interaction.

[0115] The user interaction unit is responsible for real-time interaction with users, dynamically adjusting interaction strategies based on user feedback, and providing more personalized services.

[0116] The specific implementation steps of the agent framework are as follows:

[0117] Step S41: System initialization, loading the intelligent agent framework and related modules.

[0118] Step S42: The task planning unit generates a task execution path based on user input and historical records.

[0119] Step S43: the knowledge base management unit provides knowledge support, and the user interaction unit interacts with the user in real time.

[0120] 5. Adaptation to application scenarios.

[0121] Adapt according to different application scenarios (such as smart home, smart office, smart customer service, etc.) to provide personalized interactive experience.

[0122] Application scenario adaptation: by configuring different adaptation strategies, personalized support for different scenarios can be achieved.

[0123] include:

[0124] In the smart home scenario, by interacting with the communication interface of home appliances, home appliances can be controlled according to the user's voice or image commands, such as lighting adjustment and temperature control.

[0125] In smart office scenarios, text input and voice input are combined to provide services including meeting record organization and document editing.

[0126] In intelligent customer service scenarios, user questions can be quickly identified through text and voice input, and accurate answers can be provided in combination with the knowledge base.

[0127] The specific implementation steps for application scenario adaptation are as follows:

[0128] Step S51: Load the corresponding adaptation strategy according to the application scenario;

[0129] Step S52: In the smart home scenario, control home appliances according to user instructions;

[0130] Step S53: In the smart office scenario, provide meeting record organization and document editing services;

[0131] Step S54: In the intelligent customer service scenario, customer service is provided in combination with the knowledge base.

[0132] An embodiment of the present invention also provides a multimodal agent interaction system under an agent framework, which specifically realizes multimodal agent interaction through the multimodal agent interaction method under the agent framework described in the above embodiment.

[0133] The system includes:

[0134] 1. Multimodal input module supports input in multiple modes including text, voice, and image, and automatically recognizes and converts them into a format that the system can process.

[0135] Text input is entered through devices such as keyboards or touch screens, and the system converts it into a unified text format.

[0136] Voice input is collected through a microphone and converted into text by the speech recognition unit. The speech recognition unit is based on deep learning algorithms such as recurrent neural networks (RNNs) or convolutional neural networks (CNNs), achieving high-precision speech-to-text conversion.

[0137] Image input is captured by a camera and processed by the image recognition unit, which extracts key information and converts it into a format recognizable by the system. The image recognition unit uses computer vision techniques such as object detection and image segmentation to extract key information from the image, including object location and color.

[0138] The specific implementation steps of multimodal input are as follows:

[0139] Step S11: The user inputs text via the keyboard, and the system receives and formats the text;

[0140] Step S12: The user inputs voice through the microphone, and the system performs voice recognition and converts it into text;

[0141] Step S13: The user inputs an image through the camera, and the system performs image recognition and extracts key information.

[0142] 2. The modal switching module automatically switches the interaction mode according to the user's input mode to ensure the naturalness and smoothness of the interaction.

[0143] Modal switching monitors the user's input type in real time and uses pattern recognition algorithms to determine the current input mode. For example, it can distinguish between voice input and text input by detecting characteristics such as the frequency and amplitude of the input signal, and recognize image input by analyzing the pixel information of the input data.

[0144] During the switching process, the interaction interface and feedback method are optimized according to the characteristics of the current mode to ensure the continuity and naturalness of the interaction.

[0145] The specific steps for implementing mode switching are as follows:

[0146] Step S21: The system monitors the user's input type in real time;

[0147] Step S22: determining whether the input mode is voice, text or image;

[0148] Step S23: Automatically switch to the corresponding interaction mode according to the input mode.

[0149] 3. Multimodal fusion module: It fuses input information from different modalities to provide richer and more accurate feedback.

[0150] Fuse input information from different modalities to provide richer and more accurate feedback.

[0151] The multimodal fusion module adopts advanced fusion algorithms, such as feature-level fusion and decision-level fusion.

[0152] Feature-level fusion extracts feature information from different modal inputs and maps it into the same feature space for fusion processing. For example, the acoustic features of a speech signal and the visual features of an image are fused to generate a comprehensive feature vector.

[0153] Decision-level fusion generates the final decision based on the processing results of each modality through methods such as weighted averaging and voting. In addition, this module can dynamically adjust the fusion strategy based on the user's historical interaction records and contextual information to further improve the fusion effect.

[0154] The specific implementation steps of multimodal fusion are as follows:

[0155] Step S31: collecting feature information of different modal inputs;

[0156] Step S32: Mapping the feature information to the same feature space for fusion processing;

[0157] Step S33: Generate comprehensive feedback information based on the fusion result.

[0158] 4. The intelligent agent framework module is used to achieve efficient processing and response to multimodal input. The intelligent agent framework dynamically adjusts the interaction strategy based on user input and historical interaction records to provide a more personalized interaction experience.

[0159] The intelligent agent framework adopts a modular design, mainly including a task planning unit, a knowledge base management unit and a user interaction unit.

[0160] The task planning unit generates the optimal task execution path based on user input and historical interaction records to ensure the efficiency and accuracy of the interaction.

[0161] The knowledge base management unit is responsible for storing and updating the system's knowledge base, including domain knowledge, user preferences and other information, to provide knowledge support for interaction.

[0162] The user interaction unit is responsible for real-time interaction with users, dynamically adjusting interaction strategies based on user feedback, and providing more personalized services.

[0163] The specific implementation steps of the agent framework are as follows:

[0164] Step S41: System initialization, loading the intelligent agent framework and related modules.

[0165] Step S42: The task planning unit generates a task execution path based on user input and historical records.

[0166] Step S43: the knowledge base management unit provides knowledge support, and the user interaction unit interacts with the user in real time.

[0167] 5. Application scenario adaptation module, which is used to adapt according to different application scenarios (such as smart home, smart office, smart customer service, etc.) to provide personalized interactive experience.

[0168] Adapt according to different application scenarios (such as smart home, smart office, smart customer service, etc.) to provide personalized interactive experience.

[0169] Application scenario adaptation: by configuring different adaptation strategies, personalized support for different scenarios can be achieved.

[0170] include:

[0171] In the smart home scenario, the system interacts with the communication interface of home appliances and controls home appliances based on the user's voice or image commands, such as lighting adjustment and temperature control.

[0172] In smart office scenarios, the system combines text input and voice input to provide services including meeting record organization and document editing.

[0173] In the intelligent customer service scenario, the system quickly identifies user questions through text and voice input, and provides accurate answers based on the knowledge base.

[0174] The specific implementation steps for application scenario adaptation are as follows:

[0175] Step S51: The system loads the corresponding adaptation strategy according to the application scenario;

[0176] Step S52: In the smart home scenario, control home appliances according to user instructions;

[0177] Step S53: In the smart office scenario, provide meeting record organization and document editing services;

[0178] Step S54: In the intelligent customer service scenario, customer service is provided in combination with the knowledge base.

[0179] like Figure 1 As shown in the figure, it is an architecture diagram of the multimodal intelligent agent interaction system under the intelligent agent framework. The architecture diagram shows the overall architecture of the multimodal intelligent agent interaction system, including a multimodal input module, a modal switching module, a multimodal fusion module, an intelligent agent framework and an application scenario adaptation module.

[0180] Figure 2 This is a diagram of the multimodal input module. The multimodal input module supports multiple input modalities, including text, voice, and images. Users can enter text through the keyboard, voice through the microphone, or images through the camera. The system converts this input information into a unified format for subsequent processing.

[0181] Figure 3 This is a diagram of the modal switching module. The modal switching module automatically switches the interaction mode based on the user's input mode. For example, when the user inputs via voice, the system automatically switches to voice interaction mode; when the user inputs via text, the system switches to text interaction mode. This automatic switching mechanism ensures natural and smooth interaction.

[0182] Figure 4 This is a diagram of the multimodal fusion module, which fuses input from different modalities. For example, when a user inputs both voice and image, the system combines the voice content and image information to provide more accurate and rich feedback. This fusion mechanism fully leverages human multimodal perception and improves interaction efficiency.

[0183] Figure 5 This is a diagram of the agent framework, which enables efficient processing and response to multimodal input. The agent framework dynamically adjusts interaction strategies based on user input and historical interaction records, providing a more personalized interactive experience.

[0184] Figure 6 This is a diagram of the application scenario adaptation module. The application scenario adaptation module adapts to different application scenarios to provide a personalized interactive experience. For example, in a smart home scenario, the system can combine voice and image input to achieve intelligent control of home appliances; in an intelligent customer service scenario, the system can combine text and voice input to provide more accurate customer service.

[0185] An embodiment of the present invention further provides a multimodal agent interaction device under an agent framework, comprising: at least one memory and at least one processor;

[0186] The at least one memory is configured to store a machine-readable program;

[0187] The at least one processor is used to call the machine-readable program to implement the multimodal agent interaction method under the agent framework described in the above embodiment.

[0188] An embodiment of the present invention further provides a computer-readable medium having computer instructions stored thereon. When executed by a processor, the computer instructions implement the multimodal agent interaction method under the agent framework described in the above embodiments. Specifically, a system or device equipped with a storage medium can be provided, on which software program code implementing the functions of any of the above embodiments is stored, and a computer (or CPU or MPU) of the system or device can be caused to read and execute the program code stored in the storage medium.

[0189] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute part of the present invention.

[0190] Examples of storage media for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, the program code can be downloaded from a server computer via a communication network.

[0191] In addition, it should be clear that the functions of any of the above embodiments can be achieved not only by executing the program code read by the computer, but also by enabling the operating system operating on the computer to complete part or all of the actual operations based on the instructions of the program code.

[0192] In addition, it can be understood that the program code read from the storage medium is written into the memory provided in the expansion board inserted into the computer or into the memory provided in the expansion unit connected to the computer, and then based on the instructions of the program code, the CPU installed on the expansion board or expansion unit is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above embodiments.

[0193] The present invention has been shown and described in detail above through the accompanying drawings and preferred embodiments. However, the present invention is not limited to these disclosed embodiments. Based on the above multiple embodiments, those skilled in the art can know that the code review methods in the above different embodiments can be combined to obtain more embodiments of the present invention, and these embodiments are also within the scope of protection of the present invention.

Claims

1. A multimodal agent interaction method under an agent framework, characterized in that: The implementation of this method includes the following steps: S1. Multimodal input: supports input in multiple modes including text, voice, and images, and automatically recognizes and converts them into a format that the system can process. S2, mode switching, automatically switches the interaction mode according to the user's input mode to ensure the naturalness and smoothness of the interaction; S3, multimodal fusion, fusing input information from different modalities; S4. Efficiently process and respond to multimodal inputs based on an agent-based framework. The agent-based framework dynamically adjusts interaction strategies based on user input and historical interaction records. S5. Application scenario adaptation: Adapt according to different application scenarios to provide personalized interactive experience.

2. The multimodal agent interaction method under the agent framework according to claim 1, characterized in that: The multimodal input, Text input is entered through a keyboard or touch screen device, and the system converts it into a unified text format; Voice input is collected through a microphone and converted into text information by a speech recognition unit. The speech recognition unit is based on deep learning algorithms, including recurrent neural networks (RNN) or convolutional neural networks (CNN), to achieve high-precision speech-to-text functions. Image input is collected by the camera and processed by the image recognition unit to extract key information and convert it into a format that the system can recognize. The image recognition unit uses computer vision technology, including target detection and image segmentation, to extract key information from the image, including object location and color. The specific implementation steps of multimodal input include: Step S11: The user inputs text via the keyboard, and the system receives and formats the text; Step S12: The user inputs voice through the microphone, and the system performs voice recognition and converts it into text; Step S13: The user inputs an image through the camera, and the system performs image recognition and extracts key information.

3. The multimodal agent interaction method under the agent framework according to claim 1, characterized in that: The mode switching is achieved by monitoring the user's input type in real time and using a pattern recognition algorithm to determine the current input mode; by detecting the characteristics of the input signal, including frequency and amplitude characteristics, to distinguish between voice input and text input; and by analyzing the pixel information of the input data to identify image input; During the switching process, the interactive interface and feedback method are optimized according to the characteristics of the current mode to ensure the continuity and naturalness of the interaction; The specific implementation steps of mode switching include: Step S21: The system monitors the user's input type in real time; Step S22: determining whether the input mode is voice, text or image; Step S23: Automatically switch to the corresponding interaction mode according to the input mode.

4. The multimodal agent interaction method under the agent framework according to claim 1, characterized in that: The multimodal fusion uses a fusion algorithm to fuse the input information of different modalities, including feature-level fusion and decision-level fusion; Feature-level fusion extracts feature information from different modal inputs and maps them into the same feature space for fusion processing; Decision-level fusion generates the final decision result based on the processing results of each modality through weighted average and voting methods; The specific implementation steps of multimodal fusion include: Step S31: collecting feature information of different modal inputs; Step S32: Mapping the feature information to the same feature space for fusion processing; Step S33: Generate comprehensive feedback information based on the fusion result.

5. The multimodal agent interaction method under the agent framework according to claim 4 is characterized in that: The multimodal fusion dynamically adjusts the fusion strategy according to the user's historical interaction records and context information.

6. The multimodal agent interaction method under the agent framework according to claim 1, characterized in that: The intelligent agent framework adopts a modular design, including a task planning unit, a knowledge base management unit and a user interaction unit. The task planning unit generates the optimal task execution path based on user input and historical interaction records; The knowledge base management unit is responsible for storing and updating the system's knowledge base, including domain knowledge and user preference information; The user interaction unit is responsible for real-time interaction with users, dynamically adjusting interaction strategies based on user feedback, and providing personalized services; The specific implementation steps of the agent framework include: Step S41: System initialization, loading the agent framework and related modules; Step S42: The task planning unit generates a task execution path based on the user input and historical records; Step S43: the knowledge base management unit provides knowledge support, and the user interaction unit interacts with the user in real time.

7. The multimodal agent interaction method under the agent framework according to claim 1, characterized in that: The application scenario adaptation implements personalized support for different scenarios by configuring different adaptation strategies, including: In smart home scenarios, by interacting with the communication interface of home appliances, the home appliances can be controlled according to the user's voice or image commands; In smart office scenarios, text input and voice input are combined to provide services including meeting record organization and document editing; In intelligent customer service scenarios, text and voice input can be used to quickly identify user questions and provide accurate answers based on the knowledge base. The specific implementation steps for application scenario adaptation include: Step S51: Load the corresponding adaptation strategy according to the application scenario; Step S52: In the smart home scenario, control home appliances according to user instructions; Step S53: In the smart office scenario, provide meeting record organization and document editing services; Step S54: In the intelligent customer service scenario, customer service is provided in combination with the knowledge base.

8. A multimodal agent interaction system under an agent framework, characterized in that: include: Multimodal input module supports input in multiple modes including text, voice, and images, and automatically recognizes and converts them into a format that the system can process; The modal switching module automatically switches the interaction mode according to the user's input mode to ensure the naturalness and smoothness of the interaction; Multimodal fusion module, which fuses input information from different modalities; The intelligent agent framework module is used to achieve efficient processing and response to multimodal input. The intelligent agent framework dynamically adjusts the interaction strategy based on user input and historical interaction records; Application scenario adaptation module, used to adapt according to different application scenarios and provide personalized interactive experience; The system specifically realizes multimodal intelligent agent interaction through the method described in any one of claims 1 to 7.

9. A multimodal agent interaction device under an agent framework, characterized in that: include: at least one memory and at least one processor; The at least one memory is configured to store a machine-readable program; The at least one processor is configured to call the machine-readable program to implement the method according to any one of claims 1 to 7.

10. A computer-readable medium, characterized in that The computer-readable medium stores computer instructions, which, when executed by a processor, implement the method according to any one of claims 1 to 7.

Images