Multi-mode AI interaction system and method based on mobile phone video phone

Abstract

The invention discloses a multi-mode AI interaction system and method based on a mobile phone video phone, the system comprises a communication terminal, a call center module, a multi-mode logic processing module and a dynamic content generation module, the communication terminal is connected with the call center module through an operator communication network, the call center module is used for processing incoming and outgoing calls of a large number of calls, and the multi-mode logic processing module is used for processing the dynamic content generation module. The call center module comprises an SIP (Session Initiation Protocol) gateway unit, a video interaction transmission unit and a sound interaction transmission unit, the multi-modal logic processing module is used for identifying and understanding user semantics and extracting information valuable to a current service in user videos, and the dynamic content generation module is used for generating audios and videos for interaction according to the user semantics; according to the invention, expansion is carried out by combining multi-modal logic processing and dynamic content generation based on an existing call center, so that the capability of intelligent video customer service is increased, more complex problems can be processed, and the problem that a traditional IVR system cannot interact with an AI and can only process simple services is effectively solved.

Description

Technical Field

[0001] The present invention relates to the cross - field of communication technology and artificial intelligence, and in particular to a multi - modal AI interaction system and method based on mobile video calls. Background Art

[0002] Multi - modal AI interaction in video calls relies on real - time audio - video transmission, multi - modal perception (speech, vision, text) and AI fusion technology. It reduces latency through efficient encoding and decoding and edge computing, and realizes natural interaction by combining speech recognition, computer vision (facial expression / gesture analysis) and cross - modal alignment algorithms. At the same time, it uses generative AI to create virtual avatars, real - time translation and emotion computing to enhance the immersion, and uses model lightweighting and privacy encryption to ensure terminal security. It promotes the intelligence of scenarios such as remote collaboration, education, and medical care, but faces challenges such as multi - modal synchronization and computing power optimization. In the future, it may be deeply integrated with holographic communication to reshape the human - machine collaboration experience;

[0003] When interacting through video in the prior art, it is based on the Internet and uses methods such as browsers, APPs or small programs, which requires a relatively cumbersome installation and use process. The traditional call center system only supports voice and has no intelligent processing ability, unable to realize dynamic content generation and multi - modal interaction, and the service scenarios are limited. Therefore, the present invention proposes a multi - modal AI interaction system and method based on mobile video calls to solve the problems existing in the prior art. Summary of the Invention

[0004] Aiming at the above problems, the object of the present invention is to propose a multi - modal AI interaction system and method based on mobile video calls. The multi - modal AI interaction system and method based on mobile video calls are extended by combining the existing call center with multi - modal logic processing and dynamic content generation, so that it increases the ability of intelligent video customer service, can handle more complex problems, and effectively solves the problems of the traditional IVR system that can only respond to phone buttons or simple voice instructions, cannot interact with AI, cannot understand complex semantics, cannot process video image information at the user end, and the returned content is also pre - made, can only handle simple services, and cannot have real - time interaction.

[0005] To achieve the object of the present invention, the present invention is implemented through the following technical solutions: A multi-modal AI interaction system and method based on mobile video calls, including a communication terminal, a call center module, a multi-modal logic processing module, and a dynamic content generation module. The communication terminal and the call center module are connected by an operator communication network. The call center module is used to process the incoming and outgoing calls of a large number of calls, and realize remote audio and video calls through the agent / IVP and the user. The call center module includes a SIP protocol gateway unit, a video interaction transmission unit, and a voice interaction transmission unit. The SIP protocol gateway unit is used to implement the protocol conversion between the PSTN network and the IP network, without the need for users to upgrade the terminal device. The video interaction transmission unit is used to transmit the user video and backhaul the generated digital human fusion video. The voice interaction transmission unit is used to transmit the user audio and backhaul the generated digital voice. The multi-modal logic processing module is used to recognize and understand the user semantics, and extract the information valuable to the current service in the user video. The dynamic content generation module is used to generate audio and video for interaction according to the user semantics.

[0006] A further improvement lies in that: The multi-modal logic processing module includes a speech recognition unit, a video recognition unit, and an emotion calculation unit. The speech recognition unit is used to recognize the user's voice and convert the speech into text. The video recognition unit is used to analyze the expression, gesture, and environmental information. The emotion calculation unit is used to analyze the user's emotion based on the speech spectrum and facial micro-expression, and dynamically adjust the interaction strategy.

[0007] A further improvement lies in that: The dynamic content generation module includes a speech generation unit, a digital human generation unit, and an audio-video synthesis unit. The speech generation unit is used to generate speech sounds according to the interactive result content. The digital human generation unit is used to control the digital human image, lip shape, expression, and action output. The ordinary video content generation unit is used to generate ordinary video content. The audio-video synthesis unit is used to fuse and present AI digital humans, text, images, and audio.

[0008] It includes the following steps:

[0009] Step 1, the terminal dials and answers. The user dials and answers the video call through the communication terminal and accesses the call center module;

[0010] Step 2, two-way audio and video transmission. The call center module realizes remote audio and video calls through the agent / IVP and the user. The agent / IVP end transmits the user's audio and video to the multi-modal logic processing module, and at the same time transmits the audio and video generated by the dynamic content generation module to the user's communication terminal;

[0011] Step 3: Logical analysis and processing. The multimodal logic processing module recognizes the user's voice, analyzes and understands the user's semantics, recognizes the user's video, analyzes the user's facial expressions and emotions, combines them with the user's voice for analysis, and dynamically adjusts the interaction strategy;

[0012] Step 4: Digital human audio and video generation. Generate a voice response to the question based on the logical analysis result in Step 3, and control the output of the digital human image, lip movement, and facial expression actions;

[0013] Step 5: Audio and video synthesis. Synthesize the response voice audio generated in Step 4 and the digital human image video to generate interactive content.

[0014] A further improvement lies in that: in the said Step 3, the CNN model and the capabilities of the multimodal large model are used to analyze facial expressions, gestures, and environmental information.

[0015] A further improvement lies in that: in the said Step 3, the cross-attention mechanism and the capabilities of the multimodal large model are used to integrate visual and auditory features.

[0016] A further improvement lies in that: after the digital human model is rendered on the server side in the said Step 4, the video is pushed to the mobile terminal.

[0017] The beneficial effects of the present invention are as follows: By expanding based on the existing call center combined with multimodal logic processing and dynamic content generation, the present invention increases the ability of intelligent video customer service, can handle more complex problems, effectively solves the problems of the traditional IVR system, which can only respond to phone buttons or simple voice instructions, cannot interact with AI, cannot understand complex semantics, cannot process video image information on the user side, and the returned content is also pre-made, can only handle simple services, and cannot have real-time interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is the system architecture diagram of the present invention;

[0019] Figure 2 It is the step flow chart of the present invention; DETAILED DESCRIPTION OF THE INVENTION

[0020] In order to deepen the understanding of the present invention, the present invention will be further described in detail below in conjunction with embodiments. These embodiments are only used to explain the present invention and do not constitute a limitation on the protection scope of the present invention.

[0021] Embodiment 1

[0022] According to Figure 1As shown in the figure, this embodiment provides a multimodal AI interaction system based on mobile video calls, including a communication terminal, a call center module, a multimodal logic processing module, and a dynamic content generation module. The communication terminal and the call center module are connected by an operator communication network. The call center module is used to handle the incoming and outgoing calls of a large number of phones, and realize remote audio and video calls through the agent / IVP and users. The call center module includes a SIP protocol gateway unit, a video interaction transmission unit, and a voice interaction transmission unit. The SIP protocol gateway unit is used to realize the protocol conversion between the PSTN network and the IP network, without the need for users to upgrade terminal devices, reduce deployment costs, support multi-protocol interconversion of SIP / H.323 / IMS, realize the seamless connection between traditional program-controlled switches and the IP network, and enable users to perform video AI interaction only by making and answering video calls without using an APP or a small program. The operation is relatively simple and convenient for users of different age groups to use. The video interaction transmission unit is used to transmit the user's video and backhaul the generated digital human fusion video. The voice interaction transmission unit is used to transmit the user's audio and backhaul the generated digital voice, realizing the real-time presentation of the generated audio and video to the user while facilitating data retention and improving the scalability of the system. The multimodal logic processing module is used to recognize and understand the user's semantics and extract valuable information for the current business in the user's video. The dynamic content generation module is used to generate audio and video for interaction according to the user's semantics.

[0023] The multimodal logic processing module includes a speech recognition unit, a video recognition unit, and an emotion computing unit. The speech recognition unit is used to recognize the user's voice and convert the speech into text. The video recognition unit is used to analyze the expression, gesture, and environmental information. The emotion computing unit is used to analyze the user's emotion based on the speech spectrum and facial micro-expressions, dynamically adjust the interaction strategy, and use AI to analyze speech, video, and semantics simultaneously to improve the accuracy of intention recognition, solving the problem that only voice communication can be used and only specified functions can be completed in traditional work.

[0024] The dynamic content generation module includes a speech generation unit, a digital human generation unit, a general video content generation unit, and an audio-video synthesis unit. The speech generation unit is used to generate speech sounds according to the interaction result content. The digital human generation unit is used to control the digital human image, lip shape, expression, and action output. The audio-video synthesis unit is used to fuse and present AI digital humans, text, images, audio, and video, and generate personalized audio and video in real time to replace pre-recorded content, support complex services, and break through the single interaction form of traditional voice customer service.

[0025] Embodiment 2

[0026] According to Figure 2 As shown in the figure, this embodiment provides a multimodal AI interaction method based on mobile video calls, including the following steps:

[0027] Step 1: The terminal dials and answers. The user dials and answers a video call through the communication terminal to access the call center module.

[0028] Step 2: Two-way audio and video transmission. The call center module realizes remote audio and video calls with the user through the agent / IVP. The agent / IVP end transmits the user's audio and video to the multi-modal logic processing module, and at the same time transmits the audio and video generated by the dynamic content generation module to the user's communication terminal. At the same time, it is also possible to realize one-way video transmission through server control to fully protect the user's privacy.

[0029] Step 3: Logical analysis and processing. The multi-modal logic processing module recognizes the user's speech, analyzes and understands the user's semantics, recognizes the user's video, analyzes the user's expression and emotion, merges it with the user's speech for analysis, and dynamically adjusts the interaction strategy;

[0030] When performing logical analysis, the CNN model and the capabilities of the multi-modal large model are used to analyze expressions, gestures, and environmental information, and integrate visual and auditory features.

[0031] Step 4: Digital human audio and video generation. Generate a voice response to the question according to the logical analysis result in Step 3, and control the output of the digital human image, lip shape, and expression actions;

[0032] After the digital human is rendered in real time on the server, the video is pushed to the mobile terminal.

[0033] Step 5: Audio and video synthesis. Synthesize the response voice audio and the digital human image video generated in Step 4 to generate interactive content.

[0034] The above shows and describes the basic principles, main features, and advantages of the present invention. Those skilled in the art of this industry should understand that the present invention is not limited by the above embodiments. What is described in the above embodiments and the specification only illustrates the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention will have various changes and improvements, and these changes and improvements all fall within the scope of the present invention claimed. The scope of protection claimed by the present invention is defined by the appended claims and their equivalents.

Claims

1. A multi-modal AI interaction system based on mobile video calls, characterized in that: It includes a communication terminal, a call center module, a multi-modal logic processing module, and a dynamic content generation module. The communication terminal and the call center module are connected by an operator communication network. The call center module is used to handle the incoming and outgoing calls of a large number of telephones and realize remote audio and video calls through the agent / IVP and the user. The call center module includes a SIP protocol gateway unit, a video interaction transmission unit, and a voice interaction transmission unit. The SIP protocol gateway unit is used to implement the protocol conversion between the PSTN network and the IP network without the user upgrading the terminal device. The video interaction transmission unit is used to transmit the user video and backhaul the generated digital human fusion video. The voice interaction transmission unit is used to transmit the user audio and backhaul the generated digital voice. The multi-modal logic processing module is used to recognize and understand the user semantics and extract the information valuable to the current service in the user video. The dynamic content generation module is used to generate audio and video for interaction according to the user semantics.

2. The multimodal AI interaction system based on mobile video calls according to claim 1, wherein: The multi-modal logic processing module includes a speech recognition unit, a video recognition unit, and an emotion calculation unit. The speech recognition unit is used to recognize the user's voice and convert the speech into text. The video recognition unit is used to analyze the expression, gesture, and environmental information. The emotion calculation unit is used to analyze the user's emotion based on the speech spectrum and facial micro-expression and dynamically adjust the interaction strategy.

3. A multimodal AI interaction system based on mobile video calls according to claim 1, characterized in that: The dynamic content generation module includes a speech generation unit, a digital human generation unit, a general video content generation unit, and an audio-video synthesis unit. The speech generation unit is used to generate speech sounds according to the interaction result content. The digital human generation unit is used to control the output of the digital human image, lip shape, expression, and action. The general video content generation unit is used to generate general video content. The audio-video synthesis unit is used to fuse and present the AI digital human, text, image, audio, and video.

4. A multi-modal AI interaction method based on mobile video calls according to claim 1, comprising the following steps: Step 1: Terminal dials and answers. The user dials and answers a video call through the communication terminal and accesses the call center module. Step 2: Bidirectional audio and video transmission. The call center module realizes remote audio and video calls through the agent / IVP and the user. The agent / IVP end transmits the user audio and video to the multi-modal logic processing module and simultaneously transmits the audio and video generated by the dynamic content generation module to the user communication terminal. Step 3: Logical analysis and processing. The multi-modal logic processing module recognizes the user's speech, analyzes and understands the user semantics, recognizes the user video, analyzes the user's expression, emotion, and combines it with the user's speech for analysis, and dynamically adjusts the interaction strategy. Step 4: Digital human audio and video generation. Generate a question reply speech according to the logical analysis result in Step 3 and control the output of the digital human image, lip shape, and expression action. Step 5: Audio-video synthesis. Synthesize the reply speech audio and the digital human image video generated in Step 4 to generate interaction content.

5. A multimodal AI interaction method based on mobile video calls according to claim 4, characterized in that: In Step 3, the expression, gesture, and environmental information are analyzed through the CNN model and the capabilities of the multi-modal large model itself.

6. The multimodal AI interaction method based on mobile video calls according to claim 4, wherein: In step three, the cross-attention mechanism and the capabilities of the multimodal large model are used to integrate visual and auditory features.

7. A multimodal AI interaction method based on mobile video calls according to claim 4, characterized in that: In step four, after the digital human model is rendered on the server side, the video is pushed to the mobile device.

Images