A digital picture-text content reading method and system supporting multi-modal real-time voice interaction

Abstract

The present application relates to artificial intelligence and digital reading technology, and provides a digital picture-text content reading method and system supporting multi-modal real-time voice interaction. By structuring digital picture-text content, a unified context dataset containing text, images and semantic labels is constructed; user voice input is collected and converted into text, and a semantic understanding engine is used to generate voice answers within the semantic range of digital picture-text, realizing "reading while asking" interaction. After voice synthesis, the answers are output in a low-delay real-time streaming, driving picture-text highlighting, character lip movements and actions, realizing voice and visual synchronous feedback. The system dynamically limits the semantic context based on the whole book or the current reading progress, improving the relevance of the answers, and reserving a video input module to collect facial expressions, lip movements and scene information, enhancing multi-modal perception. It is suitable for educational and interactive reading scenarios such as picture books and teaching materials, and has natural human-computer interaction, strong immersion and good scalability.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and digital reading technology, and in particular to an AI-assisted learning system and method that combines digital text and image content reading with multimodal real-time voice interaction. Background Technology

[0002] With the development of digital reading and intelligent voice technology, users have higher demands for interactive experiences with digital text and image content (such as picture books and e-books). Traditional digital reading relies heavily on text and image browsing and lacks real-time voice interaction. Although AI voice assistants and chat systems exist, they are mostly for independent content reading and cannot achieve an immersive experience of reading and asking questions based on the semantics and context of the entire book.

[0003] Current real-time voice interaction mostly adopts a three-stage process of "speech-to-text → language model → text-to-speech," but the interaction latency and the naturalness of the feedback are still insufficient. To improve the smoothness of interaction, some systems are trying voice-to-voice processing. In the future, combining video input to achieve richer multimodal interaction will become a trend.

[0004] Therefore, there is an urgent need for a comprehensive technical solution that integrates multimodal input with whole-book semantic understanding, dynamic context management, and multimodal output to fully enhance the voice interaction experience of digital graphic content. Summary of the Invention

[0005] This invention provides a method and system for reading digital text and image content that supports multimodal real-time voice interaction. Addressing the lack of immersive voice Q&A experiences in existing reading applications, it proposes a "read and talk" voice interaction architecture that integrates third-party voice communication and semantic understanding capabilities. Through decoupled collaboration between the client and backend, the following core functions are achieved:

[0006] 1. Full-book-level semantic constraint capability: The current system takes the full text or summary of the image and text content as input and injects it into the system prompt words of the semantic model interface through backend configuration to realize constraint-based semantic question answering based on the full-book content; in the future, it will support the construction of semantic tag trees or knowledge graphs to enhance context modeling capabilities and interaction accuracy.

[0007] 2. Dual-channel voice interaction process design: The backend supports configuration of three-stage voice interaction (speech-to-text → semantic generation → ...).

[0008] It offers two channel strategies: text-to-speech and voice-to-voice, and allows for personalized settings based on books or characters, improving feedback efficiency and naturalness.

[0009] 3. Multimodal real-time feedback linkage mechanism: The system generates real-time speech streams from the answers through speech synthesis services and registers remote audio spectrum callbacks to obtain speech energy parameters, driving feedback such as lip-sync components, character actions, text bubbles, and graphic highlighting of NPC models, to achieve an immersive interactive experience with synchronized speech and vision.

[0010] 4. Video input module reserved: The system reserves a user camera input channel, which can be expanded in the future to collect user facial expressions, lip movements and background environment information, and support multimodal enhancement functions such as video-to-voice.

[0011] 5. Module decoupling and flexible configuration: The system adopts a client-backend separation architecture. The client integrates the Agora SDK for voice acquisition and playback, and is responsible for displaying feedback and uploading parameters; the backend is responsible for semantic prompt injection, voice channel scheduling, content limitation strategies and synthesis management, and supports dynamic adaptation across content and roles and subsequent function expansion.

[0012] This invention relies on the TEN Framework to realize semantic question answering capabilities and integrates the Agora SDK to provide basic voice and audio-visual communication. Its innovation lies in designing a set of semantic limiting prompt strategies, dual-channel voice scheduling mechanisms and multimodal feedback-driven architectures for digital reading scenarios around existing interfaces. It can be widely used in voice reading scenarios such as picture books, textbooks, and interactive novels. Attached Figure Description

[0013] Figure 1 This is a schematic diagram of the overall architecture of the system of the present invention.

[0014] Figure 2 This is a flowchart of the voice interaction process of the present invention.

[0015] Figure 3 This is a structural diagram of the client module of the present invention, including multimodal feedback linkage relationships.

[0016] Figure 4 This is a schematic diagram illustrating the design goals of the extended functions of this invention. Detailed Implementation

[0017] The following detailed description of the digital text and image content reading system and method supporting multimodal real-time voice interaction, in conjunction with the accompanying drawings and specific embodiments, is not limited thereto.

[0018] 1. System Composition

[0019] This system mainly consists of two parts: a client and a backend.

[0020] Client

[0021] The client integrates the Agora SDK, responsible for capturing, encoding, and uploading user voice data, and receiving and playing the audio stream returned from the backend. Based on the spectral information of the received voice, the client drives the lip-syncing of the virtual character model, character movements, text bubbles, and highlighting of numerical and graphic content, achieving multimodal immersive feedback. The client is also responsible for the user interface display and related parameter settings.

[0022] Backend

[0023] The backend deploys the TEN framework, which handles core business logic such as semantic understanding, dialogue generation, and speech synthesis. The TEN framework supports dual-channel voice interaction processes, including the traditional three-stage process (speech-to-text → semantic generation → text-to-speech) and a voice-to-voice process. The backend can flexibly configure and switch the voice interaction channel used according to different application scenarios or user needs to achieve the optimal balance between performance and naturalness of interaction.

[0024] The backend is responsible for generating text or audio data and transmitting it to the client over the network. The actual audio data transmission and playback are handled by the Agora SDK integrated into the client. The client receives and plays the audio stream, driving multimodal visual feedback to enhance the user's interactive experience.

[0025] 2. Work Process

[0026] (1) Voice Acquisition and Upload

[0027] Users transmit voice through the microphone on the client, and Agora SDK is responsible for collecting and uploading the voice data to the backend in real time.

[0028] (2) Semantic understanding and response generation

[0029] The TEN framework deployed in the background generates intelligent responses for the current semantic context by calling the large language model interface based on preset system prompts, which include the full text or summary of the currently read book.

[0030] (3) Speech synthesis and transmission

[0031] The response text is synthesized into an audio stream by calling a text-to-speech service in the background, or the audio stream is generated directly through a voice-to-voice channel and transmitted to the client over the network.

[0032] (4) Client playback and multimodal feedback

[0033] The client receives and plays the audio stream in real time through the remote audio callback interface of Agora SDK, and drives the NPC lip-sync component, character actions, text bubbles and graphic highlighting based on the speech spectrum to achieve an immersive interactive experience.

[0034] 3. Module reservation and expansion

[0035] This system currently uses full-text or summary digital image and text content as the semantic context injection model. Future plans include semantic tag tree or knowledge graph construction capabilities to improve the accuracy of semantic understanding and the ability to dynamically adjust context. The system also reserves a user camera input channel to support future introduction of video-to-voice multimodal interaction functions such as facial expressions, lip-syncing, and environmental awareness, enriching the interaction methods.

[0036] 4. Technical Effects

[0037] This invention combines existing third-party voice communication and large language model technologies, and relies on the TEN framework to design a dual-channel voice interaction process and semantic constraint strategy, realizing an innovative interactive method of "reading and talking simultaneously" when reading digital text and image content. The system architecture has clear responsibilities and decoupled modules, possessing good maintainability and scalability. It is suitable for immersive voice interaction scenarios of various digital text and image content such as picture books, textbooks, and interactive novels, significantly improving the user interaction experience and educational value.

Claims

1. A method and system for reading digital text and image content that supports multimodal real-time voice interaction, characterized in that, include: (1) Obtain the complete content data of the target digital graphic content; (2) The digital graphic content is structured into a unified context dataset containing text information, image information and semantic tags; (3) Receive user voice input and convert it into text by calling the voice recognition service, or receive video input and generate voice text; (4) Based on real-time collected user voice input and structured digital graphic content context data, a real-time dialogue response limited to the digital graphic content and its semantic tag extension range is generated through a dialogue engine, enabling users to initiate voice interaction at any time while reading digital graphic content, and supporting a read-ask interaction mode. (5) The generated answer is converted into speech output through speech synthesis service, and speech spectrum information is obtained in real time by registering remote audio spectrum callback, normalization parameters are calculated, and multimodal feedback content is driven based on the parameters, including passing lip-sync values ​​to the NPC model lip-sync control component, triggering character actions, displaying text bubbles and highlighting interface text, so as to realize the synchronous linkage of real-time speech and visual feedback content. The method according to claim 1, characterized in that, The context data contains a semantic tag tree of digital graphic content, used to organize entities within the content and their associated information. The method according to claim 1, characterized in that, The dialogue engine restricts the generated results based on preset semantic filtering rules, allowing only responses that are relevant to the digital graphic content and its educational objectives. The method according to claim 1, characterized in that, The speech synthesis combines the character dubbing styles in the digital graphic content and drives the corresponding character animation. The method according to claim 1, characterized in that, It also includes a voice-to-speech processing module, which enables users' voice input to be directly mapped to voice responses. The method according to claim 1, characterized in that, The video input includes collecting the user's facial expressions, lip movements, and voice information to assist in speech understanding and generation. A digital text and image content reading system supporting multimodal real-time voice interaction, characterized in that, include: (1) Digital graphic content structuring module, used to construct a contextual dataset containing text, images and semantic tags; (2) Reading progress awareness module, used to collect the user's current reading position and dynamically adjust the semantic context range; (3) Speech recognition module, used to call external speech recognition services to convert user speech input into text content; (4) Semantic understanding and generation module, which is used to generate dialogue responses limited to the digital text and image content and its semantic tag extension range based on real-time speech recognition results and structured context of digital text and image content, so as to realize voice chat interaction initiated by users in real time during reading and improve the pertinence and real-time nature of the interaction. (5) Text-to-speech module, used to call external speech synthesis services to convert text responses into speech output; (6) Voice-to-voice module, used to call the service interface that supports voice-to-voice direct mode to realize direct mapping feedback between user voice input and voice output; (7) Video input module, used to collect user facial expressions, lip movements, voice information and in-application screen or scene content through terminal devices, and provide them to subsequent modules for semantic understanding and multimodal feedback control. The collection process can call the system camera, screen recording interface or third-party video processing services. (8) Multimodal feedback module, configured as follows: a) Register a remote audio spectrum callback interface to obtain real-time voice output signals; b) Calculate the normalized intensity parameters based on the spectral information; c) Use this parameter to continuously update the analysis values ​​of the NPC model lip-sync control component; d) Trigger NPCs to speak or freeze animation based on a set threshold, and simultaneously display text bubbles and interface highlights; It enables real-time synchronous feedback of voice output with lip movements, actions, and visual content.

Images