Intelligent stereoscopic book system based on multi-dimensional interactive experience and method thereof

Abstract

The application relates to the technical field of intelligent books, and discloses an intelligent three-dimensional book system and method based on multi-dimensional interactive experience, which comprises a content generation module, an emotion recognition module, an interactive control module and a multimedia fusion module.The content generation module is used for generating a dynamic three-dimensional scene of a three-dimensional book based on text content; the emotion recognition module generates an emotion analysis result; the interactive control module is used for analyzing voice instructions and gesture actions of a reader; the multimedia fusion module is used for selecting audio, video and picture resources matched with book content and embedding the resources into the three-dimensional scene; and the paper and electronic book integration module is used for converting paper book content into an electronic book through an OCR technology.Through the synergistic effect of the emotion recognition module and the interactive control module, the output of the content generation module and the multimedia fusion module is dynamically adjusted, personalized three-dimensional scene generation and multimedia resource embedding are realized, real-time response to user emotions and operation requirements is provided, an immersive and interactive reading experience is provided, and the problem that traditional books lack dynamic interaction is solved.

Description

Technical Field

[0001] This invention relates to the field of intelligent book technology, specifically to an intelligent 3D book system and method based on multi-dimensional interactive experience. Background Technology

[0002] Traditional book systems typically rely on fixed content and lack dynamic interactive features, failing to meet users' demands for diverse and immersive reading experiences. For example, static books can only provide two-dimensional images and text; even e-books only improve portability and storage capacity, still unable to effectively combine three-dimensional scenes, dynamic resources, and personalized user interaction. Furthermore, while traditional multimedia reading systems can embed video and audio resources, resource selection is often not targeted, failing to dynamically adjust content based on the user's emotional state or operational behavior, resulting in a monotonous experience and insufficient interactivity.

[0003] Especially in the fields of education and science popularization, users often require more personalized and dynamic content presentation formats in order to gain a deeper understanding of complex knowledge structures. However, current reading systems on the market cannot adjust scene content or multimedia resources in real time according to users' different emotional states or interests, making it difficult to achieve an immersive and highly interactive reading experience. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides an intelligent 3D book system and method based on multi-dimensional interactive experience, which solves the problems of traditional reading systems lacking dynamic interaction, being unable to adjust in real time according to the user's emotional state and operational needs, and lacking personalized experience.

[0005] To achieve the above objectives, the present invention provides the following technical solution: an intelligent 3D book system based on multi-dimensional interactive experience, comprising: The content generation module is used to generate dynamic three-dimensional scenes for stereoscopic books based on text content. The scenes include virtual reality (VR) scenes and augmented reality (AR) scenes. The module adjusts the scene content according to the analysis results of the emotion recognition module and the instructions of the interaction control module. The emotion recognition module is used to monitor the reader's emotional state. It collects facial expression data and heart rate data through sensors and generates emotion analysis results. The analysis results are used to adjust the scene content generated by the content generation module and the multimedia resources selected by the multimedia fusion module. The interactive control module is used to parse the reader's voice commands and gestures, and generate operation commands related to page turning, content querying and scene adjustment. These commands directly control the content generation module and the multimedia fusion module. The multimedia fusion module is used to select audio, video and image resources that match the book content based on the 3D scene generated by the content generation module and the analysis results of the emotion recognition module, and to embed the resources into the 3D scene. The integrated paper and e-book module is used to convert paper book content into e-books using OCR technology, and to encrypt and protect the converted e-books to ensure content synchronization and copyright security.

[0006] Preferably, the content generation module includes: Modeling unit, used to generate 3D scene models based on text descriptions; The graphics processing unit is used for real-time rendering of 3D scenes; VR generation unit, used to generate virtual reality scenes; AR generation unit, used for augmented reality content.

[0007] Preferably, the emotion recognition module includes: The recognition unit is used to recognize the reader's facial expressions using a convolutional neural network algorithm; The monitoring unit is used to monitor the reader's heart rate signal using photoplethysmography. The sentiment analysis unit is used to generate sentiment analysis results based on facial expression and heart rate data.

[0008] Preferably, the interactive control module includes: A speech recognition unit is used to parse the reader's voice commands; A gesture recognition unit is used to capture and analyze hand gestures; The instruction generation unit is used to integrate voice and gesture actions to generate input instructions.

[0009] Preferably, the multimedia fusion module includes: The semantic analysis unit is used to analyze the semantic information of the pop-up book content using natural language processing technology. The media selection unit is used to select audio, video, and image resources based on semantic analysis results; The content display unit is used to embed selected multimedia resources into a dynamic 3D scene.

[0010] Preferably, the integrated paper and e-book module includes: The text recognition unit is used to digitize the content of printed books using OCR technology; Blockchain encryption units are used to encrypt and protect electronic content.

[0011] Preferably, the system further includes an artificial intelligence creation assistance module, the module comprising: The inspiration generation unit is used to generate creative inspiration based on the reader's input. The vocabulary recommendation unit is used to recommend vocabulary based on the input content; The paragraph generation unit is used to generate a continuation paragraph based on the input content.

[0012] Preferably, the system combines the emotion analysis results generated by the emotion recognition module with the operation instructions generated by the interaction control module, and dynamically adjusts the outputs of the content generation module and the multimedia fusion module in the following ways: Based on the sentiment analysis results generated by the sentiment recognition module, adjust the display effects of dynamic objects in the 3D scene output by the content generation module, including object color, size, and movement; Based on the operation instructions generated by the interactive control module, control the perspective switching and interactive hotspots of the 3D scene in the content generation module; Based on the sentiment analysis results and operation instructions, the system dynamically selects audio, video, and image resources from the multimedia fusion module and adjusts the playback order and display mode of these resources.

[0013] Preferably, the system uses machine learning algorithms to dynamically adjust the scene output of the content generation module and the resource selection of the multimedia fusion module based on the reader's reading history and preference data, in order to enhance the reader's personalized reading experience.

[0014] The intelligent 3D book method based on multi-dimensional interactive experience includes the following steps: S1. Extract the text content from the book and generate a 3D scene through the content generation module; S2. Collect and analyze readers' facial expressions and heart rate data to generate sentiment analysis results; S3. Parse the reader's voice commands and gestures to generate operation instructions; S4. Dynamically adjust the 3D scene content based on the sentiment analysis results and operation instructions, and select audio, video and image resources; S5. Convert the content of printed books into e-books and encrypt the electronic content for protection.

[0015] This invention provides an intelligent 3D book system and method based on multi-dimensional interactive experience. It has the following beneficial effects: 1. This invention achieves personalized 3D scene generation and multimedia resource embedding by dynamically adjusting the output of the content generation module and multimedia fusion module through the synergistic effect of the emotion recognition module and the interaction control module. Therefore, it can respond to the user's emotional state and operation needs in real time, providing a more immersive and user-friendly reading experience, and solving the technical problem that traditional static books cannot be dynamically interactive.

[0016] 2. This invention utilizes multiple modules working collaboratively, combining virtual reality and augmented reality technologies to generate scenes with three-dimensional dynamic effects, embedding multimedia resources such as audio, video, and images. Users can dynamically adjust the reading content and participate in interactive activities within the scene through natural voice and gestures, significantly enhancing the immersion and enjoyment of reading.

[0017] 3. This invention, through an integrated paper and e-book module, efficiently digitizes the content of printed books and uses blockchain technology to encrypt and protect the electronic content, achieving the goals of content digitization, synchronization, and security. This function effectively solves the problems of difficult conversion between paper and e-book content and insufficient copyright protection, providing security for the multi-format dissemination of books. Attached Figure Description

[0018] Figure 1 This is a system architecture diagram of the present invention; Figure 2 This is a connection block diagram of the content generation module of the present invention; Figure 3 This is a connection block diagram of the emotion recognition module of the present invention; Figure 4 This is a connection block diagram of the interactive control module of the present invention; Figure 5 This is a connection block diagram of the multimedia fusion module of the present invention; Figure 6 This is a block diagram showing the connection of the integrated paper and electronic book module of the present invention; Figure 7 This is a connection block diagram of the artificial intelligence creation assistance module of the present invention; Figure 8 This is a flowchart of the method of the present invention. Detailed Implementation

[0019] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] To better understand the present invention, the above content will be described in detail below with reference to specific embodiments. Please see the appendix Figures 1-7 This invention provides an intelligent 3D book system based on multi-dimensional interactive experience, characterized in that it includes: The content generation module is used to generate dynamic 3D scenes for 3D books based on text content. The scenes include virtual reality (VR) scenes and augmented reality (AR) scenes. The module adjusts the scene content according to the analysis results of the emotion recognition module and the instructions of the interaction control module. The emotion recognition module is used to monitor the reader's emotional state. It collects facial expression data and heart rate data through sensors, generates emotion analysis results, and uses the analysis results to adjust the scene content generated by the content generation module and the multimedia resources selected by the multimedia fusion module. The interactive control module is used to parse the reader's voice commands and gestures, and generate operation commands related to page turning, content querying and scene adjustment. The commands directly control the content generation module and the multimedia fusion module. The multimedia fusion module is used to select audio, video, and image resources that match the book content based on the 3D scene generated by the content generation module and the analysis results of the emotion recognition module, and to embed the resources into the 3D scene. The integrated paper and e-book module is used to convert paper book content into e-books using OCR technology, and to encrypt and protect the converted e-books to ensure content synchronization and copyright security.

[0021] The content generation module includes: a modeling unit for generating 3D scene models based on text descriptions; a graphics processing unit for real-time rendering of 3D scenes; a VR generation unit for generating virtual reality scenes; and an AR generation unit for augmented reality content.

[0022] The content generation module of this invention aims to efficiently transform the text content of a book into a three-dimensional dynamic scene to achieve immersive display of virtual reality (VR) and augmented reality (AR). This module consists of a modeling unit, a graphics processing unit, a VR generation unit, and an AR generation unit. The functions of each part are organically combined to collaboratively generate and interact with the three-dimensional scene. As the core module of this invention, the content generation module works in conjunction with the emotion recognition module and the multimedia fusion module to dynamically adjust the scene content and provide a multi-dimensional user interactive experience.

[0023] It should be noted that the content generation module transforms textual descriptions into visual scene models through semantic parsing and 3D modeling of the book content. By combining these models with the real-world environment through rendering, it enhances the immersiveness and interactivity of the scene. This embodiment provides a detailed description of the specific structure and technical implementation of the content generation module.

[0024] Implementation of modeling units In this embodiment, the modeling unit is used to extract semantic information from the text content of the book and generate a corresponding three-dimensional scene model.

[0025] Specifically, the modeling unit first parses the book content using Natural Language Processing (NLP) technology to identify key scene elements (such as people, environment, and objects). For example, the technical process for semantic information extraction is as follows: The input text from the book undergoes lexical and syntactic analysis to extract the main nouns and verbs from the sentences. A pre-trained language model (such as BERT or GPT) is then used to generate semantic vectors for the text.

[0026] in, For input text, The generated semantic vector; based on the semantic vector, a clustering algorithm groups content with similar semantics into descriptive units of the same scene.

[0027] Then, based on the extracted semantic information, a 3D modeling library (such as the Blender API or Unity 3D) is invoked to construct the geometric model of the scene objects. During the modeling process, vertices and edge sets are used to represent the scene structure:

[0028] Here, is the set of vertices, and is the set of edges. A polygonal mesh is generated by connecting the 3D coordinates of the vertices with the edges. In one possible implementation, the modeling unit also supports dynamically adjusting the size, position, and rotation angle of scene objects, which is accomplished through the following matrix calculation:

[0029] in, This is the original vertex position matrix. The transformation matrix is... This is the transformed vertex position matrix. It should be noted that the 3D scene generated by the modeling unit includes not only static objects, but also dynamic objects (such as moving characters, flowing rivers, etc.), and the dynamic effects are achieved through animation keyframe technology.

[0030] Implementation of the graphics processing unit In this embodiment, the graphics processing unit is used to render the 3D scene generated by the modeling unit in real time for subsequent display and interaction.

[0031] Specifically, the graphics processing unit (GPU) renders the 3D scene using GPU hardware acceleration and ray tracing algorithms. For example, the ray tracing rendering process is as follows: Determine the viewpoint position and calculate the interaction between light and the scene based on the material properties of the scene objects; The Phong lighting model is used to simulate scene lighting effects. The lighting calculation formula is as follows:

[0032] in, , , These are the ambient light, diffuse reflection, and specular reflection coefficients, respectively. , , These represent the intensities of ambient light, point light source, and specular light, respectively. For the direction of light, The direction of the normal. The direction of the reflected light. Direction of sight Gloss index; In one possible implementation, the graphics processing unit supports real-time optimization of rendering efficiency by removing objects outside the field of view through a hierarchical culling algorithm to reduce computational overhead.

[0033] Implementation of VR generation unit In this embodiment, the VR generation unit is used to construct a fully immersive virtual reality scene and present the three-dimensional scene to the user through a head-mounted display device (such as Oculus or HTC Vive).

[0034] Specifically, the VR generation unit uses viewpoint tracking technology to determine the user's observation point in the virtual scene and dynamically updates the scene content. The mathematical formula for viewpoint transformation is as follows:

[0035] in, The transformed viewpoint position. For rotation matrix, As a translation vector, it's worth noting that the VR generation unit also supports user interaction with virtual objects. For example, users can control objects in the scene using controllers or gestures to perform operations such as scaling and rotation.

[0036] Implementation of AR generation unit In this embodiment, the AR generation unit is used to combine a 3D scene with the real environment to display augmented reality content. Specifically, the AR generation unit captures the real scene using a camera and uses a feature point matching algorithm to determine the overlay position of the virtual content and the real environment. For example, the projection position of the virtual content is calculated as follows:

[0037] in, For the projection point on the screen, This is the intrinsic parameter matrix of the camera. and These are the rotation matrix and the translation vector, respectively. For object coordinates in a 3D scene, in one possible implementation, the AR generation unit supports real-time occlusion calculation to ensure that virtual objects can correctly integrate into the real environment. In some embodiments, the content generation module can also support user-defined 3D scenes. For example, after the user inputs additional scene description information, the modeling unit will dynamically adjust the scene structure according to the input content. It should be understood that the content generation module is the key to realizing the immersive reading experience of this invention, and the 3D scene content it generates is not limited to static books, but can also be extended to fields such as science education and historical scene reproduction.

[0038] The emotion recognition module includes: a recognition unit for recognizing the reader's facial expressions using a convolutional neural network algorithm; a monitoring unit for monitoring the reader's heart rate signal using photoplethysmography; and an emotion analysis unit for generating emotion analysis results based on facial expression and heart rate data.

[0039] The emotion recognition module of this invention generates real-time analysis results of the reader's emotional state by collecting and analyzing multimodal data (such as facial expressions and heart rate signals). The module includes a recognition unit, a monitoring unit, and an emotion analysis unit, which work together to collect, process, and analyze emotion data. The technical implementation of the module is described in detail below; Implementation of the recognition unit The recognition unit uses a convolutional neural network (CNN) algorithm to identify the reader's facial expressions. A camera is used to capture facial images in real time, and the recognition unit preprocesses the image data, extracts features, and classifies expressions.

[0040] Image preprocessing The MTCNN algorithm is used to detect facial regions in images and extract facial key points (such as the positions of eyes, nose, and corners of the mouth). The facial images are aligned through geometric transformations and normalized to a fixed size (e.g., 224×224 pixels) to fit the input of the deep learning model.

[0041] Feature extraction The input image is processed through a convolutional layer to extract feature maps:

[0042] in: Indicates the first Feature maps after convolutional operations For the first The convolutional kernel weights of the layer, This is the input feature map for the previous layer. For bias, For activation functions; Facial Expression Classification The extracted feature maps are processed by a fully connected layer to output the probability of the expression category:

[0043] in: For the input image Belongs to the emoji category The probability, For the weight vector, The feature vector of the input image, For bias terms, The total number of expression categories (such as "joy", "sadness", "anger", "surprise" etc.); through the above process, the recognition unit can accurately output the expression recognition results.

[0044] Implementation of monitoring unit The monitoring unit acquires the reader's heart rate signal through photoplethysmography (PPG) and analyzes heart rate variability (HRV) to assess emotional state.

[0045] PPG signal acquisition uses a sensor (such as a fingertip sensor or a wristband) to emit light of a specific wavelength (such as a green LED) to measure changes in blood volume under the skin; the intensity change signal of reflected light is obtained through a photodetector to form a PPG waveform.

[0046] Signal preprocessing involves filtering the acquired PPG signals to remove high-frequency noise and low-frequency interference; a peak detection algorithm is used to extract the R-peak position (heartbeat peak).

[0047] HRV calculation Heart rate variability is calculated based on the R-peak interval (RR interval) using the following formula:

[0048] in, As an indicator of heart rate variability, For the first The RR interval of one heartbeat (in seconds). This is the average of all RR intervals; This represents the total number of heartbeats. Emotional mapping The relationship between HRV and emotional state: Higher HRV values ​​typically correspond to a relaxed state, while lower HRV values ​​typically correspond to a tense state. By analyzing the physiological signals of the monitoring unit, implicit emotional data can be obtained, compensating for the shortcomings of facial expression recognition.

[0049] Implementation of the sentiment analysis unit The sentiment analysis unit integrates multimodal data from the identification and monitoring units to generate comprehensive sentiment analysis results.

[0050] Multimodal data fusion The features of facial expressions and heart rate signals are weighted and fused using an attention mechanism, as shown in the following formula:

[0051] in, Feature weights The importance score of the feature. The eigenvectors of a mode, This is the fused feature vector.

[0052] Emotional state classification The fused feature vectors are input into a multilayer perceptron (MLP) for emotion classification, and the output emotion category (such as "pleasure", "sadness", "tension" etc.) and its corresponding confidence score are given.

[0053] Real-time feedback The sentiment analysis unit sends the classification results to the content generation module and the multimedia fusion module in real time to adjust the scene content and media resources.

[0054] In some embodiments, the emotion analysis unit can be extended to incorporate speech emotion recognition technology to further verify the reader's emotional state through speech signals. For example, analyzing the pitch, speed, and timbre characteristics of the speech can enhance the accuracy of emotion classification. It should be noted that the emotion recognition module of this invention is not only applicable to intelligent 3D book systems, but can also be applied to fields such as mental health assessment and personalized education.

[0055] In this invention, the interactive control module includes: a speech recognition unit for parsing the reader's speech commands; a gesture recognition unit for capturing and analyzing gesture actions; and a command generation unit for integrating speech and gesture actions to generate input commands.

[0056] In this embodiment, the interactive control module is a crucial component of the invention, enabling natural interaction between the reader and the intelligent 3D book system. The module includes a speech recognition unit, a gesture recognition unit, and a command generation unit. By parsing speech commands and gestures, this module generates operation commands for controlling the content generation module and the multimedia fusion module. It should be noted that the interactive control module is designed with real-time performance and accuracy in mind. Its implementation combines speech processing technology, gesture recognition technology, and multimodal fusion technology, enabling efficient parsing of user intent and generation of precise commands.

[0057] Implementation of the speech recognition unit In this embodiment, the speech recognition unit is used to convert the reader's speech input into usable text instructions and extract semantic information to parse the user's intent.

[0058] The input speech signal is preprocessed by being framed and then subjected to Short-Time Fourier Transform (STFT) to extract frequency domain features, as shown in the following formula:

[0059] In the formula, Frequency domain amplitude at a frequency point This represents the sampled value of the speech signal in the time domain; This represents a window function used to weight signals to reduce edge effects. Indicates frame length, and These represent the frequency index and the frame index, respectively.

[0060] Feature extraction Mel-frequency cepstral coefficients (MFCCs) are extracted as speech features. The formula for calculating Mel frequencies is as follows:

[0061] in, Indicates the Mel frequency. This represents the linear frequency (in Hz).

[0062] Speech-to-text Speech-to-text conversion is achieved by using deep neural networks (such as RNNs or Transformers) based on connection-time classification (CTC) to map speech feature sequences to target text sequences.

[0063] Semantic understanding Natural language processing techniques (such as the BERT model) are used to perform semantic analysis on text commands and generate semantic vectors. Through the above process, the speech recognition unit can accurately parse the user's voice commands, such as "next page" or "zoom in on the image".

[0064] Implementation of the gesture recognition unit The gesture recognition unit is used to capture and analyze the reader's gestures and generate corresponding motion features.

[0065] Gesture data acquisition uses a depth camera to capture hand movements and generate time-series hand images.

[0066] Keypoint detection extracts joint keypoints from hand images, represented as follows:

[0067] in, Represents the two-dimensional coordinates of the i-th joint. This indicates the total number of joints in the hand (e.g., 21 joints).

[0068] Gesture feature extraction Hand movement features are extracted using a convolutional neural network (CNN), as shown in the following formula: Action Classification Use a recurrent neural network (RNN) or Transformer model to classify time series features and output gesture categories.

[0069] Gesture categories can include "waving to turn pages", "zooming in and out of images", "clicking to confirm", etc.

[0070] Implementation of the instruction generation unit The instruction generation unit is responsible for fusing the recognition results of voice and gestures and generating the final operation instructions.

[0071] Multimodal data fusion uses an attention mechanism to weightedly fuse speech and gesture features.

[0072] Instruction generation The fused feature vectors are input into the fully connected network to generate operation instructions. The instruction format is as follows:

[0073] in, Indicates the type of user action (e.g., page turning, zooming). Indicates the target object of the operation (such as "Next Page" or "Current Image").

[0074] In some embodiments, the interaction control module can also incorporate eye-tracking technology to further supplement voice and gesture input information by collecting the user's gaze position, thereby improving the accuracy of the interaction. It should be noted that the real-time performance and accuracy of the interaction control module make it suitable not only for intelligent 3D book systems but also for natural interaction in fields such as smart homes and virtual reality.

[0075] In this invention, the multimedia fusion module includes: a semantic analysis unit, used to analyze the semantic information of the 3D book content using natural language processing technology; a media selection unit, used to select audio, video, and image resources based on the semantic analysis results; and a content display unit, used to embed the selected multimedia resources into a dynamic 3D scene.

[0076] In this embodiment, the multimedia fusion module is used to add multimedia resources such as audio, video, and images to the dynamic 3D scene of the 3D book to enrich the user's reading experience. The module includes a semantic analysis unit, a media selection unit, and a content display unit. These units work together to select and integrate multimedia resources through natural language processing technology and semantic matching algorithms, dynamically optimizing the display effect.

[0077] It should be noted that the multimedia fusion module is closely integrated with the content generation module and the emotion recognition module, and achieves personalized and multi-dimensional content presentation by adjusting the resource content and display method in real time.

[0078] Implementation of semantic analysis units The semantic analysis unit is responsible for parsing the text content of the pop-up book, extracting semantic information, and generating semantic vectors for selecting multimedia resources.

[0079] Semantic information extraction: The text content of the input 3D book is used to generate semantic vectors using a pre-trained language model (such as BERT or GPT).

[0080] in, The generated semantic vector; Enter the text content.

[0081] Semantic relation modeling Syntactic parsing algorithms are used to extract the subject-verb-object structure and keywords from the text, establishing a semantic relationship network between the content. For example, when extracting the description of the "solar system" in the book, keywords such as "planet," "orbit," and "sun" are extracted to construct semantic associations.

[0082] Multi-layer semantic representation The generated semantic vectors are hierarchically clustered to extract scene-related contextual information. For example, "Earth's rotation" is categorized under the theme of "celestial motion".

[0083] Implementation of the media selection unit The media selection unit selects audio, video, and image resources that match the book content by calculating the similarity between the text semantic vector and the multimedia resource semantic vector.

[0084] Multimedia resource semantic vector generation: Extract metadata descriptions for each multimedia resource (audio, video, image) and generate corresponding semantic vectors.

[0085] in, For the semantic vector of the resource; Metadata descriptions for resources (such as "galaxy distribution map of the Milky Way").

[0086] Semantic similarity calculation, which generates semantic vectors. and resource semantic vectors Similarity:

[0087] in, Indicates the semantic similarity between text and resources; and These are the norms of the semantic vectors.

[0088] Resource sorting and filtering Multimedia resources are sorted from highest to lowest similarity, and the most relevant resources are selected for subsequent display. For example, if the resource with the highest similarity is a video, that video resource will be loaded first.

[0089] Implementation method of content display unit The content display unit embeds selected multimedia resources into the dynamic 3D scene of the pop-up book and dynamically adjusts the display effect of the resources based on the analysis results of the emotion recognition module.

[0090] Resource embedding Bind multimedia resources (audio, video, images) to locations and objects in a 3D scene. For example, embed a video describing the Earth's rotation next to an Earth model.

[0091] Dynamic display adjustment Adjust the display attributes of the resources (such as playback speed, volume, and brightness) based on the sentiment analysis results: The formula for adjusting audio playback volume is:

[0092] in, The adjusted volume; This is the initial volume; For adjustment coefficients; It is an indicator of emotional intensity.

[0093] The formula for adjusting video playback speed is:

[0094] in, The adjusted playback speed; This is the initial playback speed; This is the speed adjustment coefficient; It is an indicator of emotional intensity.

[0095] Real-time response interaction The content display unit supports interactive operations with users. For example, after a user selects a multimedia resource using gestures, the system can enlarge the display window of that resource.

[0096] In some embodiments, the multimedia fusion module can be extended to support dynamically generated multimedia resources. For example, it can generate images or audio in real time that fit the current scene using artificial intelligence algorithms, thereby enhancing the system's flexibility and versatility. It is important to understand that the multimedia fusion module is key to achieving an immersive reading experience; its functionality is not only applicable to 3D book systems but can also be applied to fields such as virtual reality teaching and cultural heritage display.

[0097] In this invention, the integrated paper and e-book module includes: a text recognition unit for digitizing the content of paper books using OCR technology; and a blockchain encryption unit for encrypting and protecting the electronic content.

[0098] In this embodiment, the integrated paper and e-book module aims to achieve rapid digitization of paper book content and securely protect the copyright of the digitized content. The module includes a text recognition unit and a blockchain encryption unit; these units work collaboratively to ensure the integrity, accuracy, and copyright security of the digitized content.

[0099] It should be noted that this module not only supports the recognition of static text content, but also digitizes content containing images, tables, and special characters, and effectively prevents the tampering and copyright infringement of digital content during the dissemination process through blockchain technology.

[0100] Implementation of the character recognition unit The text recognition unit is used to convert the content of printed books into editable electronic text using optical character recognition (OCR) technology.

[0101] Image preprocessing Collect scanned images of printed books and preprocess them, including: Grayscale conversion: Converting a color image to a grayscale image, the formula is as follows:

[0102] in, , , These represent the red, green, and blue components of the image, respectively. These are the pixel values ​​of the grayscale image.

[0103] Binarization: The grayscale image is binarized by calculating a global threshold using the Otsu algorithm.

[0104] in, These are the pixel values ​​of the binary image; The threshold value is calculated.

[0105] Text region detection Use convolutional neural networks (such as EAST or CRNN) to locate text regions in an image and generate a set of bounding boxes.

[0106] Text recognition Character recognition is performed on the image within each bounding box using a neural network model based on Connection-Temporal Classification (CTC):

[0107] in, For the input image Convert to character sequence The probability, For time step; For the first Output characters at each time step.

[0108] Formatting Preserved For content containing tables and special symbols, additional detection models are used to identify formatting information, preserving the original layout structure of the electronic text.

[0109] Implementation of blockchain encryption units Blockchain encryption units are used to encrypt and protect electronic content, ensuring its integrity and copyright security.

[0110] Content hash generation generates a unique hash value for electronic text content, used to identify and verify the integrity of the content. The hash value calculation formula is as follows:

[0111] in, The generated hash value; For the input electronic content; It is a safe hash algorithm.

[0112] Blockchain records The generated hash value and copyright information are recorded in the blockchain. The block structure includes the following fields: Block header: Contains information such as the hash value and timestamp of the previous block; The block is the hash value of the current content (the hash value of the text content). And copyright information (such as author, date, etc.).

[0113] Content verification Each time digitized content is read, its hash value is recalculated and compared with the hash value stored in the blockchain:

[0114] in, The hash value of the current content. The original hash value stored in the blockchain. To verify the results.

[0115] Copyright Tracking Each modification and transfer of content generates a new blockchain record, forming a content transfer path to track its dissemination process.

[0116] In some embodiments, the text recognition unit also supports multilingual recognition and the parsing of mathematical formulas. For example, by training a specialized OCR model, academic documents containing mathematical expressions can be recognized.

[0117] Within the blockchain encryption unit, smart contract functionality can be further introduced for automated content authorization and distribution. It's important to understand that this module's functionality is not only applicable to intelligent 3D book systems but can also be widely applied to fields such as archival digitization and copyright protection.

[0118] In this invention, the system also includes an artificial intelligence creative assistance module, which includes: an inspiration generation unit for generating creative inspiration based on the reader's input; a vocabulary recommendation unit for recommending vocabulary based on the input; and a paragraph generation unit for generating continuation paragraphs based on the input.

[0119] In this embodiment, the AI-powered creative assistance module enhances the user's creative experience by providing inspiration generation, vocabulary recommendation, and paragraph generation functions, helping users efficiently create derivative works related to the pop-up book content. The module includes an inspiration generation unit, a vocabulary recommendation unit, and a paragraph generation unit. These units work collaboratively, comprehensively utilizing natural language processing and deep learning technologies to achieve intelligent creative assistance functions.

[0120] It should be noted that this module can dynamically generate creative suggestions based on user input, supports various text types (such as stories, poems, and reviews), and can be linked with the content generation module and multimedia integration module to expand the application scenarios of pop-up books.

[0121] Implementation of the Inspiration Generation Unit The inspiration generation unit is an important component of the AI-assisted creative module. It is responsible for analyzing the text content input by the user and generating relevant creative inspiration prompts through deep semantic analysis, providing users with a wealth of creative directions.

[0122] Semantic information extraction The inspiration generation unit first extracts semantic information from the text content input by the user. Specifically, the system receives text fragments (such as a topic, paragraph, or keyword) provided by the user and performs deep semantic analysis on the text using a pre-trained natural language processing model (such as BERT or GPT).

[0123] During semantic analysis, the system converts the input text into a high-dimensional semantic representation to capture the implicit meaning and contextual information within the text. For example, when a user inputs "space exploration," the system can identify keywords such as "space" and "exploration" and infer the core semantics of the text's topic based on the context.

[0124] The process of semantic information extraction also includes syntactic structure analysis of user text, such as decomposing subject-verb-object structures and modification relationships, so as to more accurately understand the semantic core of user input.

[0125] Theme extension After extracting semantic information, the inspiration generation unit filters relevant topics from a predefined topic database based on semantic similarity. This predefined topic database contains common topics from various creative fields, such as astronomy, natural sciences, and literature and art.

[0126] The system filters out the most relevant topics by matching the semantic representation of the input text with the semantic representation of topics in the database. This process comprehensively considers the semantic weight and contextual relevance of the input text. For example, when inputting "space exploration," the system can expand to topics such as "black hole formation," "galaxy evolution," and "extraterrestrial life." These topics are highly semantically relevant to the input content and can inspire users to engage in deeper creative work.

[0127] It should be noted that the inspiration generation unit is not limited to static topic expansion; it can also be dynamically adjusted based on the user's emotional state. For example, when the sentiment analysis module detects that the user's emotions lean towards excitement or exploration, the system may prioritize recommending more forward-looking topics, such as "the possibility of extraterrestrial civilizations" or "the hypothesis of wormhole travel technology."

[0128] Inspiration output When generating creative inspiration prompts, the system presents relevant topics in a concise and intuitive way. For example, the system can display expanded topics as a list of keywords or provide users with more specific creative directions through phrase descriptions.

[0129] For example, given the input "space exploration", the inspiration generation unit might output the following creative inspirations: "The Formation Process of Black Holes": Further discussion can be made on the physical properties of black holes, their formation conditions, and their impact on galaxies.

[0130] "The Origin and Evolution of Galaxies": Exploring the evolutionary history of galaxies from the Big Bang to the present day.

[0131] "The Possibility of Extraterrestrial Life": Discussing the possibility of extraterrestrial civilizations in conjunction with the theory of habitable zones.

[0132] In addition, the inspiration generation unit supports iterative optimization based on user feedback. If a user is not satisfied with some of the generated inspiration suggestions, they can choose to mark or filter them, and the system will further adjust the recommendation results based on the user's choices.

[0133] Extended Explanation The inspiration generation unit can be applied to various scenarios, not only for text creation support but also for knowledge expansion in the field of education. For example, in history teaching, when a user inputs "Industrial Revolution," the system can expand to topics such as "The Impact of the Industrial Revolution on Social Structure" and "The Invention and Significance of the Steam Engine," helping users to comprehensively understand the relevant knowledge.

[0134] It should be noted that the inspiration generation unit takes into account semantic relevance, contextual consistency, and user creative intent when generating prompts, ensuring that the output content has practical inspirational significance for users.

[0135] Implementation of the vocabulary recommendation unit The vocabulary recommendation unit generates a list of high-frequency or high-quality vocabulary recommendations that match the semantics by analyzing the context of the user's input.

[0136] Context Analysis Perform lexical and syntactic analysis on user-input text to extract contextual keywords and phrases.

[0137] Vocabulary scoring is performed using a probability-based language model to calculate the conditional probability of candidate words.

[0138] in, For vocabulary In context The conditional probability of its occurrence; For context and vocabulary The joint probability; Candidate keyword recommendations The recommended vocabulary list is generated by sorting words by conditional probability from high to low. For example, when a user enters "solar system", the recommended words might include "planet", "orbit", and "star".

[0139] Implementation of paragraph generation unit The paragraph generation unit uses a deep learning model to automatically generate relevant continuation paragraphs based on user input.

[0140] Text generation model Paragraph generation is performed using a pre-trained text generation model (such as GPT). The model predicts the next possible text sequence based on user input.

[0141] in, To generate paragraphs Enter text Probability under given conditions; To generate the first paragraph One word; This represents the total number of words in the generated paragraph.

[0142] Generation control Set the length of the generated paragraph The relevance threshold and semantic consistency parameters ensure that the output content matches the topic entered by the user.

[0143] In some embodiments, the AI-assisted creation module can also be extended to support multimedia generation functions, such as providing users with image or audio resources that match the text content. It should be noted that the functionality of this module is not limited to supporting the creation of 3D book systems; it can also be applied to fields such as education and cultural creation, and has broad practical value.

[0144] In this invention, the system combines the emotion analysis results generated by the emotion recognition module with the operation instructions generated by the interaction control module, and dynamically adjusts the outputs of the content generation module and the multimedia fusion module in the following ways: Based on the sentiment analysis results generated by the sentiment recognition module, adjust the display effects of dynamic objects in the 3D scene output by the content generation module, including object color, size, and movement; Based on the operation instructions generated by the interactive control module, control the perspective switching and interactive hotspots of the 3D scene in the content generation module; Based on the sentiment analysis results and operation instructions, the system dynamically selects audio, video, and image resources from the multimedia fusion module and adjusts the playback order and display mode of the resources.

[0145] In this embodiment, the system of the present invention dynamically adjusts the content generation module and the multimedia fusion module by using the emotion analysis results generated by the emotion recognition module and the operation commands generated by the interaction control module. This adjustment mechanism aims to provide a personalized and immersive interactive experience, ensuring that the system can optimize the content presentation in real time according to the user's emotional state and operational needs.

[0146] I. Adjust the output of the content generation module based on the sentiment analysis results. The emotion recognition module analyzes the user's emotional state using multimodal data such as facial expressions and heart rate, and feeds the results back to the content generation module to dynamically adjust the display effect of the 3D scene. The adjustments include the following aspects: 1. Adjusting the object's color The system adjusts the colors of dynamic objects in a scene based on the user's emotional color preferences. For example, when sentiment analysis indicates that the user is in a pleasant state, the system may adjust the colors in the scene to bright, highly saturated hues to further enhance the user's sense of pleasure; while when the user's emotions lean towards calmness or focus, the system may choose soft, neutral hues to create a more calming atmosphere.

[0147] In specific scenarios, such as when reading natural science books and describing the Earth's rotation, the system can adjust the brightness and contrast of the Earth's surface color according to the user's mood to better match the current emotional tone.

[0148] 2. Adjusting object size The system adjusts the size of objects in a scene based on sentiment analysis results. For example, when a user shows strong interest in a scene (e.g., emotional excitement), the system can enlarge and highlight the core objects in that scene to enhance visual impact and appeal.

[0149] For example, when reading astronomy-related content, if a user shows curiosity about "planetary systems," the system will dynamically zoom in on the model of a specific planet to highlight its details.

[0150] 3. Adjusting the object's actions The system dynamically adjusts the movement characteristics of objects based on the user's emotions, including movement speed, trajectory, and interaction response. For example, when sentiment analysis indicates that the user is focused, the system may reduce the movement speed of objects, giving the user more time to observe and understand the scene content; when the user is in a happy state, the system may add faster or more dynamic actions to enhance the liveliness of the scene.

[0151] II. Adjust the 3D scene according to the operation instructions generated by the interactive control module. The interactive control module receives user voice commands and gestures, parses their operational intentions, and passes the results to the content generation module for controlling the perspective switching and interactive hotspot settings of the 3D scene. The specific implementation is as follows: 1. Control of viewpoint switching The system switches the perspective of the 3D scene based on the user's input. For example, when the user points to a scene area with a gesture, the system adjusts the camera perspective to make that area the focus; when the user issues a "zoom in" command, the system dynamically zooms in on the camera to provide a more detailed view.

[0152] For example, when reading content describing the Eiffel Tower, users can request a "top-down view" or a "bottom view" via voice commands, and the system will switch scenes accordingly to meet the user's needs.

[0153] 2. Setting up interactive hotspots The system sets up interactive hotspots in areas indicated by the user to trigger additional content or functions. For example, if a user taps an object in the scene (such as a country on a globe) with a gesture, the system can set up a hotspot at that location and trigger corresponding interactive behaviors (such as playing historical information about that country or displaying geographical features).

[0154] The triggering behavior for hot topics can also be dynamically adjusted based on the user's level of interest, such as loading more relevant information or multimedia resources for areas of high interest.

[0155] III. Adjust multimedia resources based on sentiment analysis and operation instructions The system combines input from the emotion recognition module and the interactive control module to dynamically select and adjust the display method of multimedia resources (such as audio, video, and images) to ensure that the resource content is highly consistent with the scene and user needs. The adjustment methods include the following: 1. Selection of multimedia resources The system filters the most relevant resources from the multimedia database based on the user's emotional state and semantic needs. For example, when sentiment analysis shows that a user is confused or focused on a certain scene, the system prioritizes loading detailed explanation videos or animated images related to the current topic to help the user understand.

[0156] For example, when reading content about "human skeletal structure", if the user shows curiosity, the system may prioritize loading dynamic model videos that demonstrate the mechanism of skeletal movement.

[0157] 2. Adjustment of playback order The system dynamically adjusts the playback order of resources based on the user's focus and operation history. For example, when a user views a certain type of resource (such as video) multiple times, the system will prioritize resources of the same type at the beginning.

[0158] 3. Dynamic adjustment of display mode The system adjusts the display effects of multimedia resources based on sentiment analysis results. For example, the playback volume of audio resources can be dynamically adjusted according to the intensity of the user's emotions; the video playback speed can be increased when the user shows impatience to meet the user's rhythm requirements.

[0159] In scene display, image resources can be adjusted according to the position and size of the 3D scene. For example, important images can be embedded in the center of the user's field of vision, while secondary resources can be placed in the background area.

[0160] In some embodiments, the system can further optimize its adjustment strategy by incorporating the user's reading history and preferences. For example, for users who frequently browse specific topics, the system will prioritize recommending multimedia resources related to that topic while appropriately reducing the loading of other resources. It should be noted that this dynamic adjustment mechanism can be applied to various scenarios, such as virtual laboratories in education and training or immersive experiences in cultural heritage displays.

[0161] In this invention, the system uses machine learning algorithms to dynamically adjust the scene output of the content generation module and the resource selection of the multimedia fusion module based on the reader's reading history and preference data, in order to enhance the reader's personalized reading experience.

[0162] In this embodiment, machine learning algorithms are used to analyze readers' reading records and preference data to generate optimization strategies for dynamically adjusting the scene output of the content generation module and the resource selection of the multimedia fusion module. The system aims to enhance the reader's immersive reading experience through personalized recommendations and real-time adjustments.

[0163] 1. Collection of reading records and preference data The system collects readers' reading records and preference data in the following ways: Reading history: This includes the reader's reading history (such as chapters read and pages spent at a particular time), reading speed, and multimedia interaction behavior (such as clicked hotspots and played resources).

[0164] Preference data: By analyzing the content types (such as popular science and history) and changes in emotional state (such as preference for light and pleasant content) selected by readers multiple times, a user interest model is constructed.

[0165] 2. Generation of personalized adjustment strategies The system uses machine learning algorithms (such as collaborative filtering recommendation and deep learning models) to train on collected reading records and preference data to generate personalized adjustment strategies. These strategies include: Scene output adjustment: Dynamically optimize the display effect of 3D scenes, such as content arrangement, focus highlighting, color matching, etc.

[0166] Multimedia resource selection: Prioritize resource types that closely match the reader's interests.

[0167] 3. Mathematical description of dynamic adjustment Suppose the system builds a preference model for each user, describing the user's preference weights for different scenarios and resources. The goal of the system's dynamic adjustment is to maximize the user experience score, which can be expressed as:

[0168] in, Indicates the number of adjustable dynamic objects in the scene; This represents the relationship between the user preference model and the scene object. Relevant weights, This represents the optimization score of the scene object (such as visual satisfaction after color adjustment). Indicates the quantity of multimedia resources; This represents the weights associated with multimedia resources in the user preference model; This indicates the recommendation score for multimedia resources (such as the degree to which the content matches the user's interests).

[0169] The system optimizes the weights of each term in the above formula. and The output effects of the content generation module and the multimedia fusion module are adjusted in real time.

[0170] 4. Example: If a user's reading history shows that they are interested in the topic of "Earth Science", the system will prioritize loading relevant 3D model scenes (such as simulations of Earth's rotation and revolution) and highlight important details.

[0171] For multimedia resources, the system will prioritize recommending images (such as geological structure maps) and videos (such as volcanic eruption simulations) related to "Earth Science".

[0172] 5. Dynamic Feedback and Iterative Optimization In actual use, the system dynamically updates the user preference model based on the reader's real-time behavior (such as resource click frequency and page dwell time). This continuous optimization mechanism ensures that the system's recommendation results are always consistent with the user's interests.

[0173] Please see the appendix Figure 8 The operating method of the above-mentioned system of the present invention includes the following steps: S1. Extract the text content from the book and generate a 3D scene through the content generation module; S2. Collect and analyze readers' facial expressions and heart rate data to generate sentiment analysis results; S3. Parse the reader's voice commands and gestures to generate operation instructions; S4. Dynamically adjust the 3D scene content based on the sentiment analysis results and operation instructions, and select audio, video and image resources; S5. Convert the content of printed books into e-books and encrypt the electronic content for protection.

[0174] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. An intelligent 3D book system based on multi-dimensional interactive experience, characterized in that: include: The content generation module is used to generate dynamic three-dimensional scenes for stereoscopic books based on text content. The scenes include virtual reality (VR) scenes and augmented reality (AR) scenes. The module adjusts the scene content according to the analysis results of the emotion recognition module and the instructions of the interaction control module. The emotion recognition module is used to monitor the reader's emotional state. It collects facial expression data and heart rate data through sensors and generates emotion analysis results. The analysis results are used to adjust the scene content generated by the content generation module and the multimedia resources selected by the multimedia fusion module. The interactive control module is used to parse the reader's voice commands and gestures, and generate operation commands related to page turning, content querying and scene adjustment. These commands directly control the content generation module and the multimedia fusion module. The multimedia fusion module is used to select audio, video and image resources that match the book content based on the 3D scene generated by the content generation module and the analysis results of the emotion recognition module, and to embed the resources into the 3D scene. The integrated paper and e-book module is used to convert paper book content into e-books using OCR technology, and to encrypt and protect the converted e-books to ensure content synchronization and copyright security.

2. The intelligent 3D book system based on multi-dimensional interactive experience according to claim 1, characterized in that, The content generation module includes: Modeling unit, used to generate 3D scene models based on text descriptions; The graphics processing unit is used for real-time rendering of 3D scenes; VR generation unit, used to generate virtual reality scenes; AR generation unit, used for augmented reality content.

3. The intelligent 3D book system based on multi-dimensional interactive experience according to claim 1, characterized in that, The emotion recognition module includes: The recognition unit is used to recognize the reader's facial expressions using a convolutional neural network algorithm; The monitoring unit is used to monitor the reader's heart rate signal using photoplethysmography. The sentiment analysis unit is used to generate sentiment analysis results based on facial expression and heart rate data.

4. The intelligent 3D book system based on multi-dimensional interactive experience according to claim 1, characterized in that, The interactive control module includes: A speech recognition unit is used to parse the reader's voice commands; A gesture recognition unit is used to capture and analyze hand gestures; The instruction generation unit is used to integrate voice and gesture actions to generate input instructions.

5. The intelligent 3D book system based on multi-dimensional interactive experience according to claim 1, characterized in that, The multimedia fusion module includes: The semantic analysis unit is used to analyze the semantic information of the pop-up book content using natural language processing technology. The media selection unit is used to select audio, video, and image resources based on semantic analysis results; The content display unit is used to embed selected multimedia resources into a dynamic 3D scene.

6. The intelligent 3D book system based on multi-dimensional interactive experience according to claim 1, characterized in that, The integrated paper and e-book module includes: The text recognition unit is used to digitize the content of printed books using OCR technology; Blockchain encryption units are used to encrypt and protect electronic content.

7. The intelligent 3D book system based on multi-dimensional interactive experience according to claim 1, characterized in that, The system also includes an artificial intelligence-assisted creation module, which includes: The inspiration generation unit is used to generate creative inspiration based on the reader's input. The vocabulary recommendation unit is used to recommend vocabulary based on the input content; The paragraph generation unit is used to generate a continuation paragraph based on the input content.

8. The intelligent 3D book system based on multi-dimensional interactive experience according to claim 1, characterized in that, The system combines the sentiment analysis results generated by the sentiment recognition module with the operation commands generated by the interaction control module, and dynamically adjusts the outputs of the content generation module and the multimedia fusion module in the following ways: Based on the sentiment analysis results generated by the sentiment recognition module, adjust the display effects of dynamic objects in the 3D scene output by the content generation module, including object color, size, and movement; Based on the operation instructions generated by the interactive control module, control the perspective switching and interactive hotspots of the 3D scene in the content generation module; Based on the sentiment analysis results and operation instructions, the system dynamically selects audio, video, and image resources from the multimedia fusion module and adjusts the playback order and display mode of these resources.

9. The intelligent 3D book system based on multi-dimensional interactive experience according to claim 1, characterized in that, The system uses machine learning algorithms to dynamically adjust the scene output of the content generation module and the resource selection of the multimedia fusion module based on the reader's reading history and preference data, in order to enhance the reader's personalized reading experience.

10. A method for an intelligent 3D book based on multi-dimensional interactive experience, based on the intelligent 3D book system based on multi-dimensional interactive experience as described in any one of the claims, characterized in that, Includes the following steps: S1. Extract the text content from the book and generate a 3D scene through the content generation module; S2. Collect and analyze readers' facial expressions and heart rate data to generate sentiment analysis results; S3. Parse the reader's voice commands and gestures to generate operation instructions; S4. Dynamically adjust the 3D scene content based on the sentiment analysis results and operation instructions, and select audio, video and image resources; S5. Convert the content of printed books into e-books and encrypt the electronic content for protection.

Images