Child digital picture book reading system and method based on immersive interactive projection

Abstract

The invention relates to a children digital picture book reading system and method based on immersive interaction, and the system comprises a projection module, an interaction module, a voice playing module, a voice conversation module, a video record synthesis module, an intelligent evaluation module, and a digital resource module. The original picture book content of the digital resource module is projected to the wall surface and the ground through the projection module to create an immersive digital picture book reading scene; the child interacts with the original picture book content of the digital resource module in real time through the interaction module, the voice playing module and the voice dialogue module; the video recording and synthesizing module is used for creating an immersive story telling scene for children and recording story telling audios and videos of the children; and the intelligent evaluation module receives the audio and the video transmitted by the video record synthesis module, and outputs an evaluation report through intelligent analysis. The problems that in the prior art, a game form and an interaction form are single, an immersive reading scene and an intelligent evaluation system are lacked, and continuous reading interest and language ability development of children are not facilitated are solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of digital picture book reading, and in particular relates to a children's digital picture book reading system and method based on immersive interactive projection. Background Art

[0002] Picture books refer to picture story books that complement each other with text and pictures, and are reading books that express specific emotions and themes. With the increasing application of digital technology in the field of children's education, digital picture book reading has received widespread attention as a new way of reading. Digital picture book reading refers to the use of electronic devices or multimedia technology to display the content of picture books, generally presented through reading pens, reading robots, tablets, e-books, apps, etc. Existing digital picture book reading technology has stimulated children's interest in reading to a certain extent, but the game form and interactive form are relatively simple, and lack immersive reading scenes, AI intelligent voice recognition and intelligent evaluation systems, which is not conducive to children maintaining continuous reading interest and language ability development.

[0003] Therefore, it is necessary to provide a children's digital picture book reading system and method based on immersive interactive projection to solve the above technical problems. Summary of the invention

[0004] The purpose of the present invention is to provide a children's digital picture book reading system and method based on immersive interactive projection, so as to solve the problems in the prior art that the game form and interactive form are relatively single, and lack immersive reading scenes, AI intelligent voice recognition and intelligent evaluation systems, which are not conducive to children maintaining continuous reading interest and language ability development.

[0005] In order to solve the above technical problems, the technical solution of the present invention is as follows:

[0006] A children's digital picture book reading system based on immersive interactive projection, comprising:

[0007] Projection module, interactive module, voice playback module, voice dialogue module, video acquisition and synthesis module, intelligent evaluation module, digital resource module. The picture book content of the digital resource module is projected onto the wall and the ground through the projection module to create an immersive digital picture book reading scene; children interact with the picture book content of the digital resource module in real time through the interactive module, voice playback module and voice dialogue module; the video acquisition and synthesis module creates an immersive storytelling scene for children and records the audio and video of their storytelling; the intelligent evaluation module receives the audio and video transmitted by the video acquisition and synthesis module, and outputs an evaluation report through intelligent analysis.

[0008] Preferably, the projection can flexibly configure the number of projectors according to the actual space size and layout of the site. It supports multiple projection modes such as three-wall U-shaped projection, three-wall plus ground projection, etc., and creates an immersive digital picture book reading scene for children by projecting the picture book content in the digital resource module onto the wall and the ground.

[0009] Preferably, the interactive module contains a radar sensing device and a microphone. The radar sensing device captures the child's instructions of slapping, dragging, and throwing the wall and triggers corresponding interactive effects, and the microphone records the child's voice audio.

[0010] Preferably, the voice playback module is used to play the picture book content of the digital resource module, and to play the command prompt sound and interactive effect feedback prompt sound entered by the interactive module.

[0011] Preferably, the voice dialogue module records the audio input by the child through the microphone in the interactive module, and then realizes real-time dialogue between the child and the illustrated character through AI intelligent voice recognition technology and voice playback module.

[0012] Preferably, the video acquisition and synthesis module contains a camera, which captures images of children standing in the camera area, synthesizes the recorded images of children with the scene projected by the digital resource module using the matting technology, and shoots a video of the children telling stories and uploads it to the intelligent evaluation module.

[0013] Preferably, the intelligent evaluation module accesses the language model and, after receiving the video of the child telling a story transmitted by the video acquisition and synthesis module, uses AI intelligent speech recognition technology to analyze the story audio in the video, intelligently generates an evaluation report, and makes targeted optimization suggestions for the child's pronunciation and speech speed. The method in which the intelligent evaluation module can accurately evaluate is as follows: Calculation of pronunciation accuracy: Let the total number of pronunciations in the children's story audio be N, and the number of correct pronunciations be n, then the pronunciation accuracy P pronunciation It can be calculated by the following formula: This function intuitively reflects the accuracy of children's pronunciation and provides a quantitative basis for the analysis of pronunciation in the evaluation report.

[0014] Speech rate evaluation function: Assuming that the duration of the story audio is t (seconds) and the number of words in the story text is W, the speech rate S (words / second) can be calculated by the following formula: At the same time, according to the appropriate speaking speed range of children of different ages (set as S min To S max ), the speech speed deviation D can be calculated speed :

[0015] The smaller the speech speed deviation, the closer the child's speech speed is to the appropriate range. These two functions help to scientifically evaluate the child's speech speed in the evaluation report and give targeted adjustment suggestions.

[0016] Comprehensive performance scoring function: Comprehensively consider the pronunciation accuracy P pronunciation , speech speed deviation D speed and the score of the interactive game S game (full score 100 points), assign corresponding weights to each factor, and set the pronunciation accuracy weight as w i , the speech speed deviation weight is w2, the interactive game score weight is w3, and w1+w2+w3=1, then the comprehensive performance score Score can be calculated by the following formula: Score=w1×P pronunciation +w2×(1-D speed )×100+w3×S game

[0017] This comprehensive score can comprehensively and objectively reflect children's overall performance in the reading process, and provide strong support for the giving of special rewards and the making of learning suggestions.

[0018] Preferably, the digital resource module is used for storing and real-time updating of digital picture book resources.

[0019] Preferably, a children's digital picture book reading method based on immersive interactive projection comprises the following steps:

[0020] S1. Select a picture book: Open the system, select the corresponding level, then select the corresponding picture book series, and finally select the corresponding picture book. The projection module will project the corresponding picture book content onto the wall and the ground.

[0021] S2. Play interactive games while reading picture books: After starting the digital picture book reading activity, the voice playback module begins to tell the story based on the projected picture book screen. During the picture book reading process, children need to complete interactive games or have voice dialogues with picture book characters through the interactive module or voice dialogue module according to randomly appearing prompts.

[0022] S3. Immersive storytelling for children: After the voice playback module automatically plays the picture book story, an immersive small theater scene will appear on the projection screen. The child stands in front of the small theater scene. The video acquisition and synthesis module captures the child’s image and the projected small theater scene through the camera, and starts recording the video of the child telling the story. After the child finishes the story, he or she taps the "Done" button in the middle, and the video input module transmits the recorded video to the intelligent evaluation module.

[0023] S4. Output intelligent evaluation report: The intelligent evaluation module will use speech recognition AI intelligent technology to analyze the audio of children's stories. Based on the comprehensive performance score, the system will give reward effects according to the following rules:

[0024] When Score ≥ 90, trophies and medals will be given;

[0025] When 80≤Score<90, medals and red flowers will be awarded;

[0026] When 70≤Score<80, a red flower will be given;

[0027] When the Score is less than 70, the child is encouraged to keep working hard and no reward effects will be given for the time being. However, more targeted suggestions for improvement will be provided in the evaluation report.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] 1. Projecting the original picture book content on the ground and walls through the projection module can create an immersive picture book reading scene, so that children will feel as if they are in a real picture book world when they walk in, fully stimulating their interest in picture book reading;

[0030] 2. Through various forms of interactive games and voice dialogues, the fun, interactivity and playability of digital picture book reading can be effectively enhanced, which is conducive to children maintaining a strong interest in continuous reading and cultivating a good habit of loving reading from an early age;

[0031] 3. Through the video acquisition and synthesis module, it can provide children with an immersive storytelling opportunity, encourage them to dare to speak and be willing to speak, and improve their language expression ability;

[0032] 4. The intelligent evaluation module can intelligently analyze the stories told by children and generate intelligent evaluation reports, and provide targeted optimization suggestions for children's pronunciation, speaking speed, etc., which can effectively promote the development of children's language ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A structural block diagram of a children's digital picture book reading system based on immersive interactive projection provided by the present invention;

[0034] Figure 2 A flowchart of a children's digital picture book reading method based on immersive interactive projection provided by the present invention;

[0035] Figure 3 for Figure 1 The structural block diagram of the interactive module is shown. DETAILED DESCRIPTION

[0036] The present invention will be further described below in conjunction with the accompanying drawings and implementation modes.

[0037] Reference Figure 1 A children's digital picture book reading system based on immersive interactive projection includes: a projection module, an interactive module, a voice playback module, a voice dialogue module, a video acquisition and synthesis module, an intelligent evaluation module, and a digital resource module. The original picture book content of the digital resource module is projected onto the wall and the ground through the projection module to create an immersive digital picture book reading scene; children interact with the original picture book content of the digital resource module in real time through the interactive module, the voice playback module and the voice dialogue module; the video acquisition and synthesis module creates an immersive storytelling scene for children and records the audio and video of their storytelling; the intelligent evaluation module receives the audio and video transmitted by the video acquisition and synthesis module, and outputs an evaluation report through intelligent analysis.

[0038] The projection module can be configured with the number of projectors according to the actual site, and can be set up with a three-wall U-shaped projection or a three-wall plus ground projection to create an immersive picture book reading scene.

[0039] The interactive module contains a radar sensing device and a microphone. The radar sensing device captures the child's instructions of slapping, dragging, and throwing the wall and triggers corresponding interactive effects, and the microphone records the child's voice audio.

[0040] Reference Figure 2 The interactive module has designed various interactive game modes, such as: tapping a single target pattern according to the question; tapping multiple target patterns in sequence according to the question; tapping the target position according to the prompt to trigger the corresponding interactive effect or switch to the next storyline; dragging the target pattern to the specified position according to the prompt to trigger the corresponding interactive special effect; throwing the target pattern at a long distance with a sandbag according to the prompt to trigger the corresponding interactive effect; dragging the target pattern to the shadow outline according to the shadow outline to trigger the outline to show the complete pattern; adjusting the up, down, left, and right positions of the pattern by tapping the direction button according to the prompt, aiming at the target; dragging the corresponding color block to the blank space of the target according to the prompt to color the blank space; dragging the corresponding pattern to the target position according to the prompt to complete the missing picture book screen; tapping two identical target patterns continuously according to the prompt to trigger the disappearance of the pattern. Through various interactive game designs, children can play games while reading picture books. While enhancing the fun and playability of picture book reading activities, it allows children to actively participate in picture books, and their sense of substitution and interaction is stronger, and their interest in reading is stronger.

[0041] The voice playback module is used to play the picture book content of the digital resource module, and to play the command prompt sound and interactive effect feedback prompt sound entered by the interactive module. Through the voice playback module, children can independently understand the content of the picture book, and can complete related picture book game tasks according to the voice playback prompts, reducing the workload of teachers.

[0042] The voice dialogue module records the audio input by the child through the microphone in the interactive module, and then realizes the real-time dialogue between the child and the picture book character through the voice recognition AI intelligent technology and the voice playback module. Children can ask random questions around the theme of the picture book, and the picture book characters will give corresponding answers based on the questions. For example, in the digital picture book reading activity of "Going to the Dentist is Not Scary", children can ask: "What are the effective ways to protect teeth?" The picture book character gives a corresponding answer: "Eating less sweets, brushing teeth frequently in the morning and evening, drinking less beverages, and regular oral examinations can effectively protect our teeth." Through voice dialogue, the interaction between children and picture book characters can be enhanced, more knowledge related to the theme of the picture book can be expanded, and children's communication skills and understanding of voice recognition AI intelligent technology can also be strengthened.

[0043] The video acquisition and synthesis module includes a camera, which captures images of children standing in the camera area, synthesizes the recorded images of children with the small theater scene projected by the digital resource module, and captures a video of children telling stories and uploads it to the intelligent evaluation module. By creating an immersive scene, children's interest in storytelling can be fully stimulated, allowing them to dare to tell stories and be happy to tell stories in an immersive and situational atmosphere, thereby promoting the improvement of their language expression ability.

[0044] The intelligent evaluation module receives the video of children telling stories transmitted by the video acquisition and synthesis module, and uses speech recognition AI intelligent technology to analyze the story audio in the video, intelligently generates an evaluation report, and makes targeted optimization suggestions for the pronunciation and speaking speed of the children. The intelligent evaluation module will give out reward effects such as gold coins, small red flowers, certificates, trophies, medals, etc. according to the children's performance, enhancing the children's self-confidence and sense of accomplishment. At the same time, based on the analysis of the evaluation report, it can help children understand their good points and shortcomings in pronunciation, speaking speed, etc., and make targeted optimization and improvement suggestions, which is conducive to promoting the development of their language ability.

[0045] The digital resource module is used for storing and updating digital picture book resources in real time, and can upload new digital picture book contents to cooperative customers online at any time.

[0046] Reference Figure 3 , a children's digital picture book reading method based on immersive interactive projection, the operation steps of which include:

[0047] S1. Select a picture book: Open the system, select the corresponding level, then select the corresponding picture book series, and finally select the corresponding picture book. The projection module will project the corresponding picture book content onto the wall and the ground.

[0048] S2. Play interactive games while reading picture books: After starting the digital picture book reading activity, the voice playback module begins to tell the story based on the projected picture book screen. During the picture book reading process, children need to complete interactive games or have voice dialogues with picture book characters through the interactive module or voice dialogue module according to randomly appearing prompts.

[0049] S3. Immersive storytelling for children: After the voice playback module automatically plays the picture book story, an immersive small theater scene will appear on the projection screen. The child stands in front of the small theater scene. The video acquisition and synthesis module captures the child’s image and the projected small theater scene through the camera, and starts recording the video of the child telling the story. After the child finishes the story, he or she taps the "Done" button in the middle, and the video input module transmits the recorded video to the intelligent evaluation module.

[0050] S4. Output intelligent evaluation report: After receiving the video, the intelligent evaluation module uses AI intelligent speech recognition technology to analyze the audio of the story told by the child. Assume that the total number of pronunciations in the story audio is N = 200, the number of correct pronunciations is n = 180, the story audio duration is t = 120 seconds, the number of words in the story text is W = 240, and the interactive game score is S. game = 85 points, weight w1 = 0.3, w2 = 0.2, w3 = 0.5, suitable speaking speed range S min = 1.5 words / second, S max =2.5 words / second.

[0051] First calculate the pronunciation accuracy:

[0052] Then calculate the speaking rate: Then calculate the speech speed deviation

[0053] Finally, calculate the overall performance score: Score = 0.3 × 90 + 0.2 × (1-0) × 100 + 0.5 × 85 = 86.5

[0054] Based on the comprehensive performance score, the system will give out certificates and red flowers, and point out in the evaluation report that the child has accurate pronunciation, moderate speaking speed, and good performance in interactive games, while encouraging the child to continue to maintain this performance.

[0055] Further, step S2 includes:

[0056] S201, when the voice playback module plays the picture book story, the interactive module will randomly pop up the corresponding interactive game, and the child can complete the corresponding game task through various interactive methods such as patting, dragging, throwing, etc. according to the screen prompts or voice broadcast prompts;

[0057] S202, the radar sensing device in the interactive module captures the child's instructions of patting, dragging, throwing, etc. on the wall and transmits them back to the background for processing. By analyzing the child's input instructions, different feedback and interactive effects can be made to achieve real-time interaction between the child and the projection screen. If the child's instructions are correct, the corresponding interactive special effects will be triggered or the next plot will be switched. If the child's instructions are wrong, an "X" prompt will appear;

[0058] S203. When the voice playback module is playing the picture book story, the child can tap the "AI dialogue" button on the projection screen at any time through the voice dialogue module to start a voice dialogue and have a real-time intelligent dialogue with the picture book characters.

[0059] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A children's digital picture book reading system based on immersive interaction, characterized in that: include: Projection module, interactive module, voice playback module, voice dialogue module, video acquisition and synthesis module, intelligent evaluation module, digital resource module. The picture book content of the digital resource module is projected onto the wall and the ground through the projection module to create an immersive digital picture book reading scene; children interact with the picture book content of the digital resource module in real time through the interactive module, voice playback module and voice dialogue module; the video acquisition and synthesis module creates an immersive storytelling scene for children and records the audio and video of their storytelling; the intelligent evaluation module receives the audio and video transmitted by the video acquisition and synthesis module, and outputs an evaluation report through intelligent analysis.

2. The children's digital picture book reading system based on immersive interaction according to claim 1 is characterized in that: The projection can flexibly configure the number of projectors according to the actual space size and layout of the site. It supports multiple projection modes such as three-wall U-shaped projection, three-wall plus ground projection, etc. By projecting the picture book content in the digital resource module onto the wall and the ground, an immersive digital picture book reading scene is created for children.

3. The children's digital picture book reading system based on immersive interaction according to claim 1 is characterized in that: The interactive module contains a radar sensing device and a microphone. The radar sensing device captures the child's instructions of slapping, dragging, and throwing the wall and triggers corresponding interactive effects, and the microphone records the child's voice audio.

4. The children's digital picture book reading system based on immersive interaction according to claim 1 is characterized in that: The voice playback module is used to play the picture book content of the digital resource module, and to play the command prompt sound and interactive effect feedback prompt sound entered by the interactive module.

5. The children's digital picture book reading system based on immersive interaction according to claim 1, characterized in that: The voice dialogue module records the audio input by the child through the microphone in the interactive module, and then realizes real-time dialogue between the child and the illustrated character through AI intelligent voice recognition technology and voice playback module.

6. The children's digital picture book reading system based on immersive interaction according to claim 1 is characterized in that The video acquisition and synthesis module includes a camera, which captures images of children standing in the camera area, synthesizes the recorded child images with the scene projected by the digital resource module using the matting technology, and captures a video of the child telling a story and uploads it to the intelligent evaluation module.

7. The children's digital picture book reading system based on immersive interaction according to claim 1, characterized in that: After receiving the video of the child telling a story transmitted by the video acquisition and synthesis module, the intelligent evaluation module accesses the language model and uses AI intelligent speech recognition technology to analyze the story audio in the video, intelligently generates an evaluation report, and makes targeted optimization suggestions for the child's pronunciation and speaking speed. The method in which the intelligent evaluation module can accurately evaluate is as follows; Calculation of pronunciation accuracy: Let the total number of pronunciations in the children's story audio be N, and the number of correct pronunciations be n, then the pronunciation accuracy P pronunciation It can be calculated by the following formula: This function intuitively reflects the accuracy of children's pronunciation and provides a quantitative basis for the analysis of pronunciation in the evaluation report. Speech rate evaluation function: Assuming that the duration of the story audio is t (seconds) and the number of words in the story text is W, the speech rate S (words / second) can be calculated by the following formula: At the same time, according to the appropriate speaking speed range of children of different ages (set as S min To S max ), the speech speed deviation D can be calculated speed : The smaller the speech speed deviation, the closer the child's speech speed is to the appropriate range. These two functions help to scientifically evaluate the child's speech speed in the evaluation report and give targeted adjustment suggestions. Comprehensive performance scoring function: Comprehensively consider the pronunciation accuracy P pronunciation , speech speed deviation D speed and the score of the interactive game S game (full score 100 points), assign corresponding weights to each factor, and set the pronunciation accuracy weight as w i , the speech speed deviation weight is w2, the interactive game score weight is w3, and w1+w2+w3=1, then the comprehensive performance score Score can be calculated by the following formula: Score=w1×P pronunciation +w2×(1-D speed )×100+w3×S game This comprehensive score can comprehensively and objectively reflect children's overall performance in the reading process, and provide strong support for the giving of special rewards and the making of learning suggestions.

8. The children's digital picture book reading system based on AI interaction according to claim 1, characterized in that: The digital resource module is used for storing and updating digital picture book resources in real time.

9. A children's digital picture book reading method based on immersive interaction, characterized in that: The following steps are involved: S1. Select a picture book: Open the system, select the corresponding level, then select the corresponding picture book series, and finally select the corresponding picture book. The projection module will project the corresponding picture book content onto the wall and the ground. S2. Play interactive games while reading picture books: After starting the digital picture book reading activity, the voice playback module begins to tell the story based on the projected picture book screen. During the picture book reading process, children need to complete interactive games or have voice dialogues with picture book characters through the interactive module or voice dialogue module according to randomly appearing prompts. S3. Immersive storytelling for children: After the voice playback module automatically plays the picture book story, an immersive small theater scene will appear on the projection screen. The child stands in front of the small theater scene. The video acquisition and synthesis module captures the child's image and the projected small theater scene through the camera, and starts recording the video of the child telling the story. After the child finishes the story, he taps the "Done" button in the middle, and the video input module transmits the recorded video to the intelligent evaluation module. S4. Output intelligent evaluation report: The intelligent evaluation module will use voice recognition AI intelligent technology to analyze the audio of children's stories. Based on the comprehensive performance score, the system will give out reward effects according to the following rules: when Score ≥ 90, a trophy and medal will be given; when 80 ≤ Score < 90, a medal and a red flower will be given; when 70 ≤ Score < 80, a red flower will be given; when Score < 70, children will be encouraged to keep working hard, and no reward effects will be given for the time being, but more targeted improvement suggestions will be provided in the evaluation report.

Images