[0021] Such as Figure 1 to Figure 3 As shown, the education system based on augmented reality books of the present invention includes augmented reality books, intelligent display terminals, and servers.
[0022] Wherein, the augmented reality book is provided with a recognition mark pattern,
[0023] The smart display terminal is provided with a camera unit, an image recognition module, and an animation interaction and sound identification module. The camera unit is used to obtain patterns in an augmented reality book, and the image recognition module is used According to the predetermined method, the pixel information is extracted from the pattern, made into a texture, and attached to the model's material ball to obtain the model with the expected color, and display it on the screen of the intelligent display terminal, the animation interaction and voice recognition module Used for the interaction of puzzles and the playback of voice resources;
[0024] The server is used to deploy interactive programs and store related data information.
[0025] With the cooperation of external devices (smart display terminal and server), the augmented reality book can display a three-dimensional model.
[0026] In the present invention, a deeper perception experience is realized through the interaction of vision, hearing, and touch. It meets the various needs of children to discover the new world in the process of growing up, and creates an educational scene and background with complete content and convenient experience to the greatest extent. The development, production and release of AR APP programs based on Unity3D software can simultaneously develop products in IOS system and Andriod system. The interactive use of mobile devices and physical books not only enhances the fun of children’s education, but also increases the interaction between parents and children. In terms of educational experience, the interactivity of AR products is an experience that many physical books and ordinary e-books do not have. Let parents and children actively interact with the characters in the books, and establish a communication bond between parents and children through augmented reality technology.
[0027] The interactive education method based on augmented reality books includes the following steps:
[0028] Visit the server to obtain information that matches the augmented reality book;
[0029] Adjust the position of the camera unit and the augmented reality book so that the augmented reality book is within the field of view of the camera;
[0030] The smart terminal obtains the identification information of the picture, extracts the pixel information from the picture, and makes a map in a predetermined format, attaches it to the material ball, obtains the display model and displays it on the display screen of the smart terminal.
[0031] Further, the above interactive education method further includes a puzzle step: when the broken picture block is dragged to a certain area, the smart terminal displays the picture block or sends out a prompt.
[0032] Therefore, during use, in-depth human-computer interaction can be carried out according to the needs of the user, such as controlling the size and position of the object to make the object move or make a sound.
[0033] It should be noted that if the relevant steps in the interactive education method are not logically or temporally related, their sequence can be adjusted according to the user's operation/use situation.
[0034] An interactive education method based on augmented reality books,
[0035] When the user color the pattern in the augmented reality book, the smart terminal obtains the shape and color of the pattern through the camera, and superimposes it on the corresponding three-dimensional model, and displays it on the display screen of the smart terminal.
[0036] Further, when displaying the three-dimensional model, audio or animation is displayed.
[0037] Therefore, the user can observe the state of the three-dimensional model in real time during use, which is more interesting.
[0038] The present invention mainly includes the following aspects:
[0039] 1. The three-dimensional character of children's 4D picture books; 2. Real-time color recognition system; 3. Three-dimensional animation interaction and sound system recognition.
[0040] The invention uses AR technology to realize the original static two-dimensional book into a three-dimensional character that can be painted in real time for color recognition, and then cooperates with animation, sound, special effects, and realizes human-computer interaction and other functions. Let the children know, learn and play in a scene that combines virtual and reality.
[0041] The characters of children's 4D picture books are three-dimensional. The main technical principle is: respectively design 12 future professional characters and 12 transportation vehicles as the main body of the book design, use the MAYA 3D animation production software to transform the characters into 3D characters and give them animations, and match the corresponding English word knowledge through the characters , Chinese writing, Pinyin and nursery rhymes written for characters, through the dual effects of the text of the book and the voice of the APP, let children learn about the profession through the book and phonetic system.
[0042] Real-time color recognition system. The main technical principle is: the basic principle of image recognition technology: use Qualcomm SDK to use the mobile phone camera to obtain the recognition card image, and then combine the obtained shadow image information with ShanderLab to extract the pixel information in the picture according to the pre-defined four vertex positions Make a 512*512 texture map and attach it to the model's material sphere, so that the colors drawn on the drawing can be added to the model that appears, and finally displayed on the device screen.
[0043] Three-dimensional animation interaction and voice recognition system. Use C# language programming, make a mobile phone APP, process the image information recognized by the camera to make a model texture, and load it on the model material ball to realize the coloring function. The puzzle function is to realize the finger dragging the broken puzzle piece to the correct Place and display, storytelling is the playback of voice resources. The software and development environment to be used are also more suitable for research and development. Unity3D is a cross-platform game engine that has emerged in recent years. It can support multiple platforms such as PC, Android, and IOS, including almost all current mobile platform systems. Easily implement cross-platform app release.
[0044] In the above steps, the realization process of color recognition is as follows:
[0045] Color recognition realization:
[0046] Step 1: Use Maya or Max to make a 3D model;
[0047] Step 2: Use Photoshop to make texture maps, then import the model and textures into Unity, make the scene, and use Qualcomm SDK to implement the augmented reality function;
[0048] Step 3: Use C# to perform functions in unity to achieve color extraction;
[0049] Code:
[0050] CameraDevice cam = CameraDevice.Instance;
[0051] Image image = cam.GetCameraImage(m_PixelFormat);
[0052] if (image == null) {
[0053] // not on mobile devices
[0054] Debug.Log(m_PixelFormat + "image is not available yet");
[0055] }
[0056] else {
[0057] // get MVP matrix
[0058] Matrix4x4 P = GL.GetGPUProjectionMatrix(Camera.main.projectionMatrix, false);
[0059] Matrix4x4 V = Camera.main.worldToCameraMatrix;
[0060] // use plane to get texcoords for target
[0061] Matrix4x4 M = mPlane.GetComponent ().localToWorldMatrix;
[0062] Matrix4x4 MVP = P * V * M;
[0063] // screen to camera image scale
[0064] float r1 = image.Height*1.0f/image.Width;
[0065] float r2 = Screen.width*1.0f/Screen.height;
[0066] if(r1> r2) {
[0067] GetComponent ().material.SetFloat("_xScale",1.0f);
[0068] GetComponent ().material.SetFloat("_yScale",r2/r1);
[0069] }
[0070] else {
[0071] GetComponent ().material.SetFloat("_xScale",r1/r2);
[0072] GetComponent ().material.SetFloat("_yScale",1.0f);
[0073] }
[0074] GetComponent ().material.SetMatrix("_MATRIX_MVP", MVP);
[0075] mCameraTexture = new Texture2D(128, 128, TextureFormat.RGB24, false);
[0076] image.CopyToTexture(mCameraTexture);
[0077] mCameraTexture.Apply(); // very important
[0078] // set camera texture
[0079] GetComponent ().material.mainTexture = mCameraTexture;
[0080] m_CanCapture = false;
[0081] }
[0082] Step 5: At the same time, it must be used with the Shader. The shader is mainly to make the recognized color into a texture and paste it to the model:
[0083] v2f o;
[0084] float4 uvTmp;
[0085] // calculate new uv in camera image
[0086] uvTmp = mul(_MATRIX_MVP, float4(v.texcoord.x-0.5f,v.texcoord.y-0.5f,0,1));
[0087] uvTmp.x = uvTmp.x/uvTmp.w;
[0088] uvTmp.y = uvTmp.y/uvTmp.w;
[0089] uvTmp.z = uvTmp.z/uvTmp.w;
[0090] // some swap for different coordinate system
[0091] uvTmp.x = (uvTmp.x + 1f)/2.0f;
[0092] uvTmp.y = (uvTmp.y + 1f)/2.0f;
[0093] o.uv.x = ((1-uvTmp.y)-0.5f)*_xScale+0.5f;
[0094] o.uv.y = ((1-uvTmp.x)-0.5f)*_yScale+0.5f;
[0095] //The position of the vertex should not be frozen, so use
[0096] //the standard UNITY_MATRIX_MVP matrix for that.
[0097] o.pos = mul(UNITY_MATRIX_MVP, v.vertex);
[0098] return o;
[0099] Through the above procedures, the predetermined technical indicators such as color recognition and lock interaction recognition can be realized.
[0100] The invention breaks the traditional flat book, realizes three-dimensional interactive animation, breaks through the traditional painting, and enables the drawn color to be presented in the three-dimensional animation character in real time.
[0101] The single painting form of traditional picture books can no longer satisfy the interests of contemporary children. The development of digital media technology and mobile devices makes children nowadays constantly seek audio-visual stimulation at every moment in their growth process. In this context, Interactive new media early education products came into being, and the birth of AR technology perfectly meets this demand. Using AR technology, children's creations in picture books can jump out of paper and be presented on mobile devices in the form of three-dimensional animations, which greatly increases children's learning motivation.
[0102] The specific implementation cases are as follows:
[0103] The "AR book" project of the present invention adopts the B/S mode, the interactive program is deployed on the server, and the user accesses the matching page of the AR e-book through the browser and the WEB browser. The program uses a smart phone, camera or projector to adjust the relative position of the camera and the book, place the book in the camera field of view, and display the original content of the paper book on the computer screen, as well as superimpose the corresponding pictures, images, and three-dimensional Virtual content such as models and sounds.
[0104] In the process of use, the application can conduct in-depth human-computer interaction according to requirements, such as controlling the size and position of the object, making the object move, making sounds, and so on.
[0105] Among them, AR books are composed of two parts, one is a paper book that integrates AR recognition mark patterns, and the other is an application that is responsible for AR functions. In addition to the general reading function of traditional books, the paper book part can also display a three-dimensional model with the cooperation of the application part. The AR application part is responsible for obtaining real images through the camera and presenting the fusion of paper books and virtual objects on the display device. In order to facilitate readers to use the augmented reality function, the application adopts the B/S mode. Readers can easily use AR books as long as they use smartphones, cameras, and projectors.
[0106] The augmented reality application part is responsible for obtaining the image data of the book through the smart phone, identifying the marking pattern from it, loading the corresponding 3D model from the mobile phone according to the marking pattern, and fusing the image of the real book and the 3D model, and finally presenting the realistic augmented reality image on On the reader's cell phone.
[0107] The AR application part uses C# as the development language, Vuforia SDK4.2 as the tracking library, Unity3D as the three-dimensional framework, and 3D MAX and Maya are responsible for model making and conversion calls.
[0108] Step 3: Use C# to perform the function in unity to realize the color extraction function,
[0109] Step 4: Code
[0110] CameraDevice cam = CameraDevice.Instance;
[0111] Image image = cam.GetCameraImage(m_PixelFormat);
[0112] if (image == null) {
[0113] // not on mobile devices
[0114] Debug.Log(m_PixelFormat + "image is not available yet");
[0115] }
[0116] else {
[0117] // get MVP matrix
[0118] Matrix4x4 P = GL.GetGPUProjectionMatrix(Camera.main.projectionMatrix, false);
[0119] Matrix4x4 V = Camera.main.worldToCameraMatrix;
[0120] Use ShaderLab to realize the fusion of real video images and virtual 3D models, as well as important functions such as image recognition and interaction.
[0121] Design of paper books: Design the content of paper books according to the reading characteristics of different customers.
[0122] For example, children: paper books should have bright colors, large and simple pictures, simple story lines, single and repetitive sentences. It is especially worth noting that the size of the words in the book must be large fonts. Generally speaking, the width of each word should be the same as the width of an adult man's index finger. As children grow older, the language and sentence patterns in the book can gradually increase in difficulty, and the words can gradually shrink.
[0123] In addition to traditional reading, the paper part of AR books is also responsible for providing clear and recognizable marking patterns for augmented reality applications. Only by carefully arranging the content of the book, the plane content and virtual model of the AR e-book can complement each other, and the unique advantages and charm of the AR e-book can be effectively used.
[0124] Three-dimensional model and multimedia resource design: books should be based on pictures. In AR books, the 3D model is responsible for creating virtual 3D information. Only realistic 3D objects can be better integrated into the real scene, giving people a sense of reality. Therefore, the design of 3D models is an important part of the design of augmented reality e-books.
[0125] The implementation process of the foregoing embodiment is as follows:
[0126] First, perform product design and make all original design pictures into recognition pictures, then use 3D modeling software to make 3D models, then import model textures and model files into Unity3D for functional realization, and use Vuforia SDK to realize color recognition and other functions Finally, use Unity3D to release the Android version of Apk, and then combine Xcode for APP release.
[0127] Among them, the main functions of the above APP are described as follows:
[0128] The first is to reasonably arrange and produce paper books, and integrate specific marking patterns into the books to meet the needs of augmented reality books;
[0129] The second is to use 3D modeling tools proficiently, through repeated and meticulous modifications, to make realistic 3D models;
[0130] The third is to use various multimedia software to edit other media resources;
[0131] The fourth is to complete the program development and implementation of various aspects of logo recognition, three-dimensional registration, virtual and real combination, and human-computer interaction on the selected platform. Among them, the program development based on the selected platform is the technical core of the entire AR book development process;
[0132] The fifth is to integrate the previous work to make the entire project an organic whole. At the same time, various feedback and adjustments are often made to the work of the previous stage at this stage.
[0133] The applicant believes that: books carry a lot of abstract experience, especially important ways and tools for children to learn. If you can effectively and reasonably incorporate specific experiences in traditional books, and at the same time increase the attractiveness of reading through colors, animations, sounds, interactions, etc. If you can watch the "real" scene described in the book, then such a book will definitely give readers a better reading interest and experience, more direct experience, and more effective learning.
[0134] From a technical perspective, augmented reality technology not only relies on electronic devices, but also on the interaction and integration of real scenes. It has been based on hearing and vision, adding visual, tactile and even adding various elements of taste to merge with each other, meeting various requirements for children to discover a new world in the process of growing up, and creating a virtual education with complete content and convenient experience to the greatest extent. Scene and background. The emergence of AR technology largely meets the needs of multimedia interactive teaching.
[0135] Model lock function innovation: Use Qualcomm SDK to identify the 3D model, then disable Qualcomm's own code ImagetargetBehiever, display the model on the screen interface, and turn on the camera as the background of the scene. If ImagetargetBehiever is available, the model can only be displayed when the card is recognized. For other similar products such as "Little Bear Neo-Pocket Zoo Card", it only has the lift function, and no lock function. This item can increase the interactive interaction.
[0136] Real-time color recognition innovation: other similar products mostly use card types such as "Little Bear Neo-Pocket Zoo Card" instead of books. The biggest difference is that the card is a pre-made character color, and only the card information needs to be recognized during the recognition process. The color uses the original texture file. And "Tu You Ji" uses real-time color recognition, draw any color at will, and the game program controls RGB color information. The analysis can display the drawing color in real time. Achieve a higher interactive effect.
[0137] The APP has implemented some other small functions, such as puzzle games, short voice stories and other functions will continue to be developed later. Realize APP customer groups, aiming at children's picture book growth records, online painting classroom, child care, prenatal education and other functions.
[0138] With the continuous increase of network bandwidth and the continuous popularity of smartphones, and the increasing number of browsers that support augmented reality technology, augmented technology is no longer a cool-looking household graphics and visualization peripheral technology, but has become the current technological development. Core. Augmented reality can effectively integrate additional information into the real world, making people's living environment more colorful. It is precisely because of this that new uses of augmented reality are constantly emerging. Augmented reality has expanded from the forefront of scientific and technological research to various fields such as business, technology, entertainment, marketing and education. Now, more and more game developers use augmented reality technology to create a brand new entertainment experience for players. Various companies dare not lag behind and have developed various brochures, product packaging and service terminals based on augmented reality.
[0139] Carrying out the research of this project, on the one hand, it has a broad market prospect for the upgrading and transformation of traditional picture books, and produces huge social and economic benefits. On the other hand, the product of this project can expand other children's maternal and child information, drawing learning, drawing growth management, and expanding auxiliary projects for the company's existing smart phone APP platform.
[0140] The preferred embodiments of the present invention are described in detail above, but the present invention is not limited to the specific details in the above-mentioned embodiments. Within the scope of the technical concept of the present invention, various equivalent transformations can be made to the technical solutions of the present invention. These equivalent transformations All belong to the protection scope of the present invention.
