PPT enhanced display system based on augmented reality

Abstract

The present invention provides a PPT enhanced display system based on augmented reality, which relates to the field of augmented reality technology. The system is deployed on the electronic device used by the user and includes a highlighted text recognition module, an enhanced display module and a model control module; the highlighted text recognition module realizes the recognition of highlighted text in the PPT; the enhanced display module displays an encyclopedia explanation of the highlighted text and superimposes the 3D model corresponding to the highlighted text in an AR manner to realize the enhanced display of the highlighted text; the model control module provides the user with a 3D model control function corresponding to the highlighted text; when the 3D model is presented in an AR manner on a plane in the real world pointed by the camera, the user can translate, scale and rotate the 3D model through touch gestures. The system can be used as an efficient means of information supplementation in educational and office scenarios. It is not only convenient and fast, and easy to operate for users, but also can provide a rich AR experience.

Description

Technical Field

[0001] The present invention relates to the field of augmented reality technology, and in particular to a PPT enhanced display system based on augmented reality. Background Art

[0002] Augmented Reality (AR) is a technology that superimposes virtual digital information such as text and images on the real world. This technology was first proposed in the 1990s with the goal of integrating virtual information into real scenes and enabling users to interact.

[0003] In recent years, the arrival of 5G and the advancement of technologies such as artificial intelligence, rendering computing, and sensory interaction have ushered in an unprecedented surge in the development of AR technology and related industries. With the improvement of terminal product computing performance, the applications of augmented reality are becoming increasingly diverse, with widespread application in numerous fields such as gaming, entertainment, education, office work, shopping, healthcare, and engineering. The high bandwidth and low latency of 5G networks will usher in new development directions for AR. The combination of AR technology and mobile devices can bring users more new services and experiences, and is being applied in an increasing number of scenarios and fields.

[0004] AR education, combined with AR technology, transforms traditional static images into dynamic, visual, and interactive three-dimensional audio and visual content, making learning more engaging. AR technology's ability to overlay virtual information onto the real world can transform previously boring teaching material into vivid images, stimulating student interest and enabling better learning. The coexistence of virtual objects and the real world allows for the visualization of complex spatial relationships and abstract concepts, allowing for the experience of phenomena impossible in the real world. It also allows for interaction with 2D or 3D composite objects in mixed reality. These educational advantages make AR a key emerging technology in the education sector.

[0005] With the continuous development of AR technology, accelerated AR development frameworks have begun to emerge, enabling rapid development and widespread application of AR technology. Vuforia is a software development kit for augmented reality applications on computing terminals and is a widely used AR development SDK. Vuforia offers a wide range of functions, such as image recognition, 3D model recognition, multi-target combination recognition, VuMark recognition, environmental recognition, plane detection, and aerial detection. It is also compatible with current mainstream PC, Android, and iOS platforms. Its high-quality recognition technology and excellent compatibility have made Vuforia a favorite among many AR application developers.

[0006] Currently, many companies both domestically and internationally are venturing into AR education, but most focus on early childhood education. Applications targeting higher grades are still limited, leaving many unmet needs. For example, in situations requiring PowerPoint presentations, such as classrooms, meetings, and academic presentations, speakers are often limited in length and can only present a condensed version of the content. Due to varying levels of knowledge, some listeners may not fully understand everything presented or may forget the definitions of certain concepts, especially highlighted terms, which require specific attention and understanding. The time-consuming and laborious process of searching for each term can easily distract attention and affect listening efficiency. Clearly, this scenario presents a need for real-time augmented information about the real world, ideally suited for AR technology. However, no current inventions address this need. Summary of the Invention

[0007] The technical problem to be solved by the present invention is to address the deficiencies of the above-mentioned existing technologies and provide a PPT enhanced display system based on augmented reality, which enables users to quickly identify highlighted words in PPT through mobile devices such as mobile phones, and provide them with two enhanced display functions of encyclopedia explanations and 3D models to help listeners obtain supplementary knowledge of the highlighted content in PPT in real time and efficiently, thereby achieving better listening effects.

[0008] In order to solve the above technical problems, the technical solution adopted by the present invention is: a PPT enhanced display system based on augmented reality is deployed on the electronic device used by the user, including a highlighted text recognition module, an enhanced display module and a model control module; the highlighted text recognition module completes three sub-functions of capturing the camera image, intercepting the highlighted area, and calling the Baidu OCR service to recognize the highlighted text, thereby realizing the recognition of the highlighted text; the enhanced display module completes three sub-functions of accessing the Baidu Encyclopedia Web service, displaying the encyclopedia explanation of the highlighted text on the screen, and overlaying the 3D model corresponding to the highlighted text in AR mode, thereby realizing the enhanced display of the highlighted text; the model control module provides the user with the 3D model manipulation function corresponding to the highlighted text; when the 3D model is presented in AR mode on the plane in the real world aimed at by the camera, the user translates, scales and rotates the 3D model through touch gestures.

[0009] The highlighted text recognition module recognizes the highlighted text in the highlighted area based on the Unity3D engine and the OpenCV library, including the following steps:

[0010] Step 1: Capture a screenshot of the camera screen of the electronic device used by the user;

[0011] Step 1.1: Import the Vuforia package into the Unity3D engine and add the ARCamera component in the Vuforia package to the current scene of the Unity3D engine. Use the GameObject.Find function provided by the Unity3D engine to locate the ARCamera component and obtain the camera component in the ARCamera component.

[0012] Step 1.2: Define the data structure RenderTexture in Unity3D and save the camera screen screenshot rendered by ARCamera to RenderTexture;

[0013] Step 1.2.1: Get the width and height of the screen of the electronic device currently used by the user, and use this width and height to initialize the RenderTexture type object rt;

[0014] Step 1.2.2: Set the ARCamera's target texture to rt, start ARCamera rendering, and capture the current ARCamera image and save it to the object rt;

[0015] Step 1.3: Read the captured current camera screenshot from RenderTexture;

[0016] Step 1.3.1: Activate RenderTexture so that the camera screenshots saved in it can be read;

[0017] Step 1.3.2: Initialize the Texture2D object texture using the width and height of the current user's electronic device screen, in RGB24 format;

[0018] Step 1.3.3: Call the ReadPixels function in the Texture2D class to read the camera screenshot saved in RenderTexture. The screenshot range is set to start from the lower left corner and extend to the entire screen size.

[0019] Step 1.3.4: After reading the screenshot, call the Apply function of the Texture2D class to save the pixel information of the screenshot;

[0020] Step 1.4: Restore the camera render target to the screen; set the ARCamera's target texture to null, that is, point it back to the screen, and deactivate the RenderTexture and delete the Rendertexture object;

[0021] Step 1.5: Call the texture2DToMat function in the OpenCV library to convert the camera screenshot texture to the Mat type;

[0022] Step 2: Process the camera screenshot to obtain the highlighted area;

[0023] Step 2.1: Create an initial mask of the highlighted area in the camera screenshot;

[0024] Step 2.1.1: Initialize the Mat type object image using the camera screenshot texture;

[0025] Step 2.1.2: Swap the R channel and the B channel in the image; call the split function in OpenCV to read the RGB channels of the image separately, and then reassemble the target image in the correct channel order;

[0026] Step 2.1.3: Convert the color space; use the color space conversion function CvtColor in the OpenCV library to convert the target image image from the RGB color space to the HSV color space to obtain image image1;

[0027] Step 2.1.4: Specify the color range [24(H), 41(S), 44(V)] to [36(H), 255(S), 255(V)] as the target color range;

[0028] Step 2.1.5: Preliminarily filter out the highlight areas in image1;

[0029] Call the color selection function inRange in the OpenCV library, specify the color range as the target color range described in 2.1.4, preliminarily filter out the highlight area in the image, and obtain the initial mask mask1 of the highlight area;

[0030] Step 2.2: De-noise the initial mask of the highlight area so that the mask accurately covers the highlight area;

[0031] Step 2.2.1: First median filtering: Use the median filter function medianBlur in the OpenCV library to perform the first filtering on mask1, and set the filter kernel size ksize to 11 to obtain mask2;

[0032] Step 2.2.2: Call the getStucturingElement function in the OpenCV library to define two kernels, kernel1 and kernel2. Kernel1 is larger than kernel2 and both are rectangular in shape.

[0033] Step 2.2.3: Dilation noise reduction: call the dilate function with kernel 1 in the OpenCV library to process mask2; dilate the white pixels inside the white rectangle in mask2, eliminate the black text noise inside most of the rectangular blocks, and obtain mask3;

[0034] Step 2.2.4: Second median filtering: Use the median filter function medianBlur in the OpenCV library to perform a second median filter on mask3, and set the filter size ksize to 19 to obtain mask4.

[0035] Step 2.2.5: Erosion recovery: Use the erode erosion function with kernel 2 in the OpenCV library to erode mask 4 and obtain the accurate mask mask 5 of the original highlight area;

[0036] Step 2.3: Use mask5 to obtain the highlighted area in the camera screenshot;

[0037] Step 2.3.1: Use the color space conversion function CvtColor to convert the target image from RGB color space to grayscale image to obtain image2;

[0038] Step 2.3.2: Use the bitwise_and function to AND mask5 with the grayscale image image2 to obtain the highlight area image resultimage. The resultimage only retains the highlight area in the original target image, and the rest is black.

[0039] Step 2.3.3: Convert the processed highlight area image resultimage of Mat type into a byte stream;

[0040] Call the matToTexture2D function in the OpenCV library to convert the result to the Texture2D type, and then call the EncodeToPNG function in the Texture2D class to convert the highlight area image resultimag into a byte stream;

[0041] Step 3: Use OCR to identify the highlighted text in the highlight area image resultimag;

[0042] Step 3.1: Initialize the OCR object client using the Baidu OCR account and password, and set the timeout;

[0043] Step 3.2: Read the highlighted area image resultimag in byte stream mode;

[0044] Step 3.3: The client calls the AccurateBasic function of Baidu text recognition, with the highlighted area image resultimag as the parameter, to obtain the result word of the highlighted text recognition;

[0045] Step 3.4: Analyze the highlighted text recognition result resultword to obtain the number of highlighted words and the highlighted word array in the target image;

[0046] Step 3.5: Determine the number of highlighted words; if the number of highlighted words is 0, a prompt box will pop up to remind the user to re-detect. If highlighted words are detected, a highlighted word list will be generated on the left side of the screen, and the first highlighted word in the list will be automatically enhanced. When the user clicks the button corresponding to a highlighted word, the enhanced display module will be called to enhance the display of the highlighted word.

[0047] The enhanced display module implements AR enhanced display of highlighted text based on Vuforia, including the following steps:

[0048] Step S1: Enhance the display of the encyclopedia explanation corresponding to the highlighted words;

[0049] Step S1.1: Obtain the Baidu Encyclopedia HTML source file corresponding to the highlighted word key;

[0050] Step S1.1.1: Import the HtmlAgilityPack library, create the Baidu Encyclopedia URI for the highlighted word key to be queried, and use the URI to create and initialize the HttpWebRequest object and set the timeout limit;

[0051] Step S1.1.2: Crawl the HTML source file of the highlighted word to be queried; use the HttpWebRequest object in the HtmlAgilityPack library to call the GetResponse function to obtain the Baidu Encyclopedia HTML source file corresponding to the keyword to be queried, and use the result to initialize the HttpWebResponse object;

[0052] Step S1.2: Parse the HTML source file to obtain the corresponding encyclopedia explanation of the keywords;

[0053] Step S1.2.1: Call the GetResponseStream function of the HttpWebResponse object and use the return result of the GetResponseStream function to initialize the StreamReader object, setting the format to UTF-8;

[0054] Step S1.2.2: Create an HtmlDocument object and call the Load function to load the data stream from the StreamReader object;

[0055] Step S1.2.3: Call the SelectNode function of the HtmlDocument object to select the description node in the HTML file and extract the content of the content attribute in the node, which is the encyclopedia explanation corresponding to the highlighted word to be queried;

[0056] Step S1.2.4: Call the Close and Dispose functions in the StreamReader class and the Close function in the HttpWebResponse class to close the data stream;

[0057] Step S1.3: Activate the text box component on the screen and replace the text in it with the encyclopedia explanation of the highlighted word, thereby achieving Baidu Encyclopedia enhanced display of the highlighted word key;

[0058] Step S2: Enhance and display the 3D model corresponding to the highlighted words;

[0059] Step S2.1: After loading the Vuforia package, import ARCamera, delete the original main camera, add the Ground Plane Stage and Plane Finder components, and set the detection mode to automatic detection;

[0060] Step S2.2: Traverse the highlighted word list. If a prefabricated 3D model of the highlighted word is stored locally, clear the previous enhancement content first, then activate the 3D model and set it as a child object of the Ground Plane Stage. If a prefabricated 3D model of the highlighted word is not stored locally, only the encyclopedia explanation corresponding to the highlighted word is displayed.

[0061] Step S2.3: When the user points the camera at a plane, the Vuforia package automatically detects the plane in the real world and pops up an anchor box. The user clicks the area where the anchor box is located to superimpose the 3D model on the plane, thereby enhancing the display of the highlighted words through the 3D model.

[0062] The model control module realizes control and interaction of the 3D model corresponding to the highlighted word through user touch gestures, including the following steps:

[0063] Step C1: Using the Unity engine, determine whether the user has touched the 3D model corresponding to the currently highlighted word;

[0064] Call the Input.GetMouseButton(0) function in the Unity engine to determine whether the 3D model corresponding to the highlighted word at the current moment has a touch screen action. If there is a touch screen action, obtain the number of touch points. If there is only one touch point, execute step C2 to rotate the model; if there are two touch points, execute step C3 to scale the model; if there are three touch points, execute step C4 to translate the model.

[0065] Step C2: realize 3D model rotation;

[0066] Step C2.1: Call the IsPointerOverGameObject function to determine whether the current touch point is on the UI control. If the touch point is on the UI control, do not scale. If not, execute step C2.2.

[0067] Step C2.2: Initialize the two-dimensional vector deltaPos using the incremental position of the touch point, call the Unity engine's rotation API transform.Rotate, set the parameters to rotate around the Y axis, and the rotation amount to the X-axis increment deltaPos,x, to implement the single-finger rotation model;

[0068] Step C3: Implement 3D model scaling;

[0069] Step C3.1: Define global variables oldTouch1 and oldTouch2 to record the old touch points, and call Input.GetTouch() to get two new touch points newTouch1 and newTouch2;

[0070] Step C3.2: Determine whether the new touch point is in the initial state. If it is in the initial state, indicating a new two-finger operation has occurred, update the global variables oldTouch1 and oldTouch2 to record the new touch point. If it is not in the starting state, proceed to step C3.3.

[0071] Step C3.3: Call the Vector2.Distance function to obtain the distance oldDistance between the two old touch points and the distance newDistance between the two new touch points, calculate the difference between the old and new distances, and divide it by the scale factor to obtain the scaling increment scaleFactor.

[0072] Step C3.4: Get the current scale of the 3D model and initialize a new Vector object using the x-axis, y-axis, and z-axis components of the current scale plus the scale increment to obtain the new scale of the 3D model;

[0073] Step C3.5: Call the IsPointerOverGameObject function to detect whether the current touch point is on a UI control. If it is on a UI control, no zoom action will be performed. If it is not on the UI, proceed to step C3.6;

[0074] Step C3.6: Determine whether the new scale of the 3D model is within the set allowable scale range. If so, change the current scale of the 3D model to the new scale to implement pinch-to-zoom. If not, do not zoom.

[0075] Step C3.7: Assign newTouch1 and newTouch2 to oldTouch1 and oldTouch2, and update the touch point position;

[0076] Step C4: realize 3D model translation;

[0077] Step C4.1: Call the IsPointerOverGameObject function to determine whether the current touch point is on the UI control. If the touch point is on the UI control, do not translate. If not, go to step C4.2.

[0078] Step C4.2: Use the incremental position of the touch point to initialize the two-dimensional vector deltaPos, call the Unity engine's translation API transform.Translate, and set the translation scale factor. The X-axis translation amount is the X-axis increment deltaPos, x multiplied by the scale factor, and the Y-axis translation amount is the Y-axis increment deltaPos, y multiplied by the scale factor, to implement the three-finger translation model.

[0079] The beneficial effects of adopting the above technical solution are: the PPT enhanced display system based on augmented reality provided by the present invention realizes the recognition of highlighted areas based on the Unity engine and OpenCV, and the recognition efficiency is high. After installing this system, the user only needs to point the camera at the highlighted word to view the encyclopedia explanation of the highlighted word and the 3D model presented in AR, which can meet the user's requirements for real-time acquisition of enhanced information. In addition, the user can also manipulate the 3D model presented in AR through touch screen gestures, which is more vivid and intuitive than traditional 2D information. This system can be used as an efficient means of information supplement in educational and office scenarios. It is not only convenient and fast, and easy to operate for users, but also can provide a rich AR experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 A structural block diagram of a PPT enhanced presentation system based on augmented reality provided by an embodiment of the present invention;

[0081] Figure 2A flowchart of highlight area recognition provided by an embodiment of the present invention;

[0082] Figure 3 A flowchart of a PPT enhanced presentation based on augmented reality provided by an embodiment of the present invention;

[0083] Figure 4 A flowchart of user activities in the augmented reality-based PPT enhanced presentation system provided by an embodiment of the present invention;

[0084] Figure 5 This is a schematic diagram of the actual application effect of the augmented reality-based PPT enhanced presentation system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0085] The following embodiments of the present invention are described in further detail with reference to the accompanying drawings and examples. The following examples are used to illustrate the present invention but are not intended to limit the scope of the present invention.

[0086] In this embodiment, the PPT enhanced display system based on augmented reality is deployed on the electronic devices used by the user (mobile phone, tablet computer, etc.), such as Figure 1 As shown, it includes a highlighted text recognition module, an enhanced display module and a model control module; the highlighted text recognition module completes three sub-functions of capturing camera images, intercepting highlighted areas, and calling Baidu OCR (Optical Character Recognition) service to recognize highlighted text, thereby realizing the recognition of highlighted text; the enhanced display module completes three sub-functions of accessing Baidu Encyclopedia Web service, displaying the encyclopedia explanation of highlighted text on the screen, and overlaying and displaying the 3D model corresponding to the highlighted text in AR mode, thereby realizing the enhanced display of highlighted text; the model control module provides users with the 3D model manipulation function corresponding to the highlighted text; when the 3D model is presented in AR mode on a plane in the real world that the camera is aimed at, the user translates, scales and rotates the 3D model through touch gestures.

[0087] In this embodiment, the highlighted text recognition module recognizes the highlighted text in the highlighted area based on the Unity3D engine and the OpenCV library, including the following steps:

[0088] Step 1: Capture a screenshot of the camera screen of the electronic device used by the user. In this embodiment, the implementation of this function can be encapsulated as a CaptureScreen function;

[0089] Step 1.1: Import the Vuforia package into the Unity3D engine and add the ARCamera component in the Vuforia package to the current scene of the Unity3D engine. Use the GameObject.Find function provided by the Unity3D engine to locate the ARCamera component and obtain the camera component.

[0090] Step 1.2: Define the data structure RenderTexture in Unity3D and save the image rendered by ARCamera (camera screen screenshot) to RenderTexture;

[0091] Step 1.2.1: Get the width and height of the screen of the electronic device currently used by the user, and use this width and height to initialize the RenderTexture type object rt;

[0092] Step 1.2.2: Set the ARCamera's target texture to object rt, start ARCamera rendering, and capture the current ARCamera image and save it to rt;

[0093] Step 1.3: Read the captured current camera screenshot from RenderTexture;

[0094] Step 1.3.1: Activate RenderTexture so that the camera screenshots saved in it can be read;

[0095] Step 1.3.2: Initialize the Texture2D object texture using the width and height of the current user's electronic device screen, in RGB24 format;

[0096] Step 1.3.3: Call the ReadPixels function in the Texture2D class to read the camera screenshot saved in RenderTexture. The screenshot range is set to start from the lower left corner and extend to the entire screen size.

[0097] Step 1.3.4: After reading the screenshot, call the Apply function of the Texture2D class to save the pixel information of the screenshot;

[0098] Step 1.4: Restore the camera render target to the screen; set the ARCamera's target texture to null, that is, point it back to the screen, and deactivate the RenderTexture and delete the Rendertexture object;

[0099] Step 1.5: Call the texture2DToMat function in the OpenCV library to convert the camera screenshot texture to the Mat type;

[0100] Step 2: Perform image processing on the camera screenshot to obtain the highlight area. In this embodiment, the algorithm of this part can be encapsulated as a CutHighlight function;

[0101] The present invention implements an image processing algorithm based on OpenCV to accurately obtain the highlight (bright yellow) area in the PPT image. The algorithm is inspired by the mask idea of the traditional craft industry. By performing a series of processing on the original image, a black and white binary image of the same size as the original image is obtained. This image is the mask. The white area in the mask corresponds to the highlight area in the original image, and the rest is black. By making an accurate mask of the highlight area, and then using the mask to perform AND operation with the original image, the highlight area in the image is extracted. The image highlight area recognition algorithm flow chart is shown as follows: Figure 2 shown.

[0102] Step 2.1: Create an initial mask of the highlighted area in the camera screenshot;

[0103] The yellow area in the image is roughly screened as the initial mask mask1 of the image.

[0104] Step 2.1.1: In this embodiment, call the function CaptureScreen described in step 1 to capture the camera screen screenshot, and use the camera screen screenshot texture to initialize the Mat type object image;

[0105] Step 2.1.2: Swap the R and B channels in the image. We found that the R and B channel data in the image were reversed after capturing the screenshot. We called the split function in OpenCV to read the RGB channels of the image separately, and then reassembled the target image in the correct channel order.

[0106] Step 2.1.3: Convert the color space; use the color space conversion function CvtColor in the OpenCV library to convert the target image image from the RGB color space to the HSV color space to obtain image image1, which is used to subsequently preliminarily filter out the highlight areas in the image;

[0107] Step 2.1.4: Specify the color range [24(H), 41(S), 44(V)] to [36(H), 255(S), 255(V)] as the target color range; the yellow range in the HSV color space is between [26(H), 43(S), 46(V)] and [34(H), 255(S), 255(V)]. To improve the robustness of the algorithm under noise such as lighting, environment, and moiré, the yellow subspace in the HSV space is appropriately expanded.

[0108] Step 2.1.5: Preliminarily filter out the highlight areas in image1;

[0109] Call the color selection function inRange in the OpenCV library. The formula of the inRange function is as follows. The color range is specified as the target color range described in 2.1.4. The highlight area in the image is initially screened out to obtain the initial mask mask1 of the highlight area.

[0110] dst(I)=lowerb(I)0≤src(I)0≤upperb(I)0∩lowerb(I)1≤src(I)1

[0111] ≤upperb(I)1∩lowerb(I)2≤src(I)2≤upperb(I)2

[0112] Where I is the pixel in image1, src(I) i Represents the value of pixel I in the i-th channel, dst(I) represents the pixel value of pixel I after processing, lowerb(I) i Represents the i-th component of the lower bound of the color of pixel I, upperb(I) i The i-th component of the upper bound of the color of pixel I, where i = H, S, or V;

[0113] The function inRange sets the pixel value of a pixel to true (white) if it is between [26(H),43(S),46(V)] and [34(H),255(S),255(V)], otherwise it is set to false (black). After processing the target image, a binary image is obtained. The position corresponding to the highlight area in the original image is roughly a white rectangle, and the rest of the image is black except for the noise points. This is the initial mask mask1 of the highlight area.

[0114] Step 2.2: De-noise the initial mask of the highlight area so that the mask accurately covers the highlight area;

[0115] The locations corresponding to the highlighted areas in the original image in mask1 are roughly represented by multiple white rectangles (because the highlighted text in the original image is a bright yellow rectangle). Furthermore, in addition to background noise, since the text in the highlighted areas of the original image is generally black, each white rectangle in mask1 contains noise left by the black text. Noise reduction requires removing both the background noise and the text noise within the white rectangles.

[0116] Step 2.2.1: First median filtering: Use the medianBlur function in the OpenCV library to perform the first median filtering on mask1. Set the filter kernel size ksize to 11 to obtain mask2, which preliminarily removes smaller noise points in the background. The matrix representation of the kernel used by the medianBlur function during the first median filtering is shown below.

[0117]

[0118] Among them, K is an 11-order square matrix, ksize.width and ksize.height are the length and width of the kernel used by the median filter function medianBlur, respectively, both of which are 11;

[0119] Step 2.2.2: Call the getStucturingElement function in the OpenCV library to define two kernels, kernel1 and kernel2. Kernel1 is slightly larger than kernel2. In this embodiment, the kernel sizes are (20, 20) and (19, 19), respectively, and both are rectangular in shape. They are used for the subsequent dilation and corrosion noise reduction operations.

[0120] Step 2.2.3: Dilation noise reduction; the dilation function dilate provides the function of expanding the white area in the image. Call the dilation function dilate with kernel kernel1 in the OpenCV library to process mask2; the white pixels inside the white rectangle in mask2 are expanded, and the black text noise inside most of the rectangular blocks is eliminated to obtain mask3. The dilation function dilate is shown in the following formula;

[0121] dst(x,y)=max (x′,y′):element(x′,y′)≠0 src(x+x′,y+y′)

[0122] Where dst(x,y) represents the pixel value at coordinate (x,y) after processing, element(x′,y′) represents the value at coordinate (x′,y′) in kernel kernel1, which is determined by the size and shape of the kernel. src(x+x′,y+y′) represents the pixel value at coordinate (x+x′,y+y′) before processing. This function sets the value of a pixel to the maximum pixel value among nearby pixels, thereby expanding the white area and shrinking the black area.

[0123] Step 2.2.4: Second median filtering: Use the medianBlur function in the OpenCV library to perform a second median filter on mask3, with the filter size ksize set to 19, to obtain mask4. During the second median filtering, the matrix representation of the kernel used by the medianBlur function is shown below.

[0124]

[0125] Among them, K2 is a 19-order square matrix. ksize2.width and ksize2.height are the length and width of the kernel used by the medianBlur filter function, respectively, and both are 19. This step can eliminate the very few black noise points inside the white square that may not have been eliminated, to prevent them from expanding back to their original size after the next erosion operation, rendering the entire noise reduction process ineffective;

[0126] Step 2.2.5: Erosion recovery; the erode erosion function provides the function of expanding the black area in the image. Use the erode erosion function with kernel 2 in the OpenCV library to erode the mask mask4 to obtain the accurate mask mask5 of the original highlight area;

[0127] The erosion function erode is shown in the following formula;

[0128] dst(x,y)=min (x′,y′):element(x′,y′)≠0 src(x+x′,y+y′)

[0129] Here, dst(x,y) represents the pixel value at coordinates (x,y) after processing, element(x′,y′) represents the value at coordinates (x′,y′) within the kernel, which is determined by the size and shape of the kernel, and src(x+x′,y+y′) represents the pixel value at coordinates (x+x′,y+y′) before processing. This function sets the value of a pixel to the minimum value among its neighboring pixels, thereby expanding black areas and shrinking white areas.

[0130] Experiments show that the edges of the white rectangles in the initial mask are often damaged due to factors such as lighting. Therefore, kernel2 is set slightly smaller than kernel1. After the erosion operation, the white area originally expanded by the dilation function is basically restored to the size of the original highlight area. At the same time, the damaged edges can be filled, making the final mask of the original highlight area complete and accurate.

[0131] Step 2.3: Use mask5 to obtain the highlighted area in the camera screenshot;

[0132] Step 2.3.1: Use the color space conversion function CvtColor to convert the target image from RGB color space to grayscale image to obtain image2;

[0133] Step 2.3.2: Use the bitwise_and function to AND mask5 with the grayscale image image2 to obtain the highlight area image resultimage. The resultimage only retains the highlight area in the original target image, and the rest is black.

[0134] Step 2.3.3: Convert the processed highlight area image resultimage of Mat type into a byte stream;

[0135] Since the subsequent OCR reads the image byte stream and there is no function to directly convert the Mat type to a byte stream, the matToTexture2D function in the OpenCV library is called first to convert the result to the Texture2D type, and then the EncodeToPNG function in the Texture2D class is called to convert the highlight area image resultimag into a byte stream;

[0136] Step 3: Use OCR to identify the highlighted text in the highlight area image resultimag;

[0137] Step 3.1: Initialize the OCR object client using the Baidu OCR account and password, and set the timeout to 60000ms;

[0138] Step 3.2: Call the highlight area processing function CutHighlight in step 2 to obtain the processed target image, and read the highlight area image resultimag in byte stream mode;

[0139] Step 3.3: The client calls the AccurateBasic function of Baidu text recognition with the parameter resultimag to obtain the result word of the highlighted text recognition.

[0140] Step 3.4: Analyze the text recognition result resultword to obtain the number of highlighted words and the highlighted word array in the target image;

[0141] Step 3.5: Determine the number of highlighted words; if the number of highlighted words is 0, a prompt box will pop up to remind the user to re-detect; if a highlighted word is detected, a list of highlighted words will be generated on the left side of the screen, and the first highlighted word in the list will be automatically enhanced; when the user clicks the button corresponding to a highlighted word, the enhanced display module will be called to enhance the display of the highlighted word;

[0142] In this embodiment, the enhanced display module implements AR enhanced display of highlighted text based on Vuforia, including the following steps:

[0143] Step S1: Enhance the display of the encyclopedia explanation corresponding to the highlighted words. This part of the algorithm is encapsulated in the function TextAugment;

[0144] Step S1.1: Obtain the Baidu Encyclopedia HTML source file corresponding to the highlighted word key;

[0145] Step S1.1.1: Import the HtmlAgilityPack library, create the Baidu Encyclopedia URI for the highlighted word key to be queried, and use the URI to create and initialize an HttpWebRequest object, setting the timeout limit to 10000ms;

[0146] Step S1.1.2: Crawl the HTML source file of the highlighted word to be queried; use the HttpWebRequest object in the HtmlAgilityPack library to call the GetResponse function to obtain the Baidu Encyclopedia HTML source file corresponding to the keyword to be queried, and use the result to initialize the HttpWebResponse object;

[0147] Step S1.2: Parse the HTML source file to obtain the corresponding encyclopedia explanation of the keywords;

[0148] Step S1.2.1: Call the GetResponseStream function of the HttpWebResponse object and use the return result of the GetResponseStream function to initialize the StreamReader object, setting the format to UTF-8;

[0149] Step S1.2.2: Create an HtmlDocument object and call the Load function to load the data stream from the StreamReader object;

[0150] Step S1.2.3: Call the SelectNode function of the HtmlDocument object to select the description node in the HTML file and extract the content of the content attribute in the node, which is the encyclopedia explanation corresponding to the highlighted word to be queried;

[0151] Step S1.2.4: Call the Close and Dispose functions in the StreamReader class and the Close function in the HttpWebResponse class to close the data stream;

[0152] Step S1.3: Activate the text box component on the screen and replace the text in it with the encyclopedia explanation of the highlighted word, thereby achieving Baidu Encyclopedia enhanced display of the highlighted word key;

[0153] Step S2: Enhance and display the 3D model corresponding to the highlighted words;

[0154] Step S2.1: After loading the Vuforia package, import ARCamera, delete the original main camera, add the Ground Plane Stage and Plane Finder components, and set the detection mode to automatic detection;

[0155] Step S2.2: Traverse the highlighted word list. If a prefabricated 3D model of the highlighted word is stored locally, clear the previous enhancement content first, then activate the 3D model and set it as a child object of the Ground Plane Stage.

[0156] Step S2.3: When the user points the camera at a plane, the Vuforia package automatically detects the plane in the real world and pops up an anchor box. The user clicks the area where the anchor box is located to superimpose the 3D model on the plane, thereby enhancing the display of the highlighted words through the 3D model.

[0157] In this embodiment, the model control module implements control and interaction of the 3D model corresponding to the highlighted word through user touch gestures, including the following steps:

[0158] Step C1: Using the Unity engine, determine whether there is a touch screen action on the 3D model corresponding to the currently highlighted word;

[0159] Call the Input.GetMouseButton(0) function in the Unity engine to determine whether the 3D model corresponding to the highlighted word at the current moment has a touch screen action. If there is a touch screen action, obtain the number of touch points. If there is only one touch point, execute step C2 to rotate the model; if there are two touch points, execute step C3 to scale the model; if there are three touch points, execute step C4 to translate the model.

[0160] Step C2: realize 3D model rotation;

[0161] Step C2.1: Call the IsPointerOverGameObject function to determine whether the current touch point is on the UI control. If the touch point is on the UI control, do not scale. If not, execute step C2.2.

[0162] Step C2.2: Initialize the two-dimensional vector deltaPos using the incremental position of the touch point, call the Unity engine's rotation API transform.Rotate, set the parameters to rotate around the Y axis, and the rotation amount to the X-axis increment deltaPos,x, to implement the single-finger rotation model;

[0163] Step C3: Implement 3D model scaling;

[0164] Step C3.1: Define global variables oldTouch1 and oldTouch2 to record the old touch points, and call Input.GetTouch() to get two new touch points newTouch1 and newTouch2;

[0165] Step C3.2: Determine whether the new touch point is in the initial state. If it is in the initial state, indicating a new two-finger operation has occurred, update the global variables oldTouch1 and oldTouch2 to record the new touch point. If it is not in the starting state, proceed to step C3.3.

[0166] Step C3.3: Call the Vector2.Distance function to obtain the distance oldDistance between the two old touch points and the distance newDistance between the two new touch points, calculate the difference between the old and new distances, and divide it by the scale factor to obtain the scaling increment scaleFactor.

[0167] Step C3.4: Get the current scale of the 3D model, and initialize a new Vector object using the x-axis, y-axis, and z-axis components of the current scale plus the scale increment to obtain the new scale of the 3D model;

[0168] Step C3.5: Call the IsPointerOverGameObject function to detect whether the current touch point is on a UI control. If it is on a UI control, no zoom action will be performed. If it is not on the UI, proceed to step C3.6;

[0169] Step C3.6: Determine whether the new scale of the 3D model is within the set allowable scale range. If so, change the current scale of the 3D model to the new scale to implement pinch-to-zoom. If not, do not zoom.

[0170] Step C3.7: Assign newTouch1 and newTouch2 to oldTouch1 and oldTouch2, and update the touch point position;

[0171] Step C4: realize 3D model translation;

[0172] Step C4.1: Call the IsPointerOverGameObject function to determine whether the current touch point is on the UI control. If the touch point is on the UI control, do not translate. If not, go to step C4.2.

[0173] Step C4.2: Use the incremental position of the touch point to initialize the two-dimensional vector deltaPos, call the Unity engine's translation API transform.Translate, and set the translation scale factor. The X-axis translation amount is the X-axis increment deltaPos, x multiplied by the scale factor, and the Y-axis translation amount is the Y-axis increment deltaPos, y multiplied by the scale factor, to implement the three-finger translation model.

[0174] In this embodiment, the process of enhancing the display of highlighted words in PPT is as follows: Figure 3 、 4 As shown in the figure, the system interface of the PPT enhanced display system based on augmented reality is as follows Figure 5 As shown, after the system is started, the initial UI interface includes a detection button at the bottom of the screen. The user points the camera at the PPT and clicks this button to recognize the highlighted words. After the system recognition is completed, an encyclopedia explanation text box appears on the left side of the screen, and the encyclopedia explanation corresponding to the first highlighted word recognized is displayed by default. A list of highlighted words appears on the right side of the screen, and each highlighted word corresponds to a button. When the user clicks a different button, the text box will then display the encyclopedia explanation corresponding to the highlighted word selected by the user. At the same time, the screen can also enhance the display of the 3D model corresponding to the highlighted word, and the user can control and interact with the 3D model.

[0175] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A PPT enhanced presentation system based on augmented reality, deployed on an electronic device used by a user, characterized by: It includes a highlighted text recognition module, an enhanced display module, and a model control module. The highlighted text recognition module completes three sub-functions: capturing camera images, intercepting highlighted areas, and calling Baidu OCR service to recognize highlighted text, thereby realizing the recognition of highlighted text. The enhanced display module completes three sub-functions: accessing Baidu Baike Web service, displaying the encyclopedia explanation of the highlighted text on the screen, and overlaying the 3D model corresponding to the highlighted text in AR mode, thereby realizing the enhanced display of the highlighted text. The model control module provides users with the function of manipulating the 3D model corresponding to the highlighted text. When the 3D model is presented in AR mode on a plane in the real world that the camera is pointed at, the user controls and interacts with the 3D model through touch gestures. The highlighted text recognition module recognizes the highlighted text in the highlighted area based on the Unity3D engine and the OpenCV library, specifically including the following steps: Step 1: Use the Unity3D engine to capture the camera screenshot of the electronic device used by the user and convert it into Mat type; Step 2: Process the camera screenshot to obtain the highlighted area; Step 2.1: Take a screenshot of the camera image, preliminarily filter out the highlight area, and obtain the initial mask of the highlight area; Step 2.1.1: Initialize the Mat type object image using the camera screenshot texture; Step 2.1.2: Swap the R channel and the B channel in the image; call the split function in OpenCV to read the RGB channels of the image separately, and then reassemble the target image in the correct channel order; Step 2.1.3: Convert the color space; use the color space conversion function CvtColor in the OpenCV library to convert the target image image from the RGB color space to the HSV color space to obtain image image1; Step 2.1.4: Specify the color range [24(H), 41(S), 44(V)] to [36(H), 255(S), 255(V)] as the target color range; Step 2.1.5: Preliminarily filter out the highlight areas in image1; Call the color selection function inRange in the OpenCV library, specify the color range as the target color range described in 2.1.4, preliminarily filter out the highlight area in the image, and obtain the initial mask mask1 of the highlight area; Step 2.2: Denoise the initial mask of the preliminarily screened highlight area to obtain a precise mask of the preliminarily screened highlight area, so that the precise mask accurately covers the highlight area; Step 2.2.1: First median filtering: Use the median filter function medianBlur in the OpenCV library to perform the first filtering on mask1, and set the filter kernel size ksize to 11 to obtain mask2; Step 2.2.2: Call the getStucturingElement function in the OpenCV library to define two kernels, kernel1 and kernel2. Kernel1 is larger than kernel2 and both are rectangular in shape. Step 2.2.3: Dilation noise reduction: call the dilate function with kernel 1 in the OpenCV library to process mask2; dilate the white pixels inside the white rectangle in mask2, eliminate the black text noise inside most of the rectangular blocks, and obtain mask3; Step 2.2.4: Second median filtering: Use the median filter function medianBlur in the OpenCV library to perform a second median filter on mask3, and set the filter size ksize to 19 to obtain mask4. Step 2.2.5: Erosion recovery: Use the erode erosion function with kernel 2 in the OpenCV library to erode mask 4 and obtain the accurate mask mask 5 of the original highlight area; Step 2.3: Use the precise mask to obtain the highlighted area in the camera screenshot and obtain the highlighted area image; Step 2.3.1: Use the color space conversion function CvtColor to convert the target image from RGB color space to grayscale image to obtain image2; Step 2.3.2: Use the bitwise_and function to AND mask5 with the grayscale image image2 to obtain the highlight area image resultimage. The resultimage only retains the highlight area in the original target image, and the rest is black. Step 2.3.3: Convert the processed highlight area image resultimage of Mat type into a byte stream; Call the matToTexture2D function in the OpenCV library to convert the result to the Texture2D type, and then call the EncodeToPNG function in the Texture2D class to convert the highlight area image resultimag into a byte stream; Step 3: Use OCR to identify the highlighted text in the highlighted area image.

2. The augmented reality-based PPT enhanced presentation system according to claim 1, characterized in that: The enhanced display module implements AR enhanced display of highlighted text based on Vuforia, and enhances the display of encyclopedia explanations and 3D models corresponding to the highlighted words.

3. The augmented reality-based PPT enhanced presentation system according to claim 2, characterized in that: The specific method of step 3 is: Step 3.1: Initialize the OCR object client using the Baidu OCR account and password, and set the timeout; Step 3.2: Read the highlighted area image resultimag in byte stream mode; Step 3.3: The client calls the AccurateBasic function of Baidu text recognition, with the highlighted area image resultimag as the parameter, to obtain the result word of the highlighted text recognition; Step 3.4: Analyze the highlighted text recognition result resultword to obtain the number of highlighted words and the highlighted word array in the target image; Step 3.5: Determine the number of highlighted words; if the number of highlighted words is 0, a prompt box will pop up to remind the user to re-detect; if highlighted words are detected, a list of highlighted words will be generated on the left side of the screen, and the first highlighted word in the list will be automatically enhanced; When the user clicks the button corresponding to a highlighted word, the enhanced display module is called to perform enhanced display on the highlighted word.

4. The augmented reality-based PPT enhanced presentation system according to claim 3, characterized in that: The enhanced display module implements AR enhanced display of highlighted text based on Vuforia, including the following steps: Step S1: Enhance the display of the encyclopedia explanation corresponding to the highlighted words; Step S1.1: Obtain the Baidu Encyclopedia HTML source file corresponding to the highlighted word key; Step S1.1.1: Import the HtmlAgilityPack library, create the Baidu Encyclopedia URI for the highlighted word key to be queried, and use the URI to create and initialize the HttpWebRequest object and set the timeout limit; Step S1.1.2: Crawl the HTML source file of the highlighted word to be queried; use the HttpWebRequest object in the HtmlAgilityPack library to call the GetResponse function to obtain the Baidu Encyclopedia HTML source file corresponding to the keyword to be queried, and initialize the HttpWebResponse object with the Baidu Encyclopedia HTML source file corresponding to the keyword to be queried; Step S1.2: Parse the HTML source file to obtain the corresponding encyclopedia explanation of the keywords; Step S1.2.1: Call the GetResponseStream function of the HttpWebResponse object and use the return result of the GetResponseStream function to initialize the StreamReader object, setting the format to UTF-8; Step S1.2.2: Create an HtmlDocument object and call the Load function to load the data stream from the StreamReader object; Step S1.2.3: Call the SelectNode function of the HtmlDocument object to select the description node in the HTML file and extract the content of the content attribute in the node, which is the encyclopedia explanation corresponding to the highlighted word to be queried; Step S1.2.4: Call the Close and Dispose functions in the StreamReader class and the Close function in the HttpWebResponse class to close the data stream; Step S1.3: Activate the text box component on the screen and replace the text in it with the encyclopedia explanation of the highlighted word, thereby achieving Baidu Encyclopedia enhanced display of the highlighted word key; Step S2: Enhance and display the 3D model corresponding to the highlighted words; Step S2.1: After loading the Vuforia package, import ARCamera, delete the original main camera, add the GroundPlane Stage and Plane Finder components, and set the detection mode to automatic detection; Step S2.2: Traverse the highlighted word list. If a prefabricated 3D model of the highlighted word is stored locally, clear the previous enhancement content first, then activate the 3D model and set it as a child object of the Ground Plane Stage. If a prefabricated 3D model of the highlighted word is not stored locally, only the encyclopedia explanation corresponding to the highlighted word is displayed. Step S2.3: When the user points the camera at a plane, the Vuforia package automatically detects the plane in the real world and pops up a prompt anchor box. The user clicks the area where the anchor box is located to superimpose the 3D model on the plane, thereby enhancing the display of the highlighted words through the 3D model.

5. The augmented reality-based PPT enhanced presentation system according to any one of claims 2 to 4, characterized in that: The model control module uses the Unity engine to determine whether the user has touched the 3D model corresponding to the currently highlighted word. If so, the model control interaction is performed based on the number of touch points. The specific method is as follows: Step C1: Using the Unity engine, determine whether the user has touched the 3D model corresponding to the currently highlighted word; Call the Input.GetMouseButton(0) function in the Unity engine to determine whether the 3D model corresponding to the highlighted word at the current moment has a touch screen action. If there is a touch screen action, obtain the number of touch points. If there is only one touch point, execute step C2 to rotate the model; if there are two touch points, execute step C3 to scale the model; if there are three touch points, execute step C4 to translate the model. Step C2: realize 3D model rotation; Step C2.1: Call the IsPointerOverGameObject function to determine whether the current touch point is on the UI control. If the touch point is on the UI control, do not scale. If not, execute step C2.

2. Step C2.2: Initialize the two-dimensional vector deltaPos using the incremental position of the touch point, call the Unity engine's rotation API transform.Rotate, set the parameters to rotate around the Y axis, and the rotation amount to the X-axis increment deltaPos,x, to implement the single-finger rotation model; Step C3: Implement 3D model scaling; Step C3.1: Define global variables oldTouch1 and oldTouch2 to record the old touch points, and call Input.GetTouch() to get two new touch points newTouch1 and newTouch2; Step C3.2: Determine whether the new touch point is in the initial state. If it is in the initial state, indicating a new two-finger operation has occurred, update the global variables oldTouch1 and oldTouch2 to record the new touch point. If it is not in the starting state, proceed to step C3.

3. Step C3.3: Call the Vector2.Distance function to obtain the distance oldDistance between the two old touch points and the distance newDistance between the two new touch points, calculate the difference between the old and new distances, and divide it by the scale factor to obtain the scaling increment scaleFactor. Step C3.4: Get the current scale of the 3D model and initialize a new Vector object using the x-axis, y-axis, and z-axis components of the current scale plus the scale increment to obtain the new scale of the 3D model; Step C3.5: Call the IsPointerOverGameObject function to detect whether the current touch point is on a UI control. If it is on a UI control, no zoom action will be performed. If it is not on the UI, proceed to step C3.6; Step C3.6: Determine whether the new scale of the 3D model is within the set allowable scale range. If so, change the current scale of the 3D model to the new scale to implement pinch-to-zoom. If not, do not zoom. Step C3.7: Assign newTouch1 and newTouch2 to oldTouch1 and oldTouch2, and update the touch point position; Step C4: realize 3D model translation; Step C4.1: Call the IsPointerOverGameObject function to determine whether the current touch point is on the UI control. If the touch point is on the UI control, do not translate. If not, go to step C4.

2. Step C4.2: Use the incremental position of the touch point to initialize the two-dimensional vector deltaPos, call the Unity engine's translation API transform.Translate, and set the translation scale factor. The X-axis translation amount is the X-axis increment deltaPos, x multiplied by the scale factor, and the Y-axis translation amount is the Y-axis increment deltaPos, y multiplied by the scale factor, to implement the three-finger translation model.

Images