Augmented reality-ai integrated interactive learning assistance method

Abstract

The application provides an AR-AI integrated interactive learning assistance method, and relates to the technical field of intelligent teaching; the method comprises the following steps: acquiring an image containing a question content and segmenting the image to obtain at least one segmentation region; acquiring text line data in the at least one segmentation region through an image recognition technology; separately extracting text lines with different features; acquiring a voice control instruction and selecting at least one text content for recognition and analysis; the application can ensure the correctness of image recognition by performing spot checks on the text recognition result on the basis of a basic text recognition algorithm, and re-recognizing the text recognition result by using an improved image recognition algorithm according to the correctness of the text recognition result, so that the correctness of image recognition can be ensured, the text input by the user can be accurately recognized when the user is assisted in learning, the accuracy and efficiency of recognition are ensured, the use experience of the user is improved, and the user is effectively guided.

Description

Technical Field

[0001] This invention relates to the field of intelligent teaching technology, and in particular to an AR-AI integrated interactive learning assistance method. Background Technology

[0002] With the rapid development of mobile internet technology, photo recognition technology, as an efficient and convenient way to obtain information, has been widely used in various fields. In multiple scenarios such as education, life services, and medical consultation, photo recognition-based applications capture the content of images taken by users and use advanced image processing and natural language processing technologies to quickly identify and analyze the information in the images, thereby providing users with corresponding answers or solutions. These applications have greatly enriched the channels for obtaining information and improved the efficiency of problem-solving.

[0003] In complex image environments, especially when text and images overlap, the accuracy of existing image recognition technologies is often severely affected. Because text in an image may be interfered with by various factors such as background patterns, colors, and lighting conditions, the recognition algorithm has difficulty in accurately distinguishing and extracting effective text information. In addition, the close overlap of text and images may also cause confusion in the recognition area, further increasing the difficulty and error rate of recognition. This lack of accuracy in recognizing text and images in overlapping situations directly limits the application and development of image recognition technology in a wider range of scenarios.

[0004] To address the issues of interactivity and accuracy in existing photo recognition applications, particularly the challenge of recognizing overlapping text and images, technological innovation and optimization are necessary. Therefore, this invention provides an improved learning assistance system. Through an improved text recognition algorithm and process, it can accurately recognize text even when users are taking notes on books or other text-based media where text overlaps with other text, ensuring accuracy and efficiency, enhancing the user experience, and providing effective guidance. Summary of the Invention

[0005] To overcome the problem of low success rate and accuracy in recognizing overlapping characters in existing text recognition processes, this invention provides an AR-AI integrated interactive learning assistance method.

[0006] To solve the above-mentioned technical problems, the technical solution provided by the present invention is as follows:

[0007] AR-AI integrated interactive learning assistance method includes the following steps:

[0008] S11: Obtain an image containing the question content and segment the image to obtain at least one segmented region;

[0009] S12: Recognize the text texture in at least one segmented region using image recognition technology, and obtain text texture data in at least one segmented region;

[0010] S13: Extract features from text patterns of different characteristics using feature recognition and classification techniques to obtain at least one text content;

[0011] S14: Obtain the user's voice command information data, recognize the user's voice command data, and obtain voice control commands;

[0012] S15: Based on the recognition results of the voice command data, select at least one text content for recognition and analysis;

[0013] S16: Based on the recognition and analysis results of the text content, provide auxiliary guidance for the user's learning through at least one of the following formats: voice, text, video, and image.

[0014] In any of the above solutions, a preferred embodiment involves the following steps when using image recognition technology to identify text patterns within at least one segmented region and obtain text pattern data within the segmented region:

[0015] S21: Image preprocessing enhances text information in the image, making the text clearer and more legible;

[0016] S22: Feature extraction: Using edge detection and texture analysis techniques, the features of text in the image are extracted to obtain comprehensive text feature data;

[0017] S23: Image segmentation, which involves analyzing connected components in an image to find the outline of text and segmenting it based on texture features;

[0018] S24: Image post-processing, further optimizing and adjusting the segmentation results to improve the accuracy and stability of recognition.

[0019] In any of the above schemes, a preferred embodiment involves extracting text patterns of different features individually using feature recognition and classification techniques to obtain at least one text content, including the following steps:

[0020] S31: Text recognition, based on the obtained comprehensive text feature data, to recognize the text and obtain the first preset text;

[0021] S32: Randomly sample 20%-40% of the characters in the first preset text.

[0022] S33: When the sampling inspection result is qualified, the text recognition ends; when the sampling inspection result is unqualified, proceed to the next step.

[0023] S34: First, the text texture in at least one segmented region is separated using color-based separation technology. If the separation is successful, at least two color-separated text image data are obtained. Text recognition is then performed on the at least two separated text image data. The text recognition is then checked in step S32. If the check is successful, the text recognition ends; otherwise, proceed to the next step.

[0024] S35: Use clustering algorithms to extract and classify image features, and use morphological operations to separate text images to obtain text image data with at least two features. Then perform text recognition on the text image data with at least two features. Then perform verification through step S32. If the verification is successful, the text recognition ends. Otherwise, modify the clustering parameters of the clustering algorithm and perform recognition again.

[0025] In any of the above solutions, a preferred embodiment involves randomly sampling 20%-40% of the characters in the first preset text, including the following steps:

[0026] S41: Reference text selection: The text texture data in the segmented area is identified by optical character recognition technology, the text font corresponding to the text texture data is identified, and the corresponding reference text is selected according to the text content in the first preset text.

[0027] S42: Threshold selection: Based on the type of text font identified, select different similarity thresholds as the first threshold for judging the correctness of characters, and calculate the second threshold for judging the text sampling result based on image clarity, font type and image background complexity.

[0028] S42: Similarity calculation: Randomly select characters from the first preset text, the number of selected characters being 20%-40% of all characters in the first preset text, and calculate the similarity between the image corresponding to the selected character and the image of the corresponding reference text;

[0029] S43: Result judgment: Based on the calculated similarity value and the selected first threshold, determine whether the character is correct;

[0030] S44: Comprehensive judgment, summarize the judgment results of all selected characters, calculate the total accuracy, and judge whether the sampling results are qualified based on the obtained second threshold.

[0031] In any of the above schemes, a preferred approach is to extract and classify image features using a clustering algorithm and separate text images using morphological operations to obtain text image data with at least two feature separations, including the following steps:

[0032] S51: Advanced feature extraction, performing advanced feature extraction on at least one segmented region, wherein the advanced feature extraction includes local binary pattern and gray-level co-occurrence matrix;

[0033] S52: Set the number of clusters K = n + 2, where n is the number of times the segmented region is detected, and input the extracted high-level features and comprehensive text features into the K-means clustering algorithm. Use the K-means clustering algorithm to divide the pixels and pixel blocks into n + 2 clusters.

[0034] S53: Remove the clusters represented by the background image, map the remaining clusters to the segmentation region, and separate the text region and remove background noise through morphological operations to obtain text image data with at least two features.

[0035] In any of the above schemes, the preferred method, when separating the text region and removing background noise through morphological operations to obtain text image data with at least two features, includes the following:

[0036] A11: Reduces the highlighted areas in the image through erosion, removes isolated noise points, and separates objects that are in contact or close together;

[0037] A12: Expands the highlighted areas in the image through dilation, fills in holes inside objects, and smooths the edges of objects.

[0038] A13: Remove small objects by opening operations, separate objects at thin points, and smooth the boundaries of larger objects;

[0039] A14: Fill the small hole inside the foreground object using the closing operation, connect neighboring objects, and smooth their boundaries;

[0040] A15: Highlight the boundaries of an object using morphological gradient operations.

[0041] In any of the above schemes, preferably, after performing text recognition on the text image data separated from at least two features, the following steps are also included:

[0042] S61: Based on location determination, analyze the position of the text in the image and match it with the expected category or region;

[0043] S62: Check if the text content matches the image context;

[0044] S63: Analyze the semantic meaning of the text and determine whether it matches the overall theme and context of the text in the image;

[0045] S64: Extract the original image of the identified erroneous character separately, determine the possible meaning of the erroneous character based on the semantic meaning of the text, and use the possible meaning of the erroneous character as the judgment criterion to re-identify the original image of the erroneous character.

[0046] In any of the above solutions, the preferred method for acquiring user voice command information data, recognizing user voice command data, and obtaining voice control commands includes the following steps:

[0047] S71: Speech signal conversion and preprocessing, converting the acquired speech signal into a digital signal, and performing filtering, pre-emphasis, framing and windowing processing on the obtained digital signal;

[0048] S72: Speech feature extraction, which converts the preprocessed digital signal from the time domain to the frequency domain and extracts parameters that can characterize the essential features of the speech signal;

[0049] S73: Acoustic modeling, using a hidden Markov model to match the extracted feature parameters with a preset acoustic model;

[0050] S74: Language Modeling and Decoding. It models language characteristics through a recurrent neural network language model, and then uses a decoder to fuse information from the acoustic model and the language model to parse out the most likely text sequence.

[0051] In any of the above schemes, a preferred embodiment, when selecting at least one text content for recognition and analysis based on the recognition results of the voice command data, includes the following steps:

[0052] S81: Text preprocessing, which preprocesses the parsed text sequence, including noise removal, word segmentation and part-of-speech tagging.

[0053] S82: Named entity recognition, which performs named entity recognition based on text preprocessing to identify specific entities in the instructions;

[0054] S83: Intent recognition, which uses natural language processing technology to analyze the semantic content of instructions and identify the user's intent;

[0055] S84: Parameter extraction, based on the recognition of user intent, further extracts key parameters from the command;

[0056] S85: Instruction parsing and construction. Based on the user's intent and extracted parameters, it constructs an instruction parse tree or similar data structure to decompose complex instructions into a series of executable subtasks.

[0057] S86: Execute instructions.

[0058] In any of the above solutions, it is preferred that when providing auxiliary guidance to the user's learning through at least one of voice, text, video, and image formats based on the recognition and analysis results of the text content, the following steps are included:

[0059] S91: Perform corresponding operations and responses based on the user's instructions;

[0060] S92: Utilize knowledge graph technology to perform in-depth analysis of the knowledge points contained in the question content, identify and extract key knowledge points, and use machine learning algorithms to predict the difficulties that users may encounter based on their learning history and performance.

[0061] S93: Based on the obtained knowledge points and difficulties, search within a pre-set database for relevant audio, images, and videos. Sort the searched audio, images, and videos according to the set importance coefficient and the proportion coefficient of the knowledge points in the questions, and use this order as the default playback order for the user.

[0062] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0063] The above-mentioned solution of the present invention, by sampling the text recognition results based on the basic text recognition algorithm to determine the correctness of the text recognition results, and then using an improved image recognition algorithm to re-recognize based on the correctness of the text recognition results, can ensure the correctness of image recognition. Thus, when assisting users in learning, it can accurately recognize the text that users need to input, ensure the accuracy and efficiency of recognition, enhance the user experience, and provide effective guidance to users.

[0064] When recognizing overlapping text, a clustering algorithm is used to process the image. The clustering algorithm is used for preprocessing and auxiliary separation, and then a text recognition algorithm is used to recognize the text. The recognition results are then checked for compliance, and the number of clusters in the clustering algorithm is changed based on the results. After the change, recognition is performed again, which can effectively ensure the accuracy of the recognition results and solve the problem of low success rate and accuracy of recognizing overlapping text in the existing text recognition process. Attached Figure Description

[0065] Figure 1 This is a flowchart illustrating the AR-AI integrated interactive learning assistance method provided in an embodiment of the present invention.

[0066] Figure 2 This is a schematic diagram of the text texture extraction process in the AR-AI integrated interactive learning assistance method provided by an embodiment of the present invention. Detailed Implementation

[0067] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0068] Please see Figure 1-2 The present invention provides an embodiment of an AR-AI integrated interactive learning assistance method, comprising the following steps:

[0069] S11: Obtain an image containing the question content and segment the image to obtain at least one segmented region, which may specifically include:

[0070] Acquire an image of a piece of paper containing the question content and segment the paper image to obtain at least one segmented region with independent text content;

[0071] S12: Recognize the text texture within at least one segmented region using image recognition technology, and obtain text texture data within at least one segmented region, which may specifically include:

[0072] First, image preprocessing is performed to enhance the text information in the image, making the text clearer and more legible. Then, feature extraction is performed on the preprocessed image using edge detection and texture analysis techniques to extract the features of the text, obtaining comprehensive text feature data. Based on the feature extraction results, image segmentation is performed by analyzing connected components in the image to find the outlines of the text and segmenting them according to texture features. After segmentation, post-processing is performed to further optimize and adjust the segmentation results to improve the accuracy and stability of recognition.

[0073] S13: Extract features from different text textures using feature recognition and classification techniques to obtain at least one text content, which may specifically include:

[0074] First, text recognition is performed. Based on the obtained comprehensive text feature data, the text is recognized to obtain the first preset text. After text recognition, detection is performed. Random sampling is conducted on the first preset text, and 20%-40% of the characters in the first preset text are sampled. If the sampling result is qualified, the text recognition ends. If the sampling result is unqualified, deep analysis is performed: First, the text texture in at least one segmented region is separated using color-based separation technology. If the separation is successful, at least two color-separated text image data are obtained. Text recognition is performed on at least two separated text image data, and the verification is performed again. If the verification is qualified, the text recognition ends. If at least two color-separated text image data cannot be obtained or the verification is unqualified, the image is re-identified using a clustering algorithm: The image features are extracted and classified using a clustering algorithm, and the text image is separated using morphological operations to obtain at least two feature-separated text image data. Text recognition is performed on at least two feature-separated text image data, and the verification is performed again. If the verification is qualified, the text recognition ends. Otherwise, the clustering parameters of the clustering algorithm are modified, and the recognition is performed again.

[0075] S14: Obtain the user's voice command information data, recognize the user's voice command data, and obtain voice control commands;

[0076] S15: Based on the recognition results of the voice command data, select at least one text content for recognition and analysis;

[0077] S16: Based on the recognition and analysis results of the text content, provide auxiliary guidance for the user's learning through at least one of the following formats: voice, text, video, and image.

[0078] In this embodiment, by sampling the text recognition results based on the basic text recognition algorithm to determine the correctness of the text recognition results, and then using an improved image recognition algorithm to re-recognize the text based on the correctness of the text recognition results, the correctness of the image recognition can be guaranteed. This allows for accurate recognition of the text that the user needs to input when assisting the user in learning, ensuring the accuracy and efficiency of the recognition, improving the user experience, and providing effective guidance to the user.

[0079] It should be noted that the above steps are only a preferred implementation order. In the specific implementation process, some steps can be changed without affecting the overall implementation effect. In order to more clearly explain the technical solution of this application, the following content explains the solution in a preferred manner.

[0080] In this embodiment of the invention, when randomly sampling 20%-40% of the characters in a first preset text, the following steps are included:

[0081] S41: Reference text selection: The text texture data in the segmented area is identified by optical character recognition technology, the text font corresponding to the text texture data is identified, and the corresponding reference text is selected according to the text content in the first preset text.

[0082] S42: Threshold selection: Based on the type of text font identified, select different similarity thresholds as the first threshold for judging the correctness of characters, and calculate the second threshold for judging the text sampling result based on image clarity, font type and image background complexity.

[0083] S42: Similarity calculation: Randomly select characters from the first preset text, the number of selected characters being 20%-40% of all characters in the first preset text, and calculate the similarity between the image corresponding to the selected character and the image of the corresponding reference text;

[0084] S43: Result judgment: Based on the calculated similarity value and the selected first threshold, determine whether the character is correct;

[0085] S44: Comprehensive judgment, summarize the judgment results of all selected characters, calculate the total accuracy, and judge whether the sampling results are qualified based on the obtained second threshold.

[0086] Through the above steps S41-S44, in this embodiment, by combining optical character recognition (OCR) technology, the text font is automatically identified and the corresponding reference text is selected. Then, image similarity calculation is performed on 20%-40% of randomly selected characters, and the correctness is judged by dynamically setting dual thresholds based on font type, image clarity and background complexity. This method not only realizes automated monitoring of text quality, but also significantly improves the accuracy and efficiency of sampling inspection, provides reliable quality assurance for text processing, and helps to continuously optimize OCR algorithms and sampling inspection strategies.

[0087] In another optional embodiment of the present invention, when different similarity thresholds are selected as the first threshold for judging the correctness of characters based on the type of identified text font, different similarity thresholds are selected for different fonts. For example, the similarity threshold for handwritten text is 72%, the similarity threshold for simplified Song typeface text is 89%, and the similarity threshold for traditional Song typeface text is 81%. Through a large number of experiments, it has been found that when the external adjustments such as image clarity, angle, and lighting are close to ideal, the detection accuracy rate for handwritten text reaches 98.72%, the accuracy rate for simplified Song typeface text reaches 99.96%, and the accuracy rate for traditional Song typeface text reaches 99.45%.

[0088] In another optional embodiment of the present invention, step S42 is performed according to the following formula:

[0089] Calculate the second threshold used to determine the results of text sampling;

[0090] Where α is the second threshold for judging the text sampling results, τ is the image clarity, τ' is the image clarity under ideal conditions, μ is the image complexity, the value of μ ranges from 0.5 to 1, and the lower the image complexity, the larger the value of μ, and ρ is a constant parameter, the size of ρ is determined according to the font type.

[0091] In this embodiment, the ratio of image sharpness to image sharpness under ideal conditions is used as the first parameter for calculating the second threshold for judging the text sampling result. The transformation result of image complexity is used as the second parameter for calculating the second threshold for judging the text sampling result. Different benchmark thresholds are selected according to font type. The two parameters are multiplied by the benchmark threshold to obtain the final second threshold. This realizes automated and efficient monitoring and evaluation of text quality, and provides strong support for the optimization and decision-making of text processing flow.

[0092] In this embodiment of the invention, the process of extracting and classifying image features using a clustering algorithm and separating text images using morphological operations to obtain text image data with at least two separate features includes the following steps:

[0093] S51: Advanced feature extraction, performing advanced feature extraction on at least one segmented region, wherein the advanced feature extraction includes local binary pattern and gray-level co-occurrence matrix;

[0094] S52: Set the number of clusters K = n + 2, where n is the number of times the segmented region is detected, and input the extracted high-level features and comprehensive text features into the K-means clustering algorithm. Use the K-means clustering algorithm to divide the pixels and pixel blocks into n + 2 clusters.

[0095] S53: Remove the clusters represented by the background image, map the remaining clusters to the segmentation region, and separate the text region and remove background noise through morphological operations to obtain text image data with at least two features.

[0096] Through the above steps S111-S114, when recognizing overlapping text, a clustering algorithm is used to process the image. The clustering algorithm is used for preprocessing and auxiliary separation, and then a text recognition algorithm is used for text recognition. The recognition results are then checked for compliance, and the number of clusters in the clustering algorithm is changed according to the detection results. That is, the number of recognized categories starts from 3 and is incremented sequentially. After the change, recognition is performed again, which can effectively ensure the accuracy of the recognition results and solve the problem of low success rate and accuracy of recognizing overlapping text in the existing text recognition process.

[0097] In this embodiment of the invention, step S12, which involves post-processing the image after segmentation to further optimize and adjust the segmentation results in order to improve the accuracy and stability of recognition, includes the following steps:

[0098] S121: Smooth the segmentation results by using morphological operations such as erosion, dilation, opening, closing, morphological gradient, top-hat transformation, and bottom-hat transformation to remove noise and burrs from the image.

[0099] S122: Perform deduplication and merging processing on duplicate or broken characters in the segmentation results to ensure that each character is independent and complete.

[0100] In another optional embodiment of the present invention, in step S121 above, the segmented image can also be smoothed according to the coordinates of the pixels, wherein the principle formula is as follows:

[0101]

[0102] Among them, I filtered (x,y) represents the pixel value of the segmented image at coordinates (x,y), σ is the standard deviation of the Gaussian function, k is the size of the filter, i and j are loop variables used to iterate through all elements of the filter, and I(x+i,y+j) represents the pixel value of the input image at coordinates (x+i,y+j), which is an element of the filter.

[0103] In this embodiment, the k-axis of each pixel in the acquired segmented image is calculated. 2 The sum of the products of each pixel and the Gaussian function, divided by the sum of the product terms, yields the smoothed pixel value. This process can suppress noise interference in the analysis of the segmented image, making the image clearer.

[0104] In another optional embodiment of the present invention, in step S11, when acquiring an image containing the title content and segmenting the image to obtain at least one segmented region, the method further includes:

[0105] S111: Image enhancement. By adjusting parameters such as brightness, contrast, and saturation, the text information in the image is enhanced, making the text clearer and more legible.

[0106] S112: Image denoising, removes noise from the image, such as salt and pepper noise, Gaussian noise, etc., to improve image clarity;

[0107] S113: Image tilt correction, corrects the tilt angle of the image to keep the text horizontal;

[0108] S114: Image grayscale processing, converting the image into a grayscale image for subsequent text detection and segmentation.

[0109] In this embodiment, the captured image is processed through the above steps S111-S114 to weaken and eliminate image quality problems caused by shooting techniques, environmental factors, and the clarity of the shooting device, thereby improving image quality and making subsequent text detection and recognition more accurate.

[0110] In this embodiment of the invention, when separating text regions and removing background noise through morphological operations to obtain text image data with at least two features, the following methods are included:

[0111] A11: Reduces the highlighted areas in the image through erosion, removes isolated noise points, and separates objects that are in contact or close together;

[0112] A12: Expands the highlighted areas in the image through dilation, fills in holes inside objects, and smooths the edges of objects.

[0113] A13: Remove small objects by opening operations, separate objects at thin points, and smooth the boundaries of larger objects;

[0114] A14: Fill the small hole inside the foreground object using the closing operation, connect neighboring objects, and smooth their boundaries;

[0115] A15: Highlight the boundaries of an object using morphological gradient operations.

[0116] In this embodiment of the invention, after performing text recognition on text image data separated from at least two features, the following steps are also included:

[0117] S61: Based on location determination, analyze the position of the text in the image and match it with the expected category or region;

[0118] S62: Check if the text content matches the image context;

[0119] S63: Analyze the semantic meaning of the text and determine whether it matches the overall theme and context of the text in the image;

[0120] S64: Extract the original image of the identified erroneous character separately, determine the possible meaning of the erroneous character based on the semantic meaning of the text, and use the possible meaning of the erroneous character as the judgment criterion to re-identify the original image of the erroneous character.

[0121] Through steps S61-S64 above, this technical solution further enhances the ability to analyze the matching degree between text and image content, building upon text recognition. First, based on location judgment, the recognized text is precisely matched with the expected category or region in the image, improving the accuracy of text positioning. Second, it checks whether the text content matches the image context, ensuring the logical coherence of the text information. Third, it deeply analyzes the semantic meaning of the text, judging whether it matches the overall theme and context of the text in the image, further improving the accuracy and relevance of text recognition. Finally, for identified erroneous characters, by extracting their original images separately and combining them with the semantic meaning of the text, the possible meaning of the erroneous characters is inferred, and this is used as a standard for re-recognition, effectively reducing the overall text understanding deviation caused by single character recognition errors. This not only improves the accuracy and efficiency of text recognition but also enhances the understanding of the correlation between text and image content, making the text information closer to the actual context of the image. In addition, through the intelligent re-recognition mechanism for erroneous characters, the text recognition error rate is further reduced, improving the overall quality of text processing and user experience.

[0122] In this embodiment of the invention, when selecting at least one text content for recognition and analysis based on the recognition result of the voice command data, the following steps are included:

[0123] S71: Speech signal conversion and preprocessing, converting the acquired speech signal into a digital signal, and performing filtering, pre-emphasis, framing and windowing processing on the obtained digital signal;

[0124] S72: Speech feature extraction, which converts the preprocessed digital signal from the time domain to the frequency domain and extracts parameters that can characterize the essential features of the speech signal;

[0125] S73: Acoustic modeling, using a hidden Markov model to match the extracted feature parameters with a preset acoustic model;

[0126] S74: Language Modeling and Decoding. It models language characteristics through a recurrent neural network language model, and then uses a decoder to fuse information from the acoustic model and the language model to parse out the most likely text sequence.

[0127] In this embodiment of the invention, when selecting at least one text content for recognition and analysis based on the recognition result of the voice command data, the following steps are also included:

[0128] S81: Text preprocessing, which preprocesses the parsed text sequence, including noise removal, word segmentation and part-of-speech tagging.

[0129] S82: Named entity recognition, which performs named entity recognition based on text preprocessing to identify specific entities in the instructions;

[0130] S83: Intent recognition, which uses natural language processing technology to analyze the semantic content of instructions and identify the user's intent;

[0131] S84: Parameter extraction, based on the recognition of user intent, further extracts key parameters from the command;

[0132] S85: Instruction parsing and construction. Based on the user's intent and extracted parameters, it constructs an instruction parse tree or similar data structure to decompose complex instructions into a series of executable subtasks.

[0133] S86: Execute instructions.

[0134] In another optional embodiment of the present invention, when selecting at least one text content for recognition and analysis based on the recognition results of voice command data, the user may need to convert a large number of paper documents or scanned copies into editable electronic documents and perform content analysis and extraction. Through the embodiment of command parsing and construction, the user can input complex commands, such as "recognize all tables in the document and extract the data therein". The system will automatically parse the command, decompose it into multiple sub-tasks such as text recognition, table location, and data extraction, and execute them in sequence, finally outputting a structured data table.

[0135] In this embodiment of the invention, when providing auxiliary guidance to a user's learning through at least one of voice, text, video, and image formats based on the recognition and analysis results of text content, the following steps are included:

[0136] S91: Perform corresponding operations and responses based on the user's instructions;

[0137] S92: Utilize knowledge graph technology to perform in-depth analysis of the knowledge points contained in the question content, identify and extract key knowledge points, and use machine learning algorithms to predict the difficulties that users may encounter based on their learning history and performance.

[0138] S93: Based on the obtained knowledge points and difficulties, search within a pre-set database for relevant audio, images, and videos. Sort the searched audio, images, and videos according to the set importance coefficient and the proportion coefficient of the knowledge points in the questions, and use this order as the default playback order for the user.

[0139] In another optional embodiment of the present invention, the user can acquire images and view videos and graphics through an AR device. When acquiring images and viewing videos and graphics through an AR device, the user first controls the AR device or other image device through voice commands to acquire the required images. Then, text recognition is performed through the image recognition steps described above. After text recognition is completed, the user selects the required text content through voice commands and modifies, organizes, and converts the text content through voice commands. After completion, the text content is parsed. At the same time, knowledge graph technology is used to deeply analyze the knowledge points contained in the question content, identify and extract key knowledge points, and predict the difficulties that the user may encounter by combining the user's learning history and performance. The next step is to search in a pre-set database to search for relevant tutoring materials, explanation videos, demonstration animations, and related questions. While explaining and analyzing the questions for the user, the relevant tutoring materials, explanation videos, demonstration animations, and related questions found by the search are played through the AR device to deepen learning and ensure the effectiveness of the user's learning.

[0140] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An AR-AI integrated interactive learning assistance method, characterized by: It includes the following steps: S11: Obtain an image containing the question content and segment the image to obtain at least one segmented region; S12: Recognize the text texture in at least one segmented region using image recognition technology, and obtain text texture data in at least one segmented region; S13: Extract features from different text patterns using feature recognition and classification techniques to obtain at least one text content. The process includes the following steps: S31: Text recognition, based on the obtained comprehensive text feature data, to recognize the text and obtain a first preset text; S32: Randomly sample 20%-40% of the characters in the first preset text; S33: If the sampling result is qualified, the text recognition ends; if the sampling result is unqualified, proceed to the next step; S34: First, the text texture in at least one segmented region is separated using color-based separation technology. If the separation is successful, at least two color-separated text image data are obtained, and text recognition is performed on the at least two separated text image data. Then, the process continues through step S... Step 32 involves verification. If the verification is successful, the text recognition process ends; otherwise, proceed to the next step. Step 35: Image features are extracted and classified using a clustering algorithm, and the text image is separated using morphological operations to obtain text image data with at least two features. Text recognition is then performed on the text image data with at least two features, and verification is performed again using step S32. If the verification is successful, the text recognition process ends; otherwise, the clustering parameters of the clustering algorithm are modified, and recognition is performed again. The comprehensive text feature data is obtained by extracting the features of the text in the preprocessed image using edge detection and texture analysis techniques. S14: Obtain the user's voice command information data, recognize the user's voice command data, and obtain voice control commands; S15: Based on the recognition results of the voice command data, select at least one text content for recognition and analysis; S16: Based on the recognition and analysis results of the text content, provide auxiliary guidance for the user's learning through at least one of the following formats: voice, text, video, and image.

2. The AR-AI integrated interactive learning assistance method according to claim 1, characterized in that: When identifying text patterns within at least one segmented region using image recognition technology to obtain text pattern data within the segmented region, the following steps are included: S21: Image preprocessing enhances text information in the image, making the text clearer and more legible; S22: Feature extraction: Using edge detection and texture analysis techniques, the features of text in the image are extracted to obtain comprehensive text feature data; S23: Image segmentation, which involves analyzing connected components in an image to find the outline of text and segmenting it based on texture features; S24: Image post-processing, further optimizing and adjusting the segmentation results to improve the accuracy and stability of recognition.

3. The AR-AI integrated interactive learning assistance method according to claim 1, characterized in that: When randomly sampling 20%-40% of the characters in a first preset text, the following steps are included: S41: Reference text selection: The text texture data in the segmented area is identified by optical character recognition technology, the text font corresponding to the text texture data is identified, and the corresponding reference text is selected according to the text content in the first preset text. S42: Threshold selection: Based on the type of text font identified, select different similarity thresholds as the first threshold for judging the correctness of characters, and calculate the second threshold for judging the text sampling result based on image clarity, font type and image background complexity. S42: Similarity calculation: Randomly select characters from the first preset text, the number of selected characters being 20%-40% of all characters in the first preset text, and calculate the similarity between the image corresponding to the selected character and the image of the corresponding reference text; S43: Result judgment: Based on the calculated similarity value and the selected first threshold, determine whether the character is correct; S44: Comprehensive judgment, integrate the judgment results of all selected characters, calculate the total accuracy, and judge whether the sampling result is qualified based on the obtained second threshold.

4. The AR-AI integrated interactive learning assistance method according to claim 3, characterized in that: The process involves using clustering algorithms to extract and classify image features, and then using morphological operations to separate text images, resulting in text image data with at least two separate features. This includes the following steps: S51: Advanced feature extraction, performing advanced feature extraction on at least one segmented region, wherein the advanced feature extraction includes local binary pattern and gray-level co-occurrence matrix; S52: Set the number of clusters K = n + 2, where n is the number of times the segmented region is detected, and input the extracted high-level features and comprehensive text features into the K-means clustering algorithm. Use the K-means clustering algorithm to divide the pixels and pixel blocks into n + 2 clusters. S53: Remove the clusters represented by the background image, map the remaining clusters to the segmentation region, and separate the text region and remove background noise through morphological operations to obtain text image data with at least two features.

5. The AR-AI integrated interactive learning assistance method according to claim 4, characterized in that: When separating text regions and removing background noise through morphological operations to obtain text image data with at least two features, the following methods are included: A11: Reduces the highlighted areas in the image through erosion, removes isolated noise points, and separates objects that are in contact or close together; A12: Expands the highlighted areas in the image through dilation, fills in holes inside objects, and smooths the edges of objects. A13: Remove small objects by opening operations, separate objects at thin points, and smooth the boundaries of larger objects; A14: Fill the small hole inside the foreground object using the closing operation, connect neighboring objects, and smooth their boundaries; A15: Highlight the boundaries of an object using morphological gradient operations.

6. The AR-AI integrated interactive learning assistance method according to claim 5, characterized in that: After performing text recognition on text image data that has been separated from at least two features, the following steps are also included: S61: Based on location determination, analyze the position of the text in the image and match it with the expected category or region; S62: Check if the text content matches the image context; S63: Analyze the semantic meaning of the text and determine whether it matches the overall theme and context of the text in the image; S64: Extract the original image of the identified erroneous character separately, determine the possible meaning of the erroneous character based on the semantic meaning of the text, and use the possible meaning of the erroneous character as the judgment criterion to re-identify the original image of the erroneous character.

7. The AR-AI integrated interactive learning assistance method according to claim 6, characterized in that: The process of acquiring user voice command information data, recognizing user voice command data, and obtaining voice control commands includes the following steps: S71: Speech signal conversion and preprocessing, converting the acquired speech signal into a digital signal, and performing filtering, pre-emphasis, framing and windowing processing on the obtained digital signal; S72: Speech feature extraction, which converts the preprocessed digital signal from the time domain to the frequency domain and extracts parameters that can characterize the essential features of the speech signal; S73: Acoustic modeling, using a hidden Markov model to match the extracted feature parameters with a preset acoustic model; S74: Language Modeling and Decoding. It models language characteristics through a recurrent neural network language model, and then uses a decoder to fuse information from the acoustic model and the language model to parse out the most likely text sequence.

8. The AR-AI integrated interactive learning assistance method according to claim 7, characterized in that: When selecting at least one text content for recognition and analysis based on the recognition results of voice command data, the following steps are included: S81: Text preprocessing, which preprocesses the parsed text sequence, including noise removal, word segmentation and part-of-speech tagging. S82: Named entity recognition, which performs named entity recognition based on text preprocessing to identify specific entities in the instructions; S83: Intent recognition, which uses natural language processing technology to analyze the semantic content of instructions and identify the user's intent; S84: Parameter extraction, based on the recognition of user intent, further extracts key parameters from the command; S85: Instruction parsing and construction. Based on the user's intent and extracted parameters, it constructs an instruction parse tree or similar data structure to decompose complex instructions into a series of executable subtasks. S86: Execute instructions.

9. The AR-AI integrated interactive learning assistance method according to claim 8, characterized in that: When providing auxiliary guidance to users' learning through at least one of the following methods—voice, text, video, and image—based on the recognition and analysis results of text content, the process includes the following steps: S91: Perform corresponding operations and responses based on the user's instructions; S92: Utilize knowledge graph technology to perform in-depth analysis of the knowledge points contained in the question content, identify and extract key knowledge points, and use machine learning algorithms to predict the difficulties that users may encounter based on their learning history and performance. S93: Based on the obtained knowledge points and difficulties, search within a pre-set database for relevant audio, images, and videos. Sort the searched audio, images, and videos according to the set importance coefficient and the proportion coefficient of the knowledge points in the questions, and use this order as the default playback order for the user.

Images