AR-AI integrated interactive learning assistance method

Abstract

The invention provides an AR-AI integrated interactive learning assistance method, and relates to the technical field of intelligent teaching. The method comprises the following steps: acquiring an image containing question content and segmenting the image to obtain at least one segmented region; acquiring character line data in the at least one segmented region through an image recognition technology; independently extracting character lines with different characteristics; acquiring a voice control instruction and selecting at least one text content for recognition and analysis; on the basis of a basic text recognition algorithm, the text recognition result is subjected to spot check, the correctness of the text recognition result is judged, and according to the correctness of the text recognition result, an improved image recognition algorithm is adopted for re-recognition, so that the correctness of image recognition can be ensured, and the learning efficiency of a user is improved when the user is assisted in learning. The text needing to be input by the user is accurately recognized, the recognition accuracy and efficiency are guaranteed, the use experience of the user is improved, and effective tutoring is conducted on the user.

Description

Technical Field

[0001] The present invention relates to the field of intelligent teaching technology, and in particular to an AR-AI integrated interactive learning assistance method. Background Art

[0002] With the rapid development of mobile Internet technology, photo recognition technology has been widely used in various fields as an efficient and convenient way to obtain information. In multiple scenarios such as education, life services, and medical consultation, photo recognition-based applications capture the image content taken by users and use advanced image processing and natural language processing technologies to quickly identify and analyze the information in the image, thereby providing users with corresponding answers or solutions. These applications have greatly enriched the channels for information acquisition and improved the efficiency of problem solving.

[0003] In complex image environments, especially when text and images overlap, the accuracy of existing camera recognition technology is often seriously affected. Since the text in the image may be affected by various factors such as background patterns, colors, and lighting conditions, it is difficult for the recognition algorithm to accurately distinguish and extract valid text information. In addition, the close overlap of text and images may also cause confusion in the recognition area, further increasing the difficulty and error rate of recognition. This lack of recognition accuracy in the case of text and image overlap directly limits the application and development of camera recognition technology in a wider range of scenarios.

[0004] In view of the problems existing in the existing photo recognition applications in terms of interactivity and recognition accuracy, especially the recognition difficulties in the case of overlapping text and images, it is necessary to carry out technical innovation and optimization. Therefore, the present invention provides an improved learning assistance system. Through the improved text recognition algorithm and process, even if the user records notes on text carriers such as books and there is text overlap with other text content, accurate recognition of text content can still be achieved, thereby ensuring recognition accuracy and efficiency, increasing the user experience, and providing effective guidance to users. Summary of the invention

[0005] In order to overcome the problem of low recognition success rate and accuracy of overlapping text in the existing text recognition process, the present invention provides an AR-AI integrated interactive learning assistance method.

[0006] In order to solve the above technical problems, the technical solutions provided by the present invention are as follows: The AR-AI integrated interactive learning assistance method includes the following steps: S11: Acquire an image containing the topic content and segment the image to obtain at least one segmented area; S12: Recognize text patterns in at least one segmented area by using image recognition technology to obtain text pattern data in at least one segmented area; S13: extracting features of text patterns with different features separately through feature recognition technology and classification technology to obtain at least one text content; S14: Acquire the user's voice command information data, and recognize the user's voice command data to obtain the voice control command; S15: selecting at least one text content for recognition and analysis according to the recognition result of the voice command data; S16: Based on the recognition and analysis results of the text content, provide auxiliary guidance for the user's learning in at least one of voice, text, video and image forms.

[0007] In any of the above solutions, preferably, when the text pattern in at least one segmented area is recognized by image recognition technology to obtain the text pattern data in the segmented area, the following steps are included: S21: Image preprocessing, enhancing the text information in the image to make the text clearer and more legible; S22: feature extraction, extracting the features of the text in the image through edge detection technology and texture analysis technology to obtain comprehensive text feature data; S23: Image segmentation, by analyzing the connected domain in the image, finding the outline of the text and segmenting it according to the texture features; S24: Image post-processing: further optimization and adjustment of the segmentation results to improve the accuracy and stability of recognition.

[0008] In any of the above solutions, preferably, when text patterns with different features are extracted separately by feature recognition technology and classification technology to obtain at least one text content, the following steps are included: S31: text recognition, based on the obtained comprehensive text feature data, the text is recognized to obtain a first preset text; S32: Randomly check the first preset text, checking 20%-40% of the characters in the first preset text; S33: When the sampling inspection result is qualified, the text recognition is terminated; when the sampling inspection result is unqualified, the next step is entered; S34: First, the text patterns in at least one segmented area are separated by a color-based separation technology. If the separation is successful, at least two color-separated text image data are obtained, and text recognition is performed on the at least two separated text image data, and then the step S32 is performed for inspection. If the inspection is qualified, the text recognition is terminated, otherwise it goes to the next step; S35: Use a clustering algorithm to extract and classify image features, and use morphological operations to separate text images to obtain at least two feature-separated text image data, and perform text recognition on at least two feature-separated text image data, and then perform a test through step S32. If the test is qualified, the text recognition is terminated. Otherwise, the clustering parameters of the clustering algorithm are modified and recognition is performed again.

[0009] In any of the above solutions, preferably, when the first preset text is randomly sampled and 20%-40% of the characters in the first preset text are sampled, the following steps are included: S41: Reference text selection, identifying the text texture data in the segmented area by optical character recognition technology, identifying the text font corresponding to the text texture data, and selecting a corresponding reference text according to the text content in the first preset text; S42: judging threshold selection, selecting different similarity thresholds as the first threshold for judging character correctness according to the type of the identified text font, and calculating the second threshold for judging the text sampling result according to the image clarity, font type and image background complexity; S42: similarity calculation, randomly selecting characters in the first preset text, the number of selected characters is 20%-40% of all characters in the first preset text, and performing similarity calculation between images corresponding to the selected characters and images of corresponding reference texts; S43: Result determination, determining whether the character is correct based on the calculated similarity value and the selected first threshold; S44: Comprehensive judgment: Comprehensively judge the judgment results of all the selected characters, calculate the total accuracy, and judge whether the sampling result is qualified according to the obtained second threshold.

[0010] In any of the above schemes, preferably, the steps of extracting and classifying image features by using a clustering algorithm and separating text images by using morphological operations to obtain at least two feature-separated text image data include: S51: Advanced feature extraction, performing advanced feature extraction on at least one segmented region, wherein the advanced feature extraction includes local binary pattern and gray level co-occurrence matrix; S52: setting the number of clusters K=n+2, where n is the number of times the segmented area is detected, and inputting the extracted high-level features and comprehensive text feature data into a K-means clustering algorithm, and using the K-means clustering algorithm to divide pixels and pixel blocks into n+2 clusters; S53: Eliminate the clusters represented by the background image, map the remaining clusters to the segmented areas, separate the text areas and remove the background noise through morphological operations, and obtain at least two feature-separated text image data.

[0011] In any of the above solutions, preferably, when the text area is separated and the background noise is removed by morphological operation to obtain at least two feature-separated text image data, the following method is included: A11: Reduces the highlight area in the image through erosion operation, removes isolated noise points, and separates touching and close objects; A12: Dilation is used to expand the highlight areas in the image, fill the holes inside the object, and make the edges of the object smoother. A13: Remove small objects through opening operations, separate objects at thin points, and smooth the boundaries of larger objects; A14: Fill small holes inside foreground objects through closing operations, connect adjacent objects, and smooth their boundaries; A15: Highlighting object boundaries through morphological gradient operations.

[0012] In any of the above solutions, preferably, after performing text recognition on at least two feature-separated text image data, the method further includes the following steps: S61: Based on the position judgment, the position of the text in the image is analyzed and matched with the expected category or area; S62: Check whether the text content is consistent with the image context; S63: Analyze the semantic meaning of the text to determine whether it fits the overall theme and context of the text in the image; S64: extracting the original image of the recognized erroneous character separately, judging the possible meaning of the erroneous character according to the semantic meaning of the text, and re-recognizing the original image of the erroneous character based on the possible meaning of the erroneous character as a judgment criterion.

[0013] In any of the above solutions, preferably, the steps of obtaining the user's voice command information data, recognizing the user's voice command data, and obtaining the voice control command include: S71: voice signal conversion and preprocessing, converting the collected voice signal into a digital signal, and filtering, pre-emphasis, framing and windowing the obtained digital signal; S72: Speech feature extraction, converting the preprocessed digital signal from the time domain to the frequency domain, and extracting parameters that can characterize the essential features of the speech signal; S73: Acoustic modeling, using hidden Markov model to match the extracted feature parameters with the preset acoustic model; S74: Language modeling and decoding, modeling language characteristics through a recurrent neural network language model, and then using a decoder to fuse the information of the acoustic model and the language model to parse the most likely text sequence.

[0014] In any of the above solutions, preferably, when selecting at least one text content for recognition and analysis according to the recognition result of the voice instruction data, the following steps are included: S81: text preprocessing, preprocessing the parsed text sequence, wherein the preprocessing methods used include noise removal, word segmentation and part-of-speech tagging; S82: Named entity recognition, performing named entity recognition based on text preprocessing to identify specific entities in the instruction; S83: Intent recognition, using natural language processing technology to analyze the semantic content of instructions and identify the user's intention; S84: Parameter extraction: based on identifying the user's intention, further extracting key parameters in the instruction; S85: instruction parsing and construction, constructing an instruction parsing tree or similar data structure based on the user's intention and the extracted parameters, and decomposing complex instructions into a series of executable subtasks; S86: Execute instructions.

[0015] In any of the above schemes, preferably, when providing auxiliary guidance for the user's learning in at least one of voice, text, video and image forms based on the recognition and analysis results of the text content, the following steps are included: S91: Perform corresponding operations and answers according to the user's instructions; S92: Use knowledge graph technology to deeply analyze the knowledge points contained in the question content, identify and extract key knowledge points, and use machine learning algorithms to predict the difficulties that users may encounter based on their learning history and performance; S93: Based on the obtained knowledge points and difficulties, search for relevant audio, pictures and videos in a pre-set database, and sort the searched audio, pictures and videos according to the set importance coefficient and the proportion coefficient of the knowledge point in the question, and use this order as the default playback order to play for the user.

[0016] Compared with the prior art, the present invention has the following beneficial effects: The above scheme of the present invention can ensure the correctness of image recognition by sampling the text recognition results on the basis of the basic text recognition algorithm, judging the correctness of the text recognition results, and re-recognizing with the improved image recognition algorithm according to the correctness of the text recognition results, so as to accurately recognize the text that the user needs to input when assisting the user in learning, ensure the accuracy and efficiency of recognition, increase the user's experience, and provide effective guidance to the user; When recognizing overlapping text, the image is processed by using a clustering algorithm, preprocessed and assisted by the clustering algorithm, and text recognition is performed using a text recognition algorithm. The recognition results are tested for eligibility, and the number of clusters of the clustering algorithm is changed according to the test results. After the change, recognition is performed again, thereby effectively ensuring the accuracy of the recognition results and solving the problem of low recognition success rate and accuracy for overlapping text in the existing text recognition process. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a flowchart of the AR-AI integrated interactive learning assistance method provided by an embodiment of the present invention.

[0018] Figure 2 It is a flowchart of text texture extraction in the AR-AI integrated interactive learning assistance method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0020] See also Figure 1-2 The present invention provides an embodiment: an AR-AI integrated interactive learning assistance method, comprising the following steps: S11: Acquire an image containing the topic content and segment the image to obtain at least one segmented area, which may specifically include: Capturing an image of a paper containing the topic content and segmenting the paper image to obtain at least one segmented region having independent text content; S12: Recognizing text patterns in at least one segmented area by image recognition technology to obtain text pattern data in at least one segmented area may specifically include: First, image preprocessing is performed to enhance the text information in the image and make the text clearer and more recognizable; then, feature extraction is performed on the preprocessed image, and the features of the text in the image are extracted through edge detection technology and texture analysis technology to obtain comprehensive text feature data; image segmentation is performed based on the feature extraction results, and the outline of the text is found by analyzing the connected domain in the image, and segmentation is performed based on the texture features; after segmentation, image post-processing is performed to further optimize and adjust the segmentation results to improve the accuracy and stability of recognition; S13: Extracting features of text patterns with different features separately through feature recognition technology and classification technology to obtain at least one text content, which may specifically include: First, text recognition is performed, and the text is recognized based on the obtained comprehensive text feature data to obtain a first preset text; after text recognition, detection is performed, and the first preset text is randomly sampled, and 20%-40% of the characters in the first preset text are sampled; when the sampling result is qualified, the text recognition is terminated, and when the sampling result is unqualified, a deep analysis is performed: first, the text texture in at least one segmented area is separated by a color-based separation technology, and if the separation is successful, at least two color-separated text image data are obtained, and text recognition is performed on the at least two separated text image data, and the test is performed again. After the test is qualified, the text recognition is terminated; if it is impossible to separate at least two color-separated text image data or the test is unqualified, the image is recognized again by a clustering algorithm: the image features are extracted and classified by a clustering algorithm, and the text image is separated by a morphological operation to obtain at least two feature-separated text image data, and text recognition is performed on the at least two feature-separated text image data, and the test is performed again. After the test is qualified, the text recognition is terminated, otherwise the clustering parameters of the clustering algorithm are modified, and the recognition is performed again; S14: Acquire the user's voice command information data, and recognize the user's voice command data to obtain the voice control command; S15: selecting at least one text content for recognition and analysis according to the recognition result of the voice command data; S16: Based on the recognition and analysis results of the text content, provide auxiliary guidance for the user's learning in at least one of voice, text, video and image forms.

[0021] In this embodiment, by spot-checking the text recognition results on the basis of the basic text recognition algorithm, judging the correctness of the text recognition results, and re-recognizing with an improved image recognition algorithm based on the correctness of the text recognition results, the correctness of the image recognition can be guaranteed, so that when assisting users in learning, the text that users need to input can be accurately recognized, ensuring the accuracy and efficiency of recognition, increasing the user experience, and providing effective guidance to users.

[0022] It should be pointed out that the above steps are only a preferred implementation order. In the specific implementation process, some steps can be replaced without affecting the overall implementation effect. In order to more clearly explain the technical solution of this application, the following content explains this solution in a preferred manner.

[0023] In an embodiment of the present invention, when randomly sampling the first preset text and sampling 20%-40% of the characters in the first preset text, the following steps are included: S41: Reference text selection, identifying the text texture data in the segmented area by optical character recognition technology, identifying the text font corresponding to the text texture data, and selecting a corresponding reference text according to the text content in the first preset text; S42: judging threshold selection, selecting different similarity thresholds as the first threshold for judging character correctness according to the type of the identified text font, and calculating the second threshold for judging the text sampling result according to the image clarity, font type and image background complexity; S42: similarity calculation, randomly selecting characters in the first preset text, the number of selected characters is 20%-40% of all characters in the first preset text, and performing similarity calculation between images corresponding to the selected characters and images of corresponding reference texts; S43: Result determination, determining whether the character is correct based on the calculated similarity value and the selected first threshold; S44: Comprehensive judgment: Comprehensively judge the judgment results of all the selected characters, calculate the total accuracy, and judge whether the sampling result is qualified according to the obtained second threshold.

[0024] Through the above steps S41-S44, in this embodiment, by combining optical character recognition (OCR) technology, the text font is automatically identified and the corresponding reference text is selected, and then the image similarity is calculated for the randomly selected 20%-40% characters, and the double threshold is dynamically set according to the font type, image clarity and background complexity for correctness judgment. This method not only realizes the automatic monitoring of text quality, but also significantly improves the accuracy and efficiency of random inspections, provides reliable quality assurance for the text processing process, and helps to continuously optimize the OCR algorithm and random inspection strategy.

[0025] In another optional embodiment of the present invention, when different similarity thresholds are selected as the first threshold for judging the correctness of characters according to the type of the identified text font, different similarity thresholds are selected for different fonts, such as the similarity threshold for handwritten text is selected as 72%, the similarity threshold for simplified Chinese Song text is selected as 89%, and the similarity threshold for traditional Chinese Song text is selected as 81%. Through a large number of experimental statistics, it is known that when external adjustments such as image clarity, angle and lighting are close to ideal states, in the detection results under the above-mentioned similarity threshold selection, the inspection accuracy rate for handwritten text reaches 98.72%, the inspection accuracy rate for simplified Chinese Song text reaches 99.96%, and the inspection accuracy rate for traditional Chinese Song text reaches 99.45%.

[0026] In another optional embodiment of the present invention, in step S42, according to the following formula: Calculate a second threshold value for judging the text sampling result; in, is the second threshold for judging the text sampling result, is the image clarity, For the ideal image clarity, is the image complexity, The value range of is 0.5-1, and the lower the complexity of the image, The larger the value of is a constant parameter, The size is determined by the font type.

[0027] In this embodiment, by taking the ratio of image clarity to image clarity in an ideal state as the first parameter for calculating the second threshold for determining text sampling results, by taking the transformation result of image complexity as the second parameter for calculating the second threshold for determining text sampling results, and by selecting different benchmark thresholds according to the font type, the two parameters obtained are multiplied by the benchmark threshold to obtain the final second threshold, thereby achieving automated and efficient monitoring and evaluation of text quality, and providing strong support for the optimization and decision-making of text processing processes.

[0028] In an embodiment of the present invention, the image features are extracted and classified by using a clustering algorithm, and the text image is separated by using a morphological operation to obtain at least two feature-separated text image data, including the following steps: S51: Advanced feature extraction, performing advanced feature extraction on at least one segmented region, wherein the advanced feature extraction includes local binary pattern and gray level co-occurrence matrix; S52: Setting the number of clusters: Setting the number of clusters K=n+2, wherein n is the number of times the segmented area is detected, and the extracted high-level features and comprehensive text feature data are input into the K-means clustering algorithm, and the pixels and pixel blocks are divided into n+2 clusters using the K-means clustering algorithm; S53: Eliminate the clusters represented by the background image, map the remaining clusters to the segmented areas, separate the text areas and remove the background noise through morphological operations, and obtain at least two feature-separated text image data.

[0029] Through the above steps S111-S114, when overlapping text is recognized, the image is processed by adopting a clustering algorithm, preprocessing and auxiliary separation are performed using the clustering algorithm, text recognition is performed using a text recognition algorithm, and the recognition result is tested for eligibility, and the number of clusters of the clustering algorithm is changed according to the test result, that is, the number of recognized categories is superimposed in sequence starting from 3, and recognition is performed again after the change, thereby effectively ensuring the accuracy of the recognition result and solving the problem of low recognition success rate and accuracy rate for overlapping text in the existing text recognition process.

[0030] In the embodiment of the present invention, in the above step S12, after the segmentation is completed, image post-processing is performed to further optimize and adjust the segmentation result to improve the accuracy and stability of recognition, including the following steps: S121: Smoothing the segmentation result by using corrosion, expansion, opening operation, closing operation, morphological gradient operation, top hat transformation and bottom hat transformation in morphological operations to remove noise and burrs in the image; S122: performing deduplication and merging processing on repeated characters or broken characters in the segmentation result to ensure that each character is independent and complete.

[0031] In another optional embodiment of the present invention, in the above step S121, the segmented image may be smoothed according to the coordinates of the pixel points, wherein the principle formula is: ; in, is the pixel value of the segmented image at the coordinate (x, y), is the standard deviation of the Gaussian function, k is the size of the filter, i and j are loop variables used to traverse all elements of the filter, The input image is at coordinates The pixel value at is an element in the filter.

[0032] In this embodiment, the coordinates of each pixel point in the segmented image are obtained to calculate the surrounding k 2 The sum of the products of the pixels and the Gaussian function is divided by the sum of the product terms to obtain the smoothed pixel value, which can suppress the noise interference in the analysis of the segmented image and make the image clearer.

[0033] In another optional embodiment of the present invention, in step S11, when acquiring an image containing the topic content and segmenting the image to obtain at least one segmented area, the method further includes: S111: Image enhancement, by adjusting image brightness, contrast, saturation and other parameters, to enhance the text information in the image and make the text more clear and legible; S112: Image denoising, removing noise in the image, such as salt and pepper noise, Gaussian noise, etc., to improve the clarity of the image; S113: image tilt correction, correcting the image tilt angle to keep the text horizontal; S114: grayscale processing of the image, converting the image into a grayscale image for subsequent text detection and segmentation.

[0034] In this embodiment, the captured image is processed through the above steps S111-S114, and image quality problems caused by shooting technology, environmental reasons and shooting equipment clarity reasons are weakened and eliminated, thereby improving image quality and making subsequent text detection and recognition more accurate.

[0035] In an embodiment of the present invention, when the text area is separated and background noise is removed by morphological operation to obtain at least two feature-separated text image data, the following method is included: A11: Reduces the highlight area in the image through erosion operation, removes isolated noise points, and separates touching and close objects; A12: Dilation is used to expand the highlight areas in the image, fill the holes inside the object, and make the edges of the object smoother. A13: Remove small objects through opening operations, separate objects at thin points, and smooth the boundaries of larger objects; A14: Fill small holes inside foreground objects through closing operations, connect adjacent objects, and smooth their boundaries; A15: Highlighting object boundaries through morphological gradient operations.

[0036] In the embodiment of the present invention, after performing text recognition on at least two feature-separated text image data, the method further includes the following steps: S61: Based on the position judgment, the position of the text in the image is analyzed and matched with the expected category or area; S62: Check whether the text content is consistent with the image context; S63: Analyze the semantic meaning of the text to determine whether it fits the overall theme and context of the text in the image; S64: extracting the original image of the recognized erroneous character separately, judging the possible meaning of the erroneous character according to the semantic meaning of the text, and re-recognizing the original image of the erroneous character based on the possible meaning of the erroneous character as a judgment criterion.

[0037] Through the above steps S61-S64, the technical solution further enhances the analysis capability of the matching degree between text and image content on the basis of text recognition. First, based on position judgment, the recognized text is accurately matched with the expected category or area in the image, thereby improving the accuracy of text positioning. Secondly, it checks whether the text content is consistent with the image context to ensure the logical coherence of the text information. Thirdly, it deeply analyzes the semantic meaning of the text to determine whether it is consistent with the overall theme and context of the text in the image, thereby further improving the accuracy and relevance of text recognition. Finally, for the recognized erroneous characters, the possible meaning of the erroneous characters is inferred by extracting their original images separately and combining them with the semantic meaning of the text, and re-identifying them based on this as the standard, thereby effectively reducing the overall text understanding deviation caused by single character recognition errors. It not only improves the accuracy and efficiency of text recognition, but also enhances the understanding of the correlation between text and image content, making the text information closer to the actual context of the image. In addition, through the intelligent re-identification mechanism of erroneous characters, the text recognition error rate is further reduced, and the overall text processing quality and user experience are improved.

[0038] In an embodiment of the present invention, when at least one text content is selected for recognition and analysis according to the recognition result of the voice instruction data, the following steps are included: S71: voice signal conversion and preprocessing, converting the collected voice signal into a digital signal, and filtering, pre-emphasis, framing and windowing the obtained digital signal; S72: Speech feature extraction, converting the preprocessed digital signal from the time domain to the frequency domain, and extracting parameters that can characterize the essential features of the speech signal; S73: Acoustic modeling, using hidden Markov model to match the extracted feature parameters with the preset acoustic model; S74: Language modeling and decoding, modeling language characteristics through a recurrent neural network language model, and then using a decoder to fuse the information of the acoustic model and the language model to parse the most likely text sequence.

[0039] In an embodiment of the present invention, when at least one text content is selected for recognition and analysis according to the recognition result of the voice instruction data, the following steps are also included: S81: text preprocessing, preprocessing the parsed text sequence, wherein the preprocessing methods used include noise removal, word segmentation and part-of-speech tagging; S82: Named entity recognition, performing named entity recognition based on text preprocessing to identify specific entities in the instruction; S83: Intent recognition, using natural language processing technology to analyze the semantic content of instructions and identify the user's intention; S84: Parameter extraction: based on identifying the user's intention, further extracting key parameters in the instruction; S85: instruction parsing and construction, constructing an instruction parsing tree or similar data structure based on the user's intention and the extracted parameters, and decomposing complex instructions into a series of executable subtasks; S86: Execute instructions.

[0040] In another optional embodiment of the present invention, when selecting at least one text content for recognition and analysis based on the recognition results of the voice command data, the user may need to convert a large number of paper documents or scans into editable electronic documents, and perform content analysis and extraction. Through the embodiment of instruction parsing and construction, the user can input complex instructions, such as "identify all tables in the document and extract the data therein", and the system will automatically parse the instructions, decompose them into multiple sub-tasks such as text recognition, table positioning, data extraction, etc., and execute them in sequence, and finally output a structured data table.

[0041] In an embodiment of the present invention, when providing auxiliary guidance for the user's learning in at least one of voice, text, video and image forms based on the recognition and analysis results of the text content, the following steps are included: S91: Perform corresponding operations and answers according to the user's instructions; S92: Use knowledge graph technology to deeply analyze the knowledge points contained in the question content, identify and extract key knowledge points, and use machine learning algorithms to predict the difficulties that users may encounter based on their learning history and performance; S93: Based on the obtained knowledge points and difficulties, search for relevant audio, pictures and videos in a pre-set database, and sort the searched audio, pictures and videos according to the set importance coefficient and the proportion coefficient of the knowledge point in the question, and use this order as the default playback order to play for the user.

[0042] In another optional embodiment of the present invention, the user can acquire images and watch videos and graphics through an AR device. When the user acquires images and watches videos and graphics through an AR device, the user first controls the AR device or other image device through voice commands to collect the required images, and then performs text recognition through the above-mentioned image recognition steps. After the text recognition is completed, the user selects the required text content through voice commands, and modifies, organizes and converts the text content through voice commands. After completion, the text content is parsed. At the same time, the knowledge graph technology is used to deeply analyze the knowledge points contained in the question content, identify and extract key knowledge points, and combine the user's learning history and performance to predict the difficulties that the user may encounter. The next step is to search in a pre-set database for relevant tutoring materials, explanation videos, demonstration animations and related topics. When explaining and analyzing the topic for the user, the AR device plays the searched relevant tutoring materials, explanation videos, demonstration animations and related topics to deepen the learning and ensure the effectiveness of the user's learning.

[0043] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. AR-AI integrated interactive learning assistance method; characterized by: The following steps are included: S11: Acquire an image containing the topic content and segment the image to obtain at least one segmented area; S12: Recognize text patterns in at least one segmented area by using image recognition technology to obtain text pattern data in at least one segmented area; S13: extracting features of text patterns with different features separately through feature recognition technology and classification technology to obtain at least one text content; S14: Acquire the user's voice command information data, and recognize the user's voice command data to obtain the voice control command; S15: selecting at least one text content for recognition and analysis according to the recognition result of the voice command data; S16: Based on the recognition and analysis results of the text content, provide auxiliary guidance for the user's learning in at least one of voice, text, video and image forms.

2. The AR-AI integrated interactive learning assistance method according to claim 1, characterized in that: When the text pattern in at least one segmented area is recognized by image recognition technology to obtain the text pattern data in the segmented area, the following steps are included: S21: Image preprocessing, enhancing the text information in the image to make the text clearer and more legible; S22: feature extraction, extracting the features of the text in the image through edge detection technology and texture analysis technology to obtain comprehensive text feature data; S23: Image segmentation, by analyzing the connected domain in the image, finding the outline of the text and segmenting it according to the texture features; S24: Image post-processing: further optimization and adjustment of the segmentation results to improve the accuracy and stability of recognition.

3. The AR-AI integrated interactive learning assistance method according to claim 2, characterized in that: When text patterns with different features are extracted separately by using feature recognition technology and classification technology to obtain at least one text content, the following steps are included: S31: text recognition, based on the obtained comprehensive text feature data, the text is recognized to obtain a first preset text; S32: Randomly check the first preset text, checking 20%-40% of the characters in the first preset text; S33: When the sampling inspection result is qualified, the text recognition is terminated; when the sampling inspection result is unqualified, the next step is entered; S34: First, the text patterns in at least one segmented area are separated by a color-based separation technology. If the separation is successful, at least two color-separated text image data are obtained, and text recognition is performed on the at least two separated text image data, and then the step S32 is performed for inspection. If the inspection is qualified, the text recognition is terminated, otherwise it goes to the next step; S35: Use a clustering algorithm to extract and classify image features, and use morphological operations to separate text images to obtain at least two feature-separated text image data, and perform text recognition on at least two feature-separated text image data, and then perform a test through step S32. If the test is qualified, the text recognition is terminated. Otherwise, the clustering parameters of the clustering algorithm are modified and recognition is performed again.

4. The AR-AI integrated interactive learning assistance method according to claim 3, characterized in that: When randomly sampling the first preset text and sampling 20%-40% of the characters in the first preset text, the following steps are included: S41: Reference text selection, identifying the text texture data in the segmented area by optical character recognition technology, identifying the text font corresponding to the text texture data, and selecting a corresponding reference text according to the text content in the first preset text; S42: judging threshold selection, selecting different similarity thresholds as the first threshold for judging character correctness according to the type of the identified text font, and calculating the second threshold for judging the text sampling result according to the image clarity, font type and image background complexity; S42: similarity calculation, randomly selecting characters in the first preset text, the number of selected characters is 20%-40% of all characters in the first preset text, and performing similarity calculation between images corresponding to the selected characters and images of corresponding reference texts; S43: Result determination, determining whether the character is correct based on the calculated similarity value and the selected first threshold; S44: Comprehensive judgment: Comprehensively judge the judgment results of all the selected characters, calculate the total accuracy, and judge whether the sampling result is qualified according to the obtained second threshold.

5. The AR-AI integrated interactive learning assistance method according to claim 4, characterized in that: The method extracts and classifies image features by using a clustering algorithm and separates text images by using morphological operations to obtain at least two feature-separated text image data, including the following steps: S51: Advanced feature extraction, performing advanced feature extraction on at least one segmented region, wherein the advanced feature extraction includes local binary pattern and gray level co-occurrence matrix; S52: setting the number of clusters K=n+2, where n is the number of times the segmented area is detected, and inputting the extracted high-level features and comprehensive text feature data into a K-means clustering algorithm, and using the K-means clustering algorithm to divide pixels and pixel blocks into n+2 clusters; S53: Eliminate the clusters represented by the background image, map the remaining clusters to the segmented areas, separate the text areas and remove the background noise through morphological operations, and obtain at least two feature-separated text image data.

6. The AR-AI integrated interactive learning assistance method according to claim 5, characterized in that: When the text area is separated and background noise is removed by morphological operation to obtain at least two feature-separated text image data, the following method is included: A11: Reduces the highlight area in the image through erosion operation, removes isolated noise points, and separates touching and close objects; A12: Dilation is used to expand the highlight areas in the image, fill the holes inside the object, and make the edges of the object smoother. A13: Remove small objects through opening operations, separate objects at thin points, and smooth the boundaries of larger objects; A14: Fill small holes inside foreground objects through closing operations, connect adjacent objects, and smooth their boundaries; A15: Highlighting object boundaries through morphological gradient operations.

7. The AR-AI integrated interactive learning assistance method according to claim 6, characterized in that: After performing text recognition on at least two feature-separated text image data, the following steps are also included: S61: Based on the position judgment, the position of the text in the image is analyzed and matched with the expected category or area; S62: Check whether the text content is consistent with the image context; S63: Analyze the semantic meaning of the text to determine whether it fits the overall theme and context of the text in the image; S64: extracting the original image of the recognized erroneous character separately, judging the possible meaning of the erroneous character according to the semantic meaning of the text, and re-recognizing the original image of the erroneous character based on the possible meaning of the erroneous character as a judgment criterion.

8. The AR-AI integrated interactive learning assistance method according to claim 7, characterized in that: The steps of obtaining the user's voice command information data, recognizing the user's voice command data, and obtaining the voice control command include: S71: voice signal conversion and preprocessing, converting the collected voice signal into a digital signal, and filtering, pre-emphasis, framing and windowing the obtained digital signal; S72: Speech feature extraction, converting the preprocessed digital signal from the time domain to the frequency domain, and extracting parameters that can characterize the essential features of the speech signal; S73: Acoustic modeling, using hidden Markov model to match the extracted feature parameters with the preset acoustic model; S74: Language modeling and decoding, modeling language characteristics through a recurrent neural network language model, and then using a decoder to fuse the information of the acoustic model and the language model to parse the most likely text sequence.

9. The AR-AI integrated interactive learning assistance method according to claim 8, characterized in that: When at least one text content is selected for recognition and analysis according to the recognition result of the voice instruction data, the following steps are included: S81: text preprocessing, preprocessing the parsed text sequence, wherein the preprocessing methods used include noise removal, word segmentation and part-of-speech tagging; S82: Named entity recognition, performing named entity recognition based on text preprocessing to identify specific entities in the instruction; S83: Intent recognition, using natural language processing technology to analyze the semantic content of instructions and identify the user's intention; S84: Parameter extraction: based on identifying the user's intention, further extracting key parameters in the instruction; S85: instruction parsing and construction, constructing an instruction parsing tree or similar data structure based on the user's intention and the extracted parameters, and decomposing complex instructions into a series of executable subtasks; S86: Execute instructions.

10. The AR-AI integrated interactive learning assistance method according to claim 9, characterized in that: When providing auxiliary guidance for the user's learning in at least one of voice, text, video and image forms based on the recognition and analysis results of the text content, the following steps are included: S91: Perform corresponding operations and answers according to the user's instructions; S92: Use knowledge graph technology to deeply analyze the knowledge points contained in the question content, identify and extract key knowledge points, and use machine learning algorithms to predict the difficulties that users may encounter based on their learning history and performance; S93: Based on the obtained knowledge points and difficulties, search for relevant audio, pictures and videos in a pre-set database, and sort the searched audio, pictures and videos according to the set importance coefficient and the proportion coefficient of the knowledge point in the question, and use this order as the default playback order to play for the user.

Images