Dynamic adjustment of commands in augmented reality experiences

Abstract

Systems and methods for augmented reality tutoring may utilize optical character recognition, natural language processing, and / or augmented reality rendering to provide real-time notifications for completing a determined task. The systems and methods may include utilizing one or more machine-learned models trained for quantitative reasoning and may include providing multiple different user interface elements at different times.

Description

TECHNICAL FIELD

[0001] Claims to Priority This application claims priority to U.S. Application No. 17 / 969,303 filed on October 19, 2022, the entire contents of which are hereby incorporated herein by reference.

[0002] The present disclosure relates generally to error detection and corrective action communication via augmented reality experiences. More specifically, the present disclosure relates to acquiring image data, processing the image data to determine that an error exists in the image data, determining a corrective action, and providing one or more user interface elements for display to indicate the corrective action. BACKGROUND ART

[0003] Determining errors in an environment and figuring out how to correct the determined errors can be difficult. In particular, errors in text can be difficult to detect. Furthermore, if an error is not detected, the error may lead to propagation of further errors, which in turn leads to further confusion. The lack of real-time error detection may cause users to spend time on issues without understanding when and where they failed.

[0004] Furthermore, some errors and / or problems may involve difficult and complex responses to resolve the problem or error. Such difficult and complex responses may lead to user confusion. Therefore, attempts to resolve the error may lead to further errors. SUMMARY OF THE INVENTION

[0005] Aspects and advantages of embodiments of the present disclosure are set forth in part in the description that follows, or may be learned from the description, or may be learned through practice of the embodiments.

[0006] One exemplary aspect of this disclosure relates to a computing system. The system may include one or more processors and one or more non-temporary computer-readable media that, when executed by one or more processors, collectively store instructions causing the computing system to perform an operation. An operation may include acquiring image data. The image data may describe one or more images. In some embodiments, one or more images may describe an environment. An operation may include processing the image data to generate semantic data. The semantic data may describe a semantic understanding of at least a portion of one or more images. An operation may include determining an error in one or more images based at least in part on the semantic data. An operation may include determining a corrective action based on the semantic data and the error. In some embodiments, the corrective action may describe at least one of replacing the error or correcting the error. An operation may include providing a user interface element for display based on the corrective action. The user interface element may include informational data describing the corrective action.

[0007] In some embodiments, determining an error in one or more images based at least partially on semantic data may include obtaining a specific machine-learned model based on the semantic data and processing the image data with the specific machine-learned model to detect the error. The error may include a discrepancy with semantic understanding. In some embodiments, the error may include a deviation from a multipart process. The multipart process may be associated with semantic data. In some embodiments, determining a corrective action based on semantic data and the error may include detecting the location of the error in the environment, determining an errorless dataset associated with the semantic data and one or more images, and determining replacement data from the errorless dataset based on the location of the error in the environment.

[0008] In some embodiments, errors may be determined using an error detection model. The error detection model may generate text data based on optical character recognition, parse the text data based on one or more features in the environment, and process each of the parsed segments to determine the error. In some embodiments, the error detection model may be trained with multiple mathematical proofs. The error detection model may include an optical character recognition model and a natural language processing model.

[0009] In some embodiments, image data may be generated by one or more image sensors of a mobile computing device. User interface elements may be provided for display by the mobile computing device. The mobile computing device may be a smart wearable.

[0010] Other exemplary embodiments of this disclosure relate to methods implemented on a computer. A method may include acquiring image data by a computing system comprising one or more processors. The image data may describe one or more images. In some embodiments, one or more images may describe one or more pages. A method may include processing the image data with an optical character recognition model by the computing system to generate text data. In some embodiments, the text data may describe text on one or more pages. A method may include determining a prompt based on the text data by the computing system. The prompt may describe a request for a response. A method may include determining a multipart response to a prompt by the computing system. The multipart response may include a plurality of separate responses associated with the prompt. A method may include acquiring additional image data by the computing system. The additional image data may describe one or more additional images. In some embodiments, one or more additional images may describe one or more pages having user-generated text. A method may include processing the additional image data with an optical character recognition model by the computing system to generate additional text data. The additional text data may describe user-generated text on one or more pages. The method may include a computing system determining that user-generated text deviates from a multipart response, and the computing system providing a notification. The notification may describe the user-generated text that has the error.

[0011] In some embodiments, the computing system's determination that user-generated text deviates from a multipart response may include determining that the user-generated text is inconsistent with the multipart response. The computing system's determination that user-generated text deviates from a multipart response may include determining that the user-generated text lacks one or more specific features of the multipart response. In some embodiments, one or more pages may include one or more questions. User-generated text may include the user's responses to one or more questions. The method may include the computing system processing image data with a machine learning model to determine prompts and multipart responses. In some embodiments, the method may include the computing system processing additional image data with a machine learning model to determine that user-generated text deviates from a multipart response.

[0012] Other exemplary aspects of this disclosure relate to one or more non-temporary computer-readable media that, when executed by one or more computing devices, collectively store instructions causing one or more computing devices to perform an action. An action may include acquiring image data. The image data may describe one or more images. In some embodiments, one or more images may describe one or more pages. One or more pages may contain multiple characters. An action may include processing the image data to generate semantic data. The semantic data may describe the semantic understanding of at least some of the multiple characters. An action may include determining, based at least in part on the semantic data, that the multiple characters contain an error. An error may describe text that is at least one of the opposite or inaccurate of semantic understanding. An action may include determining a corrective action based on the semantic data and the error. In some embodiments, the corrective action may describe at least one of replacing the error or correcting the error. An action may include providing a user interface element for display based on the corrective action. The user interface element may include informational data describing the corrective action.

[0013] In some embodiments, a user interface element may include one or more popup elements that describe a set of sub-actions for performing a correction action. A user interface element may include an inline overlay. An inline overlay may be used to extend one or more images or at least one of one or more additional images to generate one or more extended images. In some embodiments, one or more extended images may include an inline overlay superimposed on at least a portion of one or more pages. An inline overlay may describe a correction action. In some embodiments, a user interface element may include extending one or more images to indicate the location of an error.

[0014] Other aspects of this disclosure cover a variety of systems, apparatus, non-temporary computer-readable media, user interfaces, and electronic devices.

[0015] These and other features, aspects and advantages of the various embodiments of this disclosure will be better understood by referring to the following description and the appended claims. The appended drawings incorporated herein and forming part of this specification illustrate exemplary embodiments of this disclosure and, together with modes for carrying out the invention, illustrate the relevant principles.

[0016] A detailed description of embodiments intended for those skilled in the art is given herein with reference to the accompanying drawings. [Brief explanation of the drawing]

[0017] [Figure 1A] A block diagram of an exemplary computing system performing augmented reality tutoring according to an exemplary embodiment of the present disclosure is shown. [Figure 1B] A block diagram of an exemplary computing device performing augmented reality tutoring according to an exemplary embodiment of the present disclosure is shown. [Figure 1C] A block diagram of an exemplary computing device performing augmented reality tutoring according to an exemplary embodiment of the present disclosure is shown. [Figure 2A] Examples of exemplary augmented reality experiences are shown in the exemplary embodiments of this disclosure. [Figure 2B] Examples of exemplary augmented reality experiences are shown in the exemplary embodiments of this disclosure. [Figure 2C] Examples of exemplary augmented reality experiences are shown in the exemplary embodiments of this disclosure. [Figure 2D] Examples of exemplary augmented reality experiences are shown in the exemplary embodiments of this disclosure. [Figure 2E] Examples of exemplary augmented reality experiences are shown in the exemplary embodiments of this disclosure. [Figure 3] shows a block diagram of an exemplary computing system for performing augmented reality tutoring according to exemplary embodiments of the present disclosure. [Figure 4] shows a block diagram of an exemplary augmented reality tutoring system according to exemplary embodiments of the present disclosure. [Figure 5] shows a block diagram of an exemplary augmented reality tutoring system according to exemplary embodiments of the present disclosure. [Figure 6] shows a flowchart diagram of an exemplary method for performing augmented reality tutoring according to exemplary embodiments of the present disclosure. [Figure 7] shows a flowchart diagram of an exemplary method for performing image data processing and corrective action determination according to exemplary embodiments of the present disclosure. [Figure 8] shows a flowchart diagram of an exemplary method for performing video data processing and corrective action determination according to exemplary embodiments of the present disclosure. [Figure 9] shows an example of an exemplary smart wearable for acquiring image data and providing user interface elements according to exemplary embodiments of the present disclosure.

[0018] Reference numerals repeated across the several drawings are intended to identify the same features in various implementations. [Mode for Carrying Out the Invention]

[0019] Summary In general, this disclosure covers augmented reality experiences that provide multiple augmented reality assets based on processed image data. In particular, the systems and methods disclosed herein can leverage image processing (e.g., optical character recognition and / or object recognition), semantic understanding, and / or one or more user interface elements (e.g., inline overlays, pop-up speech bubbles, and / or highlights) to provide real-time instructions for error correction and / or problem solving. In some embodiments, the systems and methods may be used for augmented reality tutoring, DIY (do-it-yourself) projects, and / or error detection and error correction.

[0020] For example, the system and method may include acquiring image data. The image data may describe one or more images. In some embodiments, one or more images may describe an environment. The system and method may include processing the image data to generate semantic data. The semantic data may describe the semantic understanding of at least a portion of one or more images. The system and method may include determining errors in one or more images based at least in part on the semantic data. In some embodiments, the system and method may include determining a corrective action based on the semantic data and the errors. The corrective action may describe at least one of replacing the error or correcting the error. The system and method may include providing user interface elements for display based on the corrective action. The user interface elements may include informational data describing the corrective action.

[0021] Image data may be acquired. The image data may describe one or more images. In some embodiments, one or more images may describe an environment. The environment may include one or more issues. For example, the environment may include questions for the user to respond to. Alternatively and / or additionally, the environment may include objects for completing a DIY project. The image data may be generated by one or more image sensors of a mobile computing device (e.g., a smartphone). In some embodiments, the mobile computing device may be a smart wearable (e.g., smart glasses).

[0022] Image data may be processed to generate semantic data. Semantic data may describe the semantic understanding of at least a portion of one or more images. In some embodiments, image data may be processed with a semantic understanding model. The semantic understanding model may include one or more machine-learned models. The semantic understanding model may include natural language processing models (e.g., one or more large language models trained on multiple examples). In some embodiments, the semantic understanding model may include machine-learned models trained to understand equations and / or other quantitative representations (e.g., language models trained for quantitative reasoning, as discussed in Dyer et al., Minerva: Solving Quantitative Reasoning Problems with Language Models, GOOGLE AI BLOG (June 30, 2022), https: / / ai.googleblog.com / 2022 / 06 / minerva-solving-quantitative-reasoning.html). Additionally and / or alternatively, image data may be processed with an optical character recognition model to generate text data, which may then be processed with a semantic understanding model. Semantic data may be based on text content, recognized objects, data structure, data layout, information structure, one or more figures, additional input data received, image capture context, image capture device type, user profile data, and / or one or more other contexts. Semantic data may include one or more queries summarizing an issue (e.g., a question) at the focal point of one or more images.

[0023] The system and method may determine errors in one or more images based at least partially on semantic data. Errors may include inconsistencies with semantic understanding. In some embodiments, errors may include deviations from a multipart process. A multipart process may be associated with semantic data. For example, a multipart process may include one or more actions to respond to a question and / or to solve a problem. Errors may be determined based on heuristics, based on acquired data, and / or based on the output of a machine-trained model. Errors may be determined based on the fact that handwritten text differs from the semantic intent of printed text. For example, semantic data may include the semantic intent of printed text and the semantic understanding of handwritten text. If the semantic understanding of handwritten text does not relate to the semantic intent of printed text, an error may be determined.

[0024] In some embodiments, determining errors in one or more images based at least partially on semantic data may include obtaining a specific machine-learned model based on the semantic data and processing the image data with the specific machine-learned model to detect errors. For example, the semantic data may describe a specific problem type (e.g., a literature analysis problem, a calculus problem, and / or an organic chemistry problem) and a problem-specific machine-learned model (e.g., a literature analysis model, a calculus model, and / or an organic chemistry model). Alternatively and / or additionally, a mathematical engine (e.g., a system of mathematical functions used to process the problem using one or more processors) may be obtained and used based on the semantic data.

[0025] Alternatively and / or additionally, determining corrective actions based on semantic data and errors may include detecting the location of errors in the environment, determining error-free datasets associated with the semantic data and one or more images, and determining replacement data from the error-free datasets based on the location of errors in the environment.

[0026] In some embodiments, errors may be determined using an error detection model. The error detection model may generate text data based on optical character recognition. The error detection model may parse the text data based on one or more features in the environment. In some embodiments, the error detection model may process each of several parsed segments to determine the error. The error detection model may be trained with several mathematical proofs. Additionally and / or alternatively, the error detection model may include an optical character recognition model and a natural language processing model.

[0027] Correction actions may be determined based on semantic data and errors. A correction action may describe at least one of the following: replacing an error or correcting an error. In some embodiments, a correction action may include indicating the location of the error in the environment and one or more actions to correctly respond to a prompt identified in the environment.

[0028] The system and method may provide user interface elements for display based on a correction action. The user interface elements may include informational data describing the correction action. In some embodiments, the user interface elements may be provided for display on a mobile computing device. The user interface elements may be provided through an augmented reality experience. The user interface elements may include prompts, inline comments, pop-up speech bubbles, and / or highlighting one or more arrows.

[0029] Additionally and / or alternatively, the system and method may process image data sequentially to determine and correct user actions in real time by one or more user interface elements provided in response to determined errors. For example, the system and method may include acquiring image data. The image data may describe one or more images. In some embodiments, one or more images may describe one or more pages. The image data may be processed with an optical character recognition model to generate text data. The text data may describe text on one or more pages. The system and method may include determining a prompt based on the text data. The prompt may describe a request for a response. The system and method may include determining a multipart response to the prompt. The multipart response may include multiple separate responses associated with the prompt. In some embodiments, the system and method may include acquiring additional image data. The additional image data may describe one or more additional images. One or more additional images may describe one or more pages having user-generated text (e.g., additional handwritten text and / or user-typed data (e.g., user-generated code and / or user-generated equations)). Additional image data may be processed by an optical character recognition model to generate additional text data. In some embodiments, the additional text data may describe user-generated text on one or more pages. The system and method may include determining that the user-generated text deviates from a multipart response and providing a notification. The notification may describe the user-generated text that has errors.

[0030] The system and method may acquire image data. The image data may describe one or more images. One or more images may describe one or more pages. In some embodiments, one or more pages may contain one or more questions. One or more pages may contain printed text and handwritten text. One or more questions may contain scientific questions including mathematical formulas, writing prompts, and / or one or more figures.

[0031] Image data may be processed by an optical character recognition (OCR) model to generate text data. The text data may describe text on one or more pages. The OCR model may include one or more pre-trained models. The OCR model may include models specifically trained on handwritten text. The text data may include recognized printed text and / or recognized handwritten text.

[0032] The system and method may determine prompts based on text data. Prompts may describe a request for a response. Prompts may be determined based on semantic understanding of text on one or more pages. Alternatively and / or additionally, prompts may be queries generated based on recognized text. Prompts may be determined based on text containing one or more keywords associated with one or more prompts and / or one or more prompt types.

[0033] The system and method may determine a multipart response to a prompt. A multipart response may include multiple separate responses associated with the prompt. A multipart response may be determined based on the output of a machine learning model, based on heuristics, based on one or more search results and / or one or more knowledge graphs received from a search engine. A multipart response may be based on the output of a machine learning model trained on one or more textbooks. For example, a machine learning model may be trained to identify a particular type of problem based on one or more identified features, and the same or separate models may be trained to generate a proof that describes how to solve a particular problem. The generated proof may be a multipart response in which each line of the proof is part of the response.

[0034] Additional image data may be obtained. The additional image data may describe one or more additional images. One or more additional images may describe one or more pages containing user-generated text (e.g., additional handwritten text and / or user-typed data (e.g., user-generated code and / or user-generated equations)). The user-generated text may include the user's responses to one or more questions.

[0035] The additional image data may be processed by an optical character recognition model to generate additional text data. This additional text data may describe user-generated text on one or more pages and / or computer screens. The user-generated text may describe the user's attempts to respond to prompts (e.g., answering questions).

[0036] User-generated text may be determined to be a deviation from a multipart response. The deviation may be such that the user-generated text is the opposite of the multipart response. For example, a multipart response may involve performing a first action and then taking a second action, while user-generated text may involve performing a first action and then taking a third action that is not equivalent to the second action.

[0037] In some embodiments, determining that user-generated text deviates from a multipart response may include determining that user-generated text is inconsistent with the multipart response. For example, user-generated text may contain a semantic intent that is inconsistent with the semantic intent of one or more parts of the multipart response.

[0038] Alternatively and / or additionally, determining that user-generated text deviates from a multipart response may include determining that the user-generated text lacks one or more specific features of the multipart response. For example, the multipart response may involve multiplying both sides of an equation by 2, while the user-generated text multiplies only one side by 2.

[0039] The system and method may provide notifications. Notifications may describe user-generated text indicating an error. Notifications may be provided by an augmented reality experience rendering one or more user interface elements for providing notifications. Notifications may describe where the error occurred and how to resolve the error.

[0040] In some embodiments, the system and method may process image data with a machine learning model to determine prompts and multipart responses. The machine learning model may be a language model trained on quantitative inference. In some embodiments, the machine learning model may be trained on one or more subjects, particularly using academic materials (e.g., textbooks and / or academic papers).

[0041] In some embodiments, the system and method may process additional image data with a machine learning model to determine if user-generated text deviates from a multipart response.

[0042] The system and method may include acquiring image data. The image data may describe one or more images. In some embodiments, one or more images may describe one or more pages. One or more pages may contain multiple characters. The image data may be processed to generate semantic data. The semantic data may describe the semantic understanding of at least some of the multiple characters. The system and method may include determining, at least in part, that the multiple characters contain errors. The errors may describe text that is at least one of the opposite or inaccurate semantic understanding. In some embodiments, the system and method may include determining a corrective action based on the semantic data and the errors. The corrective action may describe at least one of replacing the errors or correcting the errors. The system and method may include providing user interface elements for display based on the corrective action. The user interface elements may include informational data describing the corrective action.

[0043] The system and method may acquire image data. The image data may describe one or more images. One or more images may describe one or more pages. In some embodiments, one or more pages may contain multiple characters. Multiple characters may be part of a problem (e.g., a question, a writing prompt, and / or an assignment statement). Characters may include letters, numbers, and / or symbols. One or more pages may contain text, pictures, shapes, figures, and / or whitespace.

[0044] Image data can be processed to generate semantic data. Semantic data can describe the semantic understanding of at least some of multiple characters. In some embodiments, semantic data may be based on text, pictures, shapes, figures, and / or whitespace.

[0045] The system and method may determine, at least partially, that multiple characters contain errors. Errors may describe text that is either the opposite of semantic understanding or inaccurate. Inaccuracies may be determined by processing multiple characters with one or more machine learning models.

[0046] The system and method may determine corrective actions based on semantic data and errors. A corrective action may describe at least one of the following: replacing an error or correcting an error. A corrective action may include deletion actions (e.g., deleting a subset of characters) and write actions (e.g., writing one or more new characters).

[0047] The system and method may provide user interface elements for display based on correction actions. User interface elements may include informational data describing correction actions. In some embodiments, user interface elements may include one or more popup elements describing a plurality of sub-actions for performing the correction action. User interface elements may include inline overlays. Inline overlays may be used to extend one or more images or at least one of one or more additional images to generate one or more extended images. One or more extended images may include inline overlays superimposed on at least a portion of one or more pages. In some embodiments, inline overlays may describe correction actions. Additionally and / or alternatively, user interface elements may include extending one or more images to indicate the location of an error.

[0048] In some embodiments, the system and method may include sequential acquisition of image data, sequential processing of image data, sequential error diagnosis, and / or sequential generation of user interface elements for correcting errors. For example, the system and method may utilize streaming optical character recognition. User interface elements (e.g., user interface elements for notifications) may be provided in a conversational format so that multipart responses can be provided step-by-step as the user progresses through different parts of problem solving.

[0049] In some embodiments, one or more machine learning models may be trained on one or more textbooks, real-world flashcards (e.g., flashcards of foreign languages, or scientific names of structures, elements, or compounds), architectural drawings, knowledge graphs, and / or study guides. Additionally and / or alternatively, one or more machine learning models may be trained on proofs. Training may include training to fit the output of a rule engine. Alternatively and / or additionally, training may include black-box optimization. The system and method may be used for writing tasks, language learning tasks, mathematical problem-solving tasks (e.g., algebra, calculus, and / or discrete mathematics), scientific problem-solving tasks (e.g., physics, organic chemistry, biology, and / or chemistry), architectural design (e.g., tracing lines and tracking measurements), and / or surgical procedures. The system and method may determine that a prompt includes multiple requested criteria, and the user's response may be processed to determine which criteria are met and which are not.

[0050] The systems and methods disclosed herein may be initiated based on one or more inputs (e.g., voice commands, button selections, and / or visual cues).

[0051] In some embodiments, the system and method may include parsing one or more images and / or parsing text data generated by optical character recognition. The parsing may be based on lines, paragraphs, syntax, page structure, image regions, and / or one or more other features.

[0052] In some embodiments, one or more teachers may be provided with a software development kit to tailor the augmented reality tutor to their specific class, course curriculum, and / or teaching style.

[0053] The systems and methods disclosed herein offer several technical effects and advantages. For example, the systems and methods can provide an augmented reality tutoring experience. In particular, the systems and methods disclosed herein can leverage optical character recognition, natural language processing, and augmented reality rendering to provide an interactive experience for error identification and for providing multipart responses.

[0054] Another technical advantage of the systems and methods of this disclosure is their ability to leverage one or more machine learning models to understand an environment and provide a step-by-step process for completing a task. For example, the systems and methods can determine the meaning of an environment, determine prompts related to the environment, determine multipart responses related to the prompts, and sequentially collect data to ensure that the user completes actions related to the multipart responses.

[0055] Other examples of technical effects and benefits relate to improvements in computational efficiency and the functionality of computing systems. For example, the systems and methods disclosed herein can leverage the storage of determined prompts and multipart responses to continuously compare additionally acquired data with multipart responses without having to continuously redetermine the meaning of the environment for error detection.

[0056] Exemplary embodiments of this disclosure will now be described in more detail with reference to the drawings.

[0057] Exemplary devices and systems Figure 1A shows a block diagram of an exemplary computing system 100 that performs dynamic command adjustment in an augmented reality experience according to an exemplary embodiment of the present disclosure. The system 100 includes a user computing device 102, a server computing system 130, and a training computing system 150, all of which are communicably coupled via a network 180.

[0058] The user computing device 102 can be any type of computing device, such as a personal computing device (e.g., a laptop or desktop), a mobile computing device (e.g., a smartphone or tablet), a game console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0059] The computing device 102 includes one or more processors 112 and memory 114. One or more processors 112 may be any suitable processing device (e.g., a processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.), and may be a single processor or multiple processors connected in an operable manner. The memory 114 may include one or more non-temporary computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, and combinations thereof. The memory 114 may store data 116 and instructions 118 that are executed by the processors 112 to cause the user computing device 102 to perform operations.

[0060] In some embodiments, the user computing device 102 may store or include one or more semantic understanding models 120. For example, the semantic understanding models 120 may be various machine learning models, such as neural networks (e.g., deep neural networks), or other types of machine learning models, including nonlinear and / or linear models, or may otherwise include them. The neural networks may include feedforward neural networks, recurrent neural networks (e.g., long-short-term memory recurrent neural networks), convolutional neural networks, or other forms of neural networks. Exemplary semantic understanding models 120 are illustrated with reference to Figures 2A to 5.

[0061] In some embodiments, one or more semantic understanding models 120 may be received from a server computing system 130 via a network 180, stored in a user computing device memory 114, and then used by one or more processors 112 or otherwise implemented. In some embodiments, the user computing device 102 may implement multiple parallel instances of a single semantic understanding model 120 (for example, to perform parallel image data processing across multiple instances of an image of the environment).

[0062] More specifically, the system and method may utilize one or more machine learning models, which may include one or more semantic understanding models 120 capable of processing image data, text data, and / or audio data to generate semantic data associated with an environment. The semantic data may be used to detect errors in the environment and generate corrective actions to correct those errors.

[0063] Additionally or alternatively, one or more semantic understanding models 140 may be included in a server computing system 130 that communicates with a user computing device 102 according to a client-server relationship, or may be stored and implemented in other ways by the server computing system 130. For example, a semantic understanding model 140 may be implemented by the server computing system 130 as part of a web service (e.g., a tutoring service). Thus, one or more models 120 may be stored and implemented in the user computing device 102, and / or one or more models 140 may be stored and implemented in the server computing system 130.

[0064] The user computing device 102 may include one or more user input components 122 that receive user input. For example, a user input component 122 may be a touch-sensitive component (e.g., a touch-sensitive display screen or touchpad) that is sensitive to the touch of a user input object (e.g., a finger or stylus). The touch-sensitive component may function to implement a virtual keyboard. Other exemplary user input components include a microphone, a regular keyboard, or other means by which the user can provide user input.

[0065] The server computing system 130 includes one or more processors 132 and memory 134. The one or more processors 132 may be any suitable processing device (e.g., a processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and may be one processor or multiple processors connected in an operable manner. The memory 134 may include one or more non-temporary computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, and combinations thereof. The memory 134 can store data 136 and instructions 138 executed by the processors 132 to cause the server computing system 130 to perform operations.

[0066] In some embodiments, the server computing system 130 includes or is implemented by one or more server computing devices. If the server computing system 130 includes multiple server computing devices, such server computing devices may operate in a sequential computing architecture, a parallel computing architecture, or any combination thereof.

[0067] As described above, the server computing system 130 can store or otherwise include one or more machine learning-prepared semantic understanding models 140. For example, model 140 can be a variety of machine learning-prepared models or can otherwise include a variety of machine learning-prepared models. Exemplary machine learning-prepared models include neural networks or other multilayer nonlinear models. Exemplary neural networks include feedforward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Exemplary models 140 are illustrated with reference to Figures 2A to 5.

[0068] The user computing device 102 and / or the server computing system 130 can train models 120 and / or 140 by interacting with a training computing system 150 which is connected communicatively via a network 180. The training computing system 150 may be separate from the server computing system 130 or may be part of the server computing system 130.

[0069] The training computing system 150 includes one or more processors 152 and memory 154. The one or more processors 152 may be any suitable processing device (e.g., a processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and may be a single processor or multiple processors connected in an operable manner. The memory 154 may include one or more non-temporary computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, and combinations thereof. The memory 154 can store data 156 and instructions 158 executed by the processors 152 to cause the training computing system 150 to perform operations. In some embodiments, the training computing system 150 includes one or more server computing devices or is otherwise implemented by them.

[0070] The training computing system 150 may include a model trainer 160 that trains machine-learned models 120 and / or 140 stored in the user computing device 102 and / or server computing system 130 using various training or learning techniques, such as backpropagation. For example, a loss function may be backpropagated through the model(s) to update one or more parameters of the model(s) (e.g., based on the gradient of the loss function). Various loss functions can be used, such as mean squared error, likelihood loss, cross-entropy loss, hinge loss, and / or various other loss functions. Gradient descent may be used to iteratively update parameters over several training iterations.

[0071] In some embodiments, backpropagation may include truncated diachronic backpropagation. The model trainer 160 may perform several generalization techniques (e.g., weight decay, dropout, etc.) to improve the generalization ability of the trained model.

[0072] In particular, the model trainer 160 can train the semantic comprehension models 120 and / or 140 based on a set of training data 162. The training data 162 may include, for example, textbooks, flashcards, academic papers, equations, natural language, books, proofs, and / or homework keys.

[0073] In some embodiments, if the user gives consent, training examples may be provided by the user computing device 102. Thus, in such embodiments, the model 120 provided to the user computing device 102 can be trained by the training computing system 150 with user-specific data received from the user computing device 102. In some examples, this process can be referred to as model personalization.

[0074] The model trainer 160 includes computer logic used to provide a desired function. The model trainer 160 can be implemented in hardware, firmware, and / or software that control a general-purpose processor. For example, in some embodiments, the model trainer 160 includes a program file stored in a storage device, loaded into memory, and executed by one or more processors. In other embodiments, the model trainer 160 includes one or more sets of computer executable instructions stored in a RAM hard disk or a tangible computer-readable storage medium such as an optical or magnetic medium.

[0075] Network 180 can be any type of communication network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or any combination thereof, and may include any number of wired or wireless links. Generally, communications over Network 180 can be transmitted over any type of wired and / or wireless connection using a wide variety of communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encoding or formatting (e.g., HTML, XML), and / or protection methods (e.g., VPN, Secure HTTP, SSL).

[0076] The machine learning models described herein can be used for a variety of tasks, applications, and / or use cases.

[0077] In some embodiments, the input to the machine learning model(s) of this disclosure may be image data. The machine learning model(s) may process the image data to produce an output. For example, the machine learning model(s) may process the image data to produce an image recognition output (e.g., recognition of image data, latent embedding of image data, encoded representation of image data, hash of image data, etc.). As another example, the machine learning model(s) may process the image data to produce an image segmentation output. As yet another example, the machine learning model(s) may process the image data to produce an image classification output. As yet another example, the machine learning model(s) may process the image data to produce an image data modification output (e.g., modification of image data, etc.). As yet another example, the machine learning model(s) may process the image data to produce an encoded image data output (e.g., encoded representation and / or compressed representation of image data, etc.). As yet another example, the machine learning model(s) may process the image data to produce an upscaled image data output. As yet another example, the machine learning model(s) may process the image data to produce a prediction output.

[0078] In some embodiments, the input to the machine learning model(s) of this disclosure may be text or natural language data. The machine learning model(s) may process the text or natural language data to produce an output. For example, the machine learning model(s) may process natural language data to produce a language coding output. As another example, the machine learning model(s) may process text or natural language data to produce a latent text embedding output. As yet another example, the machine learning model(s) may process text or natural language data to produce a translation output. As yet another example, the machine learning model(s) may process text or natural language data to produce a classification output. As yet another example, the machine learning model(s) may process text or natural language data to produce a text segmentation output. As yet another example, the machine learning model(s) may process text or natural language data to produce a semantic intent output. As another example, a machine learning model(s) can process text or natural language data to produce upscaled text or natural language output (e.g., text or natural language data of higher quality than the input text or natural language). As yet another example, a machine learning model(s) can process text or natural language data to produce predictive output.

[0079] In some embodiments, the input to the machine learning model(s) of this disclosure may be speech data. The machine learning model(s) may process the speech data to produce an output. For example, the machine learning model(s) may process the speech data to produce a speech recognition output. As another example, the machine learning model(s) may process the speech data to produce a speech translation output. As yet another example, the machine learning model(s) may process the speech data to produce a latent embedding output. As yet another example, the machine learning model(s) may process the speech data to produce an encoded speech output (e.g., an encoded representation and / or compressed representation of the speech data). As yet another example, the machine learning model(s) may process the speech data to produce an upscaled speech output (e.g., speech data of higher quality than the input speech data). As yet another example, the machine learning model(s) may process the speech data to produce a text representation output (e.g., a text representation of the input speech data). As another example, a machine learning model(s) can process speech data and generate predictive outputs.

[0080] In some embodiments, the input to the machine learning model(s) of this disclosure may be latent coded data (e.g., a latent spatial representation of the input). The machine learning model(s) may process the latent coded data to produce an output. For example, the machine learning model(s) may process the latent coded data to produce a recognition output. As another example, the machine learning model(s) may process the latent coded data to produce a reconstruction output. As yet another example, the machine learning model(s) may process the latent coded data to produce a search output. As yet another example, the machine learning model(s) may process the latent coded data to produce a reclustered output. As yet another example, the machine learning model(s) may process the latent coded data to produce a prediction output.

[0081] In some embodiments, the input to the machine learning model(s) of this disclosure may be statistical data. The machine learning model(s) may process the statistical data to generate an output. For example, the machine learning model(s) may process the statistical data to generate a recognition output. As another example, the machine learning model(s) may process the statistical data to generate a prediction output. As yet another example, the machine learning model(s) may process the statistical data to generate a classification output. As yet another example, the machine learning model(s) may process the statistical data to generate a segmentation output. As yet another example, the machine learning model(s) may process the statistical data to generate a segmentation output. As yet another example, the machine learning model(s) may process the statistical data to generate a visualization output. As yet another example, the machine learning model(s) may process the statistical data to generate a diagnostic output.

[0082] In some embodiments, the input to the machine learning model(s) of this disclosure may be sensor data. The machine learning model(s) may process the sensor data to generate an output. For example, the machine learning model(s) may process the sensor data to generate a recognition output. As another example, the machine learning model(s) may process the sensor data to generate a prediction output. As yet another example, the machine learning model(s) may process the sensor data to generate a classification output. As yet another example, the machine learning model(s) may process the sensor data to generate a segmentation output. As yet another example, the machine learning model(s) may process the sensor data to generate a segmentation output. As yet another example, the machine learning model(s) may process the sensor data to generate a visualization output. As yet another example, the machine learning model(s) may process the sensor data to generate a diagnostic output. As yet another example, the machine learning model(s) may process the sensor data to generate a detection output.

[0083] In some cases, the input includes visual data and the task is a computer vision task. In other cases, the input includes pixel data from one or more images and the task is an image processing task. For example, an image processing task could be image classification, with the output being a set of scores, each corresponding to a different object class and representing the likelihood that one or more images depict an object belonging to that object class. An image processing task could also be object detection, with the output identifying one or more regions within one or more images and, for each region, the likelihood that the region depicts an object of interest. Another example is image segmentation, where the output defines, for each pixel in one or more images, the likelihood for each category within a given set of categories. For example, the set of categories could be foreground and background. Another example is that the set of categories could be object classes. Another example is depth estimation, where the output defines, for each pixel in one or more images, the respective depth values. As another example, the image processing task could be motion estimation, where the network input includes multiple images, and the image processing output defines the motion of the scene depicted between the images in the network input, for each pixel in the input images.

[0084] In some cases, the input may include audio data representing spoken utterances, and the task may be a speech recognition task. The output may include text output mapped to the spoken utterances.

[0085] Figure 1A shows one exemplary computing system that may be used to implement the present disclosure. Other computing systems may be used in a similar manner. For example, in some embodiments, the user computing device 102 may include a model trainer 160 and a training dataset 162. In such embodiments, the model 120 may be trained or used locally on the user computing device 102. In some such embodiments, the user computing device 102 may implement a model trainer 160 that personalizes the model 120 based on user-specific data.

[0086] Figure 1B shows a block diagram of an exemplary computing device 10 that operates according to an exemplary embodiment of the present disclosure. The computing device 10 may be a user computing device or a server computing device.

[0087] Computing device 10 contains several applications (e.g., applications 1 to N). Each application contains its own machine learning library and machine learning model(s). For example, each application may include a machine learning model. Exemplary applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, and so on.

[0088] As illustrated in Figure 1B, each application can communicate with several other components of the computing device, such as one or more sensors, a context manager, a device state component, and / or additional components. In some embodiments, each application can communicate with each device component using an API (e.g., a public API). In some embodiments, the API used by each application is specific to that application.

[0089] Figure 1C shows a block diagram of an exemplary computing device 50 that operates according to an exemplary embodiment of the present disclosure. The computing device 50 may be a user computing device or a server computing device.

[0090] The computing device 50 includes several applications (e.g., applications 1 to N). Each application communicates with a central intelligence layer. Exemplary applications include text messaging applications, email applications, dictation applications, virtual keyboard applications, and browser applications. In some embodiments, each application can communicate with the central intelligence layer (and the model(s) stored within it) using an API (e.g., a common API across all applications).

[0091] The central intelligence layer includes several machine learning models. For example, as shown in Figure 1C, each machine learning model (e.g., Model) can be provided for each application and managed by the central intelligence layer. In other embodiments, two or more applications may share a single machine learning model. For example, in some embodiments, the central intelligence layer may provide a single model (e.g., Single Model) for all applications. In some embodiments, the central intelligence layer is contained within the operating system of the computing device 50 or implemented by them in other ways.

[0092] The central intelligence layer can communicate with the central device data layer. The central device data layer can be a centralized repository of data from the computing device 50. As shown in Figure 1C, the central device data layer can communicate with several other components of the computing device, such as one or more sensors, a context manager, a device state component, and / or additional components. In some embodiments, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0093] Figure 3 shows a block diagram of an exemplary computing system 300 performing augmented reality tutoring according to an exemplary embodiment of the present disclosure. For example, the systems and methods disclosed herein may include one or more computing devices communically connected via a network 302. In some embodiments, the computing system 300 may include one or more image sensors, one or more visual displays, one or more audio sensors, one or more audio output components, one or more storage devices, and / or one or more processors. The computing system 300 may include a smart device 304 (e.g., a smartphone) and / or a smart wearable 306 (e.g., smart glasses). In some embodiments, the smart device 304 and the smart wearable 306 may be communically connected via the network 302, a Bluetooth connection, and / or other communication mediums. For example, the display, sensors, and / or processor of the smart device 304 may be used together with the smart wearable 306. In some embodiments, the smart device 304 and / or smart wearable 306 may exchange data with one or more server computing systems 308 to perform the systems and methods disclosed herein.

[0094] Exemplary system configuration and exemplary model configuration Figures 2A–2E illustrate an example of an exemplary augmented reality experience according to an exemplary embodiment of the present disclosure. An exemplary augmented reality experience may be initiated based on one or more inputs (e.g., button compression, touch input to a touchscreen, object detection or classification, and / or audio input). In response to the initiation of the augmented reality experience, one or more images 202 may be acquired. In some embodiments, audio input may be acquired.

[0095] One or more images 202 may be processed to determine a prompt 204 at a focal point of the environment (for example, as shown in Figure 2A). The focal point may be determined based on location in the environment, based on user instructions, based on detected objects, based on the user's line of sight, and / or based on one or more machine-learned parameters. The prompt 204 may be highlighted, underlined, and / or indicated by one or more other techniques.

[0096] Audio input may be processed to determine speech utterances. Speech utterances may be provided for display by closed caption user interface element 206. The augmented reality experience may provide user interface elements to indicate that data is being acquired via image sensors and / or audio sensors.

[0097] Prompt 204 may be processed to determine a multipart response for a particular prompt 204. The multipart response may be determined based on one or more machine learning parameters, based on a knowledge graph, and / or based on data retrieved from a database. Multiple user interface elements may be generated based on the set of actions associated with the multipart response.

[0098] In Figure 2B, prompt 204 remains highlighted, and a first user interface element 208 is provided for display. The first user interface element 208 may describe instructions for completing one or more actions for the first part of a multipart response.

[0099] Subsequently, additional image data may be received. The additional image data may be processed to identify new text data. The new text data may include handwritten text 210. The new text data may be processed to determine that the first part of the multipart response is complete. In response to the determination, a second user interface element 214 may be provided for display (for example, as shown in Figure 2C). The second user interface element 214 may describe instructions for completing one or more actions for the second part of the multipart response. In some embodiments, the augmented reality experience may provide a position of interest indicator 212, which may indicate a position of interest for a set of actions and / or indicate a prompt type associated with a prompt 204.

[0100] In Figure 2D, further handwritten text 216 is provided and is being provided. The system and method may determine that a second part of the multipart response has been detected, and a third user interface element 218 may be provided for display. The third user interface element 218 may describe instructions for completing one or more actions on the third part of the multipart response.

[0101] In Figure 2E, a multipart response is being performed and the final handwritten response 220 is identified. The augmented reality experience may provide a completion indicator 222 and / or one or more final user interface elements 224 indicating that the set of actions has been completed.

[0102] The system and method may then be repeated for the next identified prompt. The augmented reality experience may include additional user interface elements to provide intuitive instructions. The augmented reality experience may be provided with one or more audio outputs.

[0103] Figure 4 shows a block diagram of an exemplary augmented reality tutoring system 400 according to an exemplary embodiment of the present disclosure. In some embodiments, the augmented reality tutoring system 400 may receive image data 402 and / or additional input data 404 describing an environment and / or specific prompts, and as a result of receiving the image data 402 and additional input data 404, may provide a user interface element 420 describing one or more classifications and / or instructions for completing a task. Thus, in some embodiments, the augmented reality tutoring system 400 may include an optical character recognition model 406 capable of recognizing text in an image and a semantic understanding model 410 capable of generating semantic output 412.

[0104] For example, image data 402 may be acquired by one or more image sensors. Image data 402 may describe one or more images that depict an environment. The environment may include multiple characters that describe one or more prompts (e.g., one or more questions and / or one or more instructions).

[0105] Image data 402 may be processed by an optical character recognition model 406 to recognize one or more characters, one or more symbols, and / or one or more figures, and to generate text data 408. The text data 408 may include words, numbers, equations, figures, text structures, text layouts, syntax, and / or symbols. The optical character recognition model 406 may be trained on printed text and / or in particular to determine handwritten characters.

[0106] In some embodiments, additional input data 404 may be acquired. The additional input data 404 may be acquired by one or more additional sensors, which may include audio sensors and / or touch sensors. The additional input data 404 may be generated based on one or more selections made in voice utterance and / or the user interface.

[0107] Image data 402 and / or additional input data 404 may be processed by a semantic understanding model 410 to generate a semantic output 412. The semantic understanding model 410 may include one or more segmentation models, one or more augmentation models, one or more natural language processing models, one or more quantitative inference models, and / or one or more classification models. The semantic understanding model 410 may include one or more transformer models, one or more convolutional neural networks, one or more genetic algorithmic neural networks, one or more discriminator models, and / or recurrent neural networks. The semantic understanding model 410 may be trained on large language training datasets, quantitative inference training datasets, textbook datasets, flashcard training datasets, and / or proof datasets. The semantic understanding model 410 may be trained to determine the semantic intent of input data and to perform one or more tasks based on the semantic intent. For example, the semantic comprehension model 410 may be trained on multiple tasks, which may include input summarization, response tasks, completion tasks, diagnostic tasks, problem-solving tasks, and error detection tasks, classification tasks, and / or augmentation tasks.

[0108] Based on the semantic output 412, no further action 414 may be determined, which may lead to the process restarting. Alternatively and / or additionally, the augmented reality tutor interface 416 may be initiated based on the semantic output 412. For example, the augmented reality tutor interface 416 may be initiated based on the semantic output 412 describing an error (e.g., an inaccurate response and / or a configuration problem in the environment). Alternatively and / or additionally, the augmented reality tutor interface 416 may be initiated based on the semantic output 412 describing a threshold time that occurs without any action taking place (e.g., a threshold time that occurs without any further handwriting). In some embodiments, the augmented reality tutor interface 416 may be initiated by user input that triggers the acquisition of image data 402 and / or additional input data 404.

[0109] Informational output 418 may be determined based on semantic output 412. For example, semantic output 412 may describe an error in the environment, and informational output may include instructions for a set of actions to correct the error (e.g., corrective actions). The set of actions may be determined based on the output of a machine learning model, based on one or more knowledge graphs, and / or based on one or more search results generated by querying a database. The instructions for the set of actions may be sequentially ordered.

[0110] One or more user interface elements 420 may be generated based on the information output 418. In some embodiments, a user interface element 420 may be generated for each action in a set of actions. The user interface elements 420 may then be provided to the user. The user interface elements 420 may be provided sequentially. The process may then start again.

[0111] Figure 5 shows a block diagram of an exemplary augmented reality tutoring system 500 according to an exemplary embodiment of the present disclosure. The augmented reality tutoring system 500 is similar to the augmented reality tutoring system 400 of Figure 4, except that the augmented reality tutoring system 500 further includes an augmented reality generation block 518.

[0112] The augmented reality tutoring system 500 can acquire input data, which may include words 502, numbers 504, equations 506, figures 508, structural data 510, and / or other data 512. The other data 512 may include time data, audio data, touch data, and / or contextual data. The input data may include multimodal data and / or may be conditioned or supplemented based on profile data and / or preference data.

[0113] Input data may be processed by the semantic understanding model 514 to generate or determine prompts. Prompts may be based on semantic understanding of the input data, which may include semantic understanding of the environment. Prompts may include problems to be solved (e.g., math problems, reading comprehension problems, and / or science problems), writing prompts (e.g., analytical prompts, essay prompts, and / or creative writing prompts), and / or DIY projects (e.g., furniture assembly, appliance repair, and / or vehicle maintenance).

[0114] The prompt may be processed in the response decision block 516 to generate a response. In some embodiments, the response decision block 516 may be part of the semantic understanding model 514. The semantic understanding model 514 and / or the response decision block 516 may include one or more machine learning models. In some embodiments, the response decision block 516 may include determining a query based on the prompt and querying a database (e.g., a search engine and / or an academic database).

[0115] The response may include a multipart response containing a set of actions. The set of actions may be part of a larger corrective action to correct an error. The response may be processed by the augmented reality generation block 518 to generate multiple augmented reality user interface elements to be provided to the user. The multiple augmented reality user interface elements may contain instructions for performing multiple actions associated with the response.

[0116] Multiple augmented reality user interface elements may include inline rendering 520 (e.g., text and / or symbols provided inline with text and / or objects in the environment), popup elements 522 (e.g., speech bubbles rendered within the augmented reality display), highlight elements 524 (e.g., brightening and / or darkening multiple pixels that display the environment), animation elements 526 (e.g., animated images and / or animated text that change over time), symbols (e.g., representative indicators and / or classification symbols), and / or other user interface outputs 530 (e.g., three-dimensional augmented renderings of objects in the scene).

[0117] Figure 9 shows an example of an exemplary smart wearable 900 for acquiring image data and providing user interface elements, according to an exemplary embodiment of the present disclosure. For example, systems and methods may be implemented by the smart wearable 900. In some embodiments, the smart wearable 900 may include smart glasses. The smart wearable 900 may include one or more image sensors 902, one or more computer component shells 904, one or more displays 906, and / or one or more lenses 908. The lenses may be prescription lenses, blue light lenses, tinted lenses, and / or clear non-prescription lenses. One or more image sensors 902 may be arranged such that the acquired image data describes the environment within the user's field of view. One or more computer component shells 904 may house one or more processors, one or more communication components (e.g., Bluetooth receiver, ultra-wideband receiver, and / or WiFi receiver), one or more audio components (e.g., microphone and / or speaker), and / or one or more storage devices. One or more displays 906 may be configured to display one or more user interface elements.

[0118] For example, one or more image sensors 902 can generate image data, which can be processed by one or more processors in one or more computer component shells 904. User interface elements may be selected and / or generated based on the image data. One or more user interface elements may then be provided for display via one or more displays 906.

[0119] Exemplary Method Figure 6 shows a flowchart of an exemplary method for carrying out an exemplary embodiment of the present disclosure. While Figure 6 shows steps performed in a specific order for illustrative and explanatory purposes, the methods of the present disclosure are not limited to the order or configuration shown. Various steps of Method 600 may be omitted, rearranged, combined, and / or adapted in various ways without departing from the scope of the present disclosure.

[0120] In 602, the computing system can acquire image data. The image data can describe one or more images. In some embodiments, one or more images may describe an environment. The environment may include one or more problems. For example, the environment may include questions for a user to respond to. Alternatively and / or additionally, the environment may include objects for completing a DIY project. The image data may be generated by one or more image sensors of a mobile computing device (e.g., a smartphone). In some embodiments, the mobile computing device may be a smart wearable (e.g., smart glasses).

[0121] In 604, the computing system may process image data to generate semantic data. Semantic data may describe the semantic understanding of at least a portion of one or more images. In some embodiments, the image data may be processed by a semantic understanding model. The semantic understanding model may include one or more machine-trained models. The semantic understanding model may include natural language processing models (e.g., one or more large language models trained on multiple examples). In some embodiments, the semantic understanding model may include machine-trained models trained to understand equations and / or other quantitative representations (e.g., language models trained for quantitative reasoning, as discussed in Dyer et al., Minerva: Solving Quantitative Reasoning Problems with Language Models, GOOGLE AI BLOG (June 30, 2022), https: / / ai.googleblog.com / 2022 / 06 / minerva-solving-quantitative-reasoning.html). Additionally and / or alternatively, the image data may be processed by an optical character recognition model to generate text data, which may then be processed by a semantic understanding model. Semantic data may be based on text content, recognized objects, data structure, data layout, information structure, one or more figures, additional input data received, image capture context, image capture device type, user profile data, and / or one or more other contexts. Semantic data may include one or more queries summarizing an issue (e.g., a question) at the focal point of one or more images.

[0122] In 606, a computing system may determine an error in one or more images based at least in part on semantic data. An error may include a mismatch with semantic understanding. In some embodiments, an error may include a deviation from a multipart process. A multipart process may be associated with semantic data. For example, a multipart process may include one or more actions to answer a question and / or solve a problem. An error may be determined based on heuristics, based on acquired data, and / or based on the output of a machine-trained model. An error may be determined based on the fact that handwritten text differs from the semantic intent of printed text. For example, semantic data may include the semantic intent of printed text and the semantic understanding of handwritten text. An error may be determined if the semantic understanding of handwritten text does not relate to the semantic intent of printed text.

[0123] In some embodiments, determining errors in one or more images based at least partially on semantic data may include obtaining a specific machine-learned model based on the semantic data and processing the image data with the specific machine-learned model to detect errors. For example, the semantic data may describe a specific problem type (e.g., a literature analysis problem, a calculus problem, and / or an organic chemistry problem) and a problem-specific machine-learned model (e.g., a literature analysis model, a calculus model, and / or an organic chemistry model). Alternatively and / or additionally, a mathematical engine (e.g., a system of mathematical functions used to process the problem using one or more processors) may be obtained and used based on the semantic data.

[0124] Alternatively and / or additionally, determining corrective actions based on semantic data and errors may include detecting the location of errors in the environment, determining error-free datasets associated with the semantic data and one or more images, and determining replacement data from the error-free datasets based on the location of errors in the environment.

[0125] In some embodiments, errors may be determined using an error detection model. The error detection model may generate text data based on optical character recognition. The error detection model may parse the text data based on one or more features in the environment. In some embodiments, the error detection model may process each of several parsed segments to determine the error. The error detection model may be trained with several mathematical proofs. Additionally and / or alternatively, the error detection model may include an optical character recognition model and a natural language processing model.

[0126] In 608, the computing system may determine a corrective action based on semantic data and errors. The corrective action may describe at least one of the following: replacing an error or correcting an error. In some embodiments, the corrective action may include indicating the location of the error in the environment and one or more actions to correctly respond to a prompt identified in the environment.

[0127] In 610, the computing system may provide user interface elements for display based on correction actions. User interface elements may include informational data describing correction actions. In some embodiments, user interface elements may be provided for display by mobile computing devices. User interface elements may be provided by augmented reality experiences. User interface elements may include prompts, inline comments, pop-up speech bubbles, and / or highlighting one or more arrows.

[0128] Figure 7 shows a flowchart of an exemplary method for carrying out an exemplary embodiment of the present disclosure. While Figure 7 shows steps performed in a specific order for illustrative and explanatory purposes, the methods of the present disclosure are not limited to the order or configuration shown. Various steps of Method 700 may be omitted, rearranged, combined, and / or adapted in various ways without departing from the scope of the present disclosure.

[0129] In 702, the computing system can acquire image data. One or more images may describe one or more pages. In some embodiments, one or more pages may include one or more questions. One or more pages may include printed text and handwritten text. One or more questions may include scientific questions including mathematical formulas, writing prompts, and / or one or more figures.

[0130] In 704, the computing system may process image data with an optical character recognition model to generate text data. The text data may describe text on one or more pages. The optical character recognition model may include one or more machine-trained models. The optical character recognition model may include models trained in particular on handwritten text. The text data may include recognized printed text and / or recognized handwritten text.

[0131] In 706, a computing system can determine prompts and multipart responses to prompts based on text data. Prompts may describe requests for responses. Prompts may be determined based on semantic understanding of text on one or more pages. Alternatively and / or additionally, prompts may be queries generated based on recognized text. Prompts may be determined based on text containing one or more keywords associated with one or more prompts and / or one or more prompt types.

[0132] A multipart response may contain multiple separate responses associated with a prompt. A multipart response may be determined based on the output of a machine learning model, on heuristics, on one or more search results and / or one or more knowledge graphs received from a search engine. A multipart response may be based on the output of a machine learning model trained on one or more textbooks. For example, a machine learning model may be trained to identify a particular type of problem based on one or more identified features, and the same or separate models may be trained to generate a proof that describes how to solve a particular problem. The generated proof may be a multipart response in which each line of the proof is part of the response.

[0133] In 708, the computing system may acquire additional image data. The additional image data may describe one or more additional images. One or more additional images may describe one or more pages having user-generated text (e.g., additional handwritten text and / or user-typed data (e.g., user-generated code and / or user-generated equations)). The user-generated text may include user responses to one or more questions.

[0134] In 710, the computing system may process additional image data with an optical character recognition model to generate additional text data. The additional text data may describe user-generated text on one or more pages. The user-generated text may describe the user's attempts to respond to prompts (e.g., answering questions).

[0135] In 712, the computing system may determine that user-generated text deviates from a multipart response and provide a notification. The deviation may be such that the user-generated text is the opposite of the multipart response. For example, the multipart response may include performing a first action and then taking a second action, while the user-generated text may include performing a first action and then taking a third action that is not equivalent to the second action.

[0136] In some embodiments, determining that user-generated text deviates from a multipart response may include determining that user-generated text is inconsistent with the multipart response. For example, user-generated text may contain a semantic intent that is inconsistent with the semantic intent of one or more parts of the multipart response.

[0137] Alternatively and / or additionally, determining that user-generated text deviates from a multipart response may include determining that the user-generated text lacks one or more specific features of the multipart response. For example, the multipart response may involve multiplying both sides of an equation by 2, while the user-generated text multiplies only one side by 2.

[0138] The system and method may provide notifications. Notifications may describe user-generated text indicating an error. Notifications may be provided by an augmented reality experience rendering one or more user interface elements for providing notifications. Notifications may describe where the error occurred and how to resolve the error.

[0139] In some embodiments, the system and method may process image data with a machine learning model to determine prompts and multipart responses. The machine learning model may be a language model trained on quantitative inference. In some embodiments, the machine learning model may be trained on one or more subjects, particularly using academic materials (e.g., textbooks and / or academic papers).

[0140] In some embodiments, the system and method may process additional image data with a machine learning model to determine if user-generated text deviates from a multipart response.

[0141] Figure 8 shows a flowchart of an exemplary method for carrying out an exemplary embodiment of the present disclosure. While Figure 8 shows steps performed in a specific order for illustrative and explanatory purposes, the methods of the present disclosure are not limited to the order or configuration shown. Various steps of Method 800 may be omitted, rearranged, combined, and / or adapted in various ways without departing from the scope of the present disclosure.

[0142] In 802, the computing system can acquire video data. The image data can describe one or more images. One or more images can describe one or more pages. In some embodiments, one or more pages may contain multiple characters. The multiple characters may be part of a problem (e.g., a question, a writing prompt, and / or an assignment statement). The characters may include letters, numbers, and / or symbols. One or more pages may contain text, pictures, shapes, diagrams, and / or whitespace.

[0143] In 804, the computing system can process video data to generate recognition data.

[0144] In 806, the computing system can process recognition data to generate semantic data. The semantic data may describe the semantic understanding of at least some of the characters. In some embodiments, the semantic data may be based on text, pictures, shapes, figures, and / or whitespace.

[0145] In 808, a computing system may determine, at least partially, that several characters contain errors. Errors may describe text that is either the opposite of semantic understanding or inaccurate. Inaccuracies may be determined by processing the several characters with one or more machine learning models.

[0146] In 810, the computing system may determine a correction action based on semantic data and errors. The correction action may describe at least one of the following: replacement of an error or an action to correct an error. The correction action may also include deletion actions (e.g., deleting a subset of several characters) and write actions (e.g., writing one or more new characters).

[0147] In 812, the computing system may provide user interface elements for display based on correction actions. User interface elements may include informational data describing correction actions. In some embodiments, user interface elements may include one or more popup elements describing a plurality of sub-actions for performing the correction action. User interface elements may include inline overlays. Inline overlays may be used to extend one or more images or at least one of one or more additional images to generate one or more extended images. One or more extended images may include inline overlays superimposed on at least a portion of one or more pages. In some embodiments, inline overlays may describe correction actions. Additionally and / or alternatively, user interface elements may include extending one or more images to indicate the location of an error.

[0148] Additional disclosures The technologies described herein refer to servers, databases, software applications, and other computer-based systems, as well as the actions performed and the information transmitted to and from such systems. The inherent flexibility of computer-based systems allows for a wide variety of feasible configurations, combinations, and divisions of tasks and functions between components. For example, the processes described herein can be implemented using a single device or component, or multiple devices or components working together. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0149] While the subject matter has been described in detail with respect to various specific exemplary embodiments, each example is provided for illustrative purposes only and does not limit the disclosure. Those skilled in the art will readily be able to create modifications, changes, and equivalents to such embodiments, understanding the foregoing. Therefore, the disclosure does not exclude the inclusion of such modifications, changes, and / or additions to the subject matter, which will be readily apparent to those skilled in the art. For example, features illustrated or described as part of one embodiment may be used together with other embodiments to create yet another embodiment. Therefore, the disclosure is intended to cover such modifications, changes, and equivalents.

Claims

1. A computing system, One or more processors, The system comprises one or more non-temporary computer-readable media that, when executed by the one or more processors, collectively store instructions that cause the computing system to perform an operation, and the operation is The acquisition of image data, wherein the image data describes one or more images, and the one or more images describe an environment. Processing the image data to generate semantic data, wherein the semantic data describes the semantic understanding of at least a part of the one or more images. Determining an error in one or more images based at least partially on the semantic data, wherein the error includes a mismatch between the semantic understanding of printed text and the semantic understanding of user-generated text, or a deviation from a multipart process, wherein the multipart process is associated with the semantic data, and the deviation from the multipart process includes the user-generated text being inconsistent with the multipart response of the multipart process, or the user-generated text lacking one or more specific features of the multipart response. The process involves determining a correction action based on the semantic data and the error, wherein the correction action describes at least one of the following: replacing the error or correcting the error. The present invention provides a user interface element for display based on the correction action, wherein the user interface element includes information data describing the correction action. A computing system that includes this.

2. Determining the error in one or more images based at least partially on the semantic data is: Obtaining a specific machine learning model based on the aforementioned semantic data, To detect the aforementioned error, the image data is processed with the specific machine learning model, The system according to claim 1, including the following:

3. Determining the correction action based on the semantic data and the error is: To detect the location of the error in the environment, To determine an errorless dataset associated with the semantic data and one or more images, Based on the location of the error in the environment, replacement data is determined from the error-free dataset. The system according to claim 1, including the following:

4. The aforementioned error is determined using an error detection model, and the error detection model is Generate text data based on optical character recognition, The text data is parsed based on one or more features within the environment. The system according to claim 1, which processes each parsed segment of a plurality of parsed segments in order to determine the error.

5. The system according to claim 4, wherein the error detection model is trained on multiple mathematical proofs.

6. The system according to claim 4, wherein the error detection model includes an optical character recognition model and a natural language processing model.

7. The system according to any one of claims 1 to 6, wherein the image data is generated by one or more image sensors of a mobile computing device, and the user interface elements are provided for display by the mobile computing device.

8. The system according to claim 7, wherein the mobile computing device is a smart wearable.

9. A method implemented in a computer, The acquisition of image data by a computing system including one or more processors, wherein the image data describes one or more images, and the one or more images describe one or more pages. The computing system processes the image data with an optical character recognition model to generate text data, wherein the text data describes the text on one or more pages. The computing system determines a prompt based on the text data, wherein the prompt describes a request for a response. The computing system determines a multipart response to the prompt, wherein the multipart response includes a plurality of separate responses associated with the prompt. The computing system acquires additional image data, wherein the additional image data describes one or more additional images, and the one or more additional images describe one or more pages having user-generated text. The computing system processes the additional image data with the optical character recognition model to generate additional text data, wherein the additional text data describes the user-generated text on one or more pages. The computing system determines that the user-generated text deviates from the multipart response, and this determination includes determining that the user-generated text is inconsistent with the multipart response, or determining that the user-generated text lacks one or more specific features of the multipart response. The computing system provides a notification, the notification describing the user-generated text having an error, A method of implementation in a computer, including

10. The method according to claim 9, wherein the one or more pages include one or more questions, and the user-generated text includes user responses to the one or more questions.

11. The method according to claim 9, further comprising processing the image data with a machine learning model in order to determine the prompt and the multipart response using the computing system.

12. The method according to any one of claims 9 to 11, further comprising processing the additional image data with a machine learning model in order for the computing system to determine that the user-generated text deviates from the multipart response.

13. One or more non-temporary computer-readable media, which, when executed by one or more computing devices, collectively store instructions that cause the one or more computing devices to perform an action, wherein the action is The acquisition of image data, wherein the image data describes one or more images, the one or more images describe one or more pages, and the one or more pages contain multiple characters. Processing the image data to generate semantic data, wherein the semantic data describes the understanding of the meaning of at least some of the plurality of characters. Determining that the plurality of characters contain an error based at least partially on the semantic data, wherein the error includes at least one of the following: the semantic understanding of the user-generated text is the opposite of the semantic understanding of the printed text; or the semantic understanding of the user-generated text is inaccurate. The process involves determining a correction action based on the semantic data and the error, wherein the correction action describes at least one of the following: replacing the error or correcting the error. The present invention provides a user interface element for display based on the correction action, wherein the user interface element includes information data describing the correction action. One or more non-temporary computer-readable media, including [the specified text].

14. The user interface element comprises one or more pop-up elements describing a plurality of sub-actions for performing the correction action, one or more non-temporary computer-readable media according to claim 13.

15. One or more non-temporary computer-readable media according to claim 13, wherein the user interface element includes an inline overlay, the inline overlay is used to extend one or more images or at least one of the one or more additional images to generate one or more extended images, the one or more extended images include the inline overlay superimposed on at least a portion of the one or more pages, and the inline overlay describes the correction action.

16. One or more non-temporary computer-readable media according to any one of claims 13 to 15, wherein the user interface element extends one or more of the images to indicate the location of the error.

Images