Machine-learned text alignment prediction for providing augmented reality translation interface
By processing images through paragraph detection and alignment classification models, text alignment classification is generated, solving the localization and alignment problems of text translation in non-native language environments, and realizing the native format display of translated text and improving computational efficiency.
Patent Information
- Application Number
- CN202580026078.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2024-04-15
- Filing Date
- 2025-04-08
- Publication Date
- 2026-11-10
AI Technical Summary
When translating text in a non-native language environment, translation alone cannot provide detailed information on how the information is related, and existing technologies struggle to effectively provide text tabulation, positioning, and alignment.
A computational system is used to generate paragraph bounding boxes using a paragraph detection model. The image representation and bounding boxes are then processed by a machine learning alignment classification model to generate text alignment classification. The original text is then replaced with translated text to maintain the original format.
It provides a more comprehensive and understandable display of translated text, maintaining the alignment and position of the original text while reducing computational latency and cost.
Smart Images

Figure CN123076902A_ABST
Abstract
Description
[0001] Priority requirements
[0002] This application is based on and claims priority to U.S. Provisional Application 63 / 634,233, filed April 15, 2024. This application claims priority and benefit to each of these applications, and the entire contents of all such applications are incorporated herein by reference. Technical Field
[0003] This disclosure generally relates to text alignment prediction. More specifically, this disclosure relates to text alignment prediction for determining text localization to generate an enhanced image. Background Technology
[0004] Traversing environments with text in a user's non-native language can be challenging. Translation can bridge some of the gap by providing text in a language the user can understand. However, translation alone may not provide the full picture, as how information is presented can offer further insights. Tabulation, positioning, and alignment of text can provide detailed information about how different parts of the text relate to each other. Summary of the Invention
[0005] Aspects and advantages of embodiments of this disclosure will be set forth in part in the description which follows, or may be learned from the description or by practice of the embodiments.
[0006] One example aspect of this disclosure relates to a computational system for text alignment classification. The system may include one or more processors and one or more non-transitory computer-readable media, the one or more non-transitory computer-readable media jointly storing instructions that, when executed by the one or more processors, cause the computational system to operate. The operation may include obtaining an image comprising text. The text may be in a first language. The operation may include obtaining a text translation request associated with a translation from the first language to a second language. The operation may include processing the image to generate an image representation, and processing the image using a paragraph detection model to generate one or more paragraph bounding boxes. The one or more paragraph bounding boxes may describe the location of one or more paragraphs within the image. The operation may include processing the image representation and the one or more paragraph bounding boxes using a machine learning-based alignment classification model to generate one or more alignment classifications associated with the one or more paragraphs. The one or more alignment classifications may describe the predicted text alignment of the one or more paragraphs. The operation may include obtaining translated text. In some implementations, the translated text may describe text in a second language. The operation may include generating an augmented image based on the image, one or more alignment classifications, and the translated text. The augmented image may include the image, while the text is replaced with translated text having text alignments determined based on one or more alignment classifications.
[0007] In some implementations, processing an image using a paragraph detection model to generate one or more paragraph bounding boxes may include generating multiple paragraph bounding boxes by processing the image representation using the paragraph detection model. Multiple predicted paragraph locations may be associated with multiple different paragraphs within the image. Processing the image representation and one or more paragraph bounding boxes using a machine learning alignment classification model may include generating multiple alignment classifications associated with multiple different paragraphs by processing the image representation and multiple paragraph bounding boxes using a machine learning alignment classification model. In some implementations, generating an augmented image based on an image, one or more alignment classifications, and translated text may include determining multiple feature anchors based on multiple paragraph bounding boxes and multiple alignment classifications, and rendering the translated text based on the multiple feature anchors and multiple alignment classifications. The multiple feature anchors may describe image features within the image to be associated with the location used to render a specific paragraph. The operation may include obtaining a second image, determining that the second image includes image features, and generating a second augmented image based on the multiple feature anchors, translated text, and multiple alignment classifications. The second augmented image may include the second image, where the text in the first language is replaced with the translated text in the second language.
[0008] In some implementations, using a machine learning-based alignment classification model to process an image representation and one or more paragraph bounding boxes to generate one or more alignment classifications associated with one or more paragraphs may include performing region-of-interest pooling on visual features of the image representation based on one or more paragraph bounding boxes to generate one or more pooled visual embeddings; using convolutional blocks to process one or more pooled visual embeddings to predict shape features of one or more paragraphs; generating one or more box localization embeddings based on one or more paragraph bounding boxes and shape features; and processing one or more pooled visual embeddings and one or more box localization embeddings to generate one or more alignment classifications. The one or more alignment classifications may include at least one of right alignment, left alignment, justified alignment, top alignment, bottom alignment, or center alignment.
[0009] In some implementations, the machine learning-aligned classification model may have been trained on a training dataset that includes multiple sample image representations, multiple benchmark ground truth bounding boxes, and multiple benchmark ground truth alignment labels. The multiple sample image representations may be associated with multiple training images. Each of the multiple training images may include corresponding text. The multiple training images may include text in multiple different languages. In some implementations, the multiple alignments may include left alignment, right alignment, and justified alignment.
[0010] Another example aspect of this disclosure relates to a computer-implemented method for an augmented reality translation interface. The method may include obtaining image data describing an image from one or more image sensors of a user computing device by a computing system including one or more processors. The image may include text in a first language. The method may include obtaining a text translation request associated with a translation from the first language to a second language by the computing system. The method may include processing the image by the computing system using a paragraph detection model to generate one or more paragraph bounding boxes. The one or more paragraph bounding boxes may describe the location of one or more paragraphs within the image. The method may include processing the image data and the one or more paragraph bounding boxes by the computing system using a machine learning-based alignment classification model to generate one or more paragraph location predictions and one or more alignment classifications associated with the one or more paragraphs. The one or more alignment classifications may describe the predicted text alignment of the one or more paragraphs. The method may include obtaining translated text by the computing system. The translated text may describe text in a second language. The method may include generating an augmented image by the computing system based on the image, one or more paragraph location predictions, one or more alignment classifications, and the translated text. The augmented image may include the image, while the text is replaced with translated text having a specific text alignment determined based on one or more alignment classifications and a specific text location based on one or more paragraph location predictions.
[0011] In some implementations, the method may include determining one or more feature anchors by a computational system based on one or more paragraph position predictions and one or more alignment classifications. Image data may include a compressed version of an image. In some implementations, obtaining translated text by the computational system may include processing the image using an optical character recognition model to generate text data describing text in a first language, and processing the text data using a translation model to generate translated text. Processing text data using a translation model to generate translated text may include processing the text data to determine that the text is in a first language, and obtaining a translation model in response to determining that the text is in a first language. In some implementations, obtaining a translation model by the computational system may include determining the language preference of a particular user by the computational system, and obtaining a translation model from a translation model database by the computational system based on a translation request from a first language to a second language. Language preference may include a second language.
[0012] Another exemplary aspect of this disclosure relates to one or more non-transitory computer-readable media that jointly store instructions that, when executed by one or more computing devices, cause the one or more computing devices to operate. The operation may include obtaining training data. The training data may include sample image representations, baseline ground truth bounding boxes, and baseline ground truth alignment labels. The sample image representations may describe a particular image. The baseline ground truth bounding boxes may describe the baseline ground truth location of a paragraph within the particular image. In some implementations, the baseline ground truth alignment labels may describe the baseline ground truth text alignment of the paragraph within the particular image. The operation may include processing the sample image representations and baseline ground truth bounding boxes using an alignment classification model to generate a predicted alignment classification. In some implementations, the predicted alignment classification may describe the predicted alignment of the paragraph. The operation may include evaluating a loss function that evaluates the difference between the predicted alignment classification and the baseline ground truth alignment labels, and adjusting one or more parameters of the alignment classification model, at least in part, based on the loss function.
[0013] In some implementations, the operation may include processing the image representation using a paragraph detection model to generate predicted paragraph bounding boxes. The predicted paragraph bounding boxes may describe the predicted localization of the paragraphs. The operation may include evaluating a second loss function that assesses the difference between the predicted paragraph bounding boxes and benchmark ground truth bounding boxes, and adjusting one or more second parameters of the alignment classification model, at least in part, based on the second loss function. The benchmark ground truth bounding boxes and benchmark ground truth alignment labels may have already been generated based on user input associated with a specific image. In some implementations, the loss function may include cross-entropy loss.
[0014] Other aspects of this disclosure relate to various systems, devices, non-transitory computer-readable media, user interfaces, and electronic devices.
[0015] These and other features, aspects, and advantages of the various embodiments of this disclosure will be better understood with reference to the following description and the appended claims. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate exemplary embodiments of the disclosure and, together with the description, serve to explain the relevant principles. Attached Figure Description
[0016] Referring to the accompanying drawings, a detailed discussion of embodiments is set forth in this specification for those skilled in the art, in which:
[0017] Figure 1 A block diagram of an example alignment prediction system according to an example embodiment of the present disclosure is depicted.
[0018] Figure 2A block diagram of an example image enhancement system according to an example embodiment of the present disclosure is depicted.
[0019] Figure 3 A flowchart is depicted for an example method of performing text-aligned image enhancement according to an example embodiment of the present disclosure.
[0020] Figure 4A An illustration depicting example images according to an example embodiment of the present disclosure.
[0021] Figure 4B An illustration depicts an example enhanced output image incorporating text alignment prediction according to an exemplary embodiment of the present disclosure.
[0022] Figure 4C An illustration depicts an example enhanced image without text alignment prediction according to an exemplary embodiment of the present disclosure.
[0023] Figure 5 A block diagram depicts an example text alignment classification system according to an example embodiment of the present disclosure.
[0024] Figure 6 A block diagram of an example text alignment classification model system according to an example embodiment of the present disclosure is depicted.
[0025] Figure 7 A flowchart is depicted for an example method for generating an augmented reality interface according to an example embodiment of the present disclosure.
[0026] Figure 8 A flowchart is depicted for an example method for training an alignment classification model according to an example embodiment of the present disclosure.
[0027] Figure 9 A block diagram of an example augmented reality system according to an example embodiment of the present disclosure is depicted.
[0028] Figure 10 A block diagram of an example augmented reality translation system according to an example embodiment of the present disclosure is depicted.
[0029] Figures 11A to 11C A graphical representation of an example augmented reality translation according to an exemplary embodiment of the present disclosure is provided.
[0030] Figure 12A A block diagram of an example computing system for alignment classification according to an example embodiment of the present disclosure is depicted.
[0031] Figure 12B A block diagram of an example computing system for alignment classification according to an example embodiment of the present disclosure is depicted.
[0032] The repeated reference numerals across multiple figures are intended to identify the same features in various implementations. Detailed Implementation
[0033] Generally, this disclosure relates to text alignment prediction for augmented reality rendering of translated text. Specifically, the systems and methods disclosed herein can utilize alignment classification models to generate predicted alignment classifications of one or more text sequences within an image. For example, an augmented reality interface can acquire an image via one or more image sensors of a user computing device. An embedding model can process the image to generate an image representation. Additionally and / or alternatively, a paragraph detection model can process the image to generate one or more bounding boxes describing the location and size of text paragraphs depicted in the image. The alignment classification model can then process the image representation and one or more paragraph bounding boxes to generate a predicted alignment classification for each detected paragraph within the image. Before, after, or during alignment prediction, an optical character recognition model can process the image to generate text data describing the text within the image. A translation model can process the text data to generate translated text describing text from the image translated into different languages. The augmented reality interface can then render the translated text above and / or replace the original text in the image. The augmented reality interface can utilize predicted alignment classifications to render the text such that the translated text has the same alignment as the original text.
[0034] Text alignment prediction can be used by an augmented reality interface to provide translated text in the original text's format, which can provide a more comprehensive and understandable display of the translated text. Additionally and / or alternatively, text size, color, kerning, and / or font can be determined and utilized to generate a text rendering that replicates the style and / or format of the original text. The text can then be provided in its natural form, while simultaneously using a language understood by the user.
[0035] In some implementations, text alignment prediction can be used to determine the positioning and alignment of text to be rendered in place of existing text. This can include removing the original text and rendering new text with the same alignment in the original text's location. For example, a user might request translation of text within the environment in the viewfinder. Translated text can be generated, and the text alignment of pre-existing text can be determined. The pre-existing text can then be replaced with the translated text based on the text alignment prediction to generate an augmented image. In some implementations, the determined positioning and / or alignment can be used to generate augmented reality anchors that describe on which image features the rendered text will be rendered to achieve the augmented reality experience.
[0036] Providing an augmented reality interface for displaying text in its native format can be challenging because heuristic-based placement can lead to incorrect alignment. In particular, tabular lists and other complex formatting can increase the difficulty of determining placement. For example, heuristics alone might cause the system to always center a single line of text, while a list could include multiple single lines with left alignment and varying indentations.
[0037] The alignment classification model can be trained on labeled images (e.g., images may have been annotated by multiple different human reviewers to label paragraphs and / or their corresponding alignments) to train the model to predict the text alignment of one or more paragraphs. The alignment classification model can process the image representation and one or more paragraph bounding boxes to generate predictions. The paragraph bounding boxes can be generated using a paragraph detection model, which can be a separate model and / or part of the alignment classification model. In some implementations, the alignment classification model can be further used for indentation classification, table alignment classification, and / or other structural classifications.
[0038] Text alignment prediction can be used to provide translated text rendering in a native format, which can be rendered via augmented reality glasses and / or augmented reality applications on a user's computing device. Therefore, the user can view the environment in their preferred language. Text alignment prediction can be lightweight and can be performed in parallel with translation generation to reduce latency.
[0039] Additionally and / or alternatively, the system and methods may perform style alignment. Style alignment may include using a style recognition model to process image data to determine font, font size, animation, and / or other style features (e.g., italics, bold, etc.). The output of the style recognition model can then be used to determine which styles to apply to text rendering in the translated language. The system and methods may perform style transfer and / or select styles that are aesthetically similar to the identified styles. In some implementations, fonts and / or other styles may be adjusted to ensure that the translated text is rendered in a readable manner while maintaining predicted alignment and relative style aesthetics.
[0040] In some implementations, the systems and methods disclosed herein can adjust the size, kerning, height, width and / or other features of the translated text to ensure that the rendered text is readable while maintaining the semantic alignment of the original text.
[0041] The systems and methods disclosed herein can be used to provide augmented reality translation experiences that offer translation with native alignment and positioning while simultaneously addressing issues caused by screen-based size constraints. The systems and methods can leverage hybrid solutions to deliver readable content within limited space capacity.
[0042] The systems and methods disclosed herein provide several technical effects and benefits. As an example, the systems and methods can provide an augmented reality experience by replacing first-language text with second-language text while simultaneously delivering the text in its native format (e.g., alignment, positioning, color, font, size, etc.). Specifically, the systems and methods disclosed herein can utilize paragraph detection, text alignment classification, optical character recognition, text translation, and / or augmented reality rendering to provide an augmented reality experience for generating and delivering translated text. Paragraph detection models and / or alignment classification models can be used to acquire and process image data to generate text alignment predictions. Optical character recognition models and / or translation models can be additionally used to process image data to generate translated text. The text alignment predictions and translated text can then be used to augment the original image to replace first-language text with the translated text.
[0043] Another technical benefit of the systems and methods disclosed herein is the ability to utilize one or more machine learning models to understand text within an image and provide rendered text in a native format. For example, the systems and methods may acquire an image, process the image using an embedding model to generate an image representation and process the image using a paragraph detection model to generate paragraph bounding boxes, process the image representation and paragraph bounding boxes using an alignment classification model to generate a predicted alignment classification, and then render text (e.g., translated text) within the image based on the predicted alignment classification.
[0044] Another example of the technical effects and benefits involves improved computational efficiency and improvements in the operation of the computational system. For instance, the systems and methods disclosed herein can leverage parallel processing and / or one or more lightweight models to reduce latency and lower the computational cost for alignment prediction and image augmentation. Parallel processing can reduce latency by determining the predicted alignment classification and text translation in parallel (e.g., simultaneously) rather than serially. Computational cost can be reduced by processing visual embeddings instead of the entire image, and can be further reduced by processing only the embedding features within the paragraph bounding boxes.
[0045] Exemplary embodiments of this disclosure will now be discussed in further detail with reference to the accompanying drawings.
[0046] Figure 1A block diagram of an example alignment prediction system 10 according to an exemplary embodiment of the present disclosure is depicted. In some implementations, the alignment prediction system 10 is configured to receive and / or obtain a set of input data including an image representation 12 describing an image and bounding boxes 14 describing the positioning and size of text blocks within the image, and, as a result of receiving the input data, to generate, determine, and / or provide output data of an alignment classification 18 including predicted text alignments of text blocks within the image (e.g., predicted text alignments of text within bounding boxes in the image). Thus, in some implementations, the alignment prediction system 10 may include an alignment classification model 16 operable to process the image representation 12 and bounding boxes 14 to predict whether the text is right-aligned, left-aligned, center-aligned, justified, top-aligned, and / or bottom-aligned.
[0047] Specifically, images can be captured using one or more image sensors of a computing device. An encoder model (e.g., an embedding model) can then be used to process the images to generate image representation 12. Image representation 12 may include one or more visual embeddings describing visual features within the image. The encoder model may include one or more transformer models. Additionally and / or alternatively, the encoder model may be part of a document understanding model used to determine the type of document depicted in the image, the presence of text within the image, the reading order of the text, and / or other document processing tasks.
[0048] An image can be processed using a paragraph detection model to generate one or more bounding boxes 14 describing the location and / or size of one or more blocks of text within the image (e.g., one or more paragraph bounding boxes describing the location and / or size of one or more paragraphs depicted within the image). The one or more bounding boxes 14 may include coordinates associated with their location within the image.
[0049] Alignment classification model 16 can process image representation 12 and one or more bounding boxes 14 to generate alignment classification 18 for each text block (e.g., paragraphs or other semantic groups of text determined by the bounding boxes). Alignment classification model 16 can be a machine learning model trained on benchmark ground truth bounding boxes and / or benchmark ground truth alignment classification labels. In some implementations, alignment classification model 16 may include one or more classification heads of a document understanding model. Additionally and / or alternatively, alignment classification model 16 can generate alignment classification 18 by processing specific portions of image representation 12 based on one or more bounding boxes 14. Alignment classification model 16 can be further tuned based on document type classification, detected language classification, and / or text data describing the text within the image.
[0050] Alignment category 18 can describe whether the text within the bounding box 14 is left-aligned, right-aligned, center-aligned, top-aligned, bottom-aligned, and / or justified. Alignment category 18 can be marked with and / or indexed with information describing the positioning and / or size of the corresponding text block (e.g., the corresponding paragraph).
[0051] The image can then be enhanced using alignment categories 18 and / or one or more bounding boxes 14 to generate an enhanced image that includes rendered text in the format of the original text. The rendered text may include translated text describing the original text translated into different languages. Alternatively and / or additionally, the rendered text may include the original text rendered at a higher resolution. The enhanced image can then be provided to the user.
[0052] Figure 2 A block diagram of an example image enhancement system 200 according to an exemplary embodiment of the present disclosure is depicted. The image enhancement system 200 and... Figure 1 Similar to the alignment prediction system 10, the image enhancement system 200 also includes a translation pipeline and the generation of the enhanced image 232.
[0053] Specifically, image 220 can be captured using one or more image sensors of a user computing device (e.g., a mobile computing device and / or a smart wearable device). Image 220 may depict text, which may include text from documents, signage, and / or product labels. In some implementations, image 220 may be an image obtained from the web and / or other applications.
[0054] Image 220 can be processed using a representation generation model 222 (e.g., an encoder model) to generate image representation 212. Image representation 212 may include one or more visual embeddings (e.g., one or more image embeddings and / or one or more text embeddings) describing visual features (e.g., object features, text features, and / or environmental features) within image 220. Representation generation model 222 may include one or more transformer models, one or more convolutional models, one or more self-attention models, and / or one or more feedforward models. Additionally and / or alternatively, representation generation model 222 may be part of an encoder backbone that may include a document understanding model for document type determination, layout parsing, text parsing, object recognition, context translation, text detection, reading order determination, and / or other document processing tasks for documents depicted in the image.
[0055] Image 220 can be processed using paragraph detection model 224 to generate one or more bounding boxes 214 describing the location and / or size of one or more text blocks within the image (e.g., one or more paragraph bounding boxes describing the location and / or size of one or more paragraphs depicted within the image). The one or more bounding boxes 214 may include coordinates associated with their location within the image. Paragraph detection model 224 can perform feature detection associated with detecting text features and / or structural features. In some implementations, one or more bounding boxes 214 can be generated by processing image representation 212 using paragraph detection model 224.
[0056] Alignment classification model 216 can process image representation 212 and one or more bounding boxes 214 to generate alignment classification 218 for each text block (e.g., paragraphs or other semantic groups of text). Alignment classification model 216 can be a machine learning model trained on benchmark ground truth bounding boxes and / or benchmark ground alignment classification labels generated based on user input to annotated user interface. In some implementations, alignment classification model 216 may include one or more classification heads of an image-encoded backbone. Additionally and / or alternatively, alignment classification model 216 can generate alignment classification 218 (e.g., alignment class identifiers) by processing specific portions of image representation 212 based on one or more bounding boxes 214. Alignment classification model 216 can be further tuned based on document type classification, detected language classification, and / or text data describing the text within image 220. In some implementations, alignment classification model 216 can be configured to directly process image 220 to generate one or more predicted localizations of one or more paragraphs and one or more predicted alignment classifications 218 for one or more paragraphs.
[0057] Alignment categories 218 can describe whether the text within bounding boxes 214 is left-aligned, right-aligned, center-aligned, top-aligned, bottom-aligned, and / or justified. Alignment categories 218 can be tagged with and / or indexed with information describing the positioning and / or size of the corresponding text blocks (e.g., corresponding paragraphs). In some implementations, alignment categories and / or one or more bounding boxes 214 can be embedded in an image layout embedding.
[0058] Image 220 can then be enhanced using alignment classification 218 and / or one or more bounding boxes 214 to generate an enhanced image 232 that includes rendered text in the format of the original text. The rendered text may include translated text 230 that describes the original text translated into different languages.
[0059] Specifically, the image enhancement system 200 may include a translation pipeline that processes image 220 to generate translated text 230, which may be rendered into the enhanced image 232 based on alignment classification and / or one or more bounding boxes 214.
[0060] The training pipeline may include an optical character recognition model 226 and a translation model 228. The optical character recognition model 226 can process image 220 to generate text data describing the text recognized within image 220. The translation model 228 can then process the text data to translate the text from a first language to a second language to generate translated text 230. The optical character recognition model 226 and the translation model 228 may include machine learning models. The translation model 228 may be trained for specific language-to-language translation and / or for multiple different translation types.
[0061] Generating the enhanced image 232 may include using a machine learning-based inpainting model to replace the text of image 220 with predicted background pixels before rendering (and / or overlaying) the translated text 230. In some implementations, one or more machine learning-based inpainting models may be used to generate the enhanced image 232, including diffusion models and / or one or more other generative image models. In some implementations, generating the enhanced image 232 may include determining that the source language (e.g., a first language) of image 220 has a different directional flow than the target language (e.g., a second language) (e.g., the source language can be read from left to right, while the target language can be read from right to left). The enhanced image 232 may then be generated to account for the directional differences while maintaining the original text alignment.
[0062] Figure 3 A flowchart depicts an example method performed according to an example embodiment of this disclosure. Although Figure 3 The steps are described in a particular order for illustrative and discussion purposes, but the method of this disclosure is not limited to the order or arrangement specifically shown. The various steps of method 300 may be omitted, rearranged, combined and / or adapted in various ways without departing from the scope of this disclosure.
[0063] At point 302, the computing system can acquire an image including text. The text may be in a first language (e.g., English, Spanish, French, Portuguese, Japanese, Vietnamese, etc.). The image may be generated using one or more sensors of the user computing device. The user computing device may include a mobile computing device and / or a smart wearable device. The image may be acquired from a viewfinder application's live camera feed and / or an augmented reality interface.
[0064] At 304, the computational system can process the image to generate an image representation. The image representation may include multiple vector representations. In some implementations, the image representation may include one or more visual embeddings. The image representation can be generated by processing the image using an encoder model. The encoder model may be part of a document understanding model. The encoder model may include one or more transformer models.
[0065] At 306, the computational system can utilize a paragraph detection model to process the image to generate one or more paragraph bounding boxes. These one or more paragraph bounding boxes can describe the location of one or more paragraphs within the image. The paragraph detection model can be part of a document understanding model. The paragraph detection model can be a machine learning model trained to generate bounding boxes indicating the location and size of paragraphs (or blocks of text) within the image. In some implementations, one or more paragraph bounding boxes can be generated by utilizing the paragraph detection model to process the image representation.
[0066] In some implementations, using a paragraph detection model to process an image to generate one or more paragraph bounding boxes may include generating multiple paragraph bounding boxes by using the paragraph detection model to process the image representation. Multiple predicted paragraph locations may be associated with multiple different paragraphs within the image. The paragraph bounding boxes may include oriented bounding boxes (e.g., bounding boxes with the same orientation as the text block) and / or axis-aligned bounding boxes (e.g., bounding boxes with the same orientation as the image boundary).
[0067] At point 308, the computational system can utilize a machine learning-based alignment classification model to process the image representation and one or more paragraph bounding boxes to generate one or more alignment classifications associated with one or more paragraphs. The one or more alignment classifications can describe the predicted text alignment of one or more paragraphs. The one or more alignment classifications can include right alignment, left alignment, justified alignment, top alignment, bottom alignment, and / or center alignment. The machine learning-based alignment classification model can be trained on user-annotated data. In some implementations, the machine learning-based alignment classification model can include one or more classification heads. The one or more alignment classifications can be labeled with paragraph positioning data determined based on one or more paragraph bounding boxes.
[0068] In some implementations, the machine learning alignment classification model may have been trained on a training dataset comprising multiple sample image representations, multiple benchmark ground truth bounding boxes, and multiple benchmark ground truth alignment labels. The multiple sample image representations may be associated with multiple training images. Each of the multiple training images may include corresponding text. In some implementations, the multiple training images may include text in multiple different languages. The multiple sample image representations may be associated with multiple training images. Each of the multiple training images may include corresponding text, and the multiple training images may include text with multiple different alignments. These multiple different alignments may include left alignment, right alignment, center alignment, top alignment, bottom alignment, and / or justification.
[0069] In some implementations, using a machine learning-based alignment classification model to process the image representation and one or more paragraph bounding boxes can include generating multiple alignment classifications associated with multiple different paragraphs by using the machine learning-based alignment classification model to process the image representation and multiple paragraph bounding boxes. For example, a corresponding alignment classification can be generated for each of multiple detected paragraphs.
[0070] In some implementations, using a machine learning-based alignment classification model to process an image representation and one or more paragraph bounding boxes to generate one or more alignment classifications associated with one or more paragraphs may include performing region-of-interest pooling on the image representation (e.g., visual features of the image representation) based on one or more paragraph bounding boxes to generate one or more pooled visual embeddings, using convolutional blocks to process one or more pooled visual embeddings to predict shape features of one or more paragraphs, generating one or more box localization embeddings based on one or more paragraph bounding boxes, and processing one or more shape features and one or more box localization embeddings to generate one or more alignment classifications.
[0071] At point 310, the computational system can obtain the translated text. The translated text can describe text in a second language (e.g., Spanish, French, Portuguese, Japanese, Vietnamese, English, Norwegian, Finnish, etc.). The translated text can be obtained from a translation pipeline that may have been processed in parallel with an alignment prediction pipeline. The translation pipeline may include an optical character recognition model and a translation model.
[0072] At point 312, the computing system can generate an augmented image based on an image, one or more alignment classifications, and translated text. The augmented image may include the image, while the text is replaced with translated text having text alignments determined based on one or more alignment classifications. The augmented image can be generated using one or more augmentation models (e.g., one or more machine learning augmentation models). The augmentation model can perform repairs to remove the original text, and then text rendering can be performed based on the translated text, paragraph bounding boxes, and alignment classifications. The computing system can then provide the augmented image for display. The augmented image can be provided for display in an augmented reality interface.
[0073] In some implementations, generating an augmented image based on an image, one or more alignment classifications, and translated text may include determining multiple feature anchors based on multiple paragraph bounding boxes and multiple alignment classifications, and rendering the translated text based on the multiple feature anchors and multiple alignment classifications. The multiple feature anchors can describe image features within the image that are associated with the localization used to render a specific paragraph. For example, the multiple feature anchors could be augmented reality rendering anchors.
[0074] Additionally and / or alternatively, the computing system may obtain a second image, determine that the second image includes image features associated with multiple feature anchors, and generate a second augmented image based on the multiple feature anchors, translated text, and multiple alignment classifications. The second augmented image may include the second image, where the text in the first language is replaced with translated text in the second language.
[0075] Figure 4A An illustration of an example image 410 according to an exemplary embodiment of the present disclosure is depicted. Image 410 may be a real-world reference image depicting text from a real-world document, signage, etc. Image 410 may include text of the document in the source language (e.g., a first language). The document depicted in example image 410 includes a first portion 412 of a set of paragraphs with a first alignment format and a second portion 414 of a set of bullet points with a second alignment format.
[0076] Figure 4B An illustration depicts an example enhanced output image incorporating text alignment prediction 420 according to an exemplary embodiment of this disclosure. Specifically, the systems and methods disclosed herein can be used to translate the text of image 410 and perform text alignment classification of multiple paragraphs of a document. An enhanced image with text alignment prediction 420 can then be generated based on the translated text and the text alignment classification of the multiple paragraphs. The rendered translated text can be rendered with correct native text alignment for both the first portion 412 and the second portion 414.
[0077] Figure 4CAn illustration depicts an example enhanced image without text alignment prediction 430 according to an exemplary embodiment of the present disclosure. The translated text of the enhanced image without text alignment prediction 430 may have been rendered based on heuristics rather than machine learning text alignment prediction. The text alignment of a single-line paragraph may be centered, rather than left-aligned as depicted in the source document. Therefore, a machine learning alignment classification model can make more accurate text alignment predictions than heuristic-based predictions.
[0078] Figure 5 A block diagram of an example text alignment classification system 500 according to an exemplary embodiment of the present disclosure is depicted. Specifically, the text alignment classification system 500 may include a text alignment generator 506 for processing page layout embeddings 502 and backbone visual embeddings 504 to generate mapping data 508 for rendering text. The page layout embedding 502 may describe the layout, structure, and / or document type of a document depicted in an image. In some implementations, the page layout embedding may include a structural representation (e.g., embeddings including information about words, lines, paragraphs, equations, blocks, tables, etc., their metadata, and how the information is logically connected). The backbone visual embedding 504 may describe visual features within the image.
[0079] The text alignment generator 506 may include a preprocessing block, a text alignment model (e.g., an alignment classification model), and / or a postprocessing block. The preprocessing block may preprocess the page layout embedding 502 and the backbone visual embedding 504 to generate model input for the text alignment model. The text alignment model may process the model input to generate model output describing paragraph text alignment and paragraph positioning. The postprocessing block may process the page layout embedding 502 and the model output to generate mapping data 508. The mapping data 508 may describe where and / or how the text is rendered to recreate the text format of the source image. The mapping data 508 may include embeddings, feature anchors, labeled alignment data, and / or other data.
[0080] Figure 6 A block diagram of an example text alignment classification model system 600 according to an example embodiment of the present disclosure is depicted. The text alignment classification model system may include a variety of different configurations and / or architectures. Figure 6An example is depicted; however, the text alignment classification model may include other configurations or architectures. Specifically, the text alignment classification model system 600 may include a text alignment model 606 (e.g., an alignment classification model) to process visual embeddings 602 and paragraph bounding boxes 604 to generate alignment classes 608 (e.g., one or more alignment classifications). The text alignment model 606 may include one or more region-of-interest pooling blocks, one or more L2 normalization blocks, one or more first-order normalization blocks, one or more first ReLU blocks, one or more first linear blocks, one or more second-order normalization blocks, one or more second ReLU blocks, one or more second linear blocks, and / or one or more additional processing blocks. The text alignment model 606 may include... Figure 6 The order depicted and / or blocks (and / or layers) in various other configurations.
[0081] For example, a text alignment model may include a classification head. The classification head (e.g., an alignment classification model) can perform region-of-interest pooling on visual features of a given image (dimensions of [batch size, H, W, 786]) using paragraph bounding boxes (which are the baseline ground truth during training and predicted by the paragraph detector during inference), resulting in features of dimensions [batch size, maximum number of paragraphs, 7, 7, 768]. The classification head can leverage convolutional layers to process the pooled visual embeddings, producing features of dimensions [batch size, maximum number of paragraphs, 7, 7, 256], and can perform max pooling along the 2-axis and 3-axis to produce features of shape [batch size, maximum number of paragraphs, 256]. The classification head can add processed bounding box localization embeddings to these processed pooled visual embeddings. In some implementations, the classification head can leverage three subsequent linear layers to process the obtained features to obtain a prediction of dimensions [batch size, maximum number of paragraphs] for each of the input paragraphs.
[0082] Figure 7 A flowchart depicts an example method performed according to an example embodiment of this disclosure. Although Figure 7 The steps are described in a particular order for illustrative and discussion purposes, but the method of this disclosure is not limited to the order or arrangement specifically shown. The various steps of method 700 may be omitted, rearranged, combined and / or adapted in various ways without departing from the scope of this disclosure.
[0083] At point 702, the computing system may acquire image data describing an image from one or more image sensors of the user's computing device. The image may include text in a first language. The image data may include a compressed version of the image. The image data may include one or more image embeddings and / or one or more text embeddings. The image may depict a sign, menu, book, screen, and / or other object.
[0084] At 704, the computational system can utilize a paragraph detection model to process the image to generate one or more paragraph bounding boxes. These one or more paragraph bounding boxes can describe the location of one or more paragraphs within the image. They can also describe the size of the text blocks. Furthermore, the one or more paragraph bounding boxes may include one or more overlapping bounding boxes associated with overlapping text.
[0085] At point 706, the computing system can utilize a machine learning-based alignment classification model to process image data and one or more paragraph bounding boxes to generate one or more paragraph location predictions and one or more alignment classifications associated with one or more paragraphs. The one or more alignment classifications can describe the predicted text alignment of the one or more paragraphs. The machine learning-based alignment classification model can be stored on the user's computing device. In some implementations, the one or more paragraph location predictions can be determined based on parsed detected text and / or based on one or more paragraph bounding boxes.
[0086] At point 708, the computational system can obtain the translated text. The translated text can describe text in a second language. The translated text can be parsed and / or segmented based on image regions. The translated text can be generated by processing text in an image using one or more machine learning natural language processing models.
[0087] In some implementations, obtaining translated text may include processing images using an optical character recognition model to generate text data describing text in a first language, and processing the text data using a translation model to generate translated text. Processing text data using a translation model to generate translated text may include processing the text data to determine that the text is in a first language, and obtaining a translation model in response to determining that the text is in a first language. In some implementations, obtaining a translation model may include determining the language preference of a particular user. The language preference may include a second language. Obtaining a translation model may include obtaining a translation model from a translation model database based on a translation request from a first language to a second language.
[0088] At point 710, the computing system can generate an enhanced image based on an image, one or more paragraph position predictions, one or more alignment classifications, and translated text. The enhanced image may include the image, while the text is replaced with translated text having text alignments determined based on one or more alignment classifications and specific text locations based on one or more paragraph position predictions. The enhanced image can then be provided for display via a user computing device.
[0089] In some implementations, the computational system can determine one or more feature anchors based on one or more paragraph position predictions and one or more alignment classifications. These feature anchors can then be used for continuous augmented reality rendering.
[0090] Figure 8 A flowchart depicts an example method performed according to an example embodiment of this disclosure. Although Figure 8 The steps are described in a particular order for illustrative and discussion purposes, but the method of this disclosure is not limited to the order or arrangement specifically shown. The various steps of method 800 may be omitted, rearranged, combined and / or adapted in various ways without departing from the scope of this disclosure.
[0091] At 802, the computational system can obtain training data. Training data may include sample image representations, baseline ground truth bounding boxes, and baseline ground truth alignment labels. Sample image representations can describe a specific image. In some implementations, training data may include images instead of image representations. Image representation models can be used to process training images to generate image representations. In some implementations, baseline ground truth bounding boxes can describe the baseline ground truth location of a paragraph within a specific image. Baseline ground truth alignment labels can describe the baseline ground truth text alignment of a paragraph within a specific image. In some implementations, baseline ground truth bounding boxes and baseline ground truth alignment labels may have already been generated based on user input associated with a specific image. User input can be obtained via a dedicated annotation user interface for generating baseline ground truth bounding box and baseline ground truth alignment label selections. Multiple sample images in the training dataset may have different numbers of paragraphs, different languages, different indentations, different alignments, different document types, and / or other differences.
[0092] At 804, the computational system can utilize an alignment classification model to process the sample image representation and the baseline ground truth bounding box to generate a predicted alignment classification. The predicted alignment classification can describe the predicted alignment of paragraphs. Predicted alignment may include predicted text alignment and / or predicted indentation. The alignment classification model may include an encoder model, a paragraph detection model, and / or one or more classification heads. Alternatively and / or additionally, the computational system can utilize the alignment classification model to process the sample image representation and the predicted paragraph bounding box to generate a predicted alignment classification.
[0093] At 806, the computational system can evaluate a loss function that assesses the difference between the predicted aligned classification and the baseline true aligned label. The loss function may include cross-entropy loss. Additionally and / or alternatively, the loss function may include one or more other losses (e.g., L2 loss).
[0094] At point 808, the computational system can at least partially adjust one or more parameters of the alignment classification model based on the loss function. Adjustment may include tuning one or more parameters. Adjustment can be performed based on gradient descent generated by the loss function through backpropagation. Training loops can be performed iteratively using multiple different training triples.
[0095] In some implementations, the computational system may utilize a paragraph detection model to process the image representation to generate predicted paragraph bounding boxes. Additionally and / or alternatively, an alignment classification model may be used to generate the predicted paragraph bounding boxes. The predicted paragraph bounding boxes can describe the predicted localization of the paragraphs. The computational system may evaluate a second loss function that assesses the difference between the predicted paragraph bounding boxes and the baseline ground truth bounding boxes, and adjust one or more second parameters of the alignment classification model based at least in part on the second loss function. Additionally and / or alternatively, one or more parameters of the paragraph detection model may be adjusted based on the second loss function.
[0096] Figure 9 A block diagram of an example augmented reality system 900 according to an exemplary embodiment of the present disclosure is depicted. Specifically, the augmented reality system 900 may include multiple pipelines for processing images to generate augmented reality output. The multiple pipelines may include a translation pipeline, a format determination pipeline, and / or a document type determination pipeline.
[0097] Images can be obtained from an image sensor on a user's computing device and / or from the web and / or a second user. One or more optical character recognition models 902 can be used to process the images to generate text data describing the text identified and recognized within the images. The one or more optical character recognition models 902 may include general OCR models and / or mathematical OCR models (e.g., OCR models specifically trained for mathematical character recognition, which may include graph recognition and / or equation recognition).
[0098] The language detection block 906 can then be used to process the text data to determine the source language (e.g., the language of the text in an image) and / or the target language (e.g., the user's preferred language). The translation model 908 can then be used to process the text data based on the determined source and target languages to generate translated text.
[0099] Image and / or text data can also be processed using a content extractor block, which may include an encoder model to generate visual embeddings. A paragraph detection model 910 can be used to process the extracted content (e.g., image representation) and / or image to generate paragraph bounding boxes that can be associated with corresponding unique IDs. The paragraph bounding boxes and image representations can then be processed using a repair model 912, a text style determination model 914, and / or an alignment classification model 916. The repair model 916 can replace text pixels from the image with predicted replacement pixels. The text style determination model can determine the font, kerning, size, color, and / or other style features of the original text. The alignment classification model 916 can determine the text alignment for each element in the paragraph. The repair data, text style data, and alignment classification can then be combined with page layout embeddings and / or image representations.
[0100] Additionally and / or alternatively, the document type classification model 904 can be used to process the image to generate a document type classification. The document type classification can then be used to adjust the input of one or more other blocks.
[0101] Then, a text response 920 can be generated based on the merged data, translated text, and document type classification. The text response 920 can include what text to render, how to render the text, and where to render the text. The substantive content of the rendered text can include the translated text. Position and formatting can be determined based on text style, text alignment, and / or document type classification. An augmented image for the augmented reality interface can then be generated by using a repair model 914 for repair and then rendering the text based on the text response 920.
[0102] Text alignment models (e.g., alignment classification models) can be used in augmented reality applications (e.g., augmented reality translation interfaces) to align text in rendered translated paragraphs in the same way that text is aligned in the original input image (left-aligned, center-aligned, right-aligned, top-aligned, bottom-aligned, or justified). The alignment signal can provide visual appearance improvements in the rendering of the translated text. The model can be implemented as a lightweight classification prediction head within a document understanding framework, capable of handling document understanding visual embeddings and paragraph bounding boxes detected by a paragraph detector (e.g., a machine learning paragraph prediction model).
[0103] The systems and methods disclosed herein can generate signals for augmented reality interfaces, which capture, for each paragraph, whether the source text in an image is left-aligned, right-aligned, or center-aligned. Alignment can improve the rendering quality of translated text because alignment-based rendering better preserves the original text structure and allows users to effortlessly map translated text to the original source text.
[0104] Augmented reality translation interfaces can overlay translations onto text in images and / or real-time camera feeds. Translation overlays can become tricky if the translated text is not well aligned with the source text. Rendering translations with proper alignment can be challenging.
[0105] The system can utilize writing direction for image understanding, which can be determined by mapping the inferred source language to left-to-right or right-to-left. Translation rendering can use a similar mapping from the target language, rather than the source language, to determine how to align the translated text.
[0106] In some implementations, systems and methods can utilize heuristics to align text paragraph by paragraph. For example, alignment can be heuristic-based, including: (1) center alignment if the paragraph consists of a single line of text; (2) right alignment if the target language is RTL (e.g., Arabic); and / or (3) left alignment otherwise. However, heuristics may struggle to handle more complex alignments.
[0107] For training, benchmark real paragraphs can be labeled with alignment signals to generate a training dataset. The dataset can include multiple images with multiple paragraphs in several different languages, and each paragraph can be labeled with one of the following alignment categories: Center alignment, justify, left alignment, right alignment, top alignment, and bottom alignment .
[0108] Data can be divided into Tuning and release The part is split, where release This can be used for the final testing of the production model. Therefore, for training purposes, the remaining training examples can be split into... tuning_train and tuning_val This is to obtain a validation set for monitoring the training of the model. This can be generated while remembering the document type. tuning_train and tuning_val The segmentation ensures that the two sets are balanced relative to different document type tags. Annotated data can include word, line, and / or paragraph tags.
[0109] The annotation process may include using plugins to obtain input from human graders. The rating interface may include providing reference images for display along with options for labeling paragraphs (e.g., labeling alignment). In some implementations, the rating interface may include providing reference images for display along with options for annotating paragraph positioning and size.
[0110] During model training, the system and method can measure labeled benchmark real data. tuning_val On the segment Accuracy, Precision, Recall and F1 scoreEvaluation metrics can be aggregated by class, by language, and / or across all classes. Additional evaluation can be performed using confusion matrices and / or visualizations of predictions.
[0111] The system and method disclosed in this paper can implement the alignment classification model within the document understanding framework as a separate lightweight classification head.
[0112] Paragraph alignment signals can be included as a downstream task. The objective could be to use image and text features obtained from the frozen encoder backbone as input to a lightweight downstream classification model (e.g., an alignment classification model could be used to process image and text features to generate a predicted alignment classification signal).
[0113] In some implementations, the architecture may include an encoder model, a paragraph detector model, and a prediction head (e.g., an alignment classification model). The encoder model may process the image to generate one or more image representations and / or one or more text representations. The paragraph detection model (i.e., the paragraph detector) may process the image to generate a sequence of paragraph bounding boxes. The prediction head (e.g., an alignment classification model) may process the image representation and the paragraph bounding boxes to generate one or more predicted alignment classes. The prediction head may include a region of interest alignment layer and one or more linear layers.
[0114] An encoder model can be part of a document understanding model trained to process images and / or text to encode full-page context and hierarchical relationships between different text entities and / or generate a generalized structured representation of a document from pixels that can be seamlessly used for several downstream tasks. The document understanding model may include one or more encoder models, one or more visual decoders, and / or one or more text decoders. The encoder model may include a transformer model (e.g., a multimodal vision-text transformer with trainable visual masking terms and trainable text masking terms). In some implementations, the document understanding model may determine paragraphs, document reading order, question selection (e.g., question identification and segmentation), text alignment, and / or document type classification.
[0115] In some implementations, the document understanding model may include a multimodal transformer for encoding both images and text, and may include a box prediction head responsible for regressing bounding boxes corresponding to different parts of the text layout, such as paragraphs, sections, headings, etc. The model can process both text detected within the image and pixels of the image simultaneously. Input can be provided to a model paired with a "locality embedding," which can represent the location of the element (i.e., a word or pixel block) within a 2D reference frame of the image.
[0116] A paragraph detection model may include a multimodal encoder and one or more bounding box decoders. In some implementations, the paragraph detection model may be trained for paragraph detection, indentation detection, list detection, table detection, and / or other semantic structure detection tasks. The paragraph detection model may be part of a document understanding model and / or an alignment classification model. In some implementations, the paragraph detection model may be configured and / or trained for multimodal processing. For example, the paragraph detection model may process an image and text extracted from the image to generate paragraph bounding boxes. In some implementations, the paragraph detection model may generate oriented (or angled) bounding boxes that can be oriented based on the orientation (or angle) of the detected text. The orientation of the bounding boxes may be processed by an alignment classification model to generate an orientation-based classification (e.g., inverted left alignment).
[0117] Individual blocks and / or models can be trained and / or tuned individually or jointly. Alignment prediction can be performed in parallel with text translation, text color prediction, image inpainting prediction (e.g., replacing previous text with predicted background pixels before overlaying translated text), font prediction, and / or other tasks.
[0118] Alternatively and / or additionally, the system and method may include a single model for paragraph detection and alignment classification. For example, an image representation may be generated, and the alignment classification model may process the image representation to generate one or more paragraph bounding boxes and one or more alignment classifications (e.g., a sequence of paragraph bounding box predictions along with alignment class predictions).
[0119] In some implementations, the encoder model may be a separate model from the alignment classification model and may be interchangeable with one or more other encoder models. In some implementations, the alignment classification model may include a dense prediction head that processes the image representation to generate dense predictions of the same size as the input image, with each pixel labeled with an alignment class prediction. The alignment classification model may include one or more magnified convolutional blocks (e.g., magnified convolutional layers).
[0120] In some implementations, the encoder model may include a unimodal visual backbone and / or a portion of the unimodal visual backbone to obtain an image representation. The representation can then be pooled using region-of-interest alignment using a graphical optical character recognition paragraph bounding box, and can then be sent to a classification block for alignment prediction.
[0121] In some implementations, the encoder model may include a pre-trained image encoder and / or a multimodal embedding model.
[0122] The model may include a classification head. The classification head (e.g., an alignment classification model) can perform region-of-interest pooling on visual features of a given image (dimensions of [batch size, H, W, 786]) using paragraph bounding boxes (which are the baseline ground truth during training and predicted by the paragraph detector during inference), resulting in features of dimensions [batch size, maximum number of paragraphs, 7, 7, 768]. The classification head can leverage convolutional layers to process the pooled visual embeddings, producing features of dimensions [batch size, maximum number of paragraphs, 7, 7, 256], and can perform max pooling along the 2-axis and 3-axis to produce features of shape [batch size, maximum number of paragraphs, 256]. The classification head can add processed bounding box localization embeddings to these processed pooled visual embeddings. In some implementations, the classification head can leverage three subsequent linear layers to process the obtained features to obtain a prediction of dimensions [batch size, maximum number of paragraphs] for each of the input paragraphs.
[0123] In some implementations, the system and method can be trained using both textual and image modalities. Additionally and / or alternatively, the alignment classification model can process rotated bounding boxes instead of axis-aligned bounding boxes. In some implementations, the alignment classification model can handle document type classification to condition the model for tables, lists, and / or receipts. For example, document type classification (e.g., nutrition tables, purchase receipts, etc.) can be the input terms used to condition the model. The alignment classification model can include kernel regularizers, one or more temporary fallbacks, L2 loss regularization, batch normalization, ReLU activation, layer normalization, cross-attention blocks, self-attention blocks (e.g., multi-head self-attention models), and / or other architectural blocks or layers.
[0124] Text alignment prediction can be performed to enrich the structural representation by leveraging additional signals indicating the alignment of text within each paragraph (e.g., including information about words, lines, paragraphs, equations, blocks, tables, etc., along with their metadata and embeddings of how the information is logically connected). Text alignment prediction can include processing the output of a paragraph detection model and visual features obtained from the encoded backbone.
[0125] Text alignment prediction can include preprocessing nodes, text alignment classification models, and / or postprocessing nodes.
[0126] In the preprocessing node, the system can preprocess paragraph bounding boxes obtained from the segmentation model. These bounding boxes can be provided as bounding boxes including metadata such as [top, left, height, width, angle] and can be read from the input page layout embedding. The boxes can be preprocessed to be axis-aligned and formatted as [top, left, width, height], and can be scaled to fit within the range [0, 1]. The processed axis-aligned paragraph bounding boxes can then be provided to the alignment classification model along with the visual embedding.
[0127] Alignment classification models can include lightweight classification models built on top of the encoder backbone. They can process axis-aligned paragraph bounding boxes and visual features obtained from the encoder backbone to output the alignment class for each given paragraph.
[0128] The alignment predictions generated by the alignment classification model can then be sent to the post-processing node along with the input page layout embedding. The post-processing node can then output a mapping between a unique paragraph ID and its corresponding alignment signal, where the unique paragraph ID is obtained from the input page layout embedding (e.g., the ID of each paragraph entity).
[0129] Alignment signals can be sent in text metadata format. Alignment tags can then be post-processed into a dedicated text alignment category and matched against unique paragraph IDs defined by the page layout embedding. These matches can then be stored in a text metadata mapping format and can be the output of post-processing nodes.
[0130] The mapping can be sent to a merging operator, which can insert text alignment signals (along with other metadata signals such as translation, image restoration, and / or reading order prediction) into the input page layout embedding.
[0131] In some implementations, text alignment can be implemented as a signal on a semantic understanding platform. The alignment classification model can run on a CPU, and model execution can be performed on the backend of the semantic understanding platform.
[0132] The preprocessing block prepares the input for use in the alignment classification model. The input to the alignment classification model can include visual embeddings from the encoder backbone and paragraph bounding boxes retrieved from the input page layout embeddings. The bounding boxes can be converted to axis-aligned bounding boxes and then, together with the visual embeddings, converted to tensors.
[0133] The paragraph bounding box for each detected paragraph returned by the paragraph detection model can be sent as a rotated bounding box and stored in the input page layout embedding. Coordinates can be represented in pixels and defined using the top-left coordinates (top_left_x, top_left_y), the width and height of the box, and the rotation angle. The box can be preprocessed to be aligned with the image axes before being sent to the alignment classification model. Preprocessing can prepare the bounding boxes for region-of-interest pooling.
[0134] The axis-aligned bounding boxes can be additionally translated by offsets defined by the padding applied to the input image. For example, the original input image may undergo preprocessing before being sent to the encoder backbone, which may resize the image and apply center padding to give the input image a square size. The visual embedding obtained from the model can correspond to the padded image. However, bounding boxes given in the input page layout embedding can be provided relative to the original image (e.g., they may undergo post-processing to recover the padding and resizing operations performed on the image). The system can add offsets corresponding to the image padding to the obtained axis-aligned bounding boxes to position them in the correct part of the visual embedding. Translation of the bounding boxes can move these boxes so that they point to the part of the visual embedding that corresponds to the pixels of the original input image, rather than the padded pixels.
[0135] The preprocessing block can output a tensor map that includes visual embeddings and axis-aligned paragraph bounding boxes.
[0136] Alignment classification models can process visual embeddings and paragraph bounding boxes to generate classification labels for each of the detected paragraphs.
[0137] The alignment classification model can generate alignment predictions for each paragraph bounding box passed as input, including padding words. In the post-processing step, the system can merge the predictions with the unique paragraph IDs provided in the paragraph entity information within the page layout embedding. For example, the system can generate a mapping that associates each paragraph prediction with its corresponding unique paragraph ID. This mapping can then be sent to merge the relevant metadata with the page layout embedding.
[0138] Generally, this disclosure relates to systems and methods for achieving realistic text inpainting in augmented reality using generative models (e.g., generative adversarial networks). An example application of the proposed system is augmented reality translation. For example, a user can operate an image capture device (e.g., a camera, smartphone, etc.) to capture images of a real-world scene including real-world text (e.g., signs, restaurant menus, etc.). The real-world text can be translated into different languages. Additionally, according to one aspect of this disclosure, the captured image can be processed using a machine learning generative model to produce an augmented image. The augmented image can depict the real-world scene, while the real-world text is removed. Specifically, due to the use of a machine learning generative model, the augmented image can appear significantly more realistic, for example, compared to an image where the real-world text is simply obscured using a monochrome frame. The translated text can be combined with the augmented image to generate an output image depicting the real-world scene including the translated text. The output image can be provided for display to a user (e.g., within the viewfinder interface of a camera-enabled application), thereby providing an augmented reality experience including the translation of the real-world text.
[0139] The systems and methods disclosed herein offer several technical effects and benefits. As an example, by using machine learning-based generative models, the proposed technique can provide a more realistic and immersive augmented reality experience. Specifically, machine learning-based generative models are capable of generating augmented images in which real-world text has been removed, but which remain significantly more realistic or visually accurate compared to more naive text removal or occlusion techniques. For example, machine learning-based generative models can be specifically trained to remove text from images and replace (or “repair”) the removed text with a very realistic, consistent, and accurate background. Therefore, using machine learning-based generative models can produce final images that are more faithful to the actual real-world background while also including translated text, thereby providing a more immersive augmented reality experience. This also has the effect of not occluding useful content in the image (such as content that may contain text to be translated and included in labeled images, maps, etc.). As another example of technical effects and benefits, by using machine learning-based generative models, the proposed method is more robust to changes in camera pose, lighting, or other dynamic changes in the scene, for example, compared to fragile human algorithms. This can result in a more consistent and immersive user experience.
[0140] Figure 10 A block diagram of an example augmented reality translation system 1000 according to an exemplary embodiment of the present disclosure is depicted. The augmented reality translation system 1000 can receive an input image 1014. The input image can depict a real-world scene including real-world text at text locations. For example, the scene can include a signboard including text on the signboard. As another example, the scene can include a physical document (e.g., a menu, form, newspaper, receipt, etc.) including text on the physical document.
[0141] Furthermore, while the present disclosure describes various aspects of real-world imagery depicting real-world scenes, these methods can also be applied to virtual or synthetic imagery depicting virtual or synthetic scenes. As an example, the proposed methods can be applied to virtual cameras (e.g., virtual cameras in video games or other virtual worlds) for translating text content rendered or otherwise included within a virtual or synthetic scene.
[0142] In some implementations, the input image 1014 can be an image captured by a user using an image capture device (e.g., a camera, smartphone, etc.). For example, the user device (e.g., a smartphone, tablet, etc.) may include one or more applications that support a camera. As an example, the application may be a dedicated camera application. In other examples, various applications may include an interface that allows the user to capture images (e.g., by invoking a dedicated camera application or otherwise). When facilitating image capture by the user, the application may provide a viewfinder interface, which generally depicts the camera's field of view in real time (e.g., displays the camera's current field of view in the interface).
[0143] The augmented reality translation system 1000 may include a text detection and recognition system 1018. The text detection and recognition system 1018 can detect and recognize text within an input image 1014. As an example, the text detection and recognition system 1018 may include various components or employ various techniques, such as line detection, script recognition, optical character recognition, inferring text structure (e.g., "segmentation"), etc. These tasks can be performed using conventional methods or by employing machine learning models (such as, for example, text detection and / or recognition models).
[0144] Therefore, the text detection and recognition system 1018 can employ various techniques to generate a set of recognized text 1026 and identifiers for one or more text locations 1028. The recognized text 1026 can be expressed using tokens, characters, and / or other text representations. The text locations 1028 can be expressed as coordinate values (e.g., (x, y) values in pixel space), bounding boxes, extracted portions of the input image 1014, and / or other formats. As another example, the text locations 1028 can be used to generate a mask (e.g., a binary pixel-level mask) indicating the locations of text within the input image 1014.
[0145] As yet another example, text position 1028 can be used to extract one or more sub-parts of the text specifically depicted in the input image 1014 (e.g., to exclude parts of the input image 1014 that do not depict text). As an example, the multiple sub-parts correspond to multiple lines of text, multiple words of text, or multiple chunks of text, respectively. For example, the portion of the input image 1014 depicting text can be broken down into chunks of predefined sizes.
[0146] Augmented reality translation system 1000 may include translation system 1020. Translation system 1020 can translate identified text 1026 into translated text 1030, wherein translated text 1030 corresponds to identified text 1026 translated into a different language. Translation system 1020 may include various components or employ various techniques (such as lookup tables) or machine learning models (such as sequence-to-sequence translation models). In some examples, translation system 1020 may be performed or implemented by invoking an external translation service or server.
[0147] According to one aspect of this disclosure, the augmented reality translation system 1000 may include a machine learning generative model 1022. The machine learning generative model 1022 may process at least a portion of an input image 1014 to generate an augmented image 1032. The augmented image 1032 may depict a real-world scene, while real-world text is removed (e.g., replaced or "repaired" to an inferred or predicted background that visually and realistically matches the rest of the real-world scene's background). In some implementations, the machine learning generative model 1022 may be a convolutional neural network. In some implementations, the machine learning generative model 1022 may be trained in an adversarial arrangement and may therefore be referred to as a generative adversarial network (GAN). Figure 10 A machine learning generative model 1022 is described; however, the systems and methods disclosed herein can utilize one or more other machine learning models instead of the machine learning generative model 1022. For example, in some implementations, image generation and / or pixel generation may not be performed.
[0148] In some implementations, the machine learning generative model 1022 can process the entire input image 1014 to generate an enhanced image 1032. In other implementations, the machine learning generative model 1022 can process one or more sub-portions of the input image 1014 to generate an enhanced image 1032. For example, as described above, the text detection and recognition system 1018 can detect one or more sub-portions of the input image 1014 depicting real-world text. The machine learning generative model 1022 can process one or more sub-portions of the input image 1014 to generate one or more enhanced sub-portions. The augmented reality translation system 1012 can replace one or more sub-portions of the input image 1014 with one or more enhanced sub-portions to generate an enhanced image 1032.
[0149] Augmented reality translation system 1012 can generate an output image 1016 from an augmented image 1032 and translated text 1030. The output image 1016 can depict a real-world scene with the translated text 1030 at text location 1028. As an example, augmented reality translation system 1000 can implement a renderer 1024 for overlaying the translated text 1030 onto the augmented image 1032 at text location 1028. The renderer 1024 can be provided with information about text location 1028, attributes of the real-world text in the input image 1014 (e.g., color, font, etc.), information about camera pose / orientation, and / or other information that allows the output image 1016 to have a similar appearance and feel to the input image 1014 regarding the translated text 1030. In another example, the renderer 1024 can be a more complex model performing generative text rendering.
[0150] The augmented reality translation system 1012 (or other collaborative system) can provide an output image 1016 for display to the user. For example, the output image 1016 can be provided in the viewfinder interface of a camera-enabled application.
[0151] In some implementations, the augmented reality translation system 1000 can operate iteratively and in real-time across multiple iterations. For example, the augmented reality translation system 1000 can be executed when the user interacts with a camera-enabled application. When the application is opened, it can be as follows: Figure 10 Each image captured by the camera is processed as shown to generate a corresponding output image. The output image can be provided in the viewfinder interface. In this way, a real-time augmented reality experience can be provided. In other implementations, the proposed method can be implemented only on selected keyframes (e.g., determined by motion, blur, etc.) rather than operating on each individual image captured by the camera. The augmented image can then be tracked and correctly positioned and superimposed on the real-time camera feed until the next frame is selected. For example, the brightness of the augmented image can be adjusted by measuring the change relative to neighboring pixels at frame selection. Alternatively or additionally, the augmented reality translation system 1000 can operate "post-capture" (i.e., operate at some point after the image has been captured, rather than in real-time).
[0152] The augmented reality translation system 1000 can be implemented entirely or partially at the user device, at the server device, or at a combination of the user device and the server device.
[0153] Generative model 1022 may include an encoder-decoder architecture. The architecture may include special mobile-friendly convolutional heads (e.g., MobileNetV1 and / or MobileNetV2 convolutional blocks or layers). Additionally, to increase flow between distant layers, the architecture may include skip connections. Additionally and / or alternatively, the architecture may include dilated convolutional layers in the middle of bottleneck sections of the network, thereby enabling the model to capture global context.
[0154] In some implementations, the generative model 1022 can process the entire image 1014 (e.g., the bounding boxes of optical character recognition lines are also provided to the model). However, in other implementations, the generative model 1022 can process sub-parts of the input image 1014. For example, each sub-part can be processed by the model individually and / or in parallel.
[0155] In one example, lines of text can be segmented into fixed-size blocks and then passed to the model in batches. In other examples, each subsection can correspond to a single word. In yet another example, each subsection can correspond to an entire line of text.
[0156] Figures 11A to 11C A graphical representation of an example augmented reality translation according to an exemplary embodiment of this disclosure is provided. Specifically, Figure 11A The image shown is the initial image captured by the image capture device. Figure 11A The image depicts a book whose cover includes real-world English language text. Figure 11A It can be the original image, and Figure 11B and Figure 11C It is an enhanced image.
[0157] Figure 11B The output image shown is an augmented reality translation of real-world English text into Japanese text, displayed on a book cover. (As shown in...) Figure 11B As can be seen, aspects of the background image (e.g., cliffs and beaches) (e.g., coloring) are still preserved behind the translated Japanese text (e.g., cliffs and beaches are still visible behind the Japanese text). This demonstrates an improvement of this disclosure (e.g., compared to other techniques that rely on monochrome blocks, blurring, or do not preserve the color and / or semantic content of the background image).
[0158] Figure 11C This demonstrates that the augmented reality translation experience continues seamlessly despite changes in camera orientation relative to the scene (e.g., camera movement relative to books). This confirms the benefits of using a learned model to provide consistent output as disclosed in this publication.
[0159] Figure 12AA block diagram of an example computing system 100 for alignment classification according to an example embodiment of the present disclosure is depicted. System 100 includes a user computing system 102, a server computing system 130, and / or a third computing system 150 communicatively coupled via a network 180.
[0160] User computing system 102 may include any type of computing device, such as, for example, a personal computing device (e.g., a laptop or desktop computer), a mobile computing device (e.g., a smartphone or tablet), a game console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.
[0161] User computing system 102 includes one or more processors 112 and memory 114. The one or more processors 112 can be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and can be a single processor or multiple processors operatively connected. Memory 114 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. Memory 114 can store data 116 and instructions 118, which are executed by processor 112 to operate user computing system 102.
[0162] In some implementations, the user computing system 102 may store or include one or more machine learning models 120. For example, the machine learning model 120 may be, or may otherwise include, various machine learning models, such as neural networks (e.g., deep neural networks) or other types of machine learning models, including nonlinear and / or linear models. Neural networks may include feedforward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks, or other forms of neural networks.
[0163] In some implementations, one or more machine learning models 120 may be received from server computing system 130 via network 180, stored in user computing device memory 114, and then used or otherwise implemented by one or more processors 112. In some implementations, user computing system 102 may implement multiple parallel instances of a single machine learning model 120 (e.g., parallel machine learning model processing across multiple instances of input data and / or detected features).
[0164] More specifically, one or more machine learning models 120 may include one or more detection models, one or more classification models, one or more segmentation models, one or more augmentation models, one or more generative models, one or more natural language processing models, one or more optical character recognition models, and / or one or more other machine learning models. One or more machine learning models 120 may include one or more transformer models. One or more machine learning models 120 may include one or more neural radiation field models, one or more diffusion models, and / or one or more autoregressive language models.
[0165] One or more machine learning models 120 can be used to detect one or more object features. The detected object features can be classified and / or embedded. Classification and / or embedding can then be used to perform a search to determine one or more search results. Alternatively and / or additionally, one or more detected features can be used to determine that indicators (e.g., user interface elements indicating detected features) will be provided to indicate that features have been detected. The user can then select the indicator to allow feature classification, embedding, and / or searching to proceed. In some implementations, classification, embedding, and / or searching can be performed before the indicator is selected.
[0166] In some implementations, one or more machine learning models 120 can process image data, text data, audio data, and / or latently encoded data to generate output data, which may include image data, text data, audio data, and / or latently encoded data. One or more machine learning models 120 can perform optical character recognition, natural language processing, image classification, object classification, text classification, audio classification, context determination, action prediction, image correction, image enhancement, text enhancement, sentiment analysis, object detection, error detection, repair, video stabilization, audio correction, audio enhancement, and / or data segmentation (e.g., mask-based segmentation).
[0167] A machine learning model can be or includes one or more machine learning models or model components. Example machine learning models can include neural networks (e.g., deep neural networks). Example machine learning models can include non-linear or linear models. Example machine learning models can use other architectures to replace or attach to neural networks. Example machine learning models can include decision tree-based models, support vector machines, hidden Markov models, Bayesian networks, linear regression models, k-means clustering models, etc.
[0168] Example neural networks can include feedforward neural networks, recurrent neural networks (RNNs) (including recurrent neural networks based on long short-term memory (LSTM)), convolutional neural networks (CNNs), diffusion models, generative adversarial networks, or other forms of neural networks. Example neural networks can also be deep neural networks. Some example machine learning models may utilize attention mechanisms, such as self-attention. For example, some example machine learning models may include multi-head self-attention models.
[0169] Machine learning models can include single or multiple instances of the same model configured to operate on data from the input. Machine learning models can also include an ensemble of different models that can interact collaboratively to process data from the input. For example, machine learning models can employ a hybrid expert structure. See, for example, Zhou et al. Mixture-of-Experts with Expert Choice Routing , arXiv:2202.09368v2 (October 14, 2022).
[0170] Inputs can generally include or otherwise represent various types of data. Inputs can include one type of data or many different types of data. Outputs can be data of the same type or a different data type compared to the inputs. Outputs can include one type of data or many different types of data.
[0171] Example data types of input or output include natural language text data, software code data (e.g., source code, object code, machine code, or any other form of computer-readable instruction or programming language), machine code data (e.g., binary code, assembly code, or other forms of machine-readable instruction that can be executed directly by a computer's central processing unit), assembly code data (e.g., a low-level programming language that uses a symbolic representation of machine code instructions to program the processing unit), genetic data or other chemical or biochemical data, image data, audio data, audiovisual data, tactile data, biometric data, medical data, financial data, statistical data, geographic data, astronomical data, historical data, and generally sensor data (e.g., digital or analog values, such as voltage or other absolute or relative level measurements from real or artificial inputs, such as from audio sensors, light sensors, displacement sensors, etc.). Data can be raw or processed and can be in any format or mode.
[0172] In multimodal inputs or outputs, example combinations of data types include image data and audio data, image data and natural language data, natural language data and software code data, image data and biometric data, sensor data and medical data, etc. It should be understood that any combination of data types can exist in the input or output.
[0173] Example inputs may include one or more data types, such as the example data types noted above. Example outputs may include one or more data types, such as the example data types noted above. The data type of the input may be the same as or different from the data type of the output. It should be understood that the example data types noted above are provided for illustrative purposes only. The data types contemplated within the scope of this disclosure are not limited to those examples noted above.
[0174] Additionally or alternatively, one or more machine learning models 140 may be included in or otherwise stored and implemented by server computing system 130, which communicates with user computing system 102 according to a client-server relationship. For example, machine learning model 140 may be implemented by server computing system 130 as part of a web service (e.g., viewfinder service, visual search service, image processing service, ambient computing service, and / or overlay application service). Thus, one or more models 120 may be stored and implemented at user computing system 102, and / or one or more models 140 may be stored and implemented at server computing system 130.
[0175] User computing system 102 may also include one or more user input components 122 that receive user input. For example, user input component 122 may be a touch-sensitive component (e.g., a touch-sensitive display or touchpad) sensitive to the touch of a user input object (e.g., a finger or stylus). The touch-sensitive component may be used to implement a virtual keyboard. Other example user input components include a microphone, a traditional keyboard, or other means by which the user can provide input.
[0176] In some implementations, user computing system 102 may store and / or provide one or more user interfaces 124, which may be associated with one or more applications. The one or more user interfaces 124 may be configured to receive input and / or provide data to be displayed (e.g., image data, text data, audio data, one or more user interface elements, augmented reality experiences, virtual reality experiences, and / or other data to be displayed). User interfaces 124 may be associated with one or more other computing systems (e.g., server computing system 130 and / or third-party computing system 150). User interfaces 124 may include viewfinder interfaces, search interfaces, generative model interfaces, social media interfaces, and / or media content library interfaces.
[0177] User computing system 102 may include one or more sensors 126 and / or receive data from one or more sensors 126. The one or more sensors 126 may be housed in a housing assembly that houses one or more processors 112, memory 114, and / or one or more hardware components that may store one or more software packages and / or cause one or more software packages to run. The one or more sensors 126 may include one or more image sensors (e.g., cameras), one or more lidar sensors, one or more audio sensors (e.g., microphones), one or more inertial sensors (e.g., inertial measurement units), one or more biosensors (e.g., heart rate sensors, pulse sensors, retinal sensors, and / or fingerprint sensors), one or more infrared sensors, one or more location sensors (e.g., GPS), one or more touch sensors (e.g., conductive touch sensors and / or mechanical touch sensors), and / or one or more other sensors. The one or more sensors may be used to acquire data associated with the user's environment (e.g., images of the user's environment, records of the environment, and / or the user's location).
[0178] User computing system 102 may include user computing device 104 and / or portions thereof. User computing device 104 may include mobile computing devices (e.g., smartphones or tablets), desktop computers, laptop computers, smart wearable devices, and / or smart home appliances. Additionally and / or alternatively, the user computing system may acquire data from one or more user computing devices 104 and / or generate data using one or more user computing devices 104. For example, a smartphone camera may be used to capture image data describing the environment, and / or an overlay application of user computing device 104 may be used to track and / or process data provided to the user. Similarly, one or more sensors associated with a smart wearable device may be used to acquire data about the user and / or the user's environment (e.g., image data may be acquired using a camera housed in the user's smart glasses). Additionally and / or alternatively, data may be acquired and uploaded from other user devices that may be specifically used for data acquisition or generation.
[0179] Server computing system 130 includes one or more processors 132 and memory 134. The one or more processors 132 can be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and can be a single processor or multiple processors operatively connected. Memory 134 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. Memory 134 can store data 136 and instructions 138, which are executed by processor 132 to operate server computing system 130.
[0180] In some implementations, the server computing system 130 includes one or more server computing devices or is otherwise implemented by said one or more server computing devices. In instances where the server computing system 130 includes multiple server computing devices, such server computing devices may operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.
[0181] As described above, server computing system 130 may store or otherwise include one or more machine learning models 140. For example, model 140 may be, or can otherwise include, various machine learning models. Example machine learning models include neural networks or other multi-layered nonlinear models. Example neural networks include feedforward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Reference Figure 9 B. Discussion of example model 140.
[0182] Additionally and / or alternatively, server computing system 130 may include search engine 142 and / or be communicatively connected to search engine 142, which may be used to crawl one or more databases (and / or resources). Search engine 142 may process data from user computing system 102, server computing system 130, and / or third-party computing system 150 to determine one or more search results associated with input data. Search engine 142 may perform item-based search, tag-based search, Boolean-based search, image search, embedding-based search (e.g., nearest neighbor search), multimodal search, and / or one or more other search techniques.
[0183] Server computing system 130 may store and / or provide one or more user interfaces 144 for obtaining input data and / or providing output data to one or more users. The one or more user interfaces 144 may include one or more user interface elements, which may include input fields, navigation tools, content chips, selectable tiles, widgets, data display carousels, dynamic animations, information pop-ups, image enhancement, text-to-speech, speech-to-text, augmented reality, virtual reality, feedback loops, and / or other interface elements.
[0184] User computing system 102 and / or server computing system 130 may be trained via interactive training models 120 and / or 140 with third-party computing system 150 communicatively coupled via network 180. Third-party computing system 150 may be separate from server computing system 130 or may be part of server computing system 130. Alternatively and / or additionally, third-party computing system 150 may be associated with one or more web resources, one or more web platforms, one or more other users, and / or one or more contexts.
[0185] Examples of machine learning models may include generative models (e.g., large language models, base models, visual language models, image generation models, text-to-image models, audio generation models, and / or other generative models).
[0186] Training and / or tuning a machine learning model may include obtaining training instances. The set of training data may include multiple training instances partitioned across multiple datasets (e.g., training datasets, validation datasets, or test datasets). Training instances may be labeled or unlabeled. When training a model using an evaluation of its performance on a runtime instance (e.g., online training / learning), runtime inference may form that training instance. Example data types for training instances and various tasks associated with them are described throughout this disclosure.
[0187] Training and / or tuning may include using one or more machine learning models to process training instances to generate output. The output may be obtained directly from one or more machine learning models, or it may be a downstream result of a processing operation chain, which includes the output of one or more machine learning models.
[0188] Training and / or tuning may include receiving an evaluation signal associated with the output. A loss function can be used to obtain the evaluation signal. Various determinations of the loss can be used, such as mean squared error, likelihood loss, cross-entropy loss, hinge loss, contrastive loss, or various other loss functions. The evaluation signal can be computed using known baseline true labels (e.g., supervised learning), predicted or estimated labels (e.g., semi-supervised or self-supervised learning), or without labels (e.g., unsupervised learning). The evaluation signal can be a reward (e.g., for reinforcement learning). The reward can be computed using a machine learning reward model configured to generate a reward based on the received output. The reward can also be computed using feedback data describing human feedback to the output.
[0189] Training and / or tuning may include updating the machine learning model using an evaluation signal. For example, in some embodiments, various training or learning techniques (such as, for example, backpropagation) may be used to learn the values of the parameters of the machine learning model. For example, the evaluation signal may be backpropagated from the output (or another source of the evaluation signal) through the machine learning model to update one or more parameters of the model (e.g., based on the gradient of the evaluation signal relative to the parameter values). For example, a system containing one or more machine learning models may be trained in an end-to-end manner. Gradient descent techniques may be used to iteratively update the parameters over multiple training iterations. In some implementations, performing error backpropagation may include performing truncated backpropagation through time. Training and / or tuning may include implementing various generalization techniques (e.g., weight decay, dropout, etc.) to improve the generalization ability of the model being trained.
[0190] In some implementations, the above training loop can be used to train a machine learning model from an initial state to a fully trained state (e.g., when the model exhibits the desired performance profile, such as based on accuracy, precision, recall, etc.).
[0191] In some implementations, the training loop described above can be implemented for specific stages of the training process. For example, in some implementations, the training loop can be implemented for pre-training the machine learning model. Pre-training can include, for example, large-scale training on potentially noisy data to achieve a broad performance level base across multiple tasks / data types. In some implementations, the training loop can be implemented for fine-tuning the machine learning model. Fine-tuning can include, for example, smaller-scale training on higher-quality (e.g., labeled, curated, etc.) data. Fine-tuning can affect all or part of the parameters of the machine learning model. For example, parts of the machine learning model can be "frozen" for certain training stages. For example, parameters associated with the embedding space can be "frozen" during fine-tuning (e.g., to preserve information learned from a broader domain than that present in the fine-tuned dataset). Example fine-tuning methods include reinforcement learning. Reinforcement learning can be based on user feedback on model performance during use.
[0192] The third-party computing system 150 may include one or more processors 152 and memory 154. The one or more processors 152 may be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and may be a single processor or multiple processors operatively connected. The memory 154 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. The memory 154 may store data 156 and instructions 158, which are executed by the processor 152 to operate the third-party computing system 150. In some implementations, the third-party computing system 150 includes one or more server computing devices or is otherwise implemented by said one or more server computing devices.
[0193] Network 180 can be any type of communication network, such as a local area network (e.g., intranet), a wide area network (e.g., the Internet), or some combination thereof, and can include any number of wired or wireless links. Generally, communication over network 180 can be carried via any type of wired and / or wireless connection using a wide variety of communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, Secure HTTP, SSL).
[0194] The machine learning models described in this specification can be used for a variety of tasks, applications, and / or use cases.
[0195] In some implementations, the input to the machine learning model of this disclosure can be image data. The machine learning model can process the image data to generate output. As an example, the machine learning model can process image data to generate image recognition output (e.g., image data identification, latent embedding of image data, encoded representation of image data, hashing of image data, etc.). As another example, the machine learning model can process image data to generate image segmentation output. As another example, the machine learning model can process image data to generate image classification output. As another example, the machine learning model can process image data to generate image data modification output (e.g., image data alteration, etc.). As another example, the machine learning model can process image data to generate encoded image data output (e.g., encoded and / or compressed representation of image data, etc.). As another example, the machine learning model can process image data to generate upgraded image data output. As another example, the machine learning model can process image data to generate predictive output.
[0196] In some implementations, the input to the machine learning model of this disclosure can be text or natural language data. The machine learning model can process the text or natural language data to generate output. As an example, the machine learning model can process natural language data to generate a language-encoded output. As another example, the machine learning model can process text or natural language data to generate a latent text embedding output. As another example, the machine learning model can process text or natural language data to generate a translation output. As another example, the machine learning model can process text or natural language data to generate a classification output. As another example, the machine learning model can process text or natural language data to generate a text segmentation output. As another example, the machine learning model can process text or natural language data to generate a semantic intent output. As another example, the machine learning model can process text or natural language data to generate an upgraded text or natural language output (e.g., text or natural language data of higher quality than the input text or natural language). As another example, the machine learning model can process text or natural language data to generate a predictive output.
[0197] In some implementations, the input to the machine learning model of this disclosure can be speech data. The machine learning model can process the speech data to generate output. As an example, the machine learning model can process speech data to generate speech recognition output. As another example, the machine learning model can process speech data to generate speech translation output. As another example, the machine learning model can process speech data to generate latent embedding output. As another example, the machine learning model can process speech data to generate encoded speech output (e.g., encoded and / or compressed representations of speech data, etc.). As another example, the machine learning model can process speech data to generate upgraded speech output (e.g., speech data of higher quality than the input speech data, etc.). As another example, the machine learning model can process speech data to generate text representation output (e.g., a text representation of the input speech data, etc.). As another example, the machine learning model can process speech data to generate predictive output.
[0198] In some implementations, the input to the machine learning model of this disclosure can be sensor data. The machine learning model can process the sensor data to generate output. As an example, the machine learning model can process sensor data to generate identification output. As another example, the machine learning model can process sensor data to generate prediction output. As another example, the machine learning model can process sensor data to generate classification output. As another example, the machine learning model can process sensor data to generate segmentation output. As another example, the machine learning model can process sensor data to generate segmentation output. As another example, the machine learning model can process sensor data to generate visualization output. As another example, the machine learning model can process sensor data to generate diagnostic output. As another example, the machine learning model can process sensor data to generate detection output.
[0199] In some cases, the input includes visual data, and the task is a computer vision task. In other cases, the input includes pixel data from one or more images, and the task is an image processing task. For example, an image processing task could be image classification, where the output is a set of scores, each score corresponding to a different object class and representing the probability that one or more images depict an object belonging to that object class. An image processing task could be object detection, where the image processing output identifies one or more regions in one or more images, and for each region, identifies the probability that the region depicts an object of interest. As another example, an image processing task could be image segmentation, where the image processing output defines the probability of each category in a predetermined set of categories for each pixel in one or more images. For example, the set of categories could be foreground and background. As another example, the set of categories could be object classes. As another example, an image processing task could be depth estimation, where the image processing output defines a corresponding depth value for each pixel in one or more images. As another example, an image processing task could be motion estimation, where the network input includes multiple images, and the image processing output defines the motion of the scene depicted at that pixel between the images in the network input for each pixel in one of the input images.
[0200] In some implementations, the task can be a generative task, and one or more machine learning models (e.g., 120 and / or 140) can be configured to output content generated based on one or more inputs. For example, the inputs can be, or otherwise represent, data of one or more modalities, which encodes the context used to generate additional content.
[0201] In some implementations, the task can be a text completion task. A machine learning model can be configured to process input representing text data and generate an output representing additional text data following the input text sequence. For example, a machine learning model can be configured to generate output to complete a sentence, paragraph, or section of text following a portion of the text represented by the input.
[0202] In some implementations, the task can be an instruction-following task. A machine learning model can be configured to process inputs representing instructions for performing a function and generate outputs that advance towards a goal satisfying the instruction's function (e.g., at least one step of a multi-step process for performing the function). The output can represent data of the same or different modality as the input. For example, the input can represent text data (e.g., natural language instructions for a task to be performed), and the machine learning model can process the input to generate an output representing text data in response to the instructions (e.g., a natural language response, a programming language response, a machine language response, etc.). The input can represent image data (e.g., image-based instructions for a task to be performed, optionally accompanied by text instructions), and the machine learning model can process the input to generate an output representing text data in response to the instructions (e.g., a natural language response, a programming language response, a machine language response, etc.). One or more outputs can be generated iteratively or recursively to sequentially process and complete steps toward completing the requested function. For example, the initial output can be executed by an external system or processed by the machine learning model to complete the initial steps for performing the function. Multiple steps can be performed to obtain the final output in response to the initial instruction.
[0203] In some implementations, the task can be a question-answering task. A machine learning model can be configured to process input representing a question to be answered and generate an output that advances towards a goal of returning an answer to the question (e.g., at least one step in a multi-step process for performing the function). The output can represent data of the same or different modality as the input. For example, the input can represent text data (e.g., natural language instructions for a task to be performed), and the machine learning model can process the input to generate an output representing text data in response to the question (e.g., a natural language response, a programming language response, a machine language response, etc.). The input can represent image data (e.g., image-based instructions for a task to be performed, optionally accompanied by text instructions), and the machine learning model can process the input to generate an output representing text data in response to the question (e.g., a natural language response, a programming language response, a machine language response, etc.). One or more outputs can be generated iteratively or recursively to sequentially process and complete steps toward answering the question. For example, the initial output can be executed by an external system or processed by the machine learning model to complete the initial steps of obtaining an answer to the question (e.g., querying a database, performing calculations, executing scripts, etc.). Multiple steps can be performed to obtain the final output in response to the problem.
[0204] In some implementations, the task can be an image generation task. A machine learning model can be configured to process input representing context about a desired portion of an image's content. Context can include text data, image data, audio data, etc. The machine learning model can be configured to generate output representing image data depicting the image in context. For example, the machine learning model can be configured to generate pixel data of an image. The values of channels associated with pixels in the pixel data can be selected based on context (e.g., based on probabilities determined according to the context).
[0205] In some implementations, the task can be an audio generation task. A machine learning model can be configured to process input representing context about a desired portion of the audio content. Context can include text data, image data, audio data, etc. The machine learning model can be configured to generate output representing context-dependent audio data. For example, the machine learning model can be configured to generate waveform data in the form of an image (e.g., a spectrogram). The values of channels associated with pixels in the image can be selected based on the context. The machine learning model can be configured to generate waveform data as a sequence of discrete samples of a continuous waveform. The values of the sequence can be selected based on the context (e.g., based on probabilities determined according to the context).
[0206] In some implementations, the task can be a data generation task. A machine learning model can be configured to process inputs representing context about a desired portion of data (e.g., data from various data domains, such as sensor data, image data, multimodal data, statistical data, etc.). For example, the desired data can be synthetic data used to train other machine learning models. The context can include any data type. The machine learning model can be configured to generate outputs representing data aligned with the desired data. For example, the machine learning model can be configured to generate data values to populate a dataset. The values of data objects can be selected based on context (e.g., based on probabilities determined according to the context).
[0207] A user computing system may include multiple applications (e.g., application 1 to application N). Each application may include its own corresponding machine learning library and machine learning model. For example, each application may include a machine learning model. Example applications include text messaging applications, email applications, dictation applications, virtual keyboard applications, browser applications, etc.
[0208] Each application can communicate with multiple other components of the computing device, such as, for example, one or more sensors, a context manager, device state components, and / or additional components. In some implementations, each application can use an API (e.g., a public API) to communicate with each device component. In some implementations, the API used by each application is application-specific.
[0209] User computing system 102 may include multiple applications (e.g., applications 1 to N). Each application communicates with a central intelligence layer. Example applications include text messaging applications, email applications, dictation applications, virtual keyboard applications, browser applications, etc. In some implementations, each application may use an API (e.g., a common API across all applications) to communicate with the central intelligence layer (and the models stored therein).
[0210] The central intelligence layer may include multiple machine learning models. For example, a corresponding machine learning model (e.g., a model) may be provided for each application, and the central intelligence layer manages the respective machine learning models. In other implementations, two or more applications may share a single machine learning model. For example, in some implementations, the central intelligence layer may provide a single model (e.g., a single model) for all applications. In some implementations, the central intelligence layer is included within the operating system of the computing system 100 or otherwise implemented by the operating system.
[0211] The central intelligence layer can communicate with the central device data layer. The central device data layer can be a centralized data repository for the computing system 100. The central device data layer can communicate with multiple other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, the central device data layer can use an API (e.g., a private API) to communicate with each device component.
[0212] Figure 12BA block diagram of an example computing system 50 for alignment classification and image enhancement according to an exemplary embodiment of this disclosure is depicted. Specifically, the example computing system 50 may include one or more computing devices 52, which may be used to acquire and / or generate one or more datasets, which may be processed by a sensor processing system 60 and / or an output determination system 80 to provide feedback to a user, the feedback providing information about features in one or more acquired datasets. The one or more datasets may include image data, text data, audio data, multimodal data, latently encoded data, etc. The one or more datasets may be acquired via one or more sensors associated with one or more computing devices 52 (e.g., one or more sensors within computing devices 52). Additionally and / or alternatively, the one or more datasets may be stored data and / or retrieved data (e.g., data retrieved from web resources). For example, a user may interact with images, text, and / or other content items. Interaction with content items may then be used to generate one or more determinations.
[0213] One or more computing devices 52 may acquire and / or generate one or more datasets based on image capture, sensor tracking, data storage retrieval, content download (e.g., downloading images or other content items from web resources via the Internet), and / or via one or more other technologies. One or more datasets may be processed using a sensor processing system 60. The sensor processing system 60 may employ one or more processing techniques using one or more machine learning models, one or more search engines, and / or one or more other processing techniques. One or more processing techniques may be performed in any combination and / or individually. One or more processing techniques may be performed serially and / or in parallel. In particular, one or more datasets may be processed using a context determination block 62, which may determine the context associated with one or more content items. The context determination block 62 may identify and / or process metadata, user profile data (e.g., preferences, user search history, user browsing history, user purchase history, and / or user input data), previous interaction data, global trend data, location data, time data, and / or other data to determine a specific context associated with a user. Context can be associated with an event, a identified trend, a specific action, a specific type of data, a specific environment, and / or another context related to the user and / or the data retrieved or obtained.
[0214] The sensor processing system 60 may include an image preprocessing block 64. The image preprocessing block 64 may be used to adjust one or more values of the acquired and / or received image to prepare the image for processing by one or more machine learning models and / or one or more search engines 74. The image preprocessing block 64 may resize the image, adjust saturation values, adjust resolution, strip and / or add metadata, and / or perform one or more other operations.
[0215] In some implementations, the sensor processing system 60 may include one or more machine learning models, which may include a detection model 66, a segmentation model 68, a classification model 70, an embedding model 72, and / or one or more other machine learning models. For example, the sensor processing system 60 may include one or more detection models 66, which may be used to detect specific features in a processed dataset. In particular, one or more detection models 66 may be used to process one or more images to generate one or more bounding boxes associated with detected features in one or more images.
[0216] Additionally and / or alternatively, one or more segmentation models 68 may be used to segment one or more portions of a dataset from one or more datasets. For example, one or more segmentation models 68 may utilize one or more segmentation masks (e.g., manually generated and / or generated based on one or more bounding boxes) to segment a portion of an image, a portion of an audio file, and / or a portion of text. Segmentation may include isolating one or more detected objects and / or removing one or more detected objects from an image.
[0217] One or more classification models 70 can be used to process image data, text data, audio data, latently encoded data, multimodal data, and / or other data to generate one or more classifications. One or more classification models 70 may include one or more image classification models, one or more object classification models, one or more text classification models, one or more audio classification models, and / or one or more other classification models. One or more classification models 70 can process data to determine one or more classifications.
[0218] In some implementations, one or more embedding models 72 can be used to process data to generate one or more embeddings. For example, one or more embedding models 72 can be used to process one or more images to generate one or more image embeddings in an embedding space. The one or more image embeddings can be associated with one or more image features of one or more images. In some implementations, one or more embedding models 72 can be configured to process multimodal data to generate multimodal embeddings. The one or more embeddings can be used for classification, search, and / or learning the embedding space distribution.
[0219] The sensor processing system 60 may include one or more search engines 74, which can be used to perform one or more searches. The one or more search engines 74 may crawl one or more databases (e.g., one or more local databases, one or more global databases, one or more private databases, one or more public databases, one or more dedicated databases, and / or one or more general databases) to determine one or more search results. The one or more search engines 74 may perform feature matching, text-based search, embedding-based search (e.g., k-nearest neighbor search), metadata-based search, multimodal search, web resource search, image search, text search, and / or application search.
[0220] Additionally and / or alternatively, the sensor processing system 60 may include one or more multimodal processing blocks 76, which may be used to assist in processing multimodal data. The one or more multimodal processing blocks 76 may include generating multimodal queries and / or multimodal embeddings for processing by one or more machine learning models and / or one or more search engines 74.
[0221] The output determination system 80 can then be used to process the output of the sensor processing system 60 to determine one or more outputs to be provided to the user. The output determination system 80 may include heuristic-based determination, machine learning model-based determination, user-selection-based determination, and / or context-based determination.
[0222] Output determination system 80 can determine how and / or where to provide one or more search results in search results interface 82. Additionally and / or alternatively, output determination system 80 can determine how and / or where to provide one or more machine learning model outputs in machine learning model output interface 84. In some implementations, one or more search results and / or one or more machine learning model outputs can be provided for display via one or more user interface elements. One or more user interface elements can be overlaid on the displayed data. For example, one or more detection indicators can be overlaid on detected objects in the viewfinder. One or more user interface elements can be optional for one or more additional searches and / or one or more additional machine learning model processes. In some implementations, user interface elements can be provided as application-specific user interface elements, and / or can be uniformly provided across different applications. One or more user interface elements can include pop-up displays, interface overlays, interface tiles and / or blocks, carousels, audio feedback, animations, interactive widgets, and / or other user interface elements.
[0223] Additionally and / or alternatively, data associated with the output of sensor processing system 60 can be used to generate and / or provide augmented reality and / or virtual reality experiences 86. For example, one or more acquired datasets can be processed to generate one or more augmented reality rendering assets and / or one or more virtual reality rendering assets, which can then be used to provide augmented reality and / or virtual reality experiences 86 to a user. Augmented reality experiences can render information associated with the environment into the corresponding environment. Alternatively and / or additionally, objects associated with the processed dataset can be rendered into the user environment and / or virtual environment. Rendering dataset generation can include training one or more neural radiation field models to learn a three-dimensional representation of one or more objects.
[0224] In some implementations, one or more action prompts 88 can be determined based on the output of the sensor processing system 60. For example, search prompts, purchase prompts, generation prompts, reservation prompts, call prompts, redirection prompts, and / or one or more other prompts can be determined to be associated with the output of the sensor processing system 60. The one or more action prompts 88 can then be provided to the user via one or more selectable user interface elements. In response to the selection of one or more selectable user interface elements, a corresponding action can be performed for the corresponding action prompt (e.g., a search can be performed, a purchase application programming interface can be used, and / or another application can be opened).
[0225] In some implementations, one or more generative models 90 may be used to process one or more datasets and / or the output of the sensor processing system 60 to generate model-generated content items, which can then be provided to a user. The generation may be based on user selection and / or may be automated (e.g., based on one or more criteria, which may be associated with a threshold amount of unrecognized search results).
[0226] One or more generative models 90 may include language models (e.g., large language models and / or visual language models), image generation models (e.g., text-to-image generation models and / or image augmentation models), audio generation models, video generation models, graphics generation models, and / or other data generation models (e.g., other content generation models). One or more generative models 90 may include one or more transformer models, one or more convolutional neural networks, one or more recurrent neural networks, one or more feedforward neural networks, one or more generative adversarial networks, one or more self-attention models, one or more embedding models, one or more encoders, one or more decoders, and / or one or more other models. In some implementations, one or more generative models 90 may include one or more autoregressive models (e.g., machine learning models trained to generate predicted values based on previous behavioral data) and / or one or more diffusion models (e.g., machine learning models trained to generate predicted data based on generating and processing distributed data associated with the input data).
[0227] One or more generative models 90 can be trained to process input data and generate model-generated content items, which may include multiple predicted words, pixels, signals, and / or other data. Model-generated content items may include novel content items that differ from any pre-existing work. One or more generative models 90 may utilize learned representations, sequences, and / or probability distributions to generate content items, which may include phrases, storylines, settings, objects, characters, beats, lyrics, and / or other aspects not included in existing content items.
[0228] One or more generative models 90 may include visual language models.
[0229] Visual language models can be trained, tuned, and / or configured to process image and / or text data to generate natural language output. Visual language models can leverage pre-trained large language models (e.g., large autoregressive language models) and one or more encoders (e.g., one or more image encoders and / or one or more text encoders) to provide detailed natural language output that simulates human-written natural language.
[0230] Visual language models can be used for zero-shot image classification, few-shot image classification, adding descriptive text to images, multimodal query distillation, multimodal question answering, and / or can be tuned and / or trained for multiple different tasks. Visual language models can perform visual question answering, image descriptive text generation, feature detection (e.g., content detection (e.g., for inappropriate content)), object detection, scene recognition, and / or other tasks.
[0231] Visual language models can leverage pre-trained language models, which can then be tuned to achieve multimodality. Training and / or tuning of a visual language model can include image-text matching, masked language modeling, multimodal fusion with cross-attention, contrastive learning, prefix language model training, and / or other training techniques. For example, a visual language model can be trained to process images to generate predicted text similar to benchmark real-world text data (e.g., benchmark real-world captions for an image). In some implementations, a visual language model can be trained to replace masked terms of a natural language template with text terms describing features depicted in an input image. Alternatively and / or additionally, training, tuning, and / or model inference can include multi-layered connections of visual and text embedding features. In some implementations, a visual language model can be trained and / or tuned via jointly learning image embeddings and text embedding generation, which can include training and / or tuning a system to map embeddings to a joint feature embedding space that maps text features and image features to a shared embedding space. Joint training can include parallel embedding of image-text pairs and / or can include training with triples. In some implementations, images can be used as and / or processed as prefixes for the language model.
[0232] One or more generative models 90 may be stored on the device and / or on a server computing system. In some implementations, one or more generative models 90 may be processed on-device to determine suggested searches, suggested actions, and / or suggested prompts. One or more generative models 90 may include one or more compact visual language models, which may include fewer parameters than visual language models stored and operated by the server computing system. The compact visual language models may be trained via distillation training. In some implementations, the visual language models may process display data to generate suggestions. Display data may include a single image describing a screenshot and / or may include image data, metadata, and / or other data describing a time period prior to the currently displayed content (e.g., apps, images, videos, messages, and / or other content viewed in the past 30 seconds). The user computing device may generate and store a scrolling buffer window (e.g., 30 seconds) describing data of the content displayed during the buffering period. Data may be deleted when the time has elapsed. The scrolling buffer window data may be used to determine context, which may be used for query, content, action, and / or prompt suggestions.
[0233] In some implementations, the generative model 90 may include a machine learning sequence processing model. An example system may pass input to the sequence processing model. The sequence processing model may include one or more machine learning components. The sequence processing model may process the data from the input to obtain an input sequence. The input sequence may include one or more input elements obtained from the input. The sequence processing model may use a prediction layer to process the input sequence to generate an output sequence. The output sequence may include one or more output elements generated based on the input sequence. The system may generate output based on the output sequence.
[0234] Sequence processing models can include one or more machine learning model components configured to ingest, generate, or otherwise infer sequences of information. For example, some example sequence processing models in the text domain are referred to as “large language models” or LLMs. See, for example, the PaLM 2 technology report, Google, https: / / ai.google / static / documents / palm2techreport.pdf (nd). Other example sequence processing models can operate in other domains, such as the image domain, for example, see Dosovitskiy et al. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (An image is equivalent to 16x16 characters: large-scale image) (like a recognition transducer) arXiv:2010.11929v2 (June 3, 2021); audio domain, for example, see Agostinelli et al. MusicLM: Generating Music From TextarXiv:2301.11325v1 (January 26, 2023); Biochemical domain, for example, see Jumper et al., Highly accurate protein structure prediction with AlphaFold, 596 Nature 583 (August 26, 2021). Sequence processing models can process one or more types of data simultaneously. Sequence processing models can include relatively large models (e.g., more parameters, computationally intensive, etc.), relatively small models (e.g., fewer parameters, computationally lightweight, etc.), or both.
[0235] Generally, sequence processing models can use data from the input to obtain an input sequence. For example, the input sequence may include a representation of data from input 2 in a format understood by the sequence processing model. One or more machine learning components of the sequence processing model may ingest data from the input, parse the data into fragments compatible with the processing architecture of the sequence processing model (e.g., via "lexicalization"), and project the fragments into the input space associated with the prediction layer (e.g., via "embedding").
[0236] Sequence processing models can ingest data from input and parse the data into a sequence of elements to obtain an input sequence. For example, a portion of the input data can be broken down into segments that collectively represent the content of that portion of the input data. These segments can provide the elements of the sequence.
[0237] In some implementations, processing the input data may include lexicalization. For example, a tokenizer may process a given portion of the input source and output a series of tokens representing said portion of the input source (e.g., corresponding to input elements). Various methods can be used for lexicalization. For example, byte-pair encoding (BPE) techniques can be used to lexicalize a text input source. See, for example, Kudo et al. SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing (SentencePiece: Simple and language-independent word segmenters and desegmenters for neural text processing Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (System Demonstrations), pp. 66–71 (October 31–November 4, 2018), https: / / aclanthology.org / D18-2012.pdf. Lexicalization of image-based input sources can be achieved by extracting blocks from images and serializing them.
[0238] Generally, any data type can be serialized and processed into an input sequence.
[0239] A prediction layer can predict one or more output elements based on input elements. The prediction layer may include one or more layers with learned parameters, which manipulate and transform the input to extract higher-order meanings from the input elements and relationships between them. In this way, for example, an example prediction layer can predict new output elements based on the context provided by the input sequence.
[0240] The prediction layer can evaluate the associations between parts of the input sequence and specific output elements. These associations can inform the prediction of the likelihood that a particular output follows the input context. For example, consider the text snippet “The carpenter's toolbox was small and heavy. It was full of ___.” An example prediction layer can identify that “It” refers back to “toolbox” by determining the relationship between the corresponding embeddings. The example prediction layer can also associate “it” with attributes of the toolbox, such as “small” and “heavy”. Based on these associations, for example, the prediction layer can assign a higher probability to the word “nails” than to the word “sawdust”.
[0241] Transformers are example architectures that can be used in the prediction layer. See, for example, Vaswani et al. Attention Is All You Need? arXiv:1706.03762v7 (August 2, 2023). A transformer is an example of a machine learning model architecture that uses an attention mechanism to compute associations between items within a context window. The context window may include a sequence containing an input sequence and potentially one or more output elements. A transformer block may include one or more attention layers and one or more post-attention layers (e.g., feedforward layers, such as multilayer perceptrons).
[0242] The prediction layer can include other machine learning model architectures attached to or replacing the transformer-based architecture. For example, recurrent neural networks (RNNs) and long short-term memory (LSTM) models, as well as convolutional neural networks (CNNs), can also be used. In general, the prediction layer can utilize various artificial neural networks that can understand or generate sequences of information.
[0243] The output sequence may include or otherwise represent the same or different data types as the input sequence. For example, the input sequence may represent text data, and the output sequence may represent text data. The input sequence may represent image, audio, or audiovisual data, and the output sequence may represent text data (e.g., describing image, audio, or audiovisual data). It should be understood that the prediction layer and any other gap model components of the sequence processing model can be configured to receive multiple data types in the input sequence and output multiple data types in the output sequence.
[0244] The output sequence can have various relationships with the input sequence. The output sequence can be a continuation of the input sequence. The output sequence can be a supplement to the input sequence. The output sequence can translate, transform, expand, or otherwise modify the input sequence. The output sequence can answer, evaluate, acknowledge, or otherwise respond to the input sequence. The output sequence can implement instructions provided via the input sequence (or describe instructions for implementing said instructions).
[0245] The output sequence can be generated autoregressively. For example, in some applications, the output of one or more prediction layers can be passed through one or more output layers (e.g., softmax layers) to obtain a probability distribution of an output vocabulary (e.g., a vocabulary of text or symbols) conditioned on the set of input elements in a context window. In this way, for example, the output sequence can be generated autoregressively by sampling a possible next output element, adding that element to the context window, regenerating the probability distribution based on the updated context window, and sampling a possible next output element, etc.
[0246] The output sequence can also be generated non-autoregressively. For example, multiple output elements of the output sequence can be predicted together without explicit order conditions. See, for example, Saharia et al., Non-Autoregressive Machine Translation with Latent Alignments, arXiv:2004.07437v3 (November 16, 2020).
[0247] The output sequence may include one or more parts or elements. In the example content generation configuration, the output sequence may include multiple elements corresponding to multiple parts of the generated output sequence (e.g., text sentences, values of discrete waveforms, computer code, etc.). In the example classification configuration, the output sequence may include a single element associated with the classification output. For example, the output "vocabulary" may include the set of classes to which the input sequence will be classified. For example, a vision transformer block may pass latent state information to a multilayer perceptron, which outputs possible class values associated with the input image.
[0248] The output determination system 80 can utilize the data augmentation block 92 to process one or more datasets and / or the output of the sensor processing system 60 to generate augmented data. For example, the data augmentation block 92 can be used to process one or more images to generate one or more augmented images. Data augmentation can include data correction, data cropping, removal of one or more features, addition of one or more features, resolution adjustment, lighting adjustment, saturation adjustment, and / or other enhancements.
[0249] In some implementations, one or more datasets and / or the output of the sensor processing system 60 may be stored based on data storage block 94.
[0250] The output of the output determination system 80 can then be provided to the user via one or more output components of the user computing device 52. For example, one or more user interface elements associated with one or more outputs can be provided for display via a visual display of the user computing device 52.
[0251] The process can be performed iteratively and / or continuously. One or more user inputs to the provided user interface elements can modulate and / or affect the continuous processing loop.
[0252] This paper discusses technologies related to servers, databases, software applications, and other computer-based systems, as well as the actions taken and the information sent to and from such systems. The inherent flexibility of computer-based systems allows for a wide variety of possible configurations, combinations, and partitions of tasks and functionality between and within components. For example, the processes discussed in this paper can be implemented using a single device or component, or multiple devices or components working in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.
[0253] While various specific example embodiments of this subject matter have been described in detail, each example is provided by way of explanation and not limitation. Modifications, variations, and equivalents of such embodiments will be readily apparent to those skilled in the art upon acquiring an understanding of the foregoing. Therefore, this disclosure does not exclude such modifications, variations, and / or additions to the subject matter as will be apparent to those skilled in the art. For example, features shown or described as part of an embodiment may be used with another embodiment to produce yet another embodiment. Therefore, this disclosure is intended to cover such modifications, variations, and equivalents.
Claims
1. A computational system for text alignment classification, the system comprising: One or more processors; as well as One or more non-transitory computer-readable media, the one or more non-transitory computer-readable media collectively storing instructions that, when executed by the one or more processors, cause the computing system to operate, the operations including: Obtain an image that includes text, wherein the text is in a first language; Obtain a text translation request associated with the translation from the first language to the second language; The image is processed to generate an image representation; The image is processed using a paragraph detection model to generate one or more paragraph bounding boxes, wherein the one or more paragraph bounding boxes describe the location of one or more paragraphs within the image; The image representation and the one or more paragraph bounding boxes are processed using a machine learning alignment classification model to generate one or more alignment classifications associated with the one or more paragraphs, wherein the one or more alignment classifications describe the predicted text alignment of the one or more paragraphs; Obtain the translated text, wherein the translated text describes the text in the second language; and An enhanced image is generated based on the image, the one or more alignment classifications, and the translated text, wherein the enhanced image includes the image and the text is replaced with the translated text having text alignments determined based on the one or more alignment classifications.
2. The system of claim 1, wherein processing the image using the paragraph detection model to generate the one or more paragraph bounding boxes comprises: Multiple paragraph bounding boxes are generated by processing the image representation using the paragraph detection model, wherein multiple predicted paragraph locations are associated with multiple different paragraphs within the image; and The use of a machine learning-aligned classification model to process the image representation and the one or more paragraph bounding boxes includes: Multiple alignment classifications associated with the multiple different paragraphs are generated by processing the image representation and the multiple paragraph bounding boxes using the machine learning-based alignment classification model.
3. The system of claim 2, wherein generating the enhanced image based on the image, the one or more alignment classifications, and the translated text comprises: Multiple feature anchor points are determined based on the multiple paragraph bounding boxes and the multiple alignment classifications; as well as The translated text is rendered based on the multiple feature anchors and the multiple alignment classifications.
4. The system of claim 3, wherein the plurality of feature anchors describe image features within the image to be associated with positioning for rendering a particular paragraph.
5. The system of claim 4, wherein the operation further comprises: Obtain the second image; The second image is determined to include the image features; as well as A second enhanced image is generated based on the plurality of feature anchors, the translated text, and the plurality of alignment classifications, wherein the second enhanced image includes the second image, and the text in the first language is replaced with the translated text in the second language.
6. The system of any preceding claim, wherein processing the image representation and the one or more paragraph bounding boxes using the machine learning-based alignment classification model to generate the one or more alignment classifications associated with the one or more paragraphs comprises: Based on the one or more paragraph bounding boxes, region-of-interest pooling is performed on the visual features of the image representation to generate one or more pooled visual embeddings. Convolutional blocks are used to process the one or more pooled visual embeddings to predict the shape features of the one or more paragraphs; One or more box positioning embeddings are generated based on the one or more paragraph bounding boxes and the shape features; as well as Process the one or more pooled visual embeddings and the one or more box positioning embeddings to generate the one or more alignment classifications.
7. The system of any of the preceding claims, wherein the one or more alignment categories include at least one of right alignment, left alignment, justified alignment, top alignment, bottom alignment, or center alignment.
8. The system of any of the preceding claims, wherein the machine learning aligned classification model is trained on a training dataset comprising multiple sample image representations, multiple benchmark ground truth bounding boxes, and multiple benchmark ground truth aligned labels.
9. The system of claim 8, wherein the plurality of sample images are associated with a plurality of training images, each of the plurality of training images includes corresponding text, and wherein the plurality of training images include text in a plurality of different languages.
10. The system of claim 8, wherein the plurality of sample images are associated with a plurality of training images, each of the plurality of training images includes corresponding text, and wherein the plurality of training images include text with a plurality of different alignments, wherein the plurality of different alignments includes left alignment, right alignment, and justified alignment.
11. A computer-implemented method for an augmented reality translation interface, the method comprising: Image data describing an image is obtained from one or more image sensors of a user computing device by a computing system including one or more processors, wherein the image includes text in a first language; The computing system obtains text translation requests associated with translation from the first language to the second language; The computing system processes the image using a paragraph detection model to generate one or more paragraph bounding boxes, wherein the one or more paragraph bounding boxes describe the location of one or more paragraphs within the image; The computing system processes the image data and the one or more paragraph bounding boxes using a machine learning alignment classification model to generate one or more paragraph localization predictions and one or more alignment classifications associated with the one or more paragraphs, wherein the one or more alignment classifications describe the predicted text alignment of the one or more paragraphs; The computing system obtains translated text, wherein the translated text describes the text in the second language; as well as The computing system generates an enhanced image based on the image, the one or more paragraph position predictions, the one or more alignment classifications, and the translated text, wherein the enhanced image includes the image, and the text is replaced with the translated text having text alignment determined based on the one or more alignment classifications and specific text positioning based on the one or more paragraph positioning predictions.
12. The method of claim 11, further comprising: The computing system determines one or more feature anchors based on the one or more paragraph position predictions and the one or more alignment classifications.
13. The method of any of the preceding claims, wherein the image data includes a compressed version of the image.
14. The method of any preceding claim, wherein obtaining the translated text by the computing system comprises: The computing system processes the image using an optical character recognition model to generate text data describing the text in the first language; as well as The computing system uses a translation model to process the text data to generate the translated text.
15. The method of claim 14, wherein processing the text data by the computing system using the translation model to generate the translated text comprises: The computing system processes the text data to determine if the text is written in the first language; as well as In response to determining that the text is written in the first language, the translation model is obtained by the computing system.
16. The method of claim 15, wherein obtaining the translation model by the computing system comprises: The computing system determines the language preference of a specific user, wherein the language preference includes the second language; as well as The computing system obtains the translation model from the translation model database based on the translation request from the first language to the second language.
17. One or more non-transitory computer-readable media, the one or more non-transitory computer-readable media collectively storing instructions, the instructions causing the one or more computing devices to operate when executed by the one or more computing devices, the operations including: Obtain training data, wherein the training data includes sample image representations, reference ground truth bounding boxes, and reference ground truth alignment labels, wherein the sample image representations describe a specific image, wherein the reference ground truth bounding boxes describe the reference ground truth positioning of a paragraph within the specific image, and wherein the reference ground truth alignment labels describe the reference ground truth text alignment of the paragraph within the specific image; The sample image representation and the baseline ground truth bounding box are processed using an alignment classification model to generate a predicted alignment classification, wherein the predicted alignment classification describes the predicted alignment of the paragraph. An evaluation loss function is used to assess the difference between the predicted aligned classification and the baseline true aligned label; as well as One or more parameters of the alignment classification model are adjusted, at least in part, based on the loss function.
18. The one or more non-transitory computer-readable media of claim 17, wherein the operation further comprises: The image representation is processed using a paragraph detection model to generate predicted paragraph bounding boxes, wherein the predicted paragraph bounding boxes describe the predicted localization of the paragraph. A second loss function is evaluated, which assesses the difference between the predicted segment bounding box and the baseline true bounding box; as well as One or more second parameters of the alignment classification model are adjusted, at least in part, based on the second loss function.
19. One or more non-transitory computer-readable media as claimed in any of the preceding claims, wherein the reference true bounding box and the reference true alignment label are generated based on user input associated with the particular image.
20. One or more non-transitory computer-readable media as claimed in any of the preceding claims, wherein the loss function includes cross-entropy loss.
