Loyalty generation of output text for multi-modal applications

Abstract

Systems and techniques are described for generating and using a hallucination mitigating single-modal / multi-modal generative model. For example, a computing device may encode input data to generate an encoded representation of the input data. The computing device may obtain intermediate data including a plurality of partial statements associated with the input data, and may generate at least one complete statement associated with the input data based on the intermediate data. The computing device may encode the at least one complete statement to generate at least one encoded representation of the at least one complete statement. The computing device may generate a loyalty score based on a comparison of the encoded representation of the input data to the at least one encoded representation of the at least one complete statement. The computing device may reorder the plurality of partial statements of the intermediate data based on the loyalty score to generate reordered data.

Description

TECHNICAL FIELD

[0001] The present disclosure generally relates to generative models. For example, aspects of the present disclosure relate to systems and techniques for generating and using hallucination mitigating unimodal or multimodal generative models, or situations in which a generative model becomes convinced of an untrue fact associated with input data and generates text based on the untrue fact. BACKGROUND

[0002] Machine learning models (e.g., deep learning models such as neural networks) can be used to perform a variety of tasks, including depth estimation, detection and / or recognition (e.g., scene or object detection and / or recognition, speech recognition), pose estimation, image reconstruction, classification, three-dimensional (3D) modeling, dense regression tasks, data compression and / or decompression, image processing, etc. Machine learning models can be general purpose and can achieve high quality results in a variety of tasks.

[0003] Multimodal generative models tend to generate output text that is not faithful to the input context. For audio captions, a generative machine learning model receives audio as input and generates a related caption verbatim. For example, an audio signal can include a person stepping over leaves and then speaking at a slow pace on a sidewalk. A multimodal generative model can generate a caption for the audio that includes a hallucination. For example, the caption for the audio can be: “A person steps over some leaves and then stops to chop down some bushes.” The hallucination involves the caption portion about the person chopping down some bushes, as the action is not actually represented in the audio. SUMMARY

[0004] Systems and techniques are described herein for generating output text based on input content that can be unimodal or multimodal. For example, the systems and techniques can use a multimodal faithful decoder that can act as a guide in the sense of mitigating hallucinations in captions generated based on multimodal input or unimodal input.

[0005] According to some aspects, an apparatus for generating output text from input data is provided. The apparatus includes: one or more memories configured to store input data; and one or more processors coupled to the one or more memories and configured to: encode the input data to generate an encoded representation of the input data; obtain intermediate data comprising a plurality of partial sentences associated with the input data; generate at least one complete sentence associated with the input data based on the intermediate data; encode the at least one complete sentence to generate at least one encoded representation of the at least one complete sentence; generate a fidelity score based on a comparison of the encoded representation of the input data with the at least one encoded representation of the at least one complete sentence; and reorder the plurality of partial sentences of the intermediate data based on the fidelity score to generate reordered data.

[0006] In some aspects, a method for generating output text from input data is provided. The method includes: encoding the input data to generate an encoded representation of the input data; obtaining intermediate data comprising multiple partial sentences associated with the input data; generating at least one complete sentence associated with the input data based on the intermediate data; encoding the at least one complete sentence to generate at least one encoded representation of the at least one complete sentence; generating a fidelity score via a fidelity-guided engine based on a comparison between the encoded representation of the input data and at least one encoded representation of the at least one complete sentence; and reordering the multiple partial sentences of the intermediate data based on the fidelity score to generate reordered data.

[0007] In some aspects, a non-transitory computer-readable medium is provided, on which instructions are stored, which, when executed by one or more processors, cause one or more processors to be configured to: encode input data to generate an encoded representation of the input data; obtain intermediate data comprising a plurality of partial sentences associated with the input data; generate at least one complete sentence associated with the input data based on the intermediate data; encode the at least one complete sentence to generate at least one encoded representation of the at least one complete sentence; generate a fidelity score based on a comparison between the encoded representation of the input data and at least one encoded representation of the at least one complete sentence; and reorder the plurality of partial sentences of the intermediate data based on the fidelity score to generate reordered data.

[0008] In some aspects, an apparatus is provided, comprising: means for encoding input data to generate an encoded representation of the input data; means for obtaining intermediate data including a plurality of partial statements associated with the input data; means for generating at least one complete statement associated with the input data based on the intermediate data; means for encoding the at least one complete statement to generate at least one encoded representation of the at least one complete statement; means for generating a fidelity score via a fidelity guidance engine based on a comparison between the encoded representation of the input data and the at least one encoded representation of the at least one complete statement; and means for reordering the plurality of partial statements of the intermediate data based on the fidelity score to generate reordered data.

[0009] In some aspects, one or more of the devices described herein are, are a part of, and / or include: extended reality (XR) devices or systems (e.g., virtual reality (VR) devices, augmented reality (AR) devices, or mixed reality (MR) devices), mobile devices or wireless communication devices (e.g., mobile phones or other mobile devices), wearable devices (e.g., connected watches or other wearable devices), cameras, personal computers, laptop computers, vehicles or computing devices or components of vehicles, server computers or server devices (e.g., edge or cloud-based servers, personal computers acting as server devices, mobile devices such as mobile phones acting as server devices, XR devices acting as server devices, vehicles acting as server devices, network routers, or other devices acting as server devices), another device, or combinations thereof. In some aspects, the device may include one or more cameras for capturing one or more images. In some aspects, the device may include one or more displays for displaying one or more images, notifications, and / or other displayable data. In some aspects, the device may include one or more sensors (e.g., one or more inertial measurement units (IMUs), such as one or more gyroscopes, one or more gyroscopes or gyroscope testers, one or more accelerometers, any combination thereof, and / or other sensors).

[0010] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to define the scope of the claimed subject matter. This subject matter should be understood with reference to the appropriate portions of the entire specification, any or all drawings, and each claim.

[0011] The foregoing and other features and aspects will become more apparent from the following description, claims and accompanying drawings. Attached Figure Description

[0012] The exemplary aspects of this application are described in detail below with reference to the following figures:

[0013] Figure 1A This is a conceptual diagram illustrating natural language processing (NLP) system techniques based on some examples;

[0014] Figure 1B These are conceptual diagrams illustrating single-modal or multi-modal system technologies based on some examples;

[0015] Figure 2 This is a concept diagram illustrating examples of hallucinations in a chatbot using Natural Language Generation (NLG) based on some examples;

[0016] Figure 3A This is a block diagram of a natural language generation (NLG) system based on some examples;

[0017] Figure 3B This is a block diagram of a natural language generation (NLG) system with a decoder based on some examples, which includes a faithful guide (or guardrail) component;

[0018] Figure 4A This is a conceptual diagram of a greedy search decoding algorithm for a natural language generation (NLG) system, based on some examples;

[0019] Figure 4B This is a conceptual diagram of a beam search decoding algorithm for a natural language generation (NLG) system, based on some examples;

[0020] Figure 5 It is a conceptual diagram illustrating histograms of natural language reasoning (NLI) scores, which are examples of output text with and without illusions, representing the implication scores or natural language reasoning (NLI) scores indicating fidelity to the input content.

[0021] Figure 6 It is a block diagram of a decoder that utilizes sampling techniques and faithfully guided components in a natural language generation (NLG) system, based on some examples;

[0022] Figure 7A This is a block diagram of a faithfully guided component in a natural language generation (NLG) system based on some examples;

[0023] Figure 7B It is a block diagram of the component that determines whether to reorder phrases in a natural language generation (NLG) system based on some examples;

[0024] Figure 8A This is a block diagram of a proposed system for generating fidelity scores to reduce illusions in captions describing input data, based on some examples.

[0025] Figure 8B This is a block diagram of a proposed method for reinforcement learning of data captioning, based on some examples;

[0026] Figure 8C This section illustrates attention maps based on some examples.

[0027] Figure 9A This is a block diagram illustrating an example contrastive language-audio pre-trained (CLAP) system based on some examples;

[0028] Figure 9B This is a graph showing the kurtosis measure associated with the tail of a probability distribution based on some examples;

[0029] Figure 10 This is a flowchart illustrating an example process for natural language generation (NLG) based on some aspects;

[0030] Figure 11 This is a block diagram illustrating examples of deep learning networks based on some examples; and

[0031] Figure 12 These are illustrations of example system architectures used to implement certain aspects described in this article, based on some examples. Detailed Implementation

[0032] Certain aspects of this disclosure are provided below. Some of these aspects may be applied independently, and some may be applied in combination, as will be apparent to those skilled in the art. Specific details are set forth in the following description for purposes of explanation to provide a thorough understanding of the various aspects of this application. However, it will be apparent, however, that various aspects may be practiced without these specific details. The accompanying drawings and descriptions are not intended to be limiting.

[0033] The following description provides only exemplary aspects and is not intended to limit the scope, applicability, or configuration of this disclosure. Rather, the following description of the exemplary aspects will provide those skilled in the art with a description that can be used to implement the exemplary aspects. It should be understood that various changes can be made to the function and arrangement of the elements without departing from the scope of this application as set forth in the appended claims.

[0034] As noted above, machine learning systems (e.g., deep neural network systems or models) can be used to perform a variety of tasks, such as, but not limited to, detection and / or recognition (e.g., scene or object detection and / or recognition, face detection and / or recognition, speech recognition, etc.), depth estimation, pose estimation, image reconstruction, classification, 3D modeling, dense regression tasks, data compression and / or decompression, and image processing, etc. Furthermore, machine learning models can be general-purpose and achieve high-quality results across a wide range of tasks.

[0035] In some examples, machine learning systems can be used for natural language processing (NLP) tasks, such as natural language understanding (NLU) and / or natural language generation (NLG). Examples of NLG include systems that use trained machine learning models to generate summaries or captions of articles or other input content, chatbots, autocomplete systems, and so on. In some cases, NLG models can generate text that contains illusions or situations where the NLG model becomes convinced of false facts and generates text or speech based on false facts. For example, when attempting to generate a summary of a news article involving a car accident involving multiple people, an NLG model might produce an illusion, incorrectly stating in the output text that someone died in the accident when no one actually died. An NLG model can also produce an illusion in the captions of an audio signal whose described features are not actually contained in the audio signal.

[0036] Similar to NLG models, multimodal or unimodal models can also produce illusions when attempting to generate output text. Multimodal models, and in some cases unimodal models, can handle one or more different modes or types of input and generate output text that describes or is associated with the input data.

[0037] This paper describes systems, apparatuses (e.g., electronic devices), methods (also referred to as processes), and computer-readable media (collectively referred to herein as "systems and techniques") for generating output text based on input content using natural language generation. In some examples, the systems and techniques are configured to use greedy search, bundle search, or a combination thereof to search for possible tokens (e.g., words or portions thereof) that will be used in the output text, such as taking into account previously generated words in the output text and / or the input content, and to rank these possible tokens based on the probability that the tokens will be used.

[0038] In some aspects, the system and techniques may include a Natural Language Inference (NLI) scoring system. An NLI scoring system can generate an NLI score for a given set of possible tokens to identify how faithfully the tokens are to the input content, such as determining whether using the tokens in the output text results in a true, false, or neutral (e.g., undetermined) statement based on the input content. The system and techniques may reorder the possible tokens based on the NLI score, or otherwise factor the NLI score into the ordering of the possible tokens. The system and techniques may select tokens based on this ordering to generate the output text based on that ordering. By using the NLI scoring system, the system and techniques are configured to mitigate illusions (e.g., the "fact" that the input content is not true in the output text).

[0039] In some cases, the system and technology provide unimodal or multimodal processing. For example, the system may generate multiple tokens (e.g., words or portions thereof) based on input content (e.g., audio, text, unimodal, multimodal, and / or speech). The system may search the multiple tokens to generate a first order of the tokens based on probability. The system may generate scores for the multiple tokens via a fidelity-guided engine to generate a second order of the multiple tokens based on fidelity to the input content (e.g., whether these tokens produce true statements based on the input content). The system may generate an output caption that includes at least one token selected from the multiple tokens based on a reordering of the intermediate bundle.

[0040] In some aspects, systems and techniques can perform faithful generation of captions for multimodal applications. In one exemplary example, an apparatus for generating output text from input data includes: one or more memories configured to store the input data; and one or more processors coupled to the one or more memories and configured to: encode the input data to generate an encoded representation of the input data; obtain intermediate data comprising a plurality of partial sentences associated with the input data; and generate at least one complete sentence associated with the input data based on the intermediate data. The one or more processors coupled to the one or more memories are further configured to: encode at least one complete sentence to generate at least one encoded representation of at least one complete sentence; generate a fidelity score based on a comparison of the encoded representation of the input data with at least one encoded representation of the at least one complete sentence; and reorder the plurality of partial sentences of the intermediate data based on the fidelity score to generate reordered data. The apparatus provides fidelity guardrails or guidelines such that the output text or captions describing the input data are more faithful to the content of the input data.

[0041] While this application relates to both unimodal and multimodal systems, several examples are provided within the context of natural language processing. However, these principles apply to other modalities beyond natural language processing, whether unimodal or multimodal.

[0042] Various aspects of this application will be described with reference to the accompanying drawings.

[0043] Figure 1AThis is a conceptual diagram 100 illustrating Natural Language Processing (NLP) system technology. NLP 102 is useful in various fields, such as the Internet of Things (IoT), wearable devices, cloud computing, Software as a Service, search engines, data queries, or combinations thereof. NLP 102 includes Natural Language Understanding (NLU) 104 and Natural Language Generation (NLG) 106. NLU 104 refers to understanding the meaning of written and / or spoken language (e.g., text, speech, or a combination thereof). Examples of NLU 104 include text reasoning or email classification. NLG 106 refers to the task of generating written and / or spoken language (e.g., text, speech, or a combination thereof) from structured data, unstructured data, or a combination thereof. Examples of NLG 106 include query-centric summary generation, story generation, caption generation, news summary generation, conversational artificial intelligence (AI), or combinations thereof. In some examples, an NLP system may include a combination of NLU 104 and NLG 106, such as question answering, explanation, and subsequent summarization of content (e.g., news articles or stories) or audio or video input, or a combination thereof. In some examples, NLG 106 may include a transformer-based NLG 106.

[0044] Figure 1B A conceptual diagram illustrating the operation of model 110 is shown, which can be a unimodal or multimodal model. One or more input data can be provided to encoder 112. Input data can include text, video, images, audio, and other types of data. Other data can include air gesture data, motion data, doodle data on a touch-sensitive screen, and other data. For example, a user can perform hand movements related to American Sign Language or finger movements related to the movement of fingers on a keyboard projected onto a desktop. Gesture input can include the movement of fingers or a pen on a touch-sensitive display. This disclosure can relate to single-mode (or unimodal) input (any of these types of input) or multimodal input that can be any combination of two or more different input modalities.

[0045] Encoder 112 generates encoded data and sends it to decoder 114. Decoder 114 then generates output data 116. For example, audio can be received as input data by model 110. The audio could be a recording of a person walking through fallen leaves via a microphone. The goal of model 110 is to generate relevant captions for the audio.

[0046] In one example, when the system uses a greedy search technique to process the audio signal, the generated caption (describing the audio signal in text form) could be "A horse is trotting, and a horse is trotting." When using a bundle search technique, the caption could be "A horse is trotting, and a horse is trotting." Applying the fidelity guidance disclosed herein, for example with a weighting of 0.7, the guiding caption could be: "Footsteps tread through fallen leaves, followed by the sound of a microphone." In one example, ground-based captions could include: Original caption_1: A person walks quickly outside and then slows down; Original caption_2: A person walks quickly across a lawn and then stops to cut some weeds or shrubs; Original caption_3: A person quickly treads across the lawn, then stops and parts the grass; Original caption_4: The lawn sways as a person quickly treads across and then stops; Original caption_5: A person is rakeing the lawn. In such examples, the use of fidelity guidance features improves the final captions of the audio signal and removes the illusion of representing concepts not present in the content of the audio signal.

[0047] In another example, the audio signal could present a dog first growling and then barking. A greedy search caption could be generated from the audio input stating, "A person is snoring." The bundle search caption could be the same. However, by using the fidelity guide disclosed herein, again using a weight of 0.7 for illustrative purposes, the guide caption could be: "A dog growls and barks." In such examples, ground truth captions could include: Original caption_1: A person is trying to imitate an angry dog; Original caption_2: A person makes a strange, continuous gurgling sound; Original caption_3: Human breathing echoes before a person imitates a beast's roar; Original caption_4: A muffled breathing sound echoes, followed by a human making a strange animal growl and howl; Original caption_5: A person is trying to imitate an angry dog. As can be seen, captions generated via a fidelity guide engine as disclosed herein will improve the generated captions and reduce hallucinations.

[0048] Figure 2 This is a concept diagram 200 illustrating an example of chat 202 involving a chatbot using Natural Language Generation (NLG). Examples of hallucinations could include an NLG model becoming convinced of untrue facts and generating text or speech based on those untrue facts. Hallucinations can also refer to text that is meaningless or unfaithful to the input content on which the text is based (e.g., audio content, unimodal input, or multimodal input). For example, Figure 2The chatbot in the illustrated chat 202 exhibits hallucination 204, outputting the factually incorrect statement "Yes, I am a person" when answering the query "So you're a person?". The chatbot in chat 202 then exhibits a subsequent hallucination 206, outputting "Nope definitely not a machine, but sometimes it feels like people treat me like one when they ask me questions like that lol" when answering the query "Not a machine?". Hallucinations like hallucinations 204 and 206 can impair system performance and potentially cause security issues, especially when the system is relied upon for critical or sensitive purposes, such as providing accurate medical data, news summaries, driving guidance, or other data that users may rely on for decision-making.

[0049] Another illustrative example of hallucinations can be found in the context of news summary generation. For instance, a news article might describe a car accident involving car A driven by person A and car B driven by person B, in which person B dies. An exemplary summary generated by an NLG system, including the hallucination, reads: “Person A has died investigated by police in Florida after a car crashed into her man car.” This summary includes the hallucination that person A died, when in fact person B died. The summary may also include additional hallucinations in the form of meaningless text, such as “has died investigated by police” or “car crashed into her man car.”

[0050] The systems and techniques described in this article can be used to alleviate such hallucinations. For example, continuing with the above example of a news article, utilizing the systems and techniques can result in an improved summary or caption, "Person A is being investigated by police in Florida after her car crashed into Person B while she was driving," which does not contain any hallucinations.

[0051] Figure 3AThis is a block diagram of a Natural Language Generation (NLG) system 300. As shown, the NLG system 300 receives input text 302 at encoder 304. Encoder 304 may tokenize the input text 302 to divide it into tokens (e.g., words or parts thereof). Tokens may allow system 300 to understand the input text 302, such as through NLU 104. The NLG system 300 also includes a decoder 306. Decoder 306 can generate output text 308 by selecting tokens (e.g., words or parts thereof) to be included in output text 308 from a possible set of tokens (e.g., output from encoder 304). Decoder 306 generates a set of possible tokens for output text 308 and / or selects tokens from that set of possible tokens that can be read from input text 302 based on input text 302 and / or encoder 304. In some examples, taking into account any previously selected tokens and / or the input text 302, the decoder 306 may select the token for the output text 308 from the set of possible tokens based on which token is most likely to appear next.

[0052] Figure 3B This is a block diagram of a unimodal or multimodal system 350 having components as part of a decoder 310 that provides fidelity guidance. Similar to the NLG system 300, the unimodal or multimodal system 350 receives input text 302 at encoder 304. As previously described, encoder 304 may tokenize the input text 302 to divide the input text 302 into tokens (e.g., to understand the input text 302). While examples of tokens based on input text 302 have been described, other data may be processed by system 300 and / or system 350 when the input data is unimodal (such as audio) or multimodal (such as audio and video, video and text, or audio and gestures). The input text 302 may also represent other types of input data.

[0053] The decoder 310 of the single-modal or multi-modal system 350 includes a fidelity guidance engine. The fidelity guidance engine can generate output captions 312 by receiving and processing output data (e.g., tokens or data embedded in other structures) generated by the encoder 304. The aspects of the fidelity guidance engine of the decoder 310 are described below (e.g., Figure 6 Loyalty-boosting engine 608).

[0054] Figure 4A This is a conceptual diagram of a greedy search decoding algorithm 400 for a natural language generation (NLG) system. In some examples, the greedy search decoding algorithm 400 uses the following equation:

[0055] .

[0056] Formula 1: Greedy Search

[0057] Considering the previously generated words (y1,…,yt-1) and the activity report c generated at each step, the greedy search decoding algorithm 400 can select a token (e.g., a word or a part thereof) from the set of possible tokens at each branch based on which word is most likely to be used next. Figure 4A As illustrated, the selected tokens are indicated by the thicker lines between them. Each token (its...) Figure 4A The middle part (of the token) includes the corresponding probability (or confidence value) associated with that token. The greedy search decoding algorithm 400 selects the token with the highest probability (or confidence value) at each level. For example, in... Figure 4A In the example shown, the greedy search decoding algorithm 400 outputs the phrase "The nice woman" based on the fact that "nice" (50% probability) is more likely to follow "The" than "dog" (40% probability) or "car" (10% probability), and based on the fact that "woman" (40% probability) is more likely to follow "nice" than "house" (30% probability) or "guy" (30% probability).

[0058] Figure 4B This is a conceptual diagram of a beam search decoding algorithm 450 for a Natural Language Generation (NLG) system. Considering previously generated words and activity reports, the beam search decoding algorithm 450 explores the N tokens with the highest probabilities at each step and selects the optimal overall statement or phrase (e.g., the statement or phrase with the highest overall probability). For example, considering previously generated words, statements, phrases, and / or activity reports, the beam search decoding algorithm 450 may select the statement or phrase with the highest probability. In an illustrative example, the beam search decoding algorithm 450 may generate several statements or phrases, including “The nice woman,” “The nice guy,” “The dog has,” and “The dog and.” In an illustrative example, the beam search decoding algorithm 450 may select the statement or phrase “The nice woman” because the overall statement or phrase has a higher probability of use than other generated statements or phrases (e.g., “The nice guy,” “The dog has,” and “The dog and”) (e.g., considering previously generated words, statements, phrases, and / or activity reports).

[0059] Advances in pre-trained large language models have significantly improved their performance on conditional language generation tasks, including summarization, despite the consistent illusion. To reduce this illusion, some systems improve bundle search or use fact checkers as a post-processing step. Furthermore, faithful guidance components that also reduce the illusion can be implemented (e.g., the faithful guidance engine in decoder 310).

[0060] The system and techniques described in this paper utilize Natural Language Inference (NLI) entailment metrics to detect and prevent illusions in summary generation. An NLI-assisted bundle reordering mechanism is implemented by computing the entailment probability score between the input context and the bundles generated by the summary generation model during saliency-enhanced greedy decoding. Furthermore, a diversity metric is introduced to compare its effectiveness with that of the fundamental bundle search. For the Xsum and CNN / DM datasets, our proposed algorithm significantly outperforms fundamental bundle decoding in terms of this metric and other metrics.

[0061] Pre-trained sequence-to-sequence transformer models such as BART or Pegasus have shown substantial performance improvements in NLP tasks such as summarization, story generation, and generative question answering. Illusion is a problem that can be observed during the generation process in some cases, especially when pre-training is primarily performed on unlabeled data. During the pre-training phase, the model learns the inaccuracies of the language and the model's syntax, and may generate words irrelevant to the given input during inference time.

[0062] Some systems or techniques can mitigate or suppress hallucinations during decoding by using bundle search modifications that constrain the decoding steps to focus on the tokens supporting the input. In some examples, for NLP-based summary generation, inaccuracies in the summaries provided as training data to the ML model can cause inconsistencies (e.g., hallucinations) in the text generated by the ML model for NLG. In some examples, the relationship between hallucinations and prediction uncertainty can be leveraged by modifying the bundle search to preferentially minimize prediction uncertainty.

[0063] While using heuristics to constrain the bundle search may be slightly effective in mitigating the illusion, it can still (in some examples) benefit from manual checks that initialize the bundle search hyperparameters using complex knowledge of the dataset, task, and model. For example, PINOCCHIO uses cosine distance at each decoding step to measure the consistency of the generated words with the context. As datasets become more generative, relying solely on cosine distance and simple word-level heuristics to control bundle decoding at a factual level may become less effective.

[0064] The NLG system and techniques described in this paper for alleviating hallucinations based on Natural Language Inference (NLI) scoring (e.g., decoders with hallucination relief capabilities, such as...) Figure 3B The decoder 310 with a fidelity-guided engine overcomes the limitations of heuristics and cosine distance by reordering the model's top N predictions using a semantically matching NLP task of Natural Language Inference (NLI). NLG systems and techniques compute NLI entailment scores at each bundle decoding step to provide the model with opportunities to direct bundle trajectories towards regions, tokens, or words with fewer illusions. Greedy unrolling decoding can be used to generate each intermediate bundle while focusing on significant contextual parts. In some examples, the SummaC score metric can be used to order bundles at the sentence-level granularity.

[0065] NLI scoring can be used to detect illusions in generative summarization, as discussed later. Figure 5 The examples and discussions described herein illustrate the NLG system and techniques for mitigating illusions based on Natural Language Inference (NLI) scoring. These include an illusion mitigation component for bundle search that can modify the token-level cumulative bundle probability using NLI metrics or scores, and can compute reordering performance using diversity and summarization consistency (SummaC) score metrics on the Extreme Summary Generation (Xsum) and / or CNN / Daily Mail (DM) datasets.

[0066] NLI scores can be used to measure and / or improve the fidelity of output text to the input content. Fidelity refers to how consistent the generated output text is with respect to the input content. For example, terms, phrases, or statements in the generated output text that are inconsistent at the factual level with respect to the input content can be examples of illusory text. Other types of illusions in the generated output text, such as meaningless text, can also be unfaithful to the input content. NLI scores can be applied to mitigate the illusions of different NLG-based generative summarization generators, such as GPT-fine-tuned Seq2Seq based on recurrent neural networks (RNNs) and Seq2Seq (BertS2S) based on a transformer-based bidirectional encoder representation. In some examples, the Spearman correlation coefficient between textual implication scores and faithful summarization is highest compared to other automatic metrics such as recall-oriented point evaluation substitutes (ROUGE)-1, ROUGE-2, and BertScore (e.g., using a large model of transformer-based bidirectional encoder representation (BERT) fine-tuned on the Multi-Literature Natural Language Inference (MNLI) dataset). Therefore, measuring the NLI score implied by the text can be used to reduce hallucinations.

[0067] To measure fact inconsistency, the trained Fact Consistency Checking Model (FACTCC), the BERT base model, can be fine-tuned on summaries that generate illusions through synthetic means using semantically variable / invariant transformations (such as entity swapping, statement negation, paraphrasing, and noise injection). However, in some examples, such models may lack interpretability and / or have low general applicability to other datasets, for example, being good at detecting certain illusions. Improvements to the loss function components can improve overall fact accuracy. For example, truncating the loss by adaptively removing examples with high log loss can improve fact accuracy in the model.

[0068] Illusions exist in various downstream NLP tasks and can be measured using various metrics. A generative summary may be defined as generating an illusion if it has any textual spans that are semantically unsupported by the input content on which it is based. Illusions can be categorized into two main types—intrinsic and extrinsic. Intrinsic illusions refer to contradictions in the generative summary about the input content. For example, intrinsic illusions may include the use of incorrect pronouns, swapping names and verbs, etc. Models similar to FACTCC (e.g., trained on slight textual transformations) can be used to detect intrinsic illusions. Extrinsic illusions refer to unsupported textual spans present in the generated summary that cannot be validated using only the input content. Extrinsic illusions may arise because they exist in human-written summaries on which the model was trained during training (e.g., overfitting). For example, in Seq2Seq models such as GPT2, the percentage of illusions can be amplified or reduced by modifying the training data.

[0069] Natural Language Inference (NLI) can refer to the task of determining whether a natural language hypothesis can be inferred from given premises. Considering both premises and hypotheses, NLI computes the relationship between them in the form of three probabilities: implication, contradiction, and neutrality. In some examples, NLI algorithms may focus on one, two, or all three probabilities. For instance, in an exemplary example, an NLI system might focus on implication. For example, if the premise is "The sky looks cloudy today," and the hypothesis is "It might rain today," the NLI model will assign a higher probability to implication because the hypothesis implies the premise. Natural Language Inference (NLI) can be used to detect hallucinations.

[0070] Figure 5This is a conceptual graph 500 illustrating histograms of implication scores, or Natural Language Inference (NLI) scores, indicating fidelity to the input content, for output text with and without illusions. Textual implication can be used to detect illusions in generative summarization tasks. Intrinsic illusions can be difficult to detect because their detection may require more than just lexical matching to infer the relevance of a given word to its context.

[0071] The histogram includes histogram 502 of the text implication score for training data with illusions and histogram 504 of the text implication score for training data without illusions. Figure 5 In this context, entity-based hallucinations are counted for analytical purposes. This histogram illustrates experimental results analyzing the correlation between implied scores and entity hallucinations on 2000 randomly selected training samples from the Xsum dataset. Figure 5 It is evident that although low implication scores are frequent for both types of data (those with and without hallucinations), the distinction becomes clearer at higher implication scores. In fact, higher implication scores are correlated with a low probability of entity hallucination. This probability is also reflected in the average implication score in Table 1. The analysis demonstrates that entity-based hallucinations can be detected using NLI metrics. Therefore, introducing NLI into the bundle decoding process can help mitigate hallucinations.

[0072]

[0073] Table 1: Average implication score of Xsum training data on 2000 samples.

[0074] Figure 6 This is a block diagram of the decoder 600 (or decoder system) that uses the fidelity boot engine 608 to improve the output subtitles 618. The fidelity boot engine 608 is... Figure 3BAn example of a fidelity-guided engine for decoder 310 is provided. Decoder 600 introduces an embedding space fidelity score during the decoding process. For example, at the generation step, if a condition is met, decoder 600 can compute a fidelity score for intermediate bundle 612 along with a model probability score to reorder the intermediate bundles and generate a reordered intermediate bundle 614. A unimodal or multimodal encoded representation 602 is input to decoder 600. As shown, decoder 600 includes a transformer block 604 that can identify a set of possible tokens. Sampler 606 (e.g., a bundle search or greedy search algorithm based on using probabilities to order tokens) can be used to generate the intermediate bundle 612 provided to fidelity-guided engine 608. Taking into account the intermediate bundle 612 and the encoded representation 602, fidelity-guided engine 608 can generate the reordered intermediate bundle 614 provided to sampler 606 to generate the final bundle 616 used to generate the output caption 618. A fidelity guidance engine 608 is introduced into the decoding process of sampler 606. At one or more token generation steps (e.g., in some cases at each token generation step), decoder 600 considers the reordering of intermediate bundles 614 from fidelity guidance engine 608 and the prediction scores (intermediate bundles 612) from sampler 606. Further details regarding decoder 600 and fidelity guidance engine 608 are discussed below.

[0075] Figure 7A This is a block diagram 700 illustrating an example of a fidelity guidance engine 608. The encoded representation 602 can be one or more of the following: input image, input audio, input video, text, doodle gestures received on a touch-sensitive display, or gestures made in the air such as by hand or other object. The encoded representation 602 is provided to the fidelity guidance engine 608 and received by the fidelity scorer 702. The intermediate bundle 612 is also received by the fidelity guidance engine 608 at the greedy expansion engine 704 that generates complete statements. Figure 7A Example intermediate bundles 612 are shown, such as "idling buzz" and "oncoming buzz". In one aspect, the statements can be encoded via encoder 706 to generate embeddings, or greedily unfurled intermediate bundles can be generated without encoder 706. The embedded or greedily unfurled intermediate bundles are provided to reordering criterion engine 708. Reordering criterion engine 708 determines whether to reorder intermediate bundle 612. If the output of reordering criterion engine 708 is "yes", fidelity scorer 702 generates a score, which is provided to bundle reorderer 710, which then reorders intermediate bundle 612 to generate reordered intermediate bundle 614. If the output of reordering criterion engine 708 is "no", bundle reorderer 710 does not reorder intermediate bundle 612.

[0076] Figure 7BThis is an example Figure 7A A block diagram 750 shows further details of the reordering criterion engine 708. Greedily unfolded intermediate bundles 612 are received at the reordering criterion engine 708, and phrase extractors 712 extract phrases from the greedily unfolded intermediate bundles 612. Reordering criteria 714 can be applied to determine whether (e.g., yes or no) the bundles should be reordered. For example, if there are N greedily unfolded intermediate bundles and if more than or equal to N / 2 bundles have new tokens as noun or verb phrases within them, the output is "yes," meaning that reordering of the intermediate bundles should occur. If the output is "no," meaning that the N greedily unfolded intermediate bundles with new tokens as noun or verb phrases are less than N / 2, then reordering of the intermediate bundles will not occur. The "yes" or "no" output value is provided to the loyalty scorer 702. In one aspect, if the reordering criterion engine 708 determines that the intermediate bundle 612 should not be reordered, the bundle reorderer 710 does not reorder the bundles.

[0077] In one aspect, the algorithm used to reorder bundles can be based on model confidence, which can be used to determine bundle scores. For example, the model confidence can be based on the following equation: Model confidence = α Entropy + β Kurtosis. The bundle score can be determined as: log((1 + model confidence)) Next word probability + θ Loyalty score), where α, β, and θ are hyperparameters. Bundle score can be a weighted average of bundle probabilities to perform adaptive scoring. Cumulative probability can be equal to cumulative probability + W. Bundle scores. The values ​​of W, α, β, and / or θ can be generated based on examples from the dataset. These equations provide illustrative examples of how the bundle reorderer 710 can reorder intermediate bundles when the reordering criteria are indicated.

[0078] exist Figure 8AIn the diagram, a block diagram of decoder 800 illustrates an example solution for providing a fidelity score 707 from a multimodal input plus a text input. A text cue 802 can be provided to a fidelity-guided engine 608. A video frame 804 representing another input modality is shown. An audio spectrogram 806 is provided as yet another type of input. Each of these inputs can be encoded to generate a different type of embedding. For example, text cue 802 can be encoded by an encoder (not shown) that generates a text embedding 814. The encoder could, for example, be a contrastive language-image pre-trained (CLIP) encoder. Video frame 804 can be downsampled by downsampler 808, which provides the output to feature fusion network 810 to generate an image embedding 816. Image embedding 816 can also be generated from a CLIP encoder. Similarly, audio spectrogram 806 can be processed by a fixed-length audio encoder 812 to generate an audio embedding 818. Audio embedding 818 can be generated by a contrastive language-audio pre-trained (CLAP) encoder. Figure 9A An example CLAP procedure is shown.

[0079] exist Figure 8A In this process, the weighted averaging and normalization process includes receiving text embedding 814, image embedding 816, and audio embedding 818, and generating a weighted average that is provided to the cosine similarity component 822. A middle bundle 612 is provided to a greedy unwrap engine 704 to generate a greedily unwrapped middle bundle. The greedily unwrapped middle bundle (or complete sentence) can be encoded by encoder 824 to generate a second text embedding 826 (e.g., a CLIP embedding from a CLIP encoder such as encoder 824). The second text embedding 826 is provided to the cosine similarity component 822 to generate a fidelity score 707 based on the weighted average and the second text embedding 826. The cosine similarity component 822 can also represent a more general similarity component using methods other than cosine similarity. Alignment of the input embeddings (e.g., one or more of text embedding 814, image embedding 816, and / or audio embedding 818) is used as a method of checking the text embeddings associated with the caption.

[0080] Figure 8BThis is a conceptual diagram 850 illustrating an example of an offline strategy for reinforcement learning used for audio captioning. The process involves providing optimizations for key tokens. For example, captioning model 854 provides generated captions 858 from the original spectrogram 852. Based on the generated captions 858, masking engine 868 masks the generated captions 858 to generate noun and verb phrases or masked captions 870. Key token selector 864 selects key tokens from the generated captions 858 and provides them to conditional text decoder 866. Conditional text decoder 866 also receives masked captions 870. The generated captions 858 are also provided to fluency engine 860, which provides its output to reward function 856. Reward function 856 also receives scores from scoring engine 862. The scores can be CLAP scores as described above. The rewards generated by reward function 856 can be provided to captioning model 854 for training. Scores from scoring engine 862 can also be provided to captioning model 854 as the original spectrogram 852. Any training aspects disclosed herein can be executed in real time on a device.

[0081] In one example, the generated caption 858 could be: “The hum and vibration of an engine, accompanied by whispers, followed by a car driving by.” The generated caption 858 would describe what was found in the original spectrogram 852. The detected nouns and verb phrases could be hum, vibration, engine, people, talking, vehicle, driving by. The masked caption could be: [MASK] [MASK] of [MASK], where [MASK] [MASK] is followed by [MASK] [MASK]. The corrected caption could be: “Car roar, accompanied by people singing, followed by a car driving by.” The bonus score could be a fluency score of 0.8 and a CLAP score of 0.5. These values ​​are for illustrative purposes only. Tests have shown improvements using bundled methods such as the CLAP bundled method disclosed herein.

[0082] Figure 8C An illustration of attention graph 880 is provided, illustrating a method for optimizing key tokens. The input token [CLS] and its layer representation x are shown. 2 The correlation of 1 (e.g., the value R) 2[CLS] can be obtained by summing all possible paths through attention graph 880. Details on how to optimize key tokens can be found in "Measuring the Mixing of Contextual Information in the Transformer, TALP Research Center, Universitat Politecnicade Catalunya" by Ferrando et al., published October 22, 2022 (URL: https: / / arxiv.org / pdf / 2203.04212.pdf), which is incorporated herein by reference. The method measures token-to-token interactions within each transformer layer. The method provides an explanation of which input neurons contribute to a particular output neuron. In one example, the method disclosed in this paper uses aggregation of the layer-wise token-to-token interaction (ALTI) algorithm. The goal is to find which output neurons are most influenced by a particular set of input neurons.

[0083] Figure 9AThis is a conceptual diagram illustrating a CLAP model 900 used to generate corrected captions. CLAP is a pre-training technique for learning a shared embedding space between audio and language. The model can be trained to predict the most relevant text segments while taking audio into account without directly optimizing the task. To compare an audio clip or audio waveform 902 with the corresponding text 920, the audio encoder 906 can encode audio features 904 using, for example, a SwinTransformer and text features using RoBERTa, and project them into the CLAP embedding space. The dot product between the projected audio and text features (e.g., cosine similarity) is used as a similarity score or fidelity score 707. In one example, audio features 904 are provided as a set of audio features 904, which are fed to the audio encoder 906 at a Mel filter bank 908a. The output of the Mel filter bank is processed by a Con2D 910a or a two-dimensional convolutional layer, the output of which is fed to other encoder layers 916. The second set of audio waveforms 902 is provided to a Mel filter bank 908b and processed by a second Conv2D 910b, the output of which is provided to a merging Conv2D 912 and an attention feature fusion layer 914. The output of the attention feature fusion layer 914 can be provided to another encoder layer 916. The output of the encoder layer 916 is received at an MLP (Multilayer Perceptron) layer 918. The corresponding text 920 may include captions 922 and tags 924. Captions 922 can be provided to a text encoder 930, which sends its output to another MLP layer 932. Tags 924 are provided to a keyword-to-caption enhancement component 926, where the output of captions 928 can be provided to the text encoder 930 and the second MLP layer 932. The output of the MLP layer 918 is provided to or compared with the output of the second MLP layer 932 in matrix 934. The data in matrix 934 can be used to determine a similarity score or a fidelity score 707.

[0084] Figure 9B The example illustrates kurtosis 950, which is a measure of the tail of a probability distribution of a real-valued random variable. The x-axis represents the x-values, such as location or time, and the y-axis represents the probability density. Various distributions are shown. The leptokurtic distribution 956 has the highest value but is the narrowest distribution. The mesokurtic distribution is the constant-kurtotic distribution 954. The low-kurtotic distribution 952 has the lowest relative peak but is the widest distribution on the x-axis. The Kurt value is equal to μ⁴ / σ. 4 Where Kurt represents kurtosis, μ4 represents the fourth central moment, and σ 4 It is the standard deviation. Entropy is a measure of the impurities or randomness present in a dataset. Low entropy means that the model is deterministic in choosing the next word. Equation 958 can be applied in some cases as follows:

[0085]

[0086] In existing research on multimodal hallucinations, studies have shown what leads to unfaithful outputs, and previous work has attempted to perform architectural adjustments to improve model performance. This paper discloses a faithful CLAP-guided text generation system for audio-to-text applications. These principles can also be applied to any unimodal or multimodal data or application. The disclosed process can be readily extended to any combination of multimodal data (such as image / video / signal-to-text applications) by learning a shared embedding space (e.g., CLAP). A reinforcement learning framework is also disclosed, which has CLAP scores as a reward function and layer-by-layer updates based on token contributions. Multimodal generative AI systems tend to produce hallucinations or generate false alarms. Generative models may require guidance like the disclosed system and techniques to provide safe and faithful generation. Thus, systems such as ChatGPT, Bard, AudioGPT, and other generative models are available. The system and techniques disclosed in this paper can be utilized.

[0087] In some respects, the machine learning systems or neural networks described in this paper (e.g., decoder 310 such as in Figure 3) Figure 6 Training of one or more of the decoder 600 and various other machine learning networks described herein can be performed using online training, offline training, and / or various combinations of online and offline training. In some cases, online may refer to processing the input data during its processing (e.g., input data 302, such as in Figure 3). Figure 6 The multimodal coding representation (e.g., 602) can be used to alleviate hallucinations, for example, through the systems and techniques described herein. In some examples, offline may refer to a period of idle time or a period during which no input data is processed. Additionally, offline may be based on one or more time conditions (e.g., after a certain amount of time has elapsed, such as a day, a week, a month, etc.) and / or may be based on various other conditions, such as network and / or server availability, and various other conditions.

[0088] Figure 10This is a flowchart illustrating an example process 1000 for faithfully generating subtitles from a single-modal or multimodal input using one or more techniques described herein. In one example, process 1000 may be performed by one or more of the following: decoder 600, fidelity guidance engine 608, greedy unpacking engine 704, reordering criterion engine 708, bundle reorderer 710, fidelity scorer 702, transformer block 604, sampler 606, decoder 800, downsampler 808, feature fusion network 810, fixed-length audio encoder 812, weighted averaging and normalization component 820, cosine similarity component 822, encoder 824, and computation system 1200, or a combination thereof. For example, having Figure 12 The computing device architecture of the computing system 1200 shown can realize the computing device. Figure 10 The operation and / or this article about Figure 1B , Figure 3A , Figure 3B , Figure 6 , Figure 7A , Figure 7B , Figure 8A , Figure 8B , Figure 9A , Figure 11 and / or Figure 12 Any component and / or operation described in the diagram.

[0089] At operation 1002, a decoder system (e.g., decoder 600 or at least one subsystem thereof) can generate output text from input data and is configured to encode the input data (e.g., input audio, video, unimodal data, multimodal data, text, or encoded representation 602) to generate an encoded representation of the input data. In some examples, the input data includes input text (e.g., input text or encoded representation 602), input speech, images, video, gesture data, doodle data on a touch-sensitive display, or combinations thereof. In one aspect, the input data is multimodal data, meaning that there are two or more modalities or types of input. For example, the input data may include two or more of audio data, text data, image data, gesture data, doodle data, and video data. In another aspect, the input data may be unimodal or a single modality or type of input. The input data may be captured by at least one of an image sensor or a microphone.

[0090] In one aspect, the input data may include at least a first type of input data and a second type of input data. In the context of different types of input data, the decoder system (or at least one subsystem thereof) can generate output text from the input data and is configured to: encode the first type of input data to generate an encoded representation of the first type of input data; encode the second type of input data to generate an encoded representation of the second type of input data; and generate a combined representation of the first type of input data and the second type of input data based on the encoded representations of the first type of input data and the second type of input data.

[0091] In one aspect, in order to generate a combined representation of first-type input data and second-type input data, a decoder system (or at least one subsystem thereof) is configured to determine a weighted average of the encoded representations of the first-type input data and the encoded representations of the second-type input data.

[0092] In one aspect, the decoder system (e.g., decoder 600 or at least one subsystem thereof) is configured to normalize a combined representation of first-type and second-type input data. The first-type and second-type input data can be two or more of audio data, text data, image data, and video data.

[0093] In one aspect, the decoder system (e.g., decoder 600 or at least one subsystem thereof) is configured to generate a fidelity score based on a comparison of the combined representation with at least one encoded representation of at least one complete statement.

[0094] At operation 1004, the decoder system (e.g., decoder 600 or at least one subsystem thereof) is configured to and can obtain intermediate data comprising multiple partial statements associated with the input data. In one aspect, the intermediate data may include intermediate bundles generated using a bundle search technique. In another aspect, techniques such as... Figure 11 At least one neural network model is used to generate intermediate data. The at least one neural network model may include a transformer neural network model (e.g., Figure 6 Converter block 604).

[0095] At operation 1006, the decoder system (e.g., decoder 600 or at least one subsystem thereof) is configured to encode at least one complete statement to generate at least one encoded representation of at least one complete statement. The generation of at least one complete statement may be based on intermediate data using a greedy search technique (e.g., via greedy expansion engine 704).

[0096] At operation 1008, the decoder system (e.g., decoder 600 or at least one subsystem thereof) is configured to generate a loyalty score via a loyalty guidance engine (e.g., loyalty guidance engine 608) based on a comparison of the encoded representation of the input data with at least one encoded representation of at least one complete statement.

[0097] At operation 1010, a decoder system (e.g., decoder 600 or at least one subsystem thereof) is configured to and can reorder multiple partial sentences of intermediate data based on fidelity scores (e.g., via bundle reorderer 710) to generate reordered data (e.g., reordered intermediate bundles 614). The reordering of the multiple partial sentences of the intermediate data can be based on fidelity scores and model confidence to generate reordered data (e.g., reordered intermediate bundles). In one aspect, the decoder system can be configured to reorder the multiple partial sentences of the intermediate data based on cumulative probabilities generated by determining bundle scores based on model confidence, fidelity scores, and the probability of the next word in each of the multiple partial sentences. The decoder system can determine cumulative probabilities based on bundle scores and subsequently reorder the multiple partial sentences of the intermediate data based on the cumulative probabilities.

[0098] In one respect, model confidence can be determined based on entropy and kurtosis.

[0099] At operation 1012, the decoder system (e.g., decoder 600 or at least one subsystem thereof) is configured to downsample multiple frames of video data and fuse the coded representations of the multiple frames of video data to generate a fused representation of the video data. The coded representation of the input data may include the fused representation of the video data.

[0100] At operation 1014, the decoder system (e.g., decoder 600 or at least one of its subsystems) is configured to generate output text (e.g., output caption 618) associated with the input data based on the reordered data.

[0101] In one aspect, a non-transitory computer-readable medium has instructions stored thereon that, when executed by one or more processors, cause one or more processors to perform the operations according to any one of operations 1002 to 1014. In another example, an apparatus may include one or more components for performing the operations according to any one of operations 1002 to 1014.

[0102] In some examples, the decoder system includes: components for encoding input data to generate an encoded representation of the input data; components for obtaining intermediate data including multiple partial statements associated with the input data; components for generating at least one complete statement associated with the input data based on the intermediate data; components for encoding at least one complete statement to generate at least one encoded representation of at least one complete statement; components for generating a fidelity score via a fidelity guidance engine based on a comparison between the encoded representation of the input data and at least one encoded representation of at least one complete statement; and components for reordering the multiple partial statements of the intermediate data based on the fidelity score to generate reordered data. Components used to perform these operations may include, for example, one or more of the following: decoder 600, fidelity guidance engine 608, greedy unpacking engine 704, reordering criterion engine 708, bundle reorderer 710, fidelity scorer 702, transformer block 604, sampler 606, decoder 800, downsampler 808, feature fusion network 810, fixed-length audio encoder 812, weighted averaging and normalization component 820, cosine similarity component 822, encoder 824, and computing system 1200, or combinations thereof. For example, having Figure 12 The computing device architecture of the computing system 1200 shown can realize the computing device. Figure 10 The operation and / or this article about Figure 1B , Figure 3A , Figure 3B , Figure 6 , Figure 7A , Figure 7B , Figure 8A , Figure 8B , Figure 9A , Figure 11 and / or Figure 12 Any component and / or operation described in the diagram.

[0103] In some examples, the processes described herein (e.g., process 1000 and / or any other processes described herein) may be executed by a computing device or apparatus. In one example, process 1000 may be executed by a decoder system or decoder 600, fidelity guidance engine 608, greedy unpacking engine 704, reordering standard engine 708, bundle reorderer 710, fidelity scorer 702, transformer block 604, sampler 606, decoder 800, downsampler 808, feature fusion network 810, fixed-length audio encoder 812, weighted averaging and normalization component 820, cosine similarity component 822, encoder 824, computing system 1200, or a combination thereof. For example, having Figure 12 The computing device architecture of the computing system 1200 shown can realize the computing device. Figure 10 The operation and / or this article aboutFigure 1B , Figure 3A , Figure 3B , Figure 6 , Figure 7A , Figure 7B , Figure 8A , Figure 8B , Figure 9A , Figure 11 and / or Figure 12 Any component and / or operation described in the diagram.

[0104] Computing devices may include any suitable device, such as mobile devices (e.g., mobile phones), desktop computing devices, tablet computing devices, XR devices (e.g., VR headsets, AR headsets, AR glasses, etc.), wearable devices (e.g., connected watches or smartwatches or other wearable devices), server computers, vehicles (e.g., autonomous vehicles) or computing devices of vehicles, robotic devices, laptop computers, smart TVs, cameras, and / or any other computing device with the resource capability to perform the processes described herein (including process 1000 and / or any other processes described herein). In some cases, a computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other components configured to perform the steps of the processes described herein. In some examples, a computing device may include a display, a network interface configured to communicate and / or receive data, any combination thereof, and / or other components. The network interface may be configured to communicate and / or receive Internet Protocol (IP) based data or other types of data.

[0105] A component capable of implementing a computing device in a circuit. For example, the component may include electronic circuitry or other electronic hardware, and / or may be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuits (e.g., a microprocessor, graphics processing unit (GPU), digital signal processor (DSP), central processing unit (CPU), and / or other suitable electronic circuitry), and / or may include computer software, firmware, or any combination thereof for performing the various operations described herein, and / or may be implemented using computer software, firmware, or any combination thereof for performing the various operations described herein.

[0106] Process 1000 is illustrated as a logic flowchart, the operations of which represent a sequence of operations that can be implemented in hardware, computer instructions, or combinations thereof. In the context of computer instructions, each operation represents a computer-executable instruction stored on one or more computer-readable storage media that, when executed by one or more processors, performs the described operation. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc., that perform a specific function or implement a specific data type. The order in which the operations are described is not intended to be construed as limiting, and any number of described operations can be combined in any order and / or in parallel to implement the process.

[0107] Additionally, 1000 and / or any other process described herein may be executed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that executes jointly on one or more processors, by hardware, or a combination thereof. As noted above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program comprising multiple instructions executable by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0108] As described in this article, Figure 11 The neural network system 1100 can be implemented using one or more neural networks. Figure 11 It is possible to be Figure 11 The neural network 1100 is an exemplary example of a deep learning neural network 1100. An input layer 1120 includes input data. In one exemplary example, the input layer 1120 may include data representing pixels of an input video frame. The neural network 1100 includes multiple hidden layers 1122a, 1122b through 1122n. Hidden layers 1122a, 1122b through 1122n comprise “n” hidden layers, where “n” is an integer greater than or equal to one. Multiple hidden layers can be made to include as many layers as needed for a given application. The neural network 1100 also includes an output layer 1124, which provides the output produced by the processing performed by the hidden layers 1122a, 1122b through 1122n. In one exemplary example, the output layer 1124 may provide a classification of objects in the input video frame. The classification may include categories that identify the type of object (e.g., person, dog, cat, or other object).

[0109] Neural network 1100 is a multi-layer neural network composed of interconnected nodes. Each node can represent a piece of information. The information associated with these nodes is shared between different layers, and each layer retains information while processing it. In some cases, neural network 1100 may include a feedforward network, in which case there are no feedback connections in which the network's output is fed back into itself. In some cases, neural network 1100 may include a recurrent neural network, which may have loops that allow information to be carried across nodes as input is read.

[0110] Information can be exchanged between nodes via node-to-node interconnects between layers. Nodes in input layer 1120 can activate the node set in the first hidden layer 1122a. For example, as shown, each input node in input layer 1120 is connected to each node in the first hidden layer 1122a. Nodes in hidden layers 1122a, 1122b, to 1122n can transform information by applying an activation function to the information of each input node. The information derived from this transformation can then be passed to nodes in the next hidden layer 1122b, activating those nodes, which can then perform their own specified functions. Example functions include convolution, upsampling, data transformation, and / or any other suitable function. The output of hidden layer 1122b can then activate nodes in the next hidden layer, and so on. Finally, the output of hidden layer 1122n can activate one or more nodes in output layer 1124, providing the output at those nodes. In some cases, although a node in neural network 1100 (e.g., node 1126) is shown as having multiple output lines, the node has a single output and is shown as all lines output from the node representing the same output value.

[0111] In some cases, each node or the interconnection between nodes may have weights, which are a set of parameters derived from the training of the neural network 1100. Once the neural network 1100 is trained, it can be referred to as a trained neural network, which can be used to classify one or more objects. For example, the interconnection between nodes may represent a piece of information learned about the interconnected nodes. The interconnection may have tunable numerical weights that can be tuned (e.g., based on the training dataset), allowing the neural network 1100 to adapt to the input and learn as more and more data is processed.

[0112] The neural network 1100 is pre-trained to process features from the data in the input layer 1120 using different hidden layers 1122a, 1122b to 1122n, in order to provide an output through the output layer 1124. In an example where the neural network 1100 is used to identify objects in an image, the neural network 1100 can be trained using training data that includes both images and labels. For example, training images can be input into the network, where each training image has a label indicating the category of one or more objects in each image (basically, indicating to the network what the objects are and what features they have). In an exemplary example, the training images may include images of the number 2, in which case the label of the image could be [0 0 1 0 0 0 0 0 0 0].

[0113] In some cases, the neural network 1100 can use a training process called backpropagation to adjust the weights of its nodes. Backpropagation includes forward pass, loss function, back pass, and weight update. For each training iteration, forward pass, loss function, back pass, and parameter update are performed. The process can be repeated a certain number of iterations for each training image set until the neural network 1100 is trained well enough that the weights of each layer are accurately tuned.

[0114] For an example of identifying objects in an image, the forward pass may include passing a training image through a neural network 1100. The weights are initially randomized before training the neural network 1100. The image may include, for example, a numerical array representing pixels of the image. Each number in the array may include a value from 0 to 255 describing the intensity of the pixel at that location in the array. In one example, the array may include a 28×28×3 numerical array with 28 rows and 28 columns of pixels and 3 color components (such as red, green, and blue, or lightness and two chroma components, etc.).

[0115] For the first training iteration of the neural network 1100, the output will likely include values ​​due to the weights being randomly selected during initialization without prioritizing any particular class. For example, if the output is a vector with probabilities that an object includes different classes, the probability values ​​for each class may be equal or at least very similar (e.g., for ten possible classes, each class may have a probability value of 0.1). Using the initial weights, the neural network 1100 cannot determine low-level features and therefore cannot make an accurate determination of what the object's classification might be. A loss function can be used to analyze the error in the output. Any suitable loss function can be defined. An example of a loss function is Mean Squared Error (MSE). MSE is defined as... It calculates the sum of half the square of the base truth output (e.g., the actual answer) minus the square of the predicted output (e.g., the predicted answer). The loss can be set to equal to The value of .

[0116] For the first training image, the loss (or error) will be high because the actual value will be significantly different from the predicted output. The goal of training is to minimize the loss so that the predicted output matches the training label. The Neural Network 1100 performs backpropagation by determining which inputs (weights) contribute most to the network's loss and can adjust the weights to reduce and ultimately minimize the loss.

[0117] The derivative of the loss with respect to the weights (denoted as dL / dW, where W is the weight at a specific layer) can be calculated to determine the weights that contribute the most to the network's loss. After calculating the derivative, a weight update can be performed by updating all the weights of the filter. For example, the weights can be updated so that they change in the opposite direction of the gradient. A weight update can be expressed as... Where w represents the weight, w i Let represent the initial weights, and η represent the learning rate. The learning rate can be set to any suitable value, where a high learning rate includes larger weight updates, while a lower value indicates smaller weight updates.

[0118] In some cases, self-supervised learning can be used to train neural networks 1100.

[0119] Neural network 1100 can include any suitable deep network. An example includes a convolutional neural network (CNN), which includes an input layer and an output layer, with multiple hidden layers between the input and output layers. The following section discusses... Figure 12 An example of a CNN is described. The hidden layers of a CNN consist of a series of convolutional layers, non-linear layers, pooling layers (for downsampling), and fully connected layers. The neural network 1100 can include any other deep network besides CNNs, such as autoencoders, deep belief networks (DBNs), recurrent neural networks (RNNs), and other examples.

[0120] Figure 12 These are illustrations illustrating examples of systems used to implement certain aspects of this disclosure. Specifically, Figure 12 An example of computing system 1200 is illustrated. This computing system can be any computing device, such as a computing system, a camera system, or any component thereof, wherein the components of the system communicate with each other using connection 1205. Connection 1205 can be a physical connection using a bus, or a direct connection to processor 1212, such as in a chipset architecture. Connection 1205 can also be a virtual connection, a networking connection, or a logical connection.

[0121] In some examples, computing system 1200 is a distributed system, wherein the functions described in this disclosure may be distributed across a data center, multiple data centers, a peer-to-peer network, etc. In some examples, one or more system components described represent a plurality of such components, each performing some or all of the functions targeted by the described components. In some examples, the components may be physical or virtual devices.

[0122] Example system 1200 includes at least one processing unit (CPU or processor) 1210 and a connection 1205 that couples various system components, including system memories 1215 such as read-only memory (ROM) 1220 and random access memory (RAM) 1225, to processor 1212. Computing system 1200 may include a cache 1211 of high-speed memory that is directly connected to, closely adjacent to, or integrated into processor 1212.

[0123] Processor 1212 may include any general-purpose processor and hardware or software services, such as services 1232, 1234, and 1236 stored in storage device 1230, which are configured to control processor 1212 and dedicated processors in which software instructions are incorporated into the actual processor design. Processor 1212 may be a substantially completely independent computing system containing multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0124] To enable user interaction, the computing system 1200 includes an input device 1245 that can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphic input, a keyboard, a mouse, motion input, voice input, etc. The computing system 1200 may also include an output device 1235 that can be one or more of multiple output mechanisms. In some instances, a multimodal system allows a user to provide multiple types of input / output to communicate with the computing system 1200. The computing system 1200 may include a communication interface 1240, which typically controls and manages user input and system output.

[0125] The communication interface can perform or facilitate the receiving and / or transmitting of wired or wireless communications using wired and / or wireless transceivers, including utilizing audio jacks / plugs, microphone jacks / plugs, Universal Serial Bus (USB) ports / plugs, Apple... ® Lightning ® Ports / plugs, Ethernet ports / plugs, fiber optic ports / plugs, dedicated wired ports / plugs, Bluetooth ® Wireless signal transmission, Bluetooth ®Low-power (BLE) wireless signal transmission, IBEACON ® Wireless signal transmission, radio frequency identification (RFID) wireless signal transmission, near field communication (NFC) wireless signal transmission, dedicated short range communication (DSRC) wireless signal transmission, 1202.11 Wi-Fi wireless signal transmission, wireless local area network (WLAN) signal transmission, visible light communication (VLC), microwave access global interoperability (WiMAX), infrared (IR) wireless signal transmission, public switched telephone network (PSTN) signal transmission, integrated services digital network (ISDN) signal transmission, 3G / 4G / 5G / LTE cellular data network wireless signal transmission, ad hoc network signal transmission, radio wave signal transmission, microwave signal transmission, infrared signal transmission, visible light signal transmission, ultraviolet light signal transmission, wireless signal transmission along the electromagnetic spectrum, or those communications in some combination thereof.

[0126] The communication interface 1240 may also include one or more Global Navigation Satellite System (GNSS) receivers or transceivers for determining the location of the computing system 1200 based on one or more signals received from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the U.S. Global Positioning System (GPS), Russia's Global Navigation Satellite System (GLONASS), China's BeiDou Navigation Satellite System (BDS), and Europe's Galileo GNSS. There are no limitations on operation on any particular hardware arrangement, and therefore the basic features herein can be easily replaced to obtain improved hardware or firmware arrangements as they are developed.

[0127] Storage device 1230 may be a non-volatile and / or non-transitory and / or computer-readable storage device, and may be a hard disk or other type of computer-readable medium capable of storing data accessible by a computer, such as magnetic tape cassettes, flash memory cards, solid-state storage devices, digital multifunction disks, cassette tapes, floppy disks, flexible disks, hard disks, magnetic tapes, magnetic stripes / strips, any other magnetic storage media, flash memory, memristor memory, any other solid-state storage, CD-ROM, rewritable CD, digital video disc (DVD), Blu-ray Disc (BDD), holographic disc, another optical medium, secure digital (SD) cards, micro-secure digital (microSD) cards, Memory Stick. ®Cards, smart card chips, EMV chips, Subscriber Identity Module (SIM) cards, mini / micro / nano / micro SIM cards, another integrated circuit (IC) chip / card, random access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash EPROM, cache memory (L1 / L2 / L3 / L4 / L5 / L#), resistive random access memory (RRAM / ReRAM), phase-change memory (PCM), spin-transfer torque RAM (STT-RAM), another memory chip or cassette and / or combinations thereof.

[0128] Storage device 1230 may include software services, servers, services, etc., which enable the system to perform functions when the code defining such software is executed by processor 1212. In some examples, hardware services performing specific functions may include software components stored in a computer-readable medium connected to necessary hardware components, such as processor 1212, connection 1205, output device 1235, etc., to perform the functions. The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or carrying instructions and / or data. Computer-readable media may include non-transitory media in which data may be stored and which do not include carrier waves and / or transient electronic signals propagating wirelessly or over a wired connection. Examples of non-transitory media include, but are not limited to, magnetic disks or magnetic tapes, optical storage media (such as compact discs (CDs) or digital versatile discs (DVDs)), flash memory, memory, or memory devices. Computer-readable media may store code and / or machine-executable instructions thereon, which may represent procedures, functions, subroutines, programs, routines, subroutines, modules, software packages, classes, or any combination of instructions, data structures, or program statements. Code segments may be coupled to other code segments or hardware circuitry by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc., may be passed, forwarded, or transmitted via any suitable means, including memory sharing, message passing, token passing, network transmission, etc.

[0129] In some respects, computer-readable storage devices, media, and memories may include cables or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer-readable storage media explicitly exclude media such as power consumption, carrier signals, electromagnetic waves, and the signals themselves.

[0130] Specific details are provided in the foregoing description to provide a thorough understanding of the aspects and examples presented herein. However, those skilled in the art will understand that these aspects can be practiced without these specific details. For clarity, in some instances, the technology may be presented as comprising individual functional blocks, including devices, device components, steps, or routines in a method embodied in software or a combination of hardware and software. Additional components may be used in addition to those shown in the figures and / or described herein. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form to avoid obscuring these aspects with unnecessary details. In other cases, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary details to avoid obscuring the aspects.

[0131] The various aspects described above can be presented as processes or methods, depicted as flowcharts, diagrams, data flow diagrams, structure diagrams, or block diagrams. Although a flowchart may describe operations as a sequential process, many operations within an operation can be executed in parallel or concurrently. Furthermore, the order of operations can be rearranged. A process terminates when its operations are completed, but a process may have additional steps not included in the accompanying diagrams. A process can correspond to a method, function, process, subroutine, subprogram, etc. When a process corresponds to a function, its termination may correspond to the function returning to the calling function or the main function.

[0132] The processes and methods described in the examples above can be implemented using stored computer-executable instructions or computer-executable instructions otherwise obtainable from a computer-readable medium. Such instructions may include, for example, instructions and data that configure a general-purpose computer, special-purpose computer, or processing device to perform a function or group of functions. The portion may be accessible via a network of the computer resources used. The computer-executable instructions may be, for example, binary, intermediate format instructions such as assembly language, firmware, or source code. Examples of computer-readable media that can be used to store the instructions, the information used, and / or information created during the methods according to the described examples include disks or optical discs, flash memory, USB devices with non-volatile memory, networked storage devices, etc.

[0133] Devices implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented as software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) for performing necessary tasks may be stored in a computer-readable or machine-readable medium. A processor may perform the necessary tasks. Typical examples of form factors include laptops, smartphones, mobile phones, tablet devices, or other small form factor personal computers, personal digital assistants, rack-mounted devices, standalone devices, etc. The functionality described herein may also be embodied in peripheral devices or intercalation cards. By way of additional examples, such functionality may also be implemented on circuit boards of different chips or different processes executed on a single device.

[0134] Instructions, media for delivering such instructions, computing resources for executing them, and other structures for supporting such computing resources are example components for providing the functionality described in this disclosure.

[0135] In the foregoing description, aspects of this application have been described with reference to their specific aspects, but those skilled in the art will recognize that this application is not limited thereto. Therefore, although illustrative aspects of this application have been described in detail herein, it is to be understood that the various inventive concepts can be implemented and employed in a variety of other ways, and the appended claims are not intended to be construed as including such variations unless limited by prior art. The various features and aspects of the applications described above can be used individually or in combination. Furthermore, aspects can be utilized in any number of environments and applications beyond those described herein without departing from the broader spirit and scope of this specification. Therefore, the specification and drawings should be considered illustrative rather than restrictive. For illustrative purposes, the methods are described in a particular order. It should be understood that, in alternative aspects, the methods may be performed in a different order than described.

[0136] Those skilled in the art will understand that, without departing from the scope of this description, the less than (“<”) and greater than (“>”) symbols or terms used herein may be replaced with less than or equal to (“>”) respectively. ") and greater than or equal to (" The symbol ) is used instead.

[0137] When a component is described as being “configured” to perform certain operations, such a configuration can be achieved, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., microprocessors or other suitable electronic circuits) to perform the operations, or any combination thereof.

[0138] The phrase “coupled to” means any component that is physically connected directly or indirectly to another component, and / or any component that communicates directly or indirectly with another component (e.g., connected to another component via a wired or wireless connection and / or other suitable communication interface).

[0139] Claim language or other languages ​​that state "at least one of" and / or "one or more of" in a set indicate that one member of the set or multiple members of the set (in any combination) satisfy the claim. For example, claim language stating "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In another example, claim language stating "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, A and B and C, or any repetition is information or data (e.g., A and A, B and B, C and C, A and A and B, etc.), or any other ordering, repetition, or combination of A, B, and C. The language "at least one of" and / or "one or more of" in a set does not limit the set to the items listed in the set. For example, the language of a claim stating "at least one of A and B" or "at least one of A or B" may refer to A, B, or A and B, and may additionally include items not listed in the set of A and B. The phrases "at least one" and "one or more" are used interchangeably herein.

[0140] Claims using phrases such as "at least one processor, configured to," "at least one processor configured to," "one or more processors, configured to," or "one or more processors configured to," or other languages, indicate that one or more processors (in any combination) are capable of performing associated operations. For example, a claim stating "at least one processor, configured to: X, Y, and Z" means that a single processor can be used to perform operations X, Y, and Z; or that multiple processors are each assigned a specific subset of tasks to perform operations X, Y, and Z, such that the multiple processors together perform X, Y, and Z; or that a group of multiple processors work together to perform operations X, Y, and Z. In another example, a claim stating "at least one processor, configured to: X, Y, and Z" may mean that any single processor can perform only a subset of operations X, Y, and Z.

[0141] When referring to one or more elements that perform functions (e.g., steps of a method), one element may perform all functions, or more than one element may jointly perform these functions. When more than one element jointly performs these functions, each function does not need to be performed by every single element (e.g., different functions may be performed by different elements), and / or each function does not need to be performed by only one element as a whole (e.g., different elements may perform different sub-functions of a function). Similarly, when referring to one or more elements configured to cause another element (e.g., a device) to perform functions, one element may be configured to cause another element to perform all functions, or more than one element may be jointly configured to cause another element to perform these functions.

[0142] When referring to an entity that performs or is configured to perform functions (e.g., steps of a method) (e.g., any entity or device described herein), the entity may be configured to cause one or more elements (individually or collectively) to perform those functions. One or more components of the entity may include at least one memory, at least one processor, at least one communication interface, another component configured to perform one or more of those functions, and / or any combination thereof. When referring to an entity that performs functions, the entity may be configured to cause one component to perform all functions, or to cause more than one component to perform those functions collectively. When the entity is configured to cause more than one component to perform those functions collectively, each function does not need to be performed by every single component (e.g., different functions may be performed by different components), and / or each function does not need to be performed by only one component as a whole (e.g., different components may perform different sub-functions of a function).

[0143] The various exemplary logic blocks, modules, circuits, and algorithm steps described in conjunction with the examples disclosed herein can be implemented as electronic hardware, computer software, firmware, or combinations thereof. To clearly illustrate this interchangeability between hardware and software, various exemplary components, blocks, modules, circuits, and steps have been broadly described in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in different ways for each specific application, but such specific implementation decisions should not be construed as departing from the scope of this application.

[0144] The techniques described herein can also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques can be implemented in any of a variety of devices, such as general-purpose computers, wireless communication devices (mobile phones), or integrated circuit devices with multiple uses, including applications in wireless communication devices (mobile phones) and other devices. Any feature described as a module or component can be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, the techniques can be implemented at least in part by a computer-readable data storage medium comprising program code including instructions that, when executed, perform one or more of the methods, algorithms, and / or operations described above. The computer-readable data storage medium can form part of a computer program product, which may include packaging material. The computer-readable medium may include memory or data storage media, such as random access memory (RAM) (such as synchronous dynamic random access memory (SDRAM)), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, etc. Additionally or alternatively, the technology may be implemented at least in part by a computer-readable communication medium that carries or conveys program code in the form of instructions or data structures that can be accessed, read and / or executed by a computer, such as propagated signals or waves.

[0145] The program code can be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such processors can be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor; however, in alternatives, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Therefore, as used herein, the term "processor" may refer to any of the foregoing structures, any combination of the foregoing structures, or any other structure or means suitable for implementing the techniques described herein.

[0146] The exemplary aspects of this disclosure include:

[0147] Aspect 1. An apparatus for generating output text from input data, the apparatus comprising: one or more memories configured to store the input data; and one or more processors coupled to the one or more memories and configured to: encode the input data to generate an encoded representation of the input data; obtain intermediate data including a plurality of partial sentences associated with the input data; generate at least one complete sentence associated with the input data based on the intermediate data; encode the at least one complete sentence to generate at least one encoded representation of the at least one complete sentence; generate a fidelity score based on a comparison of the encoded representation of the input data with the at least one encoded representation of the at least one complete sentence; and reorder the plurality of partial sentences of the intermediate data based on the fidelity score to generate reordered data.

[0148] Aspect 2. The apparatus according to aspect 1, wherein the input data includes at least one of audio data, text data, image data, or video data.

[0149] Aspect 3. The apparatus according to aspect 2, wherein the input data includes two or more of the audio data, the text data, the image data, and the video data.

[0150] Aspect 4. The apparatus according to any one of Aspects 1 to 3, wherein the intermediate data includes an intermediate beam generated using a beam search technique.

[0151] Aspect 5. The apparatus according to any one of Aspects 1 to 4, wherein the one or more processors are configured to generate the at least one complete statement based on the intermediate data using a greedy search technique.

[0152] Aspect 6. The apparatus according to any one of Aspects 1 to 5, wherein the one or more processors are configured to reorder the plurality of partial statements of the intermediate data based on the fidelity score and model confidence to generate the reordered data.

[0153] Aspect 7. The apparatus according to Aspect 6, wherein the one or more processors are configured to: determine a bundle score based on the model confidence, the fidelity score, and the probability of the next word in each of the plurality of partial sentences; determine a cumulative probability based on the bundle score; and reorder the plurality of partial sentences of the intermediate data based on the cumulative probability.

[0154] Aspect 8. The apparatus according to aspect 7, wherein the one or more processors are configured to determine the model confidence based on entropy and kurtosis values.

[0155] Aspect 9. The apparatus according to any one of Aspects 1 to 8, wherein the input data includes video data, and wherein the one or more processors are configured to: downsample a plurality of frames of the video data; and fuse the coded representations of the plurality of frames of the video data to generate a fused representation of the video data, wherein the coded representation of the input data includes the fused representation of the video data.

[0156] Aspect 10. The apparatus according to any one of Aspects 1 to 9, wherein: the input data includes at least first type of input data and second type of input data; in order to encode the input data to generate the encoded representation of the input data, the one or more processors are configured to: encode the first type of input data to generate the encoded representation of the first type of input data; and encode the second type of input data to generate the encoded representation of the second type of input data; and the one or more processors are further configured to generate a combined representation of the first type of input data and the second type of input data based on the encoded representation of the first type of input data and the encoded representation of the second type of input data.

[0157] Aspect 11. The apparatus according to aspect 10, wherein, in order to generate the combined representation of the first type of input data and the second type of input data, the one or more processors are configured to: determine a weighted average of the encoded representation of the first type of input data and the encoded representation of the second type of input data.

[0158] Aspect 12. The apparatus according to any one of Aspects 10 or 11, wherein the one or more processors are configured to normalize the combined representation of the first type of input data and the second type of input data.

[0159] Aspect 13. The apparatus according to any one of Aspects 10 to 12, wherein the first type of input data and the second type of input data include two or more of audio data, text data, image data, and video data.

[0160] Aspect 14. The apparatus according to any one of Aspects 10 to 13, wherein the one or more processors are configured to generate the fidelity score based on a comparison of the combined representation with the at least one encoded representation of the at least one complete statement.

[0161] Aspect 15. The apparatus according to any one of aspects 1 to 14, wherein the one or more processors are configured to generate output text associated with the input data based on the reordered data.

[0162] Aspect 16. The apparatus according to any one of aspects 1 to 15, the apparatus further comprising at least one of an image sensor or a microphone, the image sensor or the microphone being configured to capture at least a portion of the input data.

[0163] Aspect 17. The apparatus according to any one of aspects 1 to 16, wherein the one or more processors are configured to use at least one neural network model to generate the intermediate data.

[0164] Aspect 18. The apparatus according to aspect 17, wherein the at least one neural network model comprises a transformer neural network model.

[0165] Aspect 19. A method for generating output text from input data, the method comprising: encoding the input data to generate an encoded representation of the input data; obtaining intermediate data comprising a plurality of partial sentences associated with the input data; generating at least one complete sentence associated with the input data based on the intermediate data; encoding the at least one complete sentence to generate at least one encoded representation of the at least one complete sentence; generating a fidelity score via a fidelity-guided engine based on a comparison of the encoded representation of the input data with the at least one encoded representation of the at least one complete sentence; and reordering the plurality of partial sentences of the intermediate data based on the fidelity score to generate reordered data.

[0166] Aspect 20. The method according to aspect 19, wherein the input data includes at least one of audio data, text data, image data, or video data.

[0167] Aspect 21. The method according to aspect 20, wherein the input data includes two or more of the audio data, the text data, the image data, and the video data.

[0168] Aspect 22. The method according to any one of aspects 19 to 21, wherein the intermediate data includes an intermediate bundle generated using a bundle search technique.

[0169] Aspect 23. The method according to any one of aspects 19 to 22, the method further comprising: generating the at least one complete statement based on the intermediate data using a greedy search technique.

[0170] Aspect 24. The method according to any one of Aspects 19 to 23, the method further comprising: reordering the plurality of partial statements of the intermediate data based on the fidelity score and model confidence to generate the reordered data.

[0171] Aspect 25. The method according to aspect 24, the method further comprising: determining a bundle score based on the model confidence, the fidelity score and the probability of the next word in each of the plurality of partial sentences; determining a cumulative probability based on the bundle score; and reordering the plurality of partial sentences of the intermediate data based on the cumulative probability.

[0172] Aspect 26. The method according to aspect 25, the method further comprising: determining the confidence level of the model based on entropy and kurtosis.

[0173] Aspect 27. The method according to any one of Aspects 21 to 26, the method further comprising: downsampling a plurality of frames of the video data; and fusing coded representations of the plurality of frames of the video data to generate a fused representation of the video data, wherein the coded representation of the input data includes the fused representation of the video data.

[0174] Aspect 28. The method according to any one of Aspects 19 to 27, wherein the input data includes at least a first type of input data and a second type of input data, wherein the method further comprises: encoding the first type of input data to generate an encoded representation of the first type of input data; encoding the second type of input data to generate an encoded representation of the second type of input data; and generating a combined representation of the first type of input data and the second type of input data based on the encoded representation of the first type of input data and the encoded representation of the second type of input data.

[0175] Aspect 29. The method according to aspect 28, wherein in order to generate the combined representation of the first type of input data and the second type of input data, the method further comprises: determining a weighted average of the encoded representation of the first type of input data and the encoded representation of the second type of input data.

[0176] Aspect 30. The method according to any one of Aspects 28 or 29, the method further comprising: normalizing the combined representation of the first type of input data and the second type of input data.

[0177] Aspect 31. The method according to any one of Aspects 28 to 30, wherein the first type of input data and the second type of input data include two or more of audio data, text data, image data, and video data.

[0178] Aspect 32. The method according to any one of aspects 28 to 31, the method further comprising: generating the fidelity score based on a comparison of the combined representation with the at least one encoded representation of the at least one complete statement.

[0179] Aspect 33. The method according to any one of aspects 19 to 32, the method further comprising: generating output text associated with the input data based on the reordered data.

[0180] Aspect 34. The method according to any one of aspects 19 to 33, wherein at least one of an image sensor or a microphone captures at least a portion of the input data.

[0181] Aspect 35. The method according to any one of aspects 19 to 34, the method further comprising: generating the intermediate data using at least one neural network model.

[0182] Aspect 36. The method according to aspect 35, wherein the at least one neural network model includes a transformer neural network model.

[0183] Aspect 37. A decoder comprising: one or more transformer blocks configured to receive an embedded coded representation of input data; a faithful determiner configured to receive intermediate bundles and reorder the intermediate bundles; and a sampler coupled to (a) the one or more transformer blocks and (b) the faithful determiner, configured to output an output subtitle.

[0184] Aspect 38. The decoder according to aspect 27, wherein the input data includes one or more modalities.

[0185] Aspect 39: The decoder according to aspect 38, wherein the one or more modalities include one or more of text, speech, and audio.

[0186] Aspect 40. A fidelity determiner / guide, the fidelity determiner / guide comprising: a weighted averaging and normalization component that receives one or more embeddings and generates normalized embeddings; and a similarity component that receives the normalized embeddings and text embeddings associated with captions, calculates a similarity score, and feeds the similarity score to a bundle weight sorter.

[0187] Aspect 41. The fidelity determiner / guide according to aspect 40, wherein the one or more embeddings include one or more of audio embeddings, image embeddings, and input text embeddings.

[0188] Aspect 42. The fidelity determiner / guide according to aspect 41, wherein the audio embedding is obtained from an audio spectrogram.

[0189] Aspect 43. The fidelity determiner / guide according to aspect 40, wherein the similarity component applies cosine similarity to calculate the similarity score.

[0190] Aspect 44. The fidelity determiner / guide according to aspect 41, further comprising:

[0191] A downsampler that receives video frames and generates output for obtaining the image embedding.

[0192] Aspect 45. The fidelity determiner / guide according to aspect 44, the fidelity determiner / guide further comprising: an encoder that receives caption data and generates the text embedding provided to the similarity component.

[0193] Aspect 46. A non-transitory computer-readable medium having instructions stored thereon, which, when executed by one or more processors, cause the one or more processors to perform any one of aspects 19 to 36.

[0194] Aspect 478. An apparatus comprising components for performing operations according to any one of aspects 19 to 36.

Claims

1. An apparatus for generating output text from input data, the apparatus comprising: One or more memories configured to store the input data; and One or more processors, said one or more processors being coupled to said one or more memories and configured to: The input data is encoded to generate an encoded representation of the input data; Obtain intermediate data comprising multiple partial statements associated with the input data; Based on the intermediate data, at least one complete statement associated with the input data is generated; Encode the at least one complete statement to generate at least one encoded representation of the at least one complete statement; A loyalty score is generated based on a comparison between the encoded representation of the input data and the encoded representation of the at least one complete statement. as well as The multiple partial statements in the intermediate data are reordered based on the fidelity score to generate reordered data.

2. The apparatus of claim 1, wherein the input data includes at least one of audio data, text data, image data, or video data.

3. The apparatus of claim 2, wherein the input data includes two or more of the audio data, the text data, the image data, and the video data.

4. The apparatus of claim 1, wherein the intermediate data comprises an intermediate beam generated using a beam search technique.

5. The apparatus of claim 1, wherein the one or more processors are configured to generate the at least one complete statement based on the intermediate data using a greedy search technique.

6. The apparatus of claim 1, wherein the one or more processors are configured to reorder the plurality of partial statements of the intermediate data based on the fidelity score and model confidence to generate the reordered data.

7. The apparatus of claim 6, wherein the one or more processors are configured to: The bundle score is determined based on the model confidence, the fidelity score, and the probability of the next word in each of the multiple partial sentences. The cumulative probability is determined based on the beam score; as well as The intermediate data is reordered based on the cumulative probability.

8. The apparatus of claim 7, wherein the one or more processors are configured to determine the model confidence level based on entropy and kurtosis values.

9. The apparatus of claim 1, wherein the input data includes video data, and wherein the one or more processors are configured to: Downsampling multiple frames of the video data; and The coded representations of the multiple frames of the video data are fused to generate a fused representation of the video data, wherein the coded representation of the input data includes the fused representation of the video data.

10. The apparatus according to claim 1, wherein: The input data includes at least a first type of input data and a second type of input data; In order to encode the input data to generate the encoded representation of the input data, the one or more processors are configured to: Encode the first type of input data to generate an encoded representation of the first type of input data; as well as The second type of input data is encoded to generate an encoded representation of the second type of input data; as well as The one or more processors are further configured to generate a combined representation of the first type of input data and the second type of input data based on the encoded representation of the first type of input data and the encoded representation of the second type of input data.

11. The apparatus of claim 10, wherein, in order to generate the combined representation of the first type of input data and the second type of input data, the one or more processors are configured to: Determine the weighted average of the encoded representation of the first type of input data and the encoded representation of the second type of input data.

12. The apparatus of claim 10, wherein the one or more processors are configured to normalize the combined representation of the first type of input data and the second type of input data.

13. The apparatus of claim 10, wherein the first type of input data and the second type of input data include two or more of audio data, text data, image data, and video data.

14. The apparatus of claim 10, wherein the one or more processors are configured to generate the fidelity score based on a comparison of the combined representation with the at least one encoded representation of the at least one complete statement.

15. The apparatus of claim 1, wherein the one or more processors are configured to: The output text associated with the input data is generated based on the reordered data.

16. The apparatus of claim 1, further comprising at least one of an image sensor or a microphone configured to capture at least a portion of the input data.

17. The apparatus of claim 1, wherein the one or more processors are configured to use at least one neural network model to generate the intermediate data.

18. The apparatus of claim 17, wherein the at least one neural network model comprises a transformer neural network model.

19. A method for generating output text from input data, the method comprising: The input data is encoded to generate an encoded representation of the input data; Obtain intermediate data comprising multiple partial statements associated with the input data; Based on the intermediate data, at least one complete statement associated with the input data is generated; Encode the at least one complete statement to generate at least one encoded representation of the at least one complete statement; A fidelity score is generated by a fidelity-guided engine based on a comparison between the encoded representation of the input data and the encoded representation of the at least one complete statement. as well as The multiple partial statements in the intermediate data are reordered based on the fidelity score to generate reordered data.

20. The method of claim 19, wherein the intermediate data comprises an intermediate bundle generated using a bundle search technique.

Images