Schema-less structuring of natural language

Abstract

The technology provides a flexible schema approach where an app or other service does not need to know anything about language or machine learning in order to interact with natural language associated with a user of the app. This enables tailoring for a particular application or a given task in an application. One method includes receiving a natural language input of a user (702) and then generating a dynamic schema for use with an application running on the computing system (704). The computing system applies, using a trained language model, the dynamic schema to the natural language input to generate schema data having one or more corresponding field and value pairings (706), and provides the schema data to the application (708). The application can generate an output based upon at least one of the field and value pairings (710) and present the output to the user (712).

Description

SCHEMA-LESS STRUCTURING OF NATURAL LANGUAGEBACKGROUND

[0001] Various approaches have been used for computing systems to understand spoken or written natural language. This includes pattern matching to a rule-based system, which can be effective for strict syntax and simple sentences. However, complications in sentence structure can lead to incorrect understanding of the meaning. Another approach involves training a model to a particular task. This can be used to address specific applications, such as understanding medical records, performing a customer service task, or managing a number of discrete tasks in parallel. However, this tailor-made approach may only achieve that specific goal, while also requiring particular machine learning skills from the application developer. Another approach uses large language models (LLMs), which have been widely applied to various text-related tasks. However, such models may not project their knowledge effectively, and thus external apps may not be able to effectively use the output of such models to provide relevant information to users of those apps.BRIEF SUMMARY

[0002] The technology provides a technical solution where an app or other service does not need to know anything about language or machine learning in order to interact with natural language associated with a user of the app. According to one aspect of the technology, the system provides a schema model that automatically generates an unfixed schema that the app can predict. The output of this is a (field, value) pairing. In some instances, there can be multiple values per field, including additional sets of key: value pairs A text-to-text transformer approach may be used to build the structure for the schema. This allows the app to interact with natural language from the user (e.g., a spoken command) via the schema, without requiring the app to be configured to handle such language directly. According to another aspect of the technology, the app may send an arbitrary schema to a schema model, and the schema model accordingly populates the fields of schema based on natural language received from the user. This can be used to achieve an implied understanding of the user’s mood of topics that are implied by the text, but not explicitly said. Employing an unfixed, arbitrary or otherwise dynamic schema provides system flexibility and allows tailoring for a particular application or a given task in an application. A technical benefit over conventional LLMs is that the model can be fine-tuned to manage a single task, and has less chance that the instruction in the LLM prompt will be ignored or misinterpreted due to unexpected input.

[0003] The approaches discussed herein may be employed with journal-type applications that encompass one or more verticals, such as sleep tracking, exercise, women’s health, glucose levels, hydration, etc. However, they may also be used in other types of applications that do not involve journaling.For instance, a non-journaling application can be as control input for programs, such as: “create a paintbrush of 30 pixels, in light blue and opacity of 30%” or “give me the average sales in northern Virginia over 3 months in stores with a volume of sales smaller than $1M per year”. Other examples include to-do or shopping lists, etc.

[0004] According to one aspect of the technology, a method comprises: receiving, by a computing system, a natural language input of a user; generating, by the computing system, a dynamic schema for use with an application running on the computing system; applying, by the computing system using a trained language model, the dynamic schema to the received natural language input to generate a set of schema data having one or more corresponding field and value pairings; providing the set of schema data to the application; generating, by the application, an output based upon at least one of the field and value pairings of the set of schema data; and presenting the output to the user.

[0005] In one example, the dynamic schema is an unfixed schema generated by the trained language model. In another example, the dynamic schema is an arbitrary schema generated by the application. In this case, the arbitrary schema generated by the application may be configured to elicit information corresponding to a mood of the user. Here, the output may provide a recommendation to the user based upon the information corresponding to the mood.

[0006] Alternatively or additionally to any of the above, providing the set of schema data to the application may include: parsing the set of schema data to generate a parsed set of data; and providing the parsed set of data to the application; and generating the output is based upon one or more field and value pairings of the parsed set of data. Alternatively or additionally to any of the above, the natural language input may include textual input received from the user, and the language model implements a text-to-text transformer. Alternatively or additionally to any of the above, the natural language input may include speech input received from the user, and the language model implements a speech-to-text transformer.

[0007] The language model may be a non-deterministic model. The fields in the set of schema data may be arranged in order according to a relevance criterion. Alternatively or additionally to any of the above, applying the dynamic schema to the received natural language input may include performing a recursive operation on different parts of a sentence in the natural language input to generate the set of schema data.

[0008] According to another aspect of the technology, a method comprises: receiving, by a computing system, a natural language input of a user; generating, by the computing system, a dynamic schema for use with an application running on the computing system; applying, by the computing system using a trained language model, the dynamic schema to the received natural language input to generate a set of schema data having one or more corresponding field and value pairings; and providing the set of schema data to theapplication in order for the application to either generate an output for presentation to the user based upon at least one of the field and value pairings of the set of schema data, or perform an operation based upon at least one of the field and value pairings of the set of schema data. The dynamic schema may be an unfixed schema generated by the trained language model. Alternatively, the dynamic schema may be generated based upon information received from the application.

[0009] In an example, providing the set of schema data to the application includes: parsing the set of schema data to generate a parsed set of data; and providing the parsed set of data to the application. Alternatively or additionally to any of the above, fields in the set of schema data may be arranged in order according to a relevance criterion. Alternatively or additionally to any of the above, applying the dynamic schema to the received natural language input may include performing a recursive operation on different parts of a sentence in the natural language input to generate the set of schema data.

[0010] According to a further aspect of the technology, a computer system comprises: memory configured to store a trained language model; and one or more processors operatively coupled to the memory. The one or more processors are configured to: receive, via a user interface of the computer system, a natural language input of a user; generate a dynamic schema for use with an application running on the computing system; apply, using the trained language model, the dynamic schema to the received natural language input to generate a set of schema data having one or more corresponding field and value pairings; cause the application to generate an output based upon at least one of the field and value pairings of the set of schema data; and cause the user interface to present the output to the user. In an example, the dynamic schema may be either an unfixed schema generated by the trained language model or an arbitrary schema generated by the application.

[0011] And according to yet another aspect of the technology, a computer system comprises: memory configured to store a trained language model; and one or more processors operatively coupled to the memory. The one or more processors are configured to: receive, via a user interface of the computing system, a natural language input of a user; generate a dynamic schema for use with an application; apply, using the trained language model, the dynamic schema to the received natural language input to generate a set of schema data having one or more corresponding field and value pairings; and provide the set of schema data to the application in order for the application to either generate an output for presentation to the user based upon at least one of the field and value pairings of the set of schema data, or perform an operation based upon at least one of the field and value pairings of the set of schema data.BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Fig. 1 illustrates a general approach for generating and utilizing unstructured schema in accordance with aspects of the technology.

[0013] Fig. 2 illustrates a Transformer-type architecture for use in accordance with aspects of the technology.

[0014] Figs. 3A-C illustrate unfixed schema and arbitrary schema systems in accordance with aspects of the technology.

[0015] Figs. 4A-B illustrate a recommendation example in accordance with aspects of the technology.

[0016] Figs. 5A-B illustrate another recommendation example in accordance with aspects of the technology.

[0017] Figs. 6A-B illustrate a system for use with aspects of the technology.

[0018] Figs. 7A-B illustrate example methods in accordance with aspects of the technology.DETAILED DESCRIPTION

[0019] The technology relates to systems and methods that employ an unfixed, arbitrary, otherwise dynamic schema in conjunction with neural network models, e g., large language models. Such schema may be considered dynamic, for instance, because a language model is not constrained to generate the same schema when the same or similar input is received. Moreover, when an app provides a schema to a language model, that schema may be dynamically configured according to one or more tasks to be performed by the app. Thus, the adaptable nature of the schema discussed herein is considered to be dynamic.

[0020] One technical benefit is that a model can generate the schema and the corresponding values for the schema automatically. By way of example, a sleep tracker app may receive user input such as “I was in a hotel, but it was very cold, I couldn’t sleep”. From this, the model may generate a schema such as: (activity: sleep, date-time: last night, sleep quality: bad, cause: cold), or (activity: sleep, quality: bad, location: hotel). It can be seen that the fields in both generated schemas are either fixed (like activity), covered by few representative values that the app can plan to accept (like [sleep quality, quality]) or may be less important for the app (like [cause, location]). From this, the app can use the populated schema to present a recommendation or other information to the user.

[0021] For instance, Fig. 1 illustrates air example involving a system 100 for evaluating sleep quality. This can be applicable to users for sleep schedule tracking, health monitoring, etc. The system 100 may include one or more processors 102 and memory 104 for storing data. In one example, the memory 104 may store one or more trained LLMs as well as any schema information generated by the system. A user 106 can push abutton, speak a command or otherwise initiate logging via an app on their client device 108, which may be, e.g., a smartwatch, a smart home appliance, a laptop or desktop computer, a tablet PC, a mobile phone or PDA, etc.

[0022] As shown in this example, the app starts logging information (e.g., sleep data via one or more sensors of the client device) at block 110. If user input is speech (as opposed to text), then at block 112, the app runs a speech-to-text model (e.g., an LLM) on the speech (as shown by the dashed arrow from block 110), getting the utterance text as a response. Then at block 114, the text is applied to a language (schema) model. At block 116, an unfixed or arbitrary schema is generated by the language model with (field, value) pairings. Then at block 118, the app, such as the sleep app in this example, evaluates the schema pairings to generate relevant information to the user. Here, this may include providing a sleep score according to the logged sleep information. By way of example only, the app may evaluate the logged sleep information (e.g., how long the user was awake, in a light sleep, in REM sleep and / or in deep sleep) in conjunction with the user’s natural language inputs, when selecting what information to present, how to present it, etc. Note that according to aspects of the technology, the model can work for input as speech, descriptive text or any other data source, such as a document, scraped website or even an image.EXAMPLE SYSTEMS AND METHODS

[0023] As noted above, one or more LLMs may be employed in the system 100. The LLMs may have a text-to-text transformer architecture, which can be utilized to build the structure for the schema.General Transformer Architecture

[0024] By way of example only, a general transfonner architecture is presented in Fig. 2. In particular, system 200 of Fig. 2 is implementable via a computer program by processors of one or more computers in one or more locations. The system 200 receives an input sequence 202 (e.g., a query) and processes the input sequence 202 to transduce the input sequence 202 into an output sequence 204 (e.g., an answer). The input sequence 202 has a respective network input at each of multiple input positions in an input order and the output sequence 204 has a respective network output at each of multiple output positions in an output order.

[0025] System 200 can perform any of a variety of tasks that require processing sequential inputs to generate sequential outputs. System 200 includes an attention-based sequence transduction neural network 206, which in turn includes an encoder neural network 208 and a decoder neural network 210. The encoder neural network 208 is configured to receive the input sequence 202 and generate a respective encoded representation of each of the network inputs in the input sequence. An encoded representation is a vector or other ordered collection of numeric values. The decoder neural network 210 is then configured to use the encoded representations of the network inputs to generate the output sequence 204. Generally, both the encoder 208 and the decoder 210 are attention-based. In some cases, neither the encoder nor the decoder includes any convolutional layers or any recurrent layers. The encoder neural network 208 includes anembedding layer (input embedding) 212 and a sequence of one or more encoder subnetworks 214. The encoder neural 208 network may N encoder subnetworks 214.

[0026] The embedding layer 212 is configured, for each network input in the input sequence, to map the network input to a numeric representation of the network input in an embedding space, e g , into a vector in the embedding space. The input sequence can be tokenized and then the tokens are embedded by the embedding layer. For instance, tokenization can involve splitting input text such as a sentence or a paragraph into chunks (e.g., individual words) referred to as tokens.

[0027] The embedding layer 212 then provides the numeric representations of the network inputs to the first subnetwork in the sequence of encoder subnetworks 214. The embedding layer 212 may be configured to map each network input to an embedded representation of the network input and then combine, e.g., sum or average, the embedded representation of the network input with a positional embedding of the input position of the network input in the input order to generate a combined embedded representation of the network input. In some cases, the positional embeddings are learned. As used herein, “learned” means that an operation or a value has been adjusted during the training of the sequence transduction neural network 206. In other cases, the positional embeddings may be fixed and are different for each position.

[0028] Tire combined embedded representation is then used as the numeric representation of the network input. Each of the encoder subnetworks 214 is configured to receive a respective encoder subnetwork input for each of the plurality of input positions and to generate a respective subnetwork output for each of the plurality of input positions. The encoder subnetwork outputs generated by the last encoder subnetwork in the sequence are then used as the encoded representations of the network inputs. For the first encoder subnetwork in the sequence, the encoder subnetwork input is the numeric representations generated by the embedding layer 212, and, for each encoder subnetwork other than tire first encoder subnetwork in the sequence, the encoder subnetwork input is the encoder subnetwork output of the preceding encoder subnetwork in the sequence.

[0029] Each encoder subnetwork 214 includes an encoder self-attention sub-layer 216. The encoder self-attention sub-layer 216 is configured to receive the subnetwork input for each of the plurality of input positions and, for each particular input position in the input order, apply an attention mechanism over the encoder subnetwork inputs at the input positions using one or more queries derived from the encoder subnetwork input at the particular input position to generate a respective output for the particular input position. In some cases, the attention mechanism is a multi-head attention mechanism as shown. In some implementations, each of the encoder subnetworks 214 may also include a residual connection layer that combines the outputs of the encoder self-attention sub-layer with the inputs to the encoder self-attentionsub-layer to generate an encoder self-attention residual output and a layer normalization layer that applies layer normalization to the encoder self-attention residual output. These two layers are collectively referred to as an “Add & Norm” operation in Fig. 2.

[0030] Some or all of the encoder subnetworks can also include a position-wise feed-forward layer 218 that is configured to operate on each position in the input sequence separately. In particular, for each input position, the feed-forward layer 218 is configured receive an input at the input position and apply a sequence of transformations to the input at the input position to generate an output for the input position. The inputs received by the position-wise feed-forward layer 218 can be the outputs of the layer normalization layer when the residual and layer normalization layers are included or the outputs of the encoder self-attention sub-layer 216 when the residual and layer normalization layers are not included. The transformations applied by the layer 218 will generally be the same for each input position (but different feed-forward layers in different subnetworks may apply different transformations).

[0031] In cases where an encoder subnetwork 214 includes a position-wise feed-forward layer 218 as shown, the encoder subnetwork can also include a residual connection layer that combines the outputs of the position-wise feed-forward layer with the inputs to the position-wise feed-forward layer to generate an encoder position-wise residual output and a layer normalization layer that applies layer normalization to the encoder position-wise residual output. As noted above, these two layers arc also collectively referred to as an "Add & Norm" operation. The outputs of this layer normalization layer can then be used as the outputs of the encoder subnetwork 214.

[0032] Once the encoder neural network 208 has generated the encoded representations, the decoder neural network 210 is configured to generate the output sequence in an auto-regressive manner. That is, the decoder neural network 210 generates the output sequence, by at each of a plurality of generation time steps, generating a network output for a corresponding output position conditioned on (i) tire encoded representations and (ii) network outputs at output positions preceding the output position in the output order, in particular, for a given output position, the decoder neural network generates an output that defines a probability distribution over possible network outputs at the given output position. The decoder neural network can then select a network output for the output position by sampling from the probability distribution or by selecting the network output with the highest probability.

[0033] Because the decoder neural network 210 is auto-regressive, at each generation time step, the decoder network 210 operates on the network outputs that have already been generated before the generation time step, i.e., the network outputs at output positions preceding the corresponding output position in the output order. In some implementations, to ensure this is the case during both inference and training, at each generation time step the decoder neural network 210 shifts the already generated networkoutputs right by one output order position (i.e., introduces a one position offset into the already generated network output sequence) and (as will be described in more detail below) masks certain operations so that positions can only attend to positions up to and including that position in the output sequence (and not subsequent positions) . While the remainder of the description below describes that, when generating a given output at a given output position, various components of the decoder 210 operate on data at output positions preceding the given output positions (and not on data at any other output positions), it will be understood that this type of conditioning can be effectively implemented using shifting.

[0034] The decoder neural network 210 includes an embedding layer (output embedding) 220, a sequence of decoder subnetworks 222, a linear layer 224, and a softmax layer 226. In particular, the decoder neural network can include N decoder subnetworks 222. However, while the example of Fig. 2 shows the encoder 208 and the decoder 210 including the same number of subnetworks, in some cases the encoder 208 and the decoder 210 include different numbers of subnetworks. The embedding layer 220 is configured to, at each generation time step, for each network output at an output position that precedes the current output position in the output order, map the network output to a numeric representation of the network output in the embedding space. The embedding layer 220 then provides the numeric representations of the network outputs to the first subnetwork 222 in the sequence of decoder subnetworks.

[0035] In some implementations, the embedding layer 220 is configured to map each network output to an embedded representation of the network output and combine the embedded representation of the network output with a positional embedding of the output position of the network output in the output order to generate a combined embedded representation of the network output. The combined embedded representation is then used as the numeric representation of the network output. The embedding layer 220 generates the combined embedded representation in the same manner as described above with reference to the embedding layer 212.

[0036] Each decoder subnetwork 222 is configured to, at each generation time step, receive a respective decoder subnetwork input for each of the plurality of output positions preceding the corresponding output position and to generate a respective decoder subnetwork output for each of the plurality of output positions preceding the corresponding output position (or equivalently, when the output sequence has been shifted right, each netw ork output at a position up to and including the current output position). In particular, each decoder subnetwork 222 includes two different attention sub-layers: a decoder self-attention sub-layer 228 and an encoder-decoder attention sub-layer 230. Each decoder self-attention sub-layer 228 is configured to, at each generation time step, receive an input for each output position preceding the corresponding output position and, for each of the particular output positions, apply an attention mechanism over the inputs at the output positions preceding the corresponding position using oneor more queries derived from the input at the particular output position to generate a updated representation for the particular output position. That is, the decoder self-attention sub-layer 228 applies an attention mechanism that is masked so that it does not attend over or otherwise process any data that is not at a position preceding the current output position in the output sequence.

[0037] Each encoder-decoder attention sub-layer 230, on the other hand, is configured to, at each generation time step, receive an input for each output position preceding the corresponding output position and, for each of the output positions, apply an attention mechanism over the encoded representations at the input positions using one or more queries derived from the input for the output position to generate an updated representation for the output position. Thus, the encoder-decoder attention sub-layer 230 applies attention over encoded representations while the decoder self-attention sub-layer 228 applies attention over inputs at output positions.

[0038] In the example of Fig. 2, the decoder self-attention sub-layer 228 is shown as being before the encoder-decoder attention sub-layer in the processing order within the decoder subnetwork 222. In other examples, however, the decoder self-attention sub-layer 228 may be after the encoder-decoder attention sub-layer 230 in the processing order within the decoder subnetwork 222 or different subnetworks may have different processing orders. In some implementations, each decoder subnetwork 222 includes, after the decoder self-attention sub-layer 228, after the cncodcr-dccodcr attention sub-layer 230, or after each of the two sub-layers, a residual connection layer that combines the outputs of the attention sub-layer with the inputs to the attention sub-layer to generate a residual output and a layer normalization layer that applies layer normalization to the residual output. These two layers being inserted after each of the two sub-layers, both referred to as an "Add & Norm" operation.

[0039] Some or all of the decoder subnetwork 222 also include a position-wise feed-forward layer 232 that is configured to operate in a similar manner as the position-wise feed-forward layer 218 from the encoder 208. In particular, the layer 232 is configured to, at each generation time step: for each output position preceding the corresponding output position: receive an input at the output position, and apply a sequence of transformations to the input at the output position to generate an output for the output position. The inputs received by the position-wise feed-forward layer 232 can be the outputs of the layer normalization layer (following the last attention sub-layer in the subnetwork 222) when the residual and layer normalization layers are included or the outputs of the last attention sub-layer in the subnetwork 222 when the residual and layer normalization layers are not included. In cases where a decoder subnetwork 222 includes a position-wise feed-forward layer 232, the decoder subnetwork can also include a residual connection layer that combines the outputs of the position-wise feed-forward layer with the inputs to the position-wise feed-forward layer to generate a decoder position-wise residual output and a layernormalization layer that applies layer normalization to the decoder position-wise residual output. These two layers are also collectively referred to as an "Add & Norm" operation. The outputs of this layer normalization layer can then be used as the outputs of the decoder subnetwork 222.

[0040] At each generation time step, the linear layer 224 applies a learned linear transformation to the output of the last decoder subnetwork 222 in order to project the output of the last decoder subnetwork 222 into the appropriate space for processing by the softmax layer 226. The softmax layer 226 then applies a softmax function over the outputs of the linear layer 224 to generate the probability distribution (output probabilities) 234 over the possible network outputs at the generation time step. The decoder 210 can then select a network output from the possible network outputs using the probability distribution, to output final result 204.

[0041] According to one implementation, an existing LLM can be used to generate a schema out of general language. By way of example only, Bidirectional Encoder Representations from Transformers (BERT), Language Model for Dialogue Applications (LaMDA), and / or Pathways Language Mode (PaLM) type models may be utilized. In another implementation, a compact T5 language model or similar architecture may be used to generate the schema.Schema Approaches

[0042] One approach of the technology is for the language model to automatically create an unfixed (ad hoc) schema for received natural language speech. The schema is then applied to the app's needs. This ad hoc approach can run the schema one for all consumers (e.g., different journaling tools or other verticals of a single app, or of multiple apps). This can be computationally or otherwise resource efficient. In another approach, the app provides a given schema in a predefined scheme to the language model, where the model provides responses in accordance with the predefined scheme. This is beneficial because the schema provided by tire app may be used to gather information about the implied understanding of tire user’s mood. Examples of implied fields are the mood of the user (as well as other features like anxiety, tiredness, etc ). For instance, if a meal was healthy (or salty / sugary / processed), the statement "1 ate a mango, half a pound of grapes and some watermelon" could be marked as a sugary and unprocessed meal. In another example, when the situation involves a workout / training, the system can identify what the training focused on (e.g., aerobic vs strength, leg day vs chest vs back, etc.). Here, “I did squats and deadlifts” can indicate a strength leg day. Moreover, even personal opinion can be understood with an implication. Thus, the statement “the best rock ever was Metallica” would imply that the user likes Metallica.

[0043] Fig. 3A presents a system 300 configured to implement this approach. As shown, a natural language input 302, such as speech or text from a user, is input into a language model (LLM) 304. While not shown, the natural language input 302 may undergo text to speech conversion via a conversion module,with the result being input to the LLM 304. This model 304 generates an unfixed schema 306, which may be stored in memory of the system (not shown). The unfixed schema 306 is input to an app or other program 308. In some situations, the app 308 may be a third-party application. Because the unfixed schema 306 includes processed information corresponding to the natural language input 302, the app 308 can apply that processed information to generate output 310, although this is not required. The output 310 may be, e.g., a recommendation to the user. For instance, in the case of a fitness tracker app where the unfixed schema relates to journaling of exercising or another activity, the recommendation may be to take a recovery drink, try a different type of exercise, make a modification to the exercise being performed, etc. Alternatively or additionally, the output may involve the app performing some action / operation based upon the processed information. The app can make a decision to present some or all of the information to a user. It may decide to present data derived out of the information. Or in another scenario, the app may use the information internally without presenting any output to the user. By way of example, the app may be a business-type app that evaluates the infonnation internally (such as extracting audit infonnation out of recordings).

[0044] An example of the unfixed approach is as follows. Assume the user’s natural language input is: “I went to the cinema last night”. The system could use the language model (LLM) 304 to generate any of the following unfixed schema 306:“[date-time: yesterday night, activity: entertainment, entity: cinema]”“[date: yesterday, time: night, activity: going out, entity: cinema]”“[date-time: yesterday night, activity: outing, location: cinema]”The schema can be parsed into a simple data structure like a dictionary, which may be maintained in memory of a client device (such as client device 108 of Fig. 1). The app 308 would only need to understand the simple data structure corresponding to the unfixed schema, rather than the more complex natural text from the user.

[0045] Fig. 3B presents a system 320 configured to implement an approach according to an arbitrary schema generated by an app. As shown here, natural language input 322, such as speech or text from a user, is input into a language model (LLM) 324 in the same manner as the unfixed schema approach discussed above. In this case, the model 324 receives, from an app or other program 326, an arbitrary schema 328, which may be stored in memory of the system (not shown). The arbitrary schema 328 is used by the model 324 to populate a set of schema fields 330 according to the received natural language input 322. The populated fields 330 can then be input to the app 326. Because the arbitrary schema 328 is generated by the app 326, it can be used to achieve an implied understanding of the user’s mood by selecting various schema items that can elicit information corresponding to mood.

[0046] As noted above, there can be multiple values per field, including additional sets of key: value pairs. For example, the user could log multiple meals in one sentence, each meal having multiple dishes and each dish with multiple ingredients. While the output can be a flat list with many key:value pairs that indicate this hierarchy, it could also be a hierarchy of key:value pairs, similar to JavaScript object notation (JSON).

[0047] As an example of the app-generated (arbitrary) schema, assume that the user’s input is: “I went to the cinema yesterday night, it was magical”, and assume that the schema is: “entertainment: [date (the date it happened in dd / mm / yy format), time (the 24 hour it happed ), activity (things like: going out, dating or watching), entity (like concert, venue, cinema), name (like movie name, restaurant name), mood (I was happy / sad / angry etc.)]”. In this case, the output from the model may be: ““Entertainment: [date: yesterday, time: night, activity: going out, entity: cinema, mood: amazed]”.

[0048] In the latter case, because the app is generating a tailored schema that is associated with particular aspects of the app, the user’s mood can be inferred from the user’s input. This is different from simply extracting text and identifying the mood. The implicit detection of fields such as moods / writing styles / affection / positive / negative uses the power of a trained LLM that “understands” these concepts through learning on a large corpus of data that included such concepts. The explicit schema unlocks this capability as the model gets instruction to look into those axes of implied information and not to focus only on explicit statements in the text. In view of this, while the app 326 is able to generate output 332 that may be a recommendation as discussed above, the output may be effectively enhanced based upon the mood- related information available in the populated schema fields 330. Alternatively or additionally, as discussed above the output may involve the app performing some action / operation based upon the processed information.

[0049] According to one aspect of the technology, the schema generated by the app can be used to elicit mood or other information from the natural language input. For instance, once a field in the arbitrary schema is defined such as (name, valid values, optional description, optional description of values), it can be extracted in a I way by using a dedicated prompt that asks that particular question. An example prompt is: “Given the following sentence, what [mood] can you understand I had”. In this example, valid values are only [happy, sad, apathetic, angry] . If none are found, the result returned would be NONE. Tire response can be then parsed to make sure it fits the requested schema and retired / naked as error. Most fields do not require a dedicated prompt and could use the general extractor and match its responses to the arbitrary schema the app expects. This can be done by passing over the general extractor response and semantically matching the responses to the expected API the app provided. Here, only the fields that do not haveresponses would need a prompt. Even then, there may be an easy model to predict if the general extractor would have extracted them if they existed and skip the prompt in that case.

[0050] The app may select specific items for the arbitrary schema in different ways. For instance, some apps may have a fixed schema that will match the various fields that the app stores for each joumaled / actionable item. This arbitrary schema is then written (e.g., by the app developer) into the syntax that the system provides. The system would then return the exact structure as defined in the arbitrary schema so that the information can be extracted from it.

[0051] The user input may be text or speech that is converted to text. In the first approach, without any predefined constraints the text is applied to the language model to generate (field, value) pairings for the unfixed schema. For instance, the unfixed schema for a sleep evaluation app may include pairings such as: (activity: sleep, date-time: last night, sleep quality: bad, cause: cold), or (activity: sleep, quality: bad, location: hotel). By way of example, the model may be trained to this “meta” schema of the parentheses and Field: Value structure. While a selection of field names and values may be provided, the model has leeway to improvise or otherwise unilaterally create such values. Some models, such as T5, are deterministic and return the same value each time, while other, such as LaMDA, are non-deterministic and may return a slightly different response every time. The app developer may look for the various options of field names and values to cover a set of the most common model results (e.g., the top 1-5 most common). Moreover, by default the model may return some value that it comes up with and is reasonable as a response. In cases where there is a scoped set of accepted responses (e.g., “[good, average, bad]”), the model can limit the responses to those values unless given the option to create other values, such as “[good, bad, etc.]”.

[0052] In one scenario, there may be no importance to the order of fields. However, in another scenario the order may have some relevance. For instance, in an example such as “I did 3 sets of pullups; 5 reps, 4 reps and then 2 reps”, it can be interpreted as (i) [(activity: gym, exercise: pullup, reps: 3), (activity: gym, exercise: pullup, reps: 4), (activity: gym, exercise: pullup, reps: 2)], (ii) (activity: workout, exercise: pullups, sets: 3, reps: 3, 4, 5), or (iii) (activity: gym, exercise: pullup, reps: 3, reps: 4, reps: 5). In the last case, (iii), the order would matter. According to one scenario, system could generate these according to the order they were spoken / written. The app consuming this would parse field by field to get the correct order. Thus, the system can arrange the fields according to relevance, order in which they are spoken or otherwise communicated, or based upon some other criteria.

[0053] The model-generated results (such as the unfixed schema 306 or the populated schema fields 330) may be parsed, e.g., using a regular expression (regex) or JSON parser to build a schema data structure. The data structure can then be read by the app to provide relevant information to the user. In this way, the app itself does not need to understand or process the natural language (e.g., text) of the user. The LLMoutput may be configured as (or preprocessed into) a valid JSON that a conventional JSON parser can work on. This may be suitable in many cases in the trained LLM. However, in an alternatively a simplified format of (field: value), (field, value) could be used instead. Parsing of this string into data classes can be done as part of the system offered, so the app will not need to know the model implementation details. By way of example only, parsing may be done using regex and string manipulations (search in strings, split according to pattern etc.).

[0054] In one scenario, after the schema is generated, a parsing component may parse the model output and export it in an app friendly format (e.g., JSON, dictionary, or any other data structure). If the model outputs JSON reliably then no component may be needed. The following is an example of what the (field, value) pairings may look like before and after parsing. before: (activity: eating, food: pizza, amount: 3 slices), (activity: eating, food: coke, amount: glass) after: joumaled items: List[Dict[str, str]] = [{"activity": "eating", "food": "pizza", "amount": "3 slices"}, {"activity": "eating", "food": "coke", "amount": "glass"}]

[0055] Fig. 3C illustrates another sy stem example 340 showing parsing of the schema data. Here, as with the systems in Figs. 3A-B, there is a natural language input 342, a language model 344, and an app 346. Depending on the type of schema approach, the app may generate an arbitrary schema 348 (equivalent to the approach in Fig. 3B), or the language model 344 may generate an unfixed schema (equivalent to the approach in Fig. 3A). Here, the schema is applied to the information from the natural language input 342, and a set of schema data 350 is generated by the model. The schema data 350 is then applied to a parser 352, and then parsed data is provided to the app 346, which outputs a recommendation or other information, or performs some action / operation 354 as described above. The main difference between Fig. 3B and Fig. 3C is that in the approach of Fig. 3B the app is expected to receive a string and do the parsing by itself. In the approach of Fig. 3C, the app receives an object (structs / classcs / cnums / strings / floats etc.) after it has been parsed by the system.

[0056] In one scenario, the language model is a pre-trained general language model, such as LaMDA or PaLM, which is not specifically configured to generate the schema. In another scenario, a more compact (e.g., T5) model may be used that is trained to generate the schema.

[0057] According to one aspect of the technology, one or both of fine-tuning and few-shot (or zeroshot) prompting may be performed on the model. Fine tuning is the operation of continuing the training of the model on a small selection of data that reflects on the task better than the initial training. This can make that task much better at the expense of tasks that are not in that set. As with training, the system would need a set of examples [before -> after] that reflects the real world work the model is planned to do. In this case, before would be a natural speech journaling sentence, to-do list, shopping list, logging, etc. and theoutput for each would be the items in the desired format, in our initial implementation (field: value). With this set, the model training is continued for much less iterations than the initial training. The end model will be a merge of the insights / language understanding and world knowledge of the initial training with the highly specific task to be performed.

[0058] Few-shot (and zero shot) prompting is a method in which an LLM is instructed to perform a specific operation using instructions (used in zero shot and optionally few shot) and examples (used in few shot). Typically, the text that needs analysis gets a prefix and postfix with the constant instructions and examples. The model analyzes the combined prompt and tries to answer similarly to how a person with zero background about the task would, so the instructions / examples need to leave no place for misunderstanding. An example prompt may be: “Find what meals were mentioned in the format [MEAL: DISH: INGREDIENT: AMOUNT: UNIT:], [MEAL: DISH: INGREDIENT: BARCODE: AMOUNT: UNIT:]...” By way of example, the input may be: “I've had 4 slices of a medium sized big johnny’s pepperoni pizza and half a can of diet cola for dinner”. The output may then be: [MEAL: dinner DISH: pizza INGREDIENT: big johnny’s medium pepperoni pizza AMOUNT: 4 UNIT: slice], [MEAL: dinner DISH: drink INGREDIENT: Diet Cola AMOUNT: 1 UNIT: can]. In another example, the input may be: “I've had at the bar 2 pints of stout beer, 3 handfuls of nachos and 4oz of sunflower seeds”. In this case, the output may then be: [MEAL: bar DISH: drink INGREDIENT: stout beer AMOUNT: 2 UNIT: pint], [MEAL: bar DISH: Nachos INGREDIENT: Nachos AMOUNT: 3 UNIT: handfill], [MEAL: bar DISH: Sunflower seeds AMOUNT: 4 UNIT: ounce]. According to one scenario, few shot prompting may be used to prepare data for use by the finetuning system to make the model smaller and improve the quality of this task.

[0059] In one configuration, the language model may run multiple times. Each run may be performed in order to find common fields and a hierarchy of the field-value pairs.

[0060] In one scenario, the model output may be recursive. For a complex sentence the model may need to perform multiple operations in one go: splitting the sentence, identifying the fields and values, and combining into one result. Hierarchal execution can simplify some of that operation, especially for small models. It does so by identifying subsentences that are independent from one another, and extracting by itself only fields that are common for all the sub sentences. In this scenario, the model would then run for each sub sentence recursively.

[0061] By way of example: “For lunch I ate an 8oz burger with a cup of fries and a glass of coke. For dinner I had falafel with a pint of beer.” The first result will be: (part 1: For lunch I ate an 8oz burger with a cup of fries and a glass of cola. Part 2: For dinner I had falafel with a pint of beer.). Then the model would run on part 1 and part 2 separately. With regard to part 1, the results could be: (meal: lunch, part 1 : 1 ate an8oz burger, part2: I ate a cup of fries, part3: I drank a glass of cola). Notice the output of each part if a full stand-alone sentence that does not include the common knowledge (lunch). These parts would be run again through the model. In this iteration, part 1 would be: (activity: eat, dish: burger, amount: 8, unit: oz). Running through all of the parts would lead to: (parts: [(meal: lunch, parts: [(activity: eat, dish: burger, amount: 8, unit: oz), (activity: eat, dish: fries, amount: 1, unit: cup), (activity: drink, dish: cola, amount: 1, size: cup)]), (meal: dinner, parts: [(activity: eat, dish: falafel), (activity: drink, dish: beer, amount: 1, unit: pint)])]). This tree structure can be flattened to the structure of the current model by duplicating the common elements: (meal: lunch, activity: eat, dish: burger, amount: 8, unit: oz), (meal: lunch, activity: eat, dish: fries, amount: 1, unit: cup), (meal: lunch, activity: drink, dish: cola, amount: 1, size: cup), (meal: dinner, activity: eat, dish: falafel), (meal: dinner, activity: drink, dish: beer, amount: 1, unit: pint).

[0062] Further to the descriptions above, a user may be provided with controls allowing the user to make an election as to both if and when systems, apps or other programs, or features described herein may enable collection of user information (e.g., journaling -type information relating to sleep tracking, wellness, exercise or other activities, as well as or in addition to social network, social actions, profession, a user’s preferences, or a user’s current location), and if the user is sent content or communications from a server. In addition, certain data may be treated in one or more ways before it is stored or used, so that personally identifiable information is removed. For example, a user’s identity may be treated so that no personally identifiable information can be determined for the user, or a user’s geographic location may be generalized where location information is obtained (such as to a city, ZIP code, or state level), so that a particular location of a user cannot be determined. Thus, the user may have control over what information is collected about the user, how that information is used, and what information is provided to the user.

[0063] Figs. 4A-B illustrate one example of how the app may use the schema information to provide a recommendation. As shown in view 400 of Fig. 4A, the app may be used to track sleep and provide sleep score information (e.g., within a range of poor, fair and great). Here, based upon the schema information, the app may present a suggestion to the user to “Consider going to sleep 15 minutes earlier to enhance your sleep score”, as shown in view 410 of Fig. 4B.

[0064] Figs. 5A-B illustrate another example providing a recommendation associated with an exercise app. Here, as shown in view 500 of Fig. 5A, the graphical interface of the app presents current exercise information such as cardio activity, level of workout intensity, oxygen consumption (e.g., VO2 level) or other information. In this case, based upon the schema information, the graphical interface (and / or audio information) may suggest to the user to “Increase your pace by 15 sec / mi to boost your workout effectiveness”. In this example, the interface may visually show the boost, viathe dashed arrow and hatched line. By way of example, the user’s mood, as determined by the schema processing, may indicate that theuser is feeling good about their workout so far, and this may be factored into the type of recommendation presented to the user. The interpretation of the mood can be used to select (or alter) what is recommended to the user.Example Computing Architecture

[0065] The schema-based technology discussed herein may be implemented using one or more tensor processing units (TPUs), CPUs or other computing devices in accordance with the features disclosed herein. One example computing architecture is shown in Figs. 6A and 6B. In particular, Figs. 6A and 6B are pictorial and functional diagrams, respectively, of an example system 600 that includes a plurality of computing devices and databases connected via a network. For instance, computing device(s) 602 may be implemented as a cloud-based server system. Databases 604, and 606 may store, e g., one or more schema, apps and / or trained language models, respectively. The server system may access the databases via network 608. Client devices may include one or more of a desktop computer 610 and a laptop or tablet PC 612, for instance that provide natural language input from a user. Other client devices may include handheld devices including a personal communication device such as a mobile phone or PDA 614 or a tablet 616. Another example is a wearable device 618 such as a smartwatch (or head-mounted display device).

[0066] As shown in Fig. 6B, each of the computing devices 602 and 610-618 may include one or more processors, memory, data and instructions. The memory stores information accessible by the one or more processors, including instructions and data (e.g., language models) that may be executed or otherwise used by the processor(s). The memory may be of any type capable of storing information accessible by the processor(s), including a computing device -readable medium. The memory is a non-transitory medium such as a hard-drive, memory card, optical disk, solid-state, etc. Systems may include different combinations of the foregoing, whereby different portions of the instructions and data are stored on different types of media. The instructions may be any set of instructions to be executed directly (such as machine code) or indirectly (such as scripts) by the processor(s). For example, the instructions may be stored as computing device code on the computing device-readable medium. In that regard, the terms “instructions”, “modules” and “programs” may be used interchangeably herein. The instructions may be stored in object code format for direct processing by the processor, or in any other computing device language including scripts or collections of independent source code modules that are interpreted on demand or compiled in advance.

[0067] The processors may be any conventional processors, such as commercially available CPUs, TPUs, graphical processing units (GPUs), etc. Alternatively, each processor may be a dedicated device such as an ASIC or other hardware-based processor. Although Fig. 6B functionally illustrates the processors, memory, and other elements of a given computing device as being within the same block, suchdevices may actually include multiple processors, computing devices, or memories that may or may not be stored within the same physical housing. Similarly, the memory may be a hard drive or other storage media located in a housing different from that of the processor(s), for instance in a cloud computing system of server 602. Accordingly, references to a processor or computing device will be understood to include references to a collection of processors or computing devices or memories that may or may not operate in parallel.

[0068] The computing devices may include all of the components normally used in connection with a computing device such as the processor and memory described above as well as a user interface subsystem for receiving input from a user and presenting information to the user (e.g., text, imagery, speech or other audio, videos and / or other graphical elements). The user interface subsystem may include one or more user inputs (e.g., at least one front (user) facing camera, a mouse, keyboard, touch screen and / or microphone) and one or more display devices (e.g., a monitor having a screen or any other electrical device that is operable to display information (e.g., text, imagery and / or other graphical elements). Other output devices, such as speaker(s) may also provide information to users.

[0069] The user-related computing devices (e.g., 610-618) may communicate with a back-end computing system (e.g., server 602) via one or more networks, such as network 608. The network 608, and intervening nodes, may include various configurations and protocols including short range communication protocols such as Bluetooth™, Bluetooth LE™, the Internet, World Wide Web, intranets, virtual private networks, wide area networks, local networks, private networks using communication protocols proprietary to one or more companies, Ethernet, WiFi and HTTP, and various combinations of the foregoing. Such communication may be facilitated by any device capable of transmitting data to and from other computing devices, such as modems and wireless interfaces.

[0070] In one example, computing device 602 may include one or more server computing devices having a plurality of computing devices, e.g., a load balanced server farm or cloud computing system, that exchange information with different nodes of a network for the purpose of receiving, processing and transmitting the data to and from other computing devices. For instance, computing device 602 may include one or more server computing devices that are capable of communicating with any of the computing devices 610-618 via the network 608. The computing device 602 may implement a back-end server (e.g., a cloudbased server that supports one or more apps), which receives queries from desktop computer 610, laptop / tablet PC 612, mobile phone or PDA 614, tablet 616 or wearable device 618.

[0071] One or more language models may be stored on the server or the client device(s), and the app(s) may be hosted by the server or run locally on the client device(s) . In any of these arrangements, the natural language input from the user(s) is processed by a given language model according to a schema as describedabove, in order to generate schema data that can be readily evaluated by the app in order to generate selected output.

[0072] Resultant information (e.g., populated schema fields, schema data, parsed data or other data derived from the approaches discussed herein) may be shared by the server with one or more of the client computing devices. Alternatively or additionally, the client device(s) may maintain their own databases, models, apps, etc.

[0073] Fig. 7A illustrates an exemplary method 700 in view of the above discussion. The method includes, at block 702, receiving, by a computing system, a natural language input of a user. Then at block 704 the method includes generating, by the computing system, a dynamic schema for use with an application running on the computing system. At block 706 the method includes applying, by the computing system using a trained language model, the dynamic schema to the received natural language input to generate a set of schema data having one or more corresponding field and value pairings. The set of schema data is provided to the application at block 708. Then at block 710, the method includes generating, by the application, an output based upon at least one of the field and value pairings of the set of schema data. And at block 712 the method includes presenting the output to the user.

[0074] Fig. 7B illustrates another exemplary method 720 in view of the above discussion. The method includes, at block 722, receiving, by a computing system, a natural language input of a user. Then at block 724 the method includes generating, by the computing system, a dynamic schema for use with an application running on the computing system. At block 726, the method includes applying, by the computing system using a trained language model, the dynamic schema to the received natural language input to generate a set of schema data having one or more corresponding field and value pairings. And at block 728, the method further includes providing the set of schema data to the application in order for the application to either generate an output for presentation to the user based upon at least one of the field and value pairings of the set of schema data, or perform an operation based upon at least one of the field and value pairings of the set of schema data.

[0075] Although the technology herein has been described with reference to particular embodiments, it is to be understood that these embodiments are merely illustrative of the principles and applications of the present technology. It is therefore to be understood that numerous modifications may be made to the illustrative embodiments and that other arrangements may be devised without departing from the spirit and scope of the present technology as defined by the appended claims.

Claims

CLAIMS1. A method, comprising: receiving, by a computing system, a natural language input of a user; generating, by the computing system, a dynamic schema for use with an application running on the computing system; applying, by the computing system using a trained language model, the dynamic schema to the received natural language input to generate a set of schema data having one or more corresponding field and value pairings; providing the set of schema data to the application; generating, by the application, an output based upon at least one of the field and value pairings of the set of schema data; and presenting the output to the user.

2. The method of claim 1, wherein the dynamic schema is an unfixed schema generated by the trained language model.

3. The method of claim 1, wherein the dynamic schema is an arbitrary schema generated by the application.

4. The method of claim 3, wherein the arbitrary schema generated by the application is configured to elicit information corresponding to a mood of the user.

5. The method of claim 4, wherein the output provides a recommendation to the user based upon the information corresponding to the mood.

6. The method of claim 1, wherein: providing the set of schema data to the application includes: parsing the set of schema data to generate a parsed set of data; and providing the parsed set of data to the application; and generating the output is based upon one or more field and value pairings of the parsed set of data.

7. The method of claim 1, wherein the natural language input is textual input received from the user, and the language model implements a text-to-text transformer.

8. The method of claim 1, wherein the natural language input is speech input received from the user, and the language model implements a speech-to-text transformer.

9. The method of claim 1, wherein the language model is a non-deterministic model.

10. The method of claim 1, wherein fields in the set of schema data are arranged in order according to a relevance criterion.

11. The method of claim 1 , wherein applying the dynamic schema to the received natural language input includes performing a recursive operation on different parts of a sentence in the natural language input to generate the set of schema data.

12. A method, comprising: receiving, by a computing system, a natural language input of a user; generating, by the computing system, a dynamic schema for use with an application mnning on the computing system; applying, by the computing system using a trained language model, the dynamic schema to the received natural language input to generate a set of schema data having one or more corresponding field and value pairings; and providing the set of schema data to the application in order for the application to either generate an output for presentation to the user based upon at least one of the field and value pairings of the set of schema data, or perform an operation based upon at least one of the field and value pairings of the set of schema data.

13. The method of claim 12, wherein the dynamic schema is an unfixed schema generated by the trained language model.

14. The method of claim 12, wherein the dynamic schema is generated based upon information received from the application.

15. The method of claim 12, wherein, providing the set of schema data to the application includes: parsing the set of schema data to generate a parsed set of data; andproviding the parsed set of data to the application.

16. The method of claim 12, wherein fields in the set of schema data are arranged in order according to a relevance criterion.

17. The method of claim 12, wherein applying the dynamic schema to the received natural language input includes performing a recursive operation on different parts of a sentence in the natural language input to generate the set of schema data.

18. A computer system, comprising: memory configured to store a trained language model; and one or more processors operatively coupled to the memory, the one or more processors being configured to: receive, via a user interface of the computer system, a natural language input of a user; generate a dynamic schema for use with an application running on the computing system; apply, using the trained language model, the dynamic schema to the received natural language input to generate a set of schema data having one or more corresponding field and value pairings; cause the application to generate an output based upon at least one of the field and value pairings of the set of schema data; and cause the user interface to present the output to the user.

19. The computer system of claim 18, wherein the dynamic schema is either an unfixed schema generated by the trained language model or an arbitrary schema generated by the application.

20. A computer system, comprising: memory configured to store a trained language model; and one or more processors operatively coupled to the memory, the one or more processors being configured to: receive, via a user interface of the computing system, a natural language input of a user; generate a dynamic schema for use with an application; apply, using the trained language model, the dynamic schema to the received natural language input to generate a set of schema data having one or more corresponding field and value pairings; and provide the set of schema data to the application in order for the application to either generatean output for presentation to the user based upon at least one of the field and value pairings of the set of schema data, or perform an operation based upon at least one of the field and value pairings of the set of schema data.

Images