Content generation and assessment

Abstract

The disclosure relates to a computer-implemented method that comprises receiving a content item; providing the content item as input to a trained machine learning model; receiving, as output from the trained machine learning model, an indication of an emotion the content item will evoke in a predetermined user orgroup. The method may be used as part of a content moderation system to moderate content, or as a filter of the output of a generative model. The disclosure also relates to a computer-implemented method that comprises providing, as input to a first machine learning model, an input indicative of a desired emotion; receiving, as output from the first machine learning model, one or more data items for inclusion in a prompt for a second machine learning model, the second machine learning model being a generative model; generating a prompt including the one or more data items; providing the prompt as input to the second machine learning model; and receiving, as output from the second machine learning model, a content item reflective of the desired emotion. Corresponding training techniques are also provided for these methods.

Description

CONTENT GENERATION AND ASSESSMENT

[0001] Generative artificial intelligence (Al) models that generate content, for example based on textual prompts, are becoming increasingly commonplace. Examples of such models include ChatGPT, which generates text in response to a prompt, and DALL-E, which generates images in response to a prompt. Such models are trained on massive datasets and are typically accessed over the internet via a suitable application programming interface (API). These models are typically general purpose, in the sense that they have been trained to perform a broad task, in contrast to earlier machine learning models trained to carry out more specific narrow classification or generation tasks. For example, ChatGPT is trained to carry out general text generation, rather than e.g. to act as a chatbot or some other more specific purpose.

[0002] Emotion plays a significant role in the content consumed by users. This may be in terms of the emotion displayed by the author of the content (e.g. a message that is conveying anger) and in the emotion a piece of content evokes in a user. It is therefore desirable to generate content that either evokes or avoids a particular emotion. This may be desirable e.g. to generate suitable age appropriate content, such as content that will not scare children, or content that will engage a target audience so that a suitable safety message or warning is heeded. It may also be desirable to assess the emotion that content will evoke in a user (or group of users), with such assessments finding utility in generating feedback for generation of future content, or moderating existing content. Accordingly, the present disclosure provides techniques for generating and assessing content that takes into account these issues.

[0003] According to first aspect of the disclosure, there is provided a computer- implemented method comprising:receiving a content item; providing the content item as input to a trained machine learning model; receiving, as output from the trained machine learning model, an indication of an emotion the content item will evoke in a predetermined user or group.

[0004] The trained machine learning model may have been trained on content items relating to the predetermined user or group. For example, the content items may be authored, commented on or liked by the user or group.

[0005] The indication of the emotion the content item will evoke may comprise a numerical score indicative of a level of emotion invoked. The trained machine learning model may return an indication of a plurality of emotions, suitably each represented with a numerical score indicative of a level of the emotion invoked.

[0006] The method may comprise moderating the content item based on the indication of the emotion. It will be understood "based on" in this context may refer to the emotion being one factor taken into account by a suitable content moderation system. Suitable content moderation techniques are described in detail in the applicant's co-pending International Patent Application no PCT / EP2022 / 078593, the contents of which are incorporated herein by reference in their entirety. For example, the emotion as determined may be one of the outputs of the machine learning models discussed therein, which is combined to generate a moderation result. The output of the models (including the output of the trained machine learning model) may be combined according to a combination logic. The method may comprise receiving a behaviour profile, wherein the combination logic is determined based on the behaviour profile. The content moderated may be social media content, content from organization-internal communication systems (e.g. intranets, group chats on platforms such as Teams or Slack), messages in live chats or gaming systems and so on. This technique of assessing the emotion evoked by content may assist in the accurate identification of content that is undesirable.

[0007] In further examples, the content item may be the output of a generative model. That is to say, the present techniques may act as a wrapper or filter applied to the output of a generative model. This may allow a user to be provided with content that meets certain emotional ranges or thresholds. For example, "scary" content may be filtered for sensitive users. Accordingly, the method may comprise receiving an input prompt, providing the input prompt to a generative model, and receiving the content item from the generative model.

[0008] In still further examples, the indication of the emotion the content item will evoke may act as training or fine-tuning data for a generative model. Accordingly, the method may comprise training the generative model using training data comprising the content item and the indication. The training technique may be an "online" training technique.

[0009] In some examples, the method may comprise comparing the indication of the emotion, suitably in the form of a numerical score, to one or more predetermined threshold. Accordingly, the indication can be checked against a "normal" emotional response for that user or group of users. In some examples, the thresholds may vary depending upon various parameters. For example, the thresholds may vary with the time of day, such that a user's "normal" happiness range in a morning may differ from that in an evening. Other parameters may include time of year, weather conditions etc.

[0010] In some examples, the method may be applied to a plurality of content items associated with a user (e.g. generated with, read, etc) overtime, so as to track changes in emotional state.

[0011] According to a second aspect there is provided a computer-implemented method comprising:receiving a training data set, the training data set comprising a plurality of a training content items, each training content item having a training label reflective of an emotion evoked in a user or group of users by the respective content item; training the machine learning model using the training data set to output, in response to receiving an input content data item, an indication of an emotion the input content item will evoke in the user or group of users.

[0012] The training data set may comprise any of the features defined below in relation to the fourth aspect.

[0013] According to a third aspect of the disclosure, there is provided a computer- implemented method comprising: providing, as input to a first machine learning model, an input indicative of a desired emotion; receiving, as output from the first machine learning model, one or more data items for inclusion in a prompt for a second machine learning model, the second machine learning model being a generative model; generating a prompt including the one or more data items; providing the prompt as input to the second machine learning model; and receiving, as output from the second machine learning model, a content item reflective of the desired emotion.

[0014] This technique may therefore employ a trained machine learning model (the first machine learning model) to act as a prompt generator, which is able to generate suitable input for a generative machine learning model. This effectively provides a means of leveraging generally-trained generative models to provide output that evokes a desired emotional response.

[0015] In some examples, the generative model is a model configured to generate text. Consequently, the content item may be text reflecting the desired emotion. In other examples, the generative model may generate a content item in the form of an image. However, a wide variety of generative models may be employed in conjunction with the present techniques, including video generation models, audio generation models, multimedia generation models and so on. Example models include GPT-3, GPT-3.5 turbo, GPT-4, GPT-4o, ChatGPT, and Dall-E.

[0016] The input indicative of a desired emotion may comprise user input, for example in the form of text, speech or the like (e.g. "give me an image that will make me happy"). In some examples, the desired emotion is drawn from a predetermined set of emotions, which may be a set of emotions defined according to suitable academic psychological research.

[0017] The generated prompt make comprise additional text data beyond the one or more data items generated by the first machine learning model. For example, the prompt may take the form of a template, wherein the generated data items are inserted into the template. The generated data items may form "shots" according to a one shot or few shot learning technique.

[0018] According to a fourth aspect of the disclosure, there is provided a computer- implemented method of training a first machine learning model comprising: receiving a training data set, the training data set comprising a plurality of a content items, each content item having a training label reflective of an emotion associated with the respective content item; training the first machine learning model using the training data set to output, in response to receiving an input indicative of a desired emotion, one or more data items for inclusion in a prompt to be input to a generative machine learning model,wherein, the one or more data items, when included in the prompt, cause the generative machine learning model to generate a content item reflective of the desired emotion.

[0019] Training the first machine learning model may comprise generating and inputting probing prompts to the generative machine learning model. The output of the probing prompts may be used in calculating a loss function used in training the machine learning model.

[0020] The first machine learning model may be any suitable machine learning model capable of generating or selecting data items to include in the prompt. The first machine learning model may be a neural network. The first machine learning model may be pre-trained, such that the step above of training the first machine learning model may comprise fine-tuning the first machine learning model. An example model may be a BART model.

[0021] The content items in the training data set may be social media posts or messages. The content items may relate to a particular user, for example having being authored, commented on or liked by the user. The content items may relate to a group of users. For example, the content items may have been authored, commented on or liked by users in the group.

[0022] The training data may be structured training data. For example, the training data may take the form of a graph network. Each content item may form a node of the graph network. The graph network may comprise edges linking nodes. The edges may represent a temporal relationship between posts. The edges may therefore encode a timeline of the content items.

[0023] The method may comprise extracting the training data set from a social media data store. The data store may for example comprise a data lake extracted from a social media network. Extracting the training data set may comprise extracting content items authored, commented or liked by a predetermined user or group.

[0024] The optional features of any of the aspects above may be combined in any combination. Models that act both to generate content that evokes a predetermined emotion and that also assess the emotion of input content are explicitly envisaged.

[0025] The disclosure further extends to computer systems, devices, computer- readable media and computer program products having corresponding features to the methods disclosed herein.Brief of the

[0026] To assist understanding of the present disclosure and to show how embodiments may be put into effect, reference is made by way of example to the accompanying drawings in which:

[0027] Figure 1A is a schematic diagram of an example computing system for generating content.

[0028] Figure IB is a schematic illustration of a generated prompt for a generative model.

[0029] Figure 2 is a schematic diagram of an example method of training a model to generate prompts for a generative model.

[0030] Figure 3 is a schematic diagram of an example computing system for assessing content.

[0031] Figure 4A and 4B are schematic diagrams of example computing systems incorporating the techniques of Figure 3.

[0032] Figure 5 is a schematic diagram of an example moderation system.Detailed

[0033] Figure 1 illustrates an example computing system 100.

[0034] The computer system 100 may comprise one or more computers, each having memory and one or more processors. The memory stores instructions which when executed by the processors carry out the methods discussed herein. The memory also stores, either transiently or permanently, any other information required for the operation of the system. The computer system 100 may for example take the form of a server computer, for example hosted in the cloud.

[0035] The system 100 receives an input 101 from a user U, which indicates an emotion desired in generated content. For example, the user U may provide input at a device D (e.g. a smart phone, tablet computer or personal computer), which is connected to the system 100 via a suitable network connection. The input may for example include text input, speech input (which may be converted to text by the device D or system 100), the selection of buttons or other user interface elements or a combination thereof. For example, the input may be text such as "Generate an image that will make me happy"; "Write a letter to person X that will make them happy" etc.

[0036] Upon receipt of the input 101, the system 100 provides the input 101 to trained machine learning model 110. The machine learning model 110 may be any suitable machine learning model such as a neural network. An example model may be a BERT model.

[0037] The trained machine learning model 110 is trained to generate data items to include in a prompt 121 for submission to a generative model 200. In the main, the data items comprise text strings for inclusion in a text prompt submitted to a generative model 200. However, it is to be understood that in some examples, the data items may be images, audio, video, structured data (e.g. tabular data) or any other suitable data. In this context, generating the data items may include selecting suitable data items from a plurality of stored data items. The training of the model 110 will be discussed in more detail below with respect to Figure 2.

[0038] In some examples, the data items 111 are "shots" - i.e. labelled examples of content and associated emotion. The technique may therefore be a "one-shot" or "few-shot" learning technique in which one or a small number (e.g. < 10) labelled examples are included in a prompt for a generative model 200. The shots may be selected from a store of shots, or generated from scratch by the model 110.

[0039] Once generated, the system 100 constructs the prompt 121, as represented by block 120. In the example of a text prompt, this may involve concatenating the data items 111 with other suitable text that makes up the prompt 121. For example, the system 100 may store a template prompt 121, into which the data items 111 are inserted. The generation of the prompt 121 may also involve inserting other data into the prompt 121 extracted from or determined from the input 101.

[0040] Figure IB illustrates an example prompt 121. The prompt 121 includes a preamble 121-1 which instructs the generative model 200 of its task. The prompt 121 may include an indication of the emotion desired 121-2. This may be an output of the model 110, or may be otherwise extracted from the input 101, for example using heuristics or a trained intention detection model.

[0041] The prompt 121 then includes slots 121-4a-n, which are filled by the generated data items 111. The slots 121-4 may be introduced by suitable wording 121- 3, forming part of the template. The prompt 121 may also include a footer 121-5, which may for example repeat the task required or include any other suitable information required for the generative model 200.

[0042] The system 100 then submits the constructed prompt to the generative model 200. As discussed hereinabove, the generative model 200 may generate text, audio, images, video or any other suitable content, or a combination thereof. For simplicity of explanation, in the following discussion it will be assumed that the model is a text generation model. For example the model 200 may be a large language model (LLM) such as GPT-4, provided by Open Al®. The generative model 200 may be hostedremotely from the system 200 and accessible via a suitable API. However, it may also be the case that the system 100 and model 200 are co-hosted.

[0043] In response to the submitted prompt 121, the model 200 returns generated content 131 evoking the emotion specified in input 101.

[0044] This is then delivered to the user U, as represented by box 130. For example, this may involve sending the content 131 to the user device U. In other examples, it may involve uploading the content 131 to another system (e.g. a social media network or a document repository associated with the user).

[0045] Turning to Figure 2, an example technique fortraining the model 100 will now be discussed.

[0046] Data may be initially received in a relatively unstructured form, generally indicated by reference numeral 310, for example in the form of a data lake or other data structure. The data 310 may include content items 311, such as social media posts, chat submissions, and the like. The data may be collated from different sources, such that a plurality of collections of data 310 are received.

[0047] The data 310 may be processed to extract data pertaining to a particular user. For example, content items 311 that represent content authored, commented on or liked by the user are extracted. In other examples, the content items may relate to a group of users. For example, users in a particular demographic group, a particular geographic location, or having particular interests may be extracted.

[0048] The data 310 is then processed into a structured form. For example, a graph network 320 is generated from the data 310. The graph network 320 comprises nodes 321 and edges 322. The nodes may each represent a content item 311 (and optionally also other suitable metadata pertaining the to the user and / or content item 311). The edges may encode a temporal relationship between content items 311, so that thedata forms a time series. The content items 311 / nodes 321 have associated training labels that indicate the emotion present in the content item 311 / 321.

[0049] The graph network 320 is then used to train the model 110 to generate or select data items 111 for inclusion in a prompt 121, so as to evoke the desired emotion in the user or group represented in the training data. The machine learning model 110 may be pre-trained to some extent, such that the step of training the machine learning model 110 may comprise fine-tuning the machine learning model 110. An example model may be a BART model, or any other trainable model that has an encoder capability that allows it to generate text.

[0050] In terms of the way in which the graph network 320 is utilized to train the model 110 it may for example be the case that the model is configured to receive text- only input rather than structured input in the form of graph data. Accordingly, the system 100 may parse the graph-based data and convert it into a form suitable for input to the machine learning model 110 during training. For example, the graph 320 or parts thereof are represented in a textual form interpretable by the model 110, for example by expressing the links in text or with symbols (e.g. "->") representing the links.

[0051] The training process may involve generating and inputting probing prompts to the generative machine learning model 200. This may assist in determining the effectiveness of particular data items in generating the desired response from the model 200. For example, the output of the probing prompts may be used in calculating a loss function used in training the machine learning model 100.

[0052] It will be understood that techniques exist for automatic prompt generation and retrieval-augmented generation (RAG) that can be utilized in training the model 200. For example, Lewis, Patrick, et al. "Retrieval-augmented generation for knowledge-intensive nip tasks." Advances in Neural Information Processing Systems 33 (2020): 9459-9474 sets out a general-purpose recipe for training RAG models toretrieve documents for use in input to generative models. Zhou, Yongchao, et al. "Large language models are human-level prompt engineers." arXiv preprint arXiv:2211.01910 (2022) sets out techniques for generating prompts automatically (a so called "Automatic Prompt Engineer"). The disclosures of these documents are incorporated herein by reference in their entirety.

[0053] Figure 3 illustrates another example computing system 400. The computing system 400 may have a similar structure to the computing system 100 discussed above. However, in contrast to the computing system 100, the computing system 400 is configured to assess the emotions an input piece of content 401 will evoke in a user or user group represented in training data, using a trained machine learning model 410.

[0054] The trained machine learning model 400 may be trained on training data similar to that discussed above in relation to Figure 2. For example, it may be trained on content items relating to a particular user or group, so that it can determine the emotion evoked in that user or group. For example, the content items may be authored, commented on or liked by the user or group.

[0055] As illustrated in Figure 3, the output 421 includes a numerical score indicative of a level of emotion invoked by the input content 401. In the example show, the model returns scores for each of a plurality of emotions, (emotion 1 to emotion n). The emotions of the model may be based on academic psychological research quantifying or otherwise identifying emotions experienced by humans. A first example scheme includes 27 emotions. Another example scheme includes approximately 300 emotions. Yet another scheme, approximately 30,000 emotions are captured. This emotional assessment may have a multitude of uses. Two uses are illustrated in Figures 4A and 4B.

[0056] In Figure 4A, the system 400 is employed as a filter on the content generated by a generative model 200. That is to say, the system 400 acts as a "wrapper" aroundthe generative model 200, which ensures generated content evokes desired emotion in desired ranges in the end user U. For example, "scary" content may be filtered for sensitive users. Although not shown, the method may comprise receiving an input prompt from the user and providing the input prompt to a generative model, which causes the generation of the content to be filtered.

[0057] The ranges may not be static - for example, a user's tolerance to certain content may vary over time (e.g. across a day). The ranges may be determined from the training data (i.e. based an average level of certain emotions at certain times of day) or other data.

[0058] In some examples, the system 400 provides feedback to the generative model 200. This may take the form of additional training or fine-tuning data for the generative model. Accordingly, the system may train the generative model using training data comprising the content item and the indication. The training technique may be an "online" training technique.

[0059] In Figure 4B, the system 400 is incorporated into a moderation system M. The system may take moderation actions based on the output 421. Suitable content moderation techniques are described in detail in the applicant's co-pending International Patent Application no PCT / EP2022 / 078593, the contents of which are incorporated herein by reference in their entirety. For example, the emotion as determined may be one of the outputs of the machine learning models discussed therein, which is combined to generate a moderation result. The content moderated may be social media content, content from organization-internal communication systems (e.g. intranets, group chats on platforms such as Teams or Slack), messages in live chats or gaming systems and so on. In the example shown, the system M receives a post from a social media network 5000, and in response provides moderation input to the social network 5000. This may involve hiding, deleting or reporting a post so that the user U is unable to see it.

[0060] In terms of the efficacy of the techniques discussed above with respect to Figures 1-4B, the following experiments were conducted.

[0061] Firstly, a BERT classifier was trained a set of 27 emotions, using ground truth- labelled gaming domain data. The training set comprised text training data labelled with one of the following 27 emotions: [1. Admiration, 2. Amusement, 3. Anger, 4. Annoyance, 5. Approval, 6. Caring, 7. Confusion, 8. Curiosity, 9. Desire, 10. Disappointment, 11. Disapproval, 12. Disgust, 13. Embarrassment, 14. Excitement, 15. Fear, 16. Gratitude, 17. Grief, 18. Joy, 19. Love, 20. Nervousness, 21. Optimism, 22. Pride, 23. Realization, 24. Relief, 25. Remorse, 26. Sadness, 27. Surprise], The source of the data included YouTube® comments, Instagram® and / or Facebook® messages, and ad dialogs. Each class comprised lOk-lOOk of training data items, with some training data items being furthermore synthetically generated using GPT4 to create approximately balanced number of samples for each label.

[0062] The BERT classifier was then employed as filter of output from a generative model, specifically GPT-4o, in a similar manner to as shown above in FIG. 4A. In the experiments, GPT-4o was provided with zero-shot prompts to generate content having particular emotions for tasks such as advert generation, tweet generation, header / title generation for YouTube videos. An example prompt for the advert generation task was "Create an ad script summary which evokes Emotionl, Emotion2, Emotion3 in intro, body and conclusion respectively" . GPT-4o was iteratively prompted with the temperature set so that a degree of randomness was provided in the responses. By providing the BERT classifier as a filter to the output and selecting the best result from the multiple iterations, the output was found to improve emotional awareness compared to the raw GPT-4o output by 5-10% on average.

[0063] This process was found to be further improved by employing few-shot prompts with examples retrieved from the above-mentioned training set using an RAG technique. Particularly, a further increase of performance by another 2-10% was achieved depending on the target emotion. Performance particularly improved onemotions that are less subtle such as humor, when compared to more subtle emotions like amazement. In addition, it was found that performance improved more on longer texts (i.e. >300 token length, <500 token length) than on shorter ones, likely owing to better context.

[0064] Figure 5 illustrates an example 500 of the moderation system M in more detail, and the description below discusses relevant aspects of Figure 5 illustrates another example computer system 500 for content moderation.

[0065] The system 500 includes a content analysis module 510, which may also be referred to as a client application handling module 510 herein.

[0066] Figure 5 also shows one or more client applications 501, which take the form of online platforms, such as social media platforms. The system 500 communicates with the client applications 501 over a suitable network, such as the internet.

[0067] The API 503 is a programmatic interface offered by the client application 501, via which the account of a particular user may be managed. This may include accessing messages sent or received by the user, and accessing messages otherwise visible to the user accessing the client application, such as comments on videos, photos or other media, messages visible on the timeline and so on. By managing the user's account via the API 503, offensive content may be hidden, deleted or otherwise managed so that the user accessing the client application via another channel (e.g. on an application installed on a mobile device or via a web browser) is not exposed to the offensive content.

[0068] The API 503 is an application-specific API, specific to the particular client application 501. For example, the API 503 may be the Twitter API if the client application 501 is Twitter, the Facebook API if the client application 501 is Facebook, and so on.

[0069] The content analysis module 510 may comprise a plurality of integration units511, each integration unit 511 representing an interface between the API 503 offeredby a specific platform 501 and the module 510. The integration unit 511 may effectively translate structured content received according to a format set by the API 503 creator, into a standardised format for internal processing by the module 510.

[0070] It will be appreciated that the communication format employed by the APIs 503 / integration units 511 may vary. In one example, the API receives requests and returns data encoded in JavaScript Object Notation (JSON), but other formats may be employed.

[0071] Figure 5 illustrates an input message 505 received by a first integration unit 511a of a content analysis module 510 from the API 503. The input message 505 includes a content item shared to the online platform, which is an example of input data representative of a media post or message made on the online platform. The message 505 may be received as a result of the content analysis module 510 accessing the API 503 to retrieve newly posted content items or content items that are newly visible to the account holder. This may for example occur on a regular basis (i.e. periodically). Receipt of the input message 505 to the first integration unit 511a is represented by a double struck line. The input message 505 may further comprise contextual data, including one or more of the type of post or message, the user of the account (i.e. the account holder), the user making the post, a timestamp associated with the post and any other pertinent information relating to the content item.

[0072] The receiving integration unit 511a may then perform pre-processing operations on data received via the input message, for example to extract the content item from the input message 505 or to format the data for further analysis.

[0073] The content analysis module 510 includes a data store (e.g. a database), which stores an association between a user (i.e. the account holder) and one of a plurality of behaviour profiles. Each behaviour profile represents a set of viewing preferences or tolerances of users associated with the behaviour profile, which may be taken into account by the machine learning content moderation module 530 as discussed in more detail below.

[0074] For example, one behaviour profile may be associated with users that are sports people, or users interested in a particular sport. The online culture associated with particular sports (e.g. football) may be such that particular types of language (e.g. swear words) are relatively commonplace and users engaged with such content are unlikely to be offended by that language. Another behaviour profile may represent corporate users, operating in an environment or culture where there is a low tolerance for offensive content. Yet another behaviour profile may indicate a gaming user, who may again have a different set of expectations in relation to the level of desirable content moderation, and different tolerances in terms of exposure to potentially offensive content. It will be understood that a wide variety of profiles may be defined, so as to provide different tolerance levels for different users. In some examples, a profile may be associated with only a single user, providing a bespoke level of tolerance for that user.

[0075] Upon receiving content of the input message 505, the module 510 (represented in FIG 5 by behaviour group retriever 513) may retrieve the behaviour profile associated with the user (i.e. account holder) from the database 515.

[0076] The content analysis module 510 further includes a moderation request generator 517, which generates a moderation request to be supplied to the machine learning content moderation module 530. The moderation request 520 includes the content item 521 to be moderated and the retrieved behaviour group 523.

[0077] The moderation request 520 may take the form of a call to an API 531 exposed by the machine learning content moderation module 530. For example, the API call or request may be in JSON format.

[0078] Turning to the machine learning content module 530, the module 530 forms a processing pipeline for moderating content 521, in view of the assigned behaviour group 523.

[0079] The machine learning content module 530 comprises the API 531. The API531 acts as an interface between the content analysis module 510 and the machinelearning content module 530. As discussed in more detail below, the API 531 may also be configured to receive requests from systems other than the content analysis module 510.

[0080] In one example, the API 531 is a stateless API. That is to say that the machine learning content module 530 receives moderation requests 520 via the API 531 and responds to the requests 520, but does not store any client session data or other data about previous requests 520 made by the content analysis module 510. This may be possible because the content analysis module 510 is managing the interaction with the client applications 501 and maintaining state.

[0081] A first module in the processing pipeline of the emotion Al module 530 is an Al model set 532, at which the content 521 may be received. The Al model set 532 comprises one or more Al model 533, each model 533 trained to assess a particular aspect of the received content 521. The Al model set 532 includes the model 400 discussed above in relation to Figure 3.

[0082] In one example, each Al model 533 is not supplied with the behaviour group 523 associated with the received content 521. That is to say that the Al models 533 receive the content 521 as input, but not the behaviour group.

[0083] Reference numeral 534 denotes a data store, to which respective outputs 535 of each Al model 533 may be temporarily stored. The outputs 535 of each Al model 533 may then be called by a combination module 536, which may use any suitable combination logic to output a combined value representative of the level of offensiveness of the received content 521.

[0084] In some examples, the output of a particular Al model may indicate a category and a probability. That is, a particular Al model configured to assess input data in a particular manner may provide an output which indicates a categorisation of the input data into one of two or more distinct classes. In such an example, the output probability may indicate a confidence of the Al model in assigning the input data to a particular one of the two or more distinct classes, or a probability associated with theeach of the classes, such that the class of highest probability is the assigned class. In the above example of a sentiment Al model, 'positive' and 'negative' sentiment may be respectively represented by a distinct class, into which the input data may be classified; the sentiment model may further output a probability representing the confidence level with which the sentiment model has classified the input data into a particular class. In other examples, the output of an Al model may be a numerical value, which is effectively a measure of the amount or level of a particular phenomenon (e.g. toxicity) in the input data.

[0085] In some embodiments, one or more of the Al models 533 may be based on one or more generic language models. These may be pretrained on large volumes of general text data, and thus are suitable for a wide variety of language processing tasks. The Al model is then subsequently tailored to a particular task by further training (or fine-tuning), based on a tailored training set comprising application-specific training data. Examples of such generic language models (also referred to in the art as "large language models") include BERT, GPT-3, and cohere.

[0086] In one example, the plurality of Al models 533 includes a nudity detector, such as NudeNet (https: / / pypi.org / project / NudeNet / ). The plurality of Al models 533 may include one or more machine translation models, configured to translate the content item. The translation models may be applied as a pre-processing step, with the output of the translation models forming input to the other models 533.

[0087] The combination logic applied to the outputs 535 may take the form of a series of predefined operations (e.g. rules or heuristics) applied to the output 535. The operations may include thresholds applied to particular model outputs, or mathematical operations (e.g. sums, weighted averages) are applied to combine the model outputs. The combination logic may also comprise the application of ensemble models to combine the outputs 535.

[0088] In one example, the combination module 536 may further receive an indication of the associated behaviour profile 523 as input, the indication of thebehaviour profile 523 being received from the API 531. The method by which the combination logic 536 combines the model outputs 535 may be dependent on the behaviour profile 523. In other words, the combination module may apply a different combination logic (i.e. a different series of predetermined operations), depending on the received behaviour profile. For example, the module 530 may store combination logic associated with each behaviour profile, and retrieve and apply the logic. Therefore, an output value of the combination module 536 may also be dependent on the associated behaviour profile 523. That is, an identical set of model outputs 535 may result in a different combination output if a different behaviour profile 523 is assigned.

[0089] Accordingly, the same set of Al models 533 may be used to provide automatic moderation that is tailored to a particular type of account holder. For example, the aforementioned sports profile may involve a lower weighting of an Al model detecting swearing compared to a corporate profile.

[0090] The module 530 of Figure 5 further includes a thresholding module 537. The thresholding module 537 may categorise the suitability of the content 521 for display in the associated context, based on the output value of the combination module 536. In other words, the thresholding module 537 generates a moderation result based on the output of the combination module 536.

[0091] For example, the content 521 may be deemed suitable for display (i.e. pass the moderation) if the combination output value is determined to be above a first exemplary threshold value stored in the thresholding module 537. By contrast, the content 521 may be deemed unsuitable for display to the user (i.e. fail the moderation) if the combination output value is determined to be below the first exemplary threshold value stored in the thresholding module 537.

[0092] It will be appreciated that the thresholding module 537 may perform any suitable thresholding method, and may comprise any number of predefined threshold values for sorting the content 521 into a respective number of predefined categories.Note that a 'category' in this sense may be defined by a first threshold value representing a lower bound, and / or a second threshold value representing an upper bound. That is, one or more category may be provided for one or more respective range of combination output values. For example, the moderation result may include a category that indicates the content item requires human moderation, as well as categories representing passing and failing the moderation.

[0093] Although in the above example, the output value is inversely proportional to the offensiveness of the content (i.e. higher values are more likely to be suitable content), it will be appreciated that in other examples the output value may be proportional to the offensive ness, and the thresholds adjusted accordingly.

[0094] When the thresholding module 537 generates a moderation result, the module 537 may communicate the moderation result to the API 531. This communicative step is represented by an arrow denoted 538 in the example of Figure 5. On receipt of the communication 538 of the action, the API 531 may send a response message 540 to the moderation request generator 517, the message indicating the moderation result.

[0095] Upon receiving the response message 540, the moderation request generator may communicate with the application-specific API 503, sending an action instruction message 541, indicative of the action, to the relevant client application 501. The relevant client application 501 (online platform) may then put the action into effect. The moderation request generator 517 may send the action instruction 541 via the relevant integration unit 511a, which may format the action instruction message 541 for receipt by the API 503 at the client application 501.

[0096] For example, where the content item is a comment or post that would appear on the user's timeline, it may be hidden from view if the content item fails moderation. If the content item is a comment on or responsive to other content posted by the user (e.g. a comment on an Instagram or Facebook post or the like), the comment may be deleted if it fails moderation. In some examples, the action taken by the contentanalysis module 510 may include blocking, hiding or reporting content items originating from the author of the content item. It will be appreciated that the relevant APIs 503 and associated applications 501 may offer differing mechanisms for dealing with offensive content and authors of offensive content.

[0097] Where the content item passes moderation, it may be the case that the content item is explicitly approved or released so that it can be accessed by the user. In other examples, it may be that no further action is taken. That is to say, content items may be accessible by the user until they fail moderation, at which point action may be taken as set out above.

[0098] In examples where the moderation result is indicative of a content item requiring human review, the content item may be held until it has been reviewed by a moderator, for example via a suitable dashboard.

[0099] In some examples, the module 510 and module 530 are executed on different computers or computer systems. For example, each module 510, 530 may be hosted on different server computers or computing platforms. For example, the module 530 and / or module 510 may be executed in a cloud computing environment. This may enable autoscaling of resources. In other examples, the module 530 and 510 may be disposed on different or the same server computers.

[0100] As discussed above, in some examples the module 530 may be accessible by external platforms or computer systems, via the API 531. Accordingly, a third-party platform may be provided with moderation results. This configuration may be particularly useful for integration with existing systems and applications that include an ability to generate content items, but which do not have their own APIs similar to the APIs 503. This includes applications such as workplace or organisation internal communication systems, gaming systems or other applications that include live chat or messaging, and so on.

[0101] In some examples, the technique of Figure 3 may be applied to a plurality of content items associated with a user (e.g. generated by, read, interacted with etc) over time, so as to track changes in emotional state.

[0102] It will be understood that the processor or processing system or circuitry referred to herein may in practice be provided by a single chip or integrated circuit or plural chips or integrated circuits, optionally provided as a chipset, an applicationspecific integrated circuit (ASIC), field-programmable gate array (FPGA), digital signal processor (DSP), graphics processing units (GPUs), etc. The chip or chips may comprise circuitry (as well as possibly firmware) for embodying at least one or more of a data processor or processors, a digital signal processor or processors, baseband circuitry and radio frequency circuitry, which are configurable so as to operate in accordance with the exemplary embodiments. In this regard, the exemplary embodiments may be implemented at least in part by computer software stored in (non-transitory) memory and executable by the processor, or by hardware, or by a combination of tangibly stored software and hardware (and tangibly stored firmware).

[0103] Reference is made herein to data storage for storing data. This may be provided by a single device or by plural devices. Suitable devices include for example a hard disk and non-volatile semiconductor memory (e.g. a solid-state drive or SSD).

[0104] Although at least some aspects of the embodiments described herein with reference to the drawings comprise computer processes performed in processing systems or processors, the invention also extends to computer programs, particularly computer programs on or in a carrier, adapted for putting the invention into practice. The program may be in the form of non-transitory source code, object code, a code intermediate source and object code such as in partially compiled form, or in any other non-transitory form suitable for use in the implementation of processes according to the invention. The carrier may be any entity or device capable of carrying the program. For example, the carrier may comprise a storage medium, such as a solid- state drive (SSD) or other semiconductor-based RAM; a ROM, for example a CD ROMor a semiconductor ROM; a magnetic recording medium, for example a floppy disk or hard disk; optical memory devices in general; etc.

[0105] Terms such as 'component', 'module' or 'unit' used herein may include, but are not limited to, a hardware device, such as circuitry in the form of discrete or integrated components, a Field Programmable Gate Array (FPGA) or Application Specific Integrated Circuit (ASIC), which performs certain tasks or provides the associated functionality. In some embodiments, the described elements may be configured to reside on a tangible, persistent, addressable storage medium and may be configured to execute on one or more processors. These functional elements may in some embodiments include, by way of example, components, such as software components, object-oriented software components, class components and task components, processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, and variables. Although the example embodiments have been described with reference to the components, modules and units discussed herein, such functional elements may be combined into fewer elements or separated into additional elements.

[0106] The examples described herein are to be understood as illustrative examples of embodiments of the invention. Further embodiments and examples are envisaged.

[0107] Any feature described in relation to any one example or embodiment may be used alone or in combination with other features. In addition, any feature described in relation to any one example or embodiment may also be used in combination with one or more features of any other of the examples or embodiments, or any combination of any other of the examples or embodiments. Furthermore, equivalents and modifications not described herein may also be employed within the scope of the invention, which is defined in the claims and / or summary section.

Claims

CLAIMS1. A computer-implemented method comprising: receiving a content item; providing the content item as input to a trained machine learning model; receiving, as output from the trained machine learning model, an indication of an emotion the content item will evoke in a predetermined user or group.

2. The method of claim 1, wherein the trained machine learning model has been trained on content items relating to the predetermined user or group.

3. The method of claim 1 or claim 2, wherein the indication of the emotion the content item will evoke comprises a numerical score indicative of a level of emotion invoked.

4. The method of any preceding claim, comprising moderating the content item based on the indication of the emotion.

5. The method of claim 4, comprising: analysing the content item using a plurality of moderation machine learning models, the plurality of moderation machine learning models including the trained machine learning model; combining outputs of the plurality of machine learning models to generate a moderation result indicating whether the content item contains offensive content.

6. The method of claim 4, comprising: receiving a behaviour profile;determining, based on the behaviour profile, a combination logic to apply to the outputs of the plurality of machine learning models to generate the moderation result.

7. The method of any of claims 1-3, wherein the content item is an output of a generative model.

8. The method of claim 7, comprising: receiving an input prompt; providing the input prompt to a generative model, and receiving the content item from the generative model.

9. The method of any of claims 1-3, comprising training a generative model using training data comprising the content item and the indication.

10. A computer-implemented method comprising: receiving a training data set, the training data set comprising a plurality of a training content items, each training content item having a training label reflective of an emotion evoked in a user or group of users by the respective content item; training the machine learning model using the training data set to output, in response to receiving an input content data item, an indication of an emotion the input content item will evoke in the user or group of users.

11. A computer-implemented method comprising: providing, as input to a first machine learning model, an input indicative of a desired emotion;receiving, as output from the first machine learning model, one or more data items for inclusion in a prompt for a second machine learning model, the second machine learning model being a generative model; generating a prompt including the one or more data items; providing the prompt as input to the second machine learning model; and receiving, as output from the second machine learning model, a content item reflective of the desired emotion.

12. The method of claim 11, wherein the generative model is a model configured to generate text, and the content item is text reflecting the desired emotion.

13. The method of claim 11 or 12, wherein the prompt takes the form of a template, wherein the generated data items are inserted into the template.

14. A computer- implemented method of training a first machine learning model comprising: receiving a training data set, the training data set comprising a plurality of a content items, each content item having a training label reflective of an emotion associated with the respective content item; training the first machine learning model using the training data set to output, in response to receiving an input indicative of a desired emotion, one or more data items for inclusion in a prompt to be input to a generative machine learning model, wherein, the one or more data items, when included in the prompt, cause the generative machine learning model to generate a content item reflective of the desired emotion.

15. The method of claim 14, wherein training the first machine learning model comprise generating and inputting probing prompts to the generative machinelearning model, the output of the probing prompts being used in calculating a loss function used in training the machine learning model.

16. The method of claim 14 or 15, wherein the content items in the training data set are social media posts or messages relating to a user or a group of users.

17. The method of any of claims 14 to 16, wherein the training data comprises a graph network, wherein: each content item forms a node of the graph network; the graph network comprise edges linking nodes, representing a temporal relationship between content items.

18. A computer system comprising a processor and a memory storing instructions, which when executed by the processor, cause the processor to carry out the method of any preceding claim.

19. A non-transitory computer-readable medium storing instructions, which when executed by a processor, cause the processor to carry out the method of any of claims I to 17.

Images