Trained highlighting for human checking of marketing materials

Abstract

Method of training an automatic highlighting system for supporting a human checker of automatically generated visual marketing materials, comprising: presenting to the human checker a piece of automatically generated visual marketing material to be checked; obtaining from the human checker at least one label representing a result of the checking of the presented piece of automatically generated visual marketing material; for one or more areas within the presented piece of automatically generated visual marketing material, obtaining at least one human attention indication regarding the human checker's visual attention to the respective area during the checking; and training the automatic highlighting system using at least: the presented piece of automatically generated visual marketing material, the obtained at least one label, and the at least one human attention indication obtained for at least one of the one or more areas within the presented piece of automatically generated visual marketing material.

Description

[0001] Title: Trained highlighting for human checking of marketing materials

[0002] TECHNICAL FIELD

[0003] The present disclosure relates to: a method of training an automatic highlighting system for supporting a human checker of automatically generated visual marketing materials; a training system configured to perform the method; a computer readable storage medium; an automatic highlighting system trained using the method; a use of the automatic highlighting system; and a dataset for training the automatic highlighting system.

[0004] BACKGROUND

[0005] It is anticipated that visual marketing materials such as visual advertising materials will increasingly originate from generative artificial intelligence (GenAI) systems. While such systems have many advantages, there are also some potential disadvantages, notably including a risk of unintentional generation of undesired visual content. For example, some generated content may be considered offensive, harmful or illegal. Also, in some cases, generated content may not sufficiently reflect brand values, desired attributes or a desired themes. It is anticipated that at least some of this risk will persist for some time, despite efforts to improve the GenAI systems in this respect.

[0006] A known mitigation strategy involves one or more human checkers checking the generated materials for any undesired content. More generally, human checkers may be enlisted as a quality control measure. Typically, the human checker is presented with a piece of generated visual marketing material and is tasked to evaluate the presented piece against one or more quality criteria. The checker then reports the result of their checking, e.g. to a system, so that appropriate action can be taken. In a basic scenario, the human checker may report an ‘accept’ or ‘reject’ result of their checking, based on which the subject piece of marketing material is subsequently accepted or rejected for further use.

[0007] Compared to the speed and efficiency of automated generation of visual marketing materials, human checking of such generated materials can be relatively slow and inefficient. This can result in the human checking being regarded as a bottle neck in marketing operations. Meanwhile, because of limited reliability of known automatic checking systems, mere automatic checking is considered unsuitable to sufficiently prevent unacceptable marketing materials from being published.

[0008] SUMMARY

[0009] There is a desire to improve the speed and efficiency of checking of automatically generated visual marketing materials visual marketing materials, in particular while maintaining or improving quality and reliability of the checking.

[0010] An aspect provides a method of training an automatic highlighting system for supporting a human checker of automatically generated visual marketing materials. The method comprises (a.) presenting, using an electronic display screen, to the human checker a piece of automatically generated visual marketing material to be checked by the human checker. The method comprises (b.) obtaining, via an electronic input module, from the human checker an electronically coded data item comprising at least one label representing a result of the checking of the presented piece of automatically generated visual marketing material. The method comprises (c.) for one or more areas within the presented piece of automatically generated visual marketing material, obtaining, via the electronic input module, at least one human attention indication regarding the human checker’s visual attention to the respective area during the checking. The method comprises (d.) training the automatic highlighting system using at least: the presented piece of automatically generated visual marketing material, the obtained at least one label, and the at least one human attention indication obtained for at least one of the one or more areas within the presented piece of automatically generated visual marketing material.

[0011] Advantageously, an automatic highlighting system trained in this way may be employed to automatically highlight one or more portions in a piece of visual marketing material that may be particularly relevant for the human checker to direct their attention to when performing their checking. In this way, the human checker can be supported to perform their checking work more efficiently, in particular without inhibiting quality and reliability of the checking. Importantly, the highlighting system preferably does not hide or obscure any part of the material to be checked from the human checker, so that the human checker can still perform a complete checking of the material.

[0012] By taking both the human attention indication and the obtained label into account in the training, the trained highlighting system can be more inclined to highlight areas that appear to be associated with a ‘reject’ or other critical label as opposed to areas that do not appear to be associated with such a critical label, such as areas that the human checker would not be expected to pay much attention to and / or that are in a piece of material expected to receive a non-critical label, e.g. ‘accept’. The detailed description below provides relevant examples in this respect.

[0013] Optionally, further at least one highlighting output of one or more earlier versions of the automatic highlighting system is used for the training, in particular wherein the at least one highlighting output relates to at least one similar or same piece of automatically generated visual marketing material. Optionally, the at least one human attention indication used for the training is compared to the at least one highlighting output, in particular wherein a result of the comparison is used for the training. Optionally, the at least one highlighting output comprises, for one or more areas within the piece of automatically generated visual marketing material, at least one highlighting indication for the respective area.

[0014] In this way, the training be made more effective and / or efficient. For example, the training can take into account possible discrepancies between the highlighting output of the earlier version on the one hand and the obtained labels and attention indications on the other hand, allowing the training to be focused on minimizing such discrepancies. The earlier version of the highlighting system may have been trained in various ways, including using the method of training as described herein. Thus, the training may be at least partly recursive. An initial version of the highlighting system may have been trained without using human attention indications, for example using a dataset of stock images showing examples of either acceptable or unacceptable visual elements that are labelled as such in the dataset.

[0015] Optionally, the at least one highlighting indication is used for the presentation to the human checker of the piece of automatically generated visual marketing material, in particular wherein the at least one highlighting indication is represented in the presentation. In this way, the human checker can be supported in their checking by an earlier version of the highlighting system while also training data for training a later version of the highlighting system can be obtained.

[0016] Optionally, the presentation to the human checker of the piece of automatically generated visual marketing material is free from any representation of any highlighting indication. In this way, the attention indications and label can be obtained in a particularly unbiased manner.

[0017] Optionally, the at least one human attention indication is obtained using a sensing system configured to sense at least one visual attention parameter of the human checker in relation to the one or more areas within the presented piece of automatically generated visual marketing material. Optionally, the at least one visual attention parameter is representative of a duration for which the human checker looks at the respective area. Optionally, the sensing system comprises an eye tracking system. In this way, the at least one human attention indication can be obtained in a particularly effective and efficient manner. Alternatively or additionally, for example, some or all of the at least one human attention indication may be obtained from so-called manual input, e.g. data indicating where the human checker moved a cursor and / or clicked or tapped during the checking.

[0018] Optionally, the obtained at least one label comprises an indication of an acceptability level of the presented piece of automatically generated visual marketing material according to the human checker. For example, as alluded to above, the piece of material may be labelled as ‘accept’ or ‘reject’, wherein it shall be appreciated that other words and / or symbols may be used to indicate the same or a similar meaning. Further and / or other acceptability levels may be possible, including e.g. intermediate levels such as ‘doubtful’ or ‘unclear’.

[0019] Optionally, the obtained at least one label comprises an indication of a content characterization of at least part of the presented piece of automatically generated visual marketing material according to the human checker. Such a content characterization can take many forms, including one or more descriptive keywords, phrases and / or sentences, and may generally be more descriptive than the above mentioned acceptability level.

[0020] Optionally, the combination of activities a, b and c is performed for a plurality of pieces of automatically generated visual marketing material, wherein results of the combinations of activities a, b and c are aggregated across the plurality of pieces of automatically generated visual marketing material, wherein in activity d the aggregated results are used for the training. In this way, the training can be performed using one or more batches of training data.

[0021] Optionally, the piece of automatically generated visual marketing material relates to at least one of a beverage product and a beverage service. A further aspect provides a training system configured to perform the method as described herein. A further aspect provides a computer readable storage medium having stored thereon instructions which, when executed by a computer, cause the computer to perform the method as described herein. A further aspect provides an automatic highlighting system for supporting a human checker of automatically generated visual marketing materials, trained using a method as described herein and / or comprising a training system as described herein.

[0022] A further aspect provides a use of an automatic highlighting system as described herein for supporting a human checker of automatically generated visual marketing materials. Advantages of, and options for, said further aspects correspond to those described above for the method of training. It shall be appreciated that the training system and the automatic highlighting system may be at least partly combined, although it is also possible to use separate systems. The optional sensing system described elsewhere herein may be part of the training system.

[0023] A further aspect provides a dataset for training an automatic highlighting system for supporting a human checker of automatically generated visual marketing materials, obtained using a method comprising activities a, b and c of the method as described herein. The dataset comprises, for a plurality of presented pieces of automatically generated marketing material: the presented piece of automatically generated visual marketing material, or at least a reference for retrieval thereof; the obtained at least one label, or at least a reference for retrieval thereof; and the at least one human attention indication obtained for at least one of the one or more areas within the presented piece of automatically generated visual marketing material, or at least a reference for retrieval thereof. Such a dataset may be regarded as forming an intermediate product in the context of the training method, thereby providing corresponding advantages and having corresponding options associated therewith. DETAILED DESCRIPTION

[0024] In the following, aspects of the disclosure will be explained further using examples and references to accompanying figures. The figures are schematic and merely show examples. In the figures, corresponding elements are provided with corresponding reference signs.

[0025] Fig. 1 shows a process flow diagram;

[0026] Fig. 2A shows a possible piece of visual marketing material without any highlighting;

[0027] Fig. 2B shows the piece of Fig. 2A with possible highlighting;

[0028] Fig. 2C shows the piece of Fig. 2A with possible highlighting that is partly different from the highlighting in Fig. 2B; and

[0029] Fig. 3 shows a diagram of a possible training and / or highlighting system, here comprising at least one processor 1, at least one storage 2, at least one sensing device 3, at least one input device 4, and at least one output device 5.

[0030] A method of training an automatic highlighting system, a use of the trained system and related other aspects have been described in the summary section above. Fig. 1 shows a process flow diagram related to a possible implementation of such training and use. The process flow diagram will be explained here below, merely as an example of possible implementations of described aspects and options.

[0031] A human checker H, also called human user herein, is shown in Fig. 1 at a workstation, here comprising a monitor as output device 5, a keyboard and mouse as input devices 4, and an eye tracking system as sensing device 3. The workstation may also comprise a processor 1 and / or storage 2, but these may alternatively or additionally be more remote from the user, e.g. in a computer network to which the workstation is operatively connected. Algorithms and a model indicated in Fig. 1 may be implemented using such processor 1 and storage 2. Automatically generated visual marketing materials 6, also called generated media herein, may be stored using the storage 2. Fig. 2A shows a possible example of a piece of automatically generated visual marketing material 6, here depicting various visual elements including people, products, graphical elements and text elements.

[0032] Returning to Fig. 1, during use and / or training a piece of material 6 to be checked, e.g. retrieved from a database of such materials, may be presented to the human checker H via output device 5. The human checker H may be tasked, e.g. prompted or instructed, to check the presented piece according to one or more predefined criteria or other rules, and to provide via input device 4 at least one label representing a result of their checking. Such at least one label is also called classification data herein. Primarily, such at least one label may be used, e.g. in a downstream automatic process, to either allow or prevent the checked piece of material to proceed towards publication in the context of marketing activities (not illustrated in Fig. 1). The same at least one label may be used in a secondary manner as basis for and / or part of training data for training an automatic highlighting system. In some scenarios it is also possible that the at least one label is only used for training purposes and not for operational decisions regarding publication.

[0033] While the human checker H performs their checking, a sensing device 3 such as an eye tracking system may be used to obtain sensing data. The sensing data may be used to obtain at least one human attention indication, also called user attention data herein, regarding the human checker’s visual attention to one or more areas within the presented piece of material. As indicated in Fig. 1, a user attention algorithm 7 may be used to obtain the user attention data from the sensing data. The user attention data may subsequently be used as further basis for and / or part of the training data. As indicated in Fig. 1, a training data algorithm 8 may be used to obtain training data using the user attention data and classification data. When sufficient data has been collected for training, e.g. after multiple pieces of material have been checked by the human checker H, a model training algorithm 9 may be used to train an automatic highlighting system, in particular a model 10 to be used as part of the highlighting system. Such is also called highlighting model herein. The highlighting model 10, once trained, may be configured to take a piece of visual marketing material 6 as input, and to return highlighting data for the respective piece of material.

[0034] The highlighting data, also called highlighting output herein, may be used to highlight one or more areas in a piece of marketing material 6 when presented to the human checker H, for example using one or more bounding boxes B, of which examples are shown in Figs. 2B and 2C. Alternative or additional types of visual highlighting may be used, for example silhouettes may be used instead of rectangular boxes, arrows or other markers may be used, and / or a heatmap representation may be used. The highlighting data may be presented to the human checker H via output device 5, together with, e.g. overlaid on, the respective piece of marketing material.

[0035] As alluded to in the summary section above, it shall be appreciated that highlighting data may or may not be presented to the human checker during collection of training data. When highlighting data is presented during collection of training data, the shown highlighting data may be output by an earlier version of the highlighting model 10 than the version to be trained using the training data being collected. Similarly, when the human checker H is being supported by automatic highlighting in their checking of materials, there may or may not be an ongoing collection of training data.

[0036] Progressive training of the highlighting model 10 may result in the model’s highlighting data, e.g. shown as bounding boxed B, for a same piece of marketing material 6 evolving or otherwise changing over time. For example, an earlier version of the model 10 may return bounding boxes B as shown in Fig. 2B, whereas a later version of the model 10 may return bounding boxes B as shown in Fig. 2C. As can be seen in these figures, bounding boxes B or other highlighting data may thus be added or removed or changed over progressive instances of the model 10. This may for example occur due to the human checker’s attention during collection of training data being directed substantially differently compared to what is suggested by the earlier version of the highlighting data.

[0037] Although in some scenario’s a same piece of marketing material may be checked multiple times, it may be preferred to avoid such rework. Nevertheless, the progressive training can benefit from taking earlier highlighting data into account, even if this relates to materials that will not have to be checked again. This is because the model 10 and the training thereof are designed to be able to generalize over sufficiently large and diverse sets of training data.

[0038] In some implementations, the training data algorithm, or more generally the method of training, associates the obtained label or labels for a checked piece of marketing material with one or more areas of the piece to which the human checker H attended relatively strongly, e.g. for a relatively large portion of their checking time and / or a relatively long absolute duration. This may be particularly relevant in case of a critical label such as ‘reject’. In this way, the model 10 may be trained to mimic such associations and to produce highlighting data accordingly. As explained in the summary section, an initial version of the model 10 may be trained using appropriately labelled stock examples of visual elements. The initial model may then subsequently be refined, e.g. in a progressive manner, by additional training or other retraining using attention data and classification data from the human checker H as described.

[0039] As explained in the summary section above, by taking both the human attention indication, also called user attention data, and the obtained label, also called classification data, into account in the training, the trained highlighting system can be more inclined to highlight areas that appear to be associated with a ‘reject’ or other critical label as opposed to areas that do not appear to be associated with such a critical label, such as areas that the human checker would not be expected to pay much attention to and / or that are in a piece of material expected to receive a non-critical label, e.g. ‘accept’.

[0040] For example, given a first area that the human checker attended to for 70% of their checking time in a piece of material labelled ‘reject’, and a second area that the human checker attended to for 70% of their checking time in a piece of material labelled ‘accept’, the trained highlighting system may be more inclined to highlight areas similar to the first area than areas similar to the second area. As a further example, given a first area that the human checker attended to for 70% of their checking time and a second area that the human checker attended to for 10% of their checking time, both in a same piece of material labelled ‘reject’, the trained highlighting system may be more inclined to highlight areas similar to the first area than areas similar to the second area.

[0041] As a still further example, given a first area that the human checker attended to for 70% of their checking time and a second area that the human checker attended to for 10% of their checking time, both in a same piece of material labelled ‘accept’, the trained highlighting system may be less inclined to highlight areas similar to the second area than areas similar to the first area.

[0042] Various metadata for a piece of visual marketing material may be used together with the piece itself throughout the activities described. Such metadata may for example include one or more keywords, phrases and / or sentences, e.g. generated along with the generation of the piece of marketing material itself, given in a prompt used as input for the generating, and / or entered by the human checker H. The highlighting model 10 may be configured to output such metadata along with the highlighting data, e.g. as suggestions to the human checker H.

[0043] In some scenarios, the obtained sensing data and / or use attention data may additionally be used for quality control purposes with respect to the human checker H. For example, such data may be used to determine if the human checker H checked all parts of the piece of visual marketing material.

[0044] Although references to examples and figures have been provided herein, these do not limit the scope of the disclosure as determined by the claims. Within said scope, many variations, combinations and extensions are possible, as shall be appreciated by the skilled person having the benefit of the present disclosure.

Claims

Claims1. Method of training an automatic highlighting system for supporting a human checker of automatically generated visual marketing materials, comprising: a. presenting, using an electronic display screen, to the human checker a piece of automatically generated visual marketing material to be checked by the human checker; b. obtaining, via an electronic input module, from the human checker an electronically coded data item comprising at least one label representing a result of the checking of the presented piece of automatically generated visual marketing material; c. for one or more areas within the presented piece of automatically generated visual marketing material, obtaining, via the electronic input module, at least one human attention indication regarding the human checker’s visual attention to the respective area during the checking; and d. training the automatic highlighting system using at least: the presented piece of automatically generated visual marketing material, the obtained at least one label, and the at least one human attention indication obtained for at least one of the one or more areas within the presented piece of automatically generated visual marketing material.

2. Method according to claim 1, wherein further at least one highlighting output of one or more earlier versions of the automatic highlighting system is used for the training, in particular wherein the at least one highlighting output relates toat least one similar or same piece of automatically generated visual marketing material.

3. Method according to claim 2, wherein the at least one human attention indication used for the training is compared to the at least one highlighting output, in particular wherein a result of the comparison is used for the training.

4. Method according to any of claims 2 - 3, wherein the at least one highlighting output comprises, for one or more areas within the piece of automatically generated visual marketing material, at least one highlighting indication for the respective area.

5. Method according to claim 4, wherein the at least one highlighting indication is used for the presentation to the human checker of the piece of automatically generated visual marketing material, in particular wherein the at least one highlighting indication is represented in the presentation.

6. Method according to any of the preceding claims, wherein the at least one human attention indication is obtained using a sensing system configured to sense at least one visual attention parameter of the human checker in relation to the one or more areas within the presented piece of automatically generated visual marketing material.

7. Method according to claim 6, wherein the at least one visual attention parameter is representative of a duration for which the human checker looks at the respective area.

8. Method according to claim 6 or 7, wherein the sensing system comprises an eye tracking system.

9. Method according to any of the preceding claims, wherein the obtained at least one label comprises an indication of an acceptability level of the presented piece of automatically generated visual marketing material according to the human checker.

10. Method according to any of the preceding claims, wherein the obtained at least one label comprises an indication of a content characterization of at least part of the presented piece of automatically generated visual marketing material according to the human checker.

11. Method according to any of the preceding claims, wherein the combination of activities a, b and c is performed for a plurality of pieces of automatically generated visual marketing material, wherein results of the combinations of activities a, b and c are aggregated across the plurality of pieces of automatically generated visual marketing material, wherein in activity d the aggregated results are used for the training.

12. Method according to any of the preceding claims, wherein the piece of automatically generated visual marketing material relates to at least one of a beverage product and a beverage service.

13. Training system configured to perform the method according to any of the preceding claims.

14. Computer readable storage medium having stored thereon instructions which, when executed by a computer, cause the computer to perform the method according to any of claims 1 - 12.

15. Automatic highlighting system for supporting a human checker of automatically generated visual marketing materials, trained using a method according to any of claims 1 - 12 and / or comprising a training system according to claim 13.

16. Use of an automatic highlighting system according to claim 15 for supporting a human checker of automatically generated visual marketing materials.

17. Dataset for training an automatic highlighting system for supporting a human checker of automatically generated visual marketing materials, obtained using a method comprising activities a, b and c of the method according to any of claims 1 - 12, wherein the dataset comprises, for a plurality of presented pieces of automatically generated marketing material: the presented piece of automatically generated visual marketing material, or at least a reference for retrieval thereof; the obtained at least one label, or at least a reference for retrieval thereof; and the at least one human attention indication obtained for at least one of the one or more areas within the presented piece of automatically generated visual marketing material, or at least a reference for retrieval thereof.

Images