Apparatus, systems and methods for visual description

Abstract

A data processing apparatus 200b comprises a captioning model 200b to receive gameplay telemetry data indicative of one or more in-game properties for a video game session. The captioning model includ

Claims

1. A data processing apparatus comprising:a captioning model to receive gameplay telemetry data indicative of one or more in-game properties for a session of a video game, the captioning model comprising an artificial neural network (ANN) trained to output caption data comprising one or more captions in dependence upon a learned mapping between gameplay telemetry data and caption data, one or more of the captions comprising one or more words for providing a visual description for the session of the video game; andoutput circuitry to output one or more of the captions.

2. The data processing apparatus according to claim 1, wherein the ANN is trained using training data comprising gameplay telemetry data and corresponding labels associated with captions comprising words providing a visual description of video images associated with the gameplay telemetry data.

3. The data processing apparatus according to claim 2, wherein at least some of the training data comprises manually labelled gameplay telemetry data.

4. The data processing apparatus according to any to claim 2 or claim 3, wherein at least some of the training data comprises automatically labelled gameplay telemetry data comprising labels associated with captions obtained, by a video captioning model, for the video images associated with the gameplay telemetry data.

5. The data processing apparatus according to claim 4, wherein the video captioning model comprises an artificial neural network (ANN) trained to output caption data comprising one or more captions in dependence upon a learned mapping between video images and caption data.

6. The data processing apparatus according to any preceding claim, wherein the captioning model is configured to receive recorded gameplay telemetry data for a recorded session of the video game and input at least some of the recorded gameplay telemetry data to the ANN.

7. The data processing apparatus according to any preceding claim, wherein the captioning model is configured to receive streamed gameplay telemetry data for a live session of the video game and input at least some of the streamed gameplay telemetry data to the ANN.

8. The data processing apparatus according to any preceding claim, wherein the captioning model is configured to receive respective streamed gameplay telemetry data for each of a pluralityof respective instances of one or more video games and to output respective caption data for each of the plurality of respective instances of the one or more video games.

9. The data processing apparatus according to any preceding claim, wherein the captioning model comprises one or more from the list consisting of:a first ANN trained using training data associated with a first video game;a second ANN trained using training data associated with a second video game different from the first video game;a third ANN trained using training data associated with a plurality of related video games of a same video game series; anda fourth ANN trained using training data associated with a plurality of video games of a same video game genre.

10. The data processing apparatus according to any preceding claim, wherein the captioning model is configured to receive the gameplay telemetry data and associated metadata indicative of at least one of a video game title, video game series and video game genre for the video game, and the captioning model is configured to input the received gameplay telemetry data to a respective ANN selected from a plurality of ANNs in dependence on the associated metadata.

11. The data processing apparatus according to any preceding claim, wherein the gameplay telemetry data is indicative of one or more in-game properties comprising one or more from the list consisting of:at least one of a type and a name for one or more in-game objects;a position of one or more in-game objects;a velocity for one or more in-game objects;a health status for one or more in-game characters; anda score associated with at least one of a character and a team.

12. The data processing apparatus according to any preceding claim, comprising processing circuitry to execute the video game and generate video images and the gameplay telemetry data.

13. The data processing apparatus according to any one of claims 1 to 11, comprising processing circuitry to execute the video game in accordance with inputs from a virtual agent and generate the gameplay telemetry data, and error detecting circuitry to detect one or more errors associated with the session of the video game in dependence on one or more of the captions.

14. A computer implemented method comprising:inputting gameplay telemetry data indicative of one or more in-game properties for a session of a video game to a captioning model, the captioning model comprising an artificial neural network (ANN) trained to output caption data comprising one or more captions in dependence upon a learned mapping between gameplay telemetry data and caption data; and5 outputting, by the ANN, caption data comprising one or more captions, one or more of thecaptions comprising one or more words for providing a visual description for the session of the video game.

15. Computer software which when executed by a computer causes the computer to perform 10 the method according to claim 14.

Images