A contextualized augmented reality system

Abstract

A contextualized augmented reality system has a control computer, user mobile devices each with an inertial measurement unit, an installer mobile device with a UWB unit for distance measurement, and UWB beacons (31-33) on a stage 1. The control computer performs a mapping for an auditorium (10) and manages in real time interactivity of user mobile devices in the auditorium during a performance. The mapping is for each auditorium seat, and the control computer receives a seat identifier and records a distance between a mobile device at that seat and each beacon on the stage. For the real time AR interactivity, the control computer records a performance timeline with a series of interactivity windows in which a user should execute an action on their mobile device while it is orientated towards a prop on the stage. Each user mobile device in real time during a performance monitors the user actions and determines if each occurs when the device is orientated towards a certain prop on the stage and during a certain time window. These processor-intensive operations are performed only by the user mobile device using software of the system and its inertial measurement unit. There is real time uploading of scores, but this requires little bandwidth.

Description

[0001] “A Contextualized Augmented Reality System”

[0002] Introduction

[0003] The present invention relates to auditorium performances, and particularly to a technical system for providing augmented reality (“AR”) interactivity for audience members during a performance.

[0004] US2019 / 0060741 (Kristinu Contreras) describes a computer-implemented method for detecting a user’s footsteps, in which acceleration readings are averaged and normalized and arrays of inertial data pairs are generated.

[0005] The present invention is directed towards providing a system with technical features implemented in software and hardware to provide contextualized augmented reality interactivity in an auditorium.

[0006] Summary of the Invention

[0007] In various examples of the invention we describe a contextualized augmented reality system that has a control computer, user mobile devices each with an inertial measurement unit, an installer mobile device with an Ultra Wide Band (UWB) unit for distance measurement, and UWB beacons on a stage. The control computer performs a mapping for an auditorium and manages in real time interactivity of user mobile devices in the auditorium during a performance. The mapping is for each auditorium seat, and the control computer receives a seat identifier and records a distance between a mobile device at that seat and each beacon on the stage. For the real time AR interactivity, the control computer records a performance timeline with a series of interactivity windows in which a user should execute an action on their mobile device while it is orientated towards a prop on the stage. Each user mobile device in real time during a performance monitors the user actions and determines if each occurs when the device is orientated towards a certain prop on the stage and during a certain time window. These processor-intensive operations are performed only by the user mobile deice using software of the system and its inertial measurement unit. There is real time uploading of scores, but this requires little bandwidth.

[0008] In use, the system operates in preferred examples in the following major steps:

[0009] The auditorium is mapped for each seat position relative to the stage, and for prop positions on the stage. This is done using UWB measurements of an installer mobile device. A 3D map is provided. A performance is generated with time windows from a performance start time, each window being associated with a prop stage position.

[0010] At the start of the performance the user mobile device is synchronized by verifying the seat position, on the basis that the user device should capture an image of the stage which is predicted by the 3D map.

[0011] During the performance the user mobile device tracks the planned performance prop positions using object-oriented technology and determines according to its inertial measurement unit if it has moved to be pointing to a prop during the associated time window to, for example, shoot bats flying from a Pandora’s Box on the stage. The Box is real and is at its pre-planned physical position, and the bats are images generated by the user mobile device App.

[0012] We describe a contextualized augmented reality system comprising a control computer, a plurality of user mobile devices each with an inertial measurement unit, a least one installer mobile device with a UWB unit for distance measurement, and UWB beacons, wherein the control computer is configured to perform a mapping process for an auditorium, and to manage in real time interactivity of user mobile devices in the auditorium during a performance, in which: for the mapping process the control computer is configured to, for each of a plurality of auditorium seats: receive a seat identifier, measure a distance between an installer mobile device at that seat location and each beacon on the stage, and according to said measurements generate a 3D map of the stage relative to the seats, for real time performance interactivity, the system is configured to: by the control computer, record a performance comprising 3D positions above the stage of props including a plurality of prop-position pairs, in which each propposition pair is associated with interactivity windows in which a user should perform an action with their mobile device while it is orientated towards a prop on the stage during that time window, by the control computer, download the 3D map and the performance to each user mobile device performance application which registers with a seat identifier for a performance, and by each user mobile device, in real time during a performance, monitoring user actions on their mobile device and determining if required actions occur when the mobile device is orientated towards a prop on the stage and during a time window associated with that prop for that action.

[0013] In some preferred examples, the installer mobile device is configured to determine a calibration setting for a beacon and the installer mobile device by comparing each of a plurality of UWB measurements with physically measured distances relative to a beacon; and wherein the installer mobile device is configured to use said calibration settings when determining distances between the installer mobile device and each beacon when at each seat location.

[0014] In some preferred examples, there are a plurality of beacons, and the installer mobile device is configured to determine a calibration setting for each beacon, in which there is at least one front beacon at a (0, 0, 0) position on a stage floor at a front of the stage, and there is another beacon set back from the front beacon by a Z value and at a height Y relative to the stage floor, and there is a beacon on each lateral side of the front beacon.

[0015] In some preferred examples, each user mobile device is configured to perform a synchronization before each performance by verifying the seat position.

[0016] In some preferred examples, each user mobile device is configured to perform said synchronization by capturing a visual image of the stage from the seat position and comparing the stage image with an expected reference stage image in the 3D map.

[0017] In some preferred examples, each user mobile device is configured to record a starting device orientation at the start of each performance and to subsequently use its inertial measurement unit to monitor change in orientation from the starting orientation to determine if the device is at a correct orientation during each interactivity window.

[0018] In some preferred examples, each user mobile device is configured to store and display video content during a performance, the content being related to the prop positions and interactivity time windows.

[0019] In some preferred examples, the content provides video images of a virtual target adjacent the prop, and the user action is pressing a button on the mobile device to simulate shooting of the virtual target and a success score is recorded if the user device is aligned with the location adjacent the prop and the action occurs within the associated time window. In some preferred examples, each user mobile device is configured to record and upload to the control computer the success or failure of each interactivity action for its associated seat identifier.

[0020] In some preferred examples, each user mobile device is configured to, during a performance, only communicate scores with the control computer in real time, the determination of scores being by local processing on the user mobile device.

[0021] In some preferred examples, each user mobile device is configured to verify a seat position and to verify orientation of the user mobile device relative to a prop position by performing object- oriented processing in which an Augmented Reality AR camera executed n software by the user mobile device is located at the seat position, with a stage object being a child of a seat object and having an offset provided by the 3D map.

[0022] In some preferred examples, said offset is in both 3D position terms and in terms of pitch, yaw, and roll values.

[0023] We also describe a method of auditorium mapping and performing contextualized augmented reality interaction in the auditorium, the method being performed by a system of any preceding claim, and the method comprising steps of: performing a mapping process by the control computer for each of a plurality of auditorium seats: receiving a seat identifier, measuring a distance between the installer mobile device at that seat location and each beacon on the stage, and according to said measurements generating a 3D map of the stage relative to the seats, performing real time performance interactivity in which: the control computer records a performance comprising 3D positions above the stage of props including a plurality of prop-position pairs, in which each propposition pair is associated with interactivity windows in which a user should perform an action with their mobile device while it is orientated towards a prop on the stage during that time window, the control computer downloading the 3D map and the performance to each user mobile device performance application which registers with a seat identifier for a performance, and each user mobile device, in real time during a performance, monitoring user actions on their mobile device and determining if required actions occur when the mobile device is orientated towards a prop on the stage and during a time window associated with that prop for that action.

[0024] In some preferred examples, the installer mobile device determines a calibration setting for a beacon and the installer mobile device by comparing each of a plurality of UWB measurements with physically measured distances relative to a beacon; and the installer mobile device uses said calibration settings when determining distances between the installer mobile device and each beacon when at each seat location.

[0025] In some preferred examples, there are a plurality of beacons, and the installer mobile device determines a calibration setting for each beacon, in which there is at least one front beacon at a (0, 0, 0) position on a stage floor at a front of the stage, and there is another beacon set back from the front beacon by a Z value and at a height Y relative to the stage floor, and there is a beacon on each lateral side of the front beacon.

[0026] In some preferred examples, each user mobile device performs a synchronization before each performance by verifying the seat position.

[0027] In some preferred examples, each user mobile device performs said synchronization by capturing a visual image of the stage from the seat position and comparing the stage image with an expected reference stage image in the 3D map.

[0028] In some preferred examples, each user mobile device records a starting device orientation at the start of each performance and subsequently uses its inertial measurement unit to monitor change in orientation from the starting orientation to determine if the device is at a correct orientation during each interactivity window.

[0029] In some preferred examples, each user mobile device stores and displays video content during a performance, the content being related to the prop positions and interactivity time windows. In some preferred examples, the content provides video images of a virtual target adjacent the prop, and the user action is pressing a button on the mobile device to simulate shooting of the virtual target and a success score is recorded if the user device is aligned with the location adjacent the prop and the action occurs within the associated time window.

[0030] In some preferred examples, each user mobile device records and uploads to the control computer the success or failure of each interactivity action for its associated seat identifier.

[0031] In some preferred examples, each user mobile device, during a performance, only communicate scores with the control computer in real time, the determination of scores being by local processing on the user mobile device.

[0032] In some preferred examples, each user mobile device verifies a seat position and verifies orientation of the user mobile device relative to a prop position by performing object-oriented processing in which an Augmented Reality AR camera executed in software by the user mobile device is located at the seat position, with a stage object being a child of a seat object and having an offset provided by the 3D map.

[0033] In some preferred examples, said offset is in both 3D position terms and in terms of pitch, yaw, and roll values.

[0034] We also describe s contextualized augmented reality system comprising a control computer, a plurality of user mobile devices each with an inertial measurement unit, a least one installer mobile device with a UWB unit for distance measurement, and UWB beacons, wherein the control computer is configured to perform a mapping for an auditorium, and to manage in real time interactivity of user mobile devices in the auditorium during a performance, in which: for the mapping the control computer is configured to, for each of a plurality of auditorium seats: receive a seat identifier, measure a distance between an installer mobile device at that seat and each beacon on the stage, and according to said measurements generate a 3D map of the stage relative to the seats, for the real time performance interactivity, the system is configured to: by the control computer, record a performance comprising 3D positions above the stage of props each prop-position pair being associated with interactivity windows in which a user should perform an action with their mobile device while it is orientated towards a prop on the stage during that time window, by the control computer download the 3D map and the performance to each user mobile device performance application which registers with a seat identifier for a performance, and by each user mobile device, in real time during a performance, monitoring user actions on their mobile device and determining if required actions occur when the mobile device is orientated towards a prop on the stage and during a time window associated with that prop for that action.

[0035] In some preferred examples, the installer mobile device is configured to determine a calibration setting for a beacon and a mobile device by comparing each of a plurality of UWB measurement with actual measured distances relative to a beacon; and wherein the installer mobile device is configured to use said calibration settings when determining distances between the installer mobile device and each beacon when at each seat location.

[0036] In some preferred examples, there are a plurality of beacons, and the installer mobile device is configured to determine a calibration setting for each beacon, in which there is at least one beacon at a (0, 0, 0) position on a stage floor at a front of the stage, and there is another beacon set back from by a Z value from the front of the and at a height Y relative to the stage floor, there is a beacon on each lateral side of the stage.

[0037] In some preferred examples, wherein each user mobile device is configured to perform a synchronization before each performance by verifying the seat position.

[0038] In some preferred examples, each user mobile device is configured to perform said synchronization by capturing a visual image of the stage from the seat position and comparing the stage image with an expected reference stage image in the 3D map.

[0039] In some preferred examples, each user mobile device is configured to record a starting device orientation at the start of each performance and to subsequently use its inertial measurement unit to monitor change in orientation from the starting orientation to determine if the device is at the correct orientation during each interactivity window. In some preferred examples, each user mobile device is configured to store and display video content during a performance, the content being related to the prop positions and interactivity time windows.

[0040] In some preferred examples, the content provides video images of a virtual target adjacent the prop, and the user action is pressing a button on the mobile device to simulate shooting of the virtual target.

[0041] In some preferred examples, each user mobile device is configured to record and upload to the control computer the success or failure of each interactivity action for its associated seat identifier.

[0042] In some preferred examples, each user mobile device is configured to, during a performance, only communicate scores with the control computer in real time, the determination of scores being by local processing on the user mobile device.

[0043] In some preferred examples, each user mobile device is configured to verify a seat position and to verify orientation of the user mobile device relative to a prop position by performing object- oriented processing in which an Augmented reality AR camera is located at the seat position, with a stage object being a child of a seat object and having an offset provided by the 3D map.

[0044] In some preferred examples, said offset is in both 3D position terms and in terms of pitch, yaw, and roll values.

[0045] Detailed Description of the Invention

[0046] The invention will be more clearly understood from the following description of some embodiments thereof, given by way of example only with reference to the accompanying drawings in which:

[0047] Fig. 1 is a diagrammatic perspective view of an auditorium, showing the stage and the position of anchor beacons of a contextualized augmented reality system of the invention, and Fig. 2 is a plan view;

[0048] Fig. 3 is a flow diagram for operation of data processors of the system, primarily a software application and ultra wide band (UBW) functionality on an installer device / smartphone, UWB beacons at different positions on a stage, and a control computer, for system mapping of a particular auditorium; and

[0049] Fig. 4 is a flow diagram for operation of the system real time for interactivity during a performance in the mapped auditorium.

[0050] A contextualized augmented reality system of the invention is a technical platform to provide an AR enhanced stage show in an auditorium to include contextualised AR gaming features with audience participation and interactivity.

[0051] The system primarily performs the following functions:

[0052] (a) Initially mapping the auditorium using anchor beacons, at least one installer device / smartphone with UWB functionality, and a laptop computer on site with an interface to communicate with central servers.

[0053] (b) Managing real time audience participation individually for each audience member, using dedicated software on a UWB-enabled smartphone of each user (audience member), and the control computer in real time communication with these smartphones. For this aspect, advantageously the majority of the processing is done locally on each smartphone in order to minimise the bandwidth required for real time communication with the servers. In some examples some of the processing is also done by a computer in the auditorium, especially for live performances.

[0054] The control computer manages the AR functions for multiple auditoriums, each one being registered with the control computer and being mapped with the mapping data being stored for each. Overall, the digital data processors which implement the system are part of the user smartphones with the executable code being provided by a software application which is registered with the control computer, and by digital data processors in the control computer (which may be a bank of servers located remotely or in the Cloud). The system further comprises anchor beacons on the stage of each auditorium during the mapping phase.

[0055] In summary, the mapping associates each auditorium seat location with the space above the stage. Then during a performance, the control computer triggers on each user smartphone: physical presence of an “asset” or “portal” or “prop” on the stage at a particular 3D location, and a time window for a user interaction by making an input to their smartphone such as pressing a button when the smartphone is oriented correctly towards the prop.

[0056] The user interaction may be to align with the space adjacent the prop location. Each successful interaction is recorded as a score. This invention is not concerned with the nature of the game / performance, rather it concerns the technical aspects of the system to manage the interactivity.

[0057] At the performance level the interactivity may, for example, be shooting bats which fly out of a “Pandora’s Box”; the box is the prop physically on the stage and the smartphone generates video of bats flying out of it if the smartphone is correctly aligned with the space above the box. The actor opens the Pandora’s Box prop on the stage at a particular time as set for the performance, or on-the-fly when instructed to by an auditorium computer under control of performance director. The user’s smartphone, executing the software of the system’s application for this performance, can compute using its in-built inertial measurement unit, if the smartphone is directed at this particular space above the Box and generates a display of bats flying out of the Box. If the user is pointing the smartphone correctly, as if taking a photo of the focused space above the Box, and if this is done during the correct pre-appointed time window, the App records a score and uploads it to the control computer. The latter maintains a score for each audience member.

[0058] The system achieves this despite the fact that one user’s seat is for example on the left-hand side of the front row whereby the bats are at a 63° angle and 5 m distance, but another user is located in the back row on the right and sees them at -44° and from 13 m. This is because the mapping phase maps each seat position of the auditorium to the space above the stage and each user smartphone in real time uses its inertial measurement unit to track the smartphone orientation to determine if the orientation is correct for alignment with the space above the prop. The performance data downloaded by the control computer gives the App the correct interactivity time window.

[0059] The system uses Ultra Wide Band Chips in the user devices for the initial calibration and recording of anchor beacons on the stage, as described in more detail below.

[0060] Mapping

[0061] Referring to Fig. 3 in a mapping process 50 an installer inserts details of the auditorium stage in step 51 into a 1 control computer (preferably portable such as a laptop computer) which is programmed to perform calibration in conjunction with installer smartphones (or other mobile device which is UWB enabled, such as a tablet). These details include X, Y, Z position coordinates on the stage for a datum beacon 30, at the front centre of the stage 1. In step 52 beacons 30-33 are placed on the stage as follows:

[0062] Middle centre, datum (0, 0, 0), front centre of stage.

[0063] Front right 31 at a known elevation above the (X-Z) plane of the stage.

[0064] Front left 32, at the same distance from the front of the stage and the same distance from the datum beacon 30 and at the same elevation as the beacon 31.

[0065] Rear centre, 33, and again at the same elevation as the beacons 31 and 32.

[0066] Each beacon 30-33 is capable of responding to UWB radiation emitted by the installer device when it is located at each seat 10 of the auditorium.

[0067] In step 53 the installer device is placed at a known and measured distance from a beacon, and in step 54 a UWB reading is taken for the distance as measured by the UWB function of the device. The device compares the known distance with the measured distance and performs an internal distance calibration. This is repeated for each of 5 positions, each further from the beacon as indicated by the step 55. In step 56 the device determines, using all of the five readings, a calibration setting for this beacon, and as indicated by step 60 this is repeated for each of the four beacons 30-31. This provides a calibration setting for the installer smartphone UWB function. This is a value which can subsequently be used by each user smartphone, taking the physical layout of the auditorium into account.

[0068] Next, in step 61, the installer device is placed in a seat and pointed at the centre of the stage. The App then determines in step 62 a desired orientation for the device to point at the four beacon locations. It may, depending on the software configuration, also determine correct installer device orientations for other positions in 3D above the stage, these orientations being derived from the orientations for the known beacon 30-33 positions. The orientations are determined using the inertial measurement unit of the device, which has accelerometers and gyroscopes to determine the orientation as the installer points the device at the particular above-stage positions. Thet installer smartphone UWB function determines a distance from each beacon, and the mapping software relates this to an X, Y, and Z seat position and to rotation coordinates of roll, pitch, and yaw values.

[0069] Steps 61 and 62 are repeated for each seat, as indicted by the step 63.

[0070] The orientation data determined by the installer device for each seat is uploaded to the on-site control computer, which consolidates this data to provide a full mapping for the auditorium.

[0071] Example Calibration Method

[0072] Extend a measuring tape all the way along a flat surface, and preferably mark down every 10 cm.

[0073] Place a beacon 30-33 down on one end of the tape, at 0 cm.

[0074] In a UWB-Test application on the installer device change a beaconlD identifier global value to the identifier of the beacon.

[0075] Starting at 10 cm, put the installer device down so that it's facing the selected beacon and start the test application.

[0076] Record the true distance from the selected beacon to the installer device.

[0077] The installer device displays five main items as follows:

[0078] Sample Size, the number of measurements taken so far (maximum 20). New samples are added in a queue once 20 samples have been reached.

[0079] Average Distance is the average of the 20 samples collected.

[0080] Plus Offset, the current distance measurement + the offset value that this calibration is trying to determine. This value is used to check if the calibration was successful.

[0081] Read distance is the latest UWB range value. This number will appear to change frequently.

[0082] Clear Samples. This is a button. When pressed is clears out all the samples currently held.

[0083] Once 20 samples has been collected the Average Distance value is stable.

[0084] The installer device records this value beside the 10 cm value. The Average Distance value will go up and down. It usually stays within a range of + / - 5 cm.

[0085] The technician then moves the installer back 10 cm and repeats the above steps. This sequence is then repeated for each phone position until at 5 meters.

[0086] The installer device software then determines the difference between each true distance value and it's corresponding UWB measurement and averages the results to obtain an average error. This is the offset value. This is inserted into the test app, which checks to see how close the 'Plus offset' value is to the true value at various ranges.

[0087] The above steps are repeated for each beacon in the set, to check that the offset works for them too.

[0088] Performance Generation

[0089] The control computer generates games using inputs by game designers. From the technical viewpoint, the aspects of game generation relevant to the invention is that the game defines a time series from time zero including a sequence of AR events, each AR event having a time window of a number of seconds and a 3D position for the prop. In the above example, a box is placed by an actor at a specific pre-determined location on the stage and it is there at a particular time. The game coordinates which are downloaded to the user device specify the location and the time window for this event. The game software which is developed provides position coordinates for the prop and the window for the activity of bats flying out of the box. After it is downloaded to the user device the device App manages the performance by knowing these details, and requiring that the device be pointed to this position and the user pressing an activation button during this event time window and with the device orientated correctly in order for a score to be recorded.

[0090] Where the performance is a cinema film then the prop positions (items displayed at a particular location on a screen) and the time windows are known in advance very accurately. However, for live performances it is envisaged that, while the prop positions will be pre-set (for example, on a table at a particular position) the time windows may be provided on-the-fly by a director in the auditorium, by wireless communication with the user devices.

[0091] Real Time Performance Management

[0092] In a process 80 a user activates the game App on their mobile device, and this sets up a session with the control computer. This session starts with uploading of the seat identifier in step 81, upon which the control computer downloads to the user device in step 82 the event data sequence from a nominated performance start time.

[0093] The App then automatically performs a test by synchronising the device position relative to the stage. The downloaded mapping data allows the device App to determine the view of the stage from the particular seat position. This view may be generated by the user device App, or by the control computer and downloaded. The user points the phone / device towards the stage and takes a photo, and the App verifies that the image of the stage corresponds to what is expected for this mapped seat position. Moreover, the synchronization of step 83 involves the device App anchoring the stage in software to the seat position, the stage object becoming a child object of the seat position object in the object-oriented paradigm. The stage has an object which is made a child of an AR camera object and an offset of the seat position is added to the stage position. Also, once synchronized, in step 84 the user device App relates the seat position to that of each prop for the performance using the object-oriented processing as set out below.

[0094] Then, the App uses a feed from its inertial measurement unit of deltas from the stage home orientation and from the position of each prop to track orientation of the device during the performance in real time. The performance start time is known by the device, and it performs processing in steps 90, 91, and 92 to determine if the user activates a “shot” when the device is being held at the current orientation and during the relevant window for an event. For example, the box may be placed at a 1 m height (Y value) and with (X, Z) coordinates of (3 m, 2 m) during a time window of 12 minutes and 30 seconds into the performance time. The App performs locally on the user device the computing which is required to determine that the user performs the action with the orientation of the user device pointed to this box location and during this time window. If the activation is correct the device records a score in step 93. These steps are repeated for each AR interactivity window of the performance, as indicated by the decision step 94.

[0095] Example Performance Interactivity

[0096] Step 1

[0097] The user mobile device App generates a display with the performance code and prompts the user to enter a seat number.

[0098] The user enters his / her seat number in the relevant text field.

[0099] After entering the seat number, the user presses the Next button.

[0100] After pressing the Next button, the control computer checks whether the user has entered a valid seat number. The control computer checks the seat number via a JSON file (SeatsList.json) and also reads the coordinate position in 3D world space of that seat number. If the seat number is validated, then an AR camera will automatically go to the position retrieved from the JSON file. TheAR camera will be placed on that seat number’s position. For example, if Hl has been entered and its coordinate value in 3D space is (x: -4.85, y: 3.29, z: -8.24) the AR camera will placed at that position.

[0101] Seat Hl Position: Position: X= -4.85, Y = 3.29, Z = -8.24 (relative to the front and centre of stage (mapping beacon 30) (0, 0, 0) position.

[0102] Rotation: X = 0, Y = 0, Z = 0 (pitch, roll, yaw)

[0103] AR Camera Position:

[0104] Position: X= -4.85, Y = 3.29, Z = -8.24

[0105] Rotation: X = 0, Y = 98.719, Z = 0

[0106] Step 2

[0107] After placing the AR camera to that seat number’s coordinate the device App calculates the distance (offset) between the AR camera and the stage:

[0108] Offset = stageObj. position - playerTransform. position

[0109] Step 3

[0110] After calculating the distance, the device App makes the stage object an object-oriented child of the AR camera and adds an offset to the stage position which is calculated in step number 2 so whenever the AR camera moves or the device moves the stage will be moved along with the AR camera: stageObj .parent = arCamera; stageObj .localPositi on = stageObj. position + offset

[0111] Step 4

[0112] When the user chooses his / her appropriate position for synchronising the stage then he / she will press a Sync button. Then, the device App makes the parent of the stage null, and then creates an anchor of the stage’s position. After creating the anchor, the stage is changed to be a child of that anchor position to prevent drifting of the stage. So, if anchor moves from its position, it will be automatically replaced to its position: reSynching = false; print(“ Anchor Setting”); stageObj .parent = null;

[0113] GameManager.Instance.stageFinalPosition = stageObj .position;

[0114] GameManager.Instance.stageFinalRotation = stageObj. rotation; GameManager Instance. cameraAngles = session. transform. rotation;

[0115] UNITY EDITOR GameManager.Instance. anchor = GameManager.Instance.anchorManager.AddAnchor(new

[0116] Pose(stageObj .position, stageObj .rotation)); switch (GameManager.Instance. gameName) {

[0117] Case GameName. TranningShoot: targetShootGame.gameObj ect. SetActive(true); targetShootGame.targetCanvas.SetActive(true); targetShootGame.TopBasedParent.SetActive(true);

[0118] UNITY EDITOR targetShootGame.lastPosition =

[0119] GameManager.Instance. anchor.transform. position; targetShootGame.lastRotation =

[0120] GameManager.Instance. anchor.transform. rotation; stageObj .parent = GameManager.Instance. anchor.transform;

[0121] StartCoroutine(targetShootGame.onStartBtnClick()); break;

[0122] Step 5 Six props A, B, C, D, E and F are placed on the stage based on stage’s size percentage and also set size of the props.

[0123] Step 6

[0124] Stage size: Width 13m, Depth 8.3m, Height 6.87m.

[0125] Prop A is placed 70 % away from the front of the stage which is 5.81 and occupying 20 % of the size of the stage:

[0126] X = 0, Y = 2.37, Z = 5.81 Prop B on the right side of the stage in the front corner of the stage and it is occupying 1 meter area of the stage (diameter of portal B is 1 meter):

[0127] Position: X = -6, Y = 1.634, Z = 2.75

[0128] Rotation: X = -99.246, Y = -74.72699, Z = 5.287003

[0129] Prop C is placed in the totally opposite position of the portal B and it is also occupying 1 meter area of the stage, (diameter of portal c is 1 meter).

[0130] Position: X= 6, Y = 1.63, Z = 2.75

[0131] Rotation: X = -105.865, Y = 45.464, Z = 16.922

[0132] Prop D is placed 75 % away from the front of the stage which is 6.225 and 20 % away from the right side of the stage 2.6 and occupying 10 % of the size of the stage.

[0133] Position: X = -2.6, Y = 1.63, Z = 6.225

[0134] Rotation: X = -105.865, Y = 0, Z = -16.922

[0135] Prop E is placed 75 % away from the front of the stage which is 6.225 and 75 % away from the right side of the stage 3.25 and occupying 10 % of the size of the stage.

[0136] Position: X = 3.25, Y = 1.63, Z = 6.225

[0137] Rotation: X = -105.865, Y = 0, Z = 16.922

[0138] Prop F is placed in front of the stage and 75 % away from the right side of stage 3.25 and occupying 10 % and to the height of the size of the stage. Which is 11.42

[0139] Position: X = 3.25, Y = 11.42, Z = 0

[0140] Rotation: X = -105.865, Y = 0, Z = 16.922

[0141] The installer device takes advantage of its ultra wide band (UWB) component’s accuracy to capture the positions of where a smartphone would be held by an audience member while in each seat. The system can be used during blocking as well, to register where props will be. So, calibration for each auditorium can include registration of the locations of props in relation to the stage, and of each seat.

[0142] After turning on the App and entering their seat number on the first screen, the user is instructed to hold their device parallel to the stage. Once they've done so and clicked a confirm button, the camera will turn on and the user will be able to see the virtual world. If during the show the user has to put their device away temporarily, e.g. to go to the bathroom or a phone call, the user can repeat the above process to get back to the show.

[0143] As noted above the auditorium has four UWB beacons 30-33 for calibration. This is preferred because it achieves full mapping of the space above the stage without many beacons. The precise distance measurements from the four UWB beacons 30-33 determine a 3D coordinate using an implementation of the algorithm made by: https: / / github . com / akshayb6 / trilaterati on-in-3 d

[0144] In the above example the mapping is performed by a laptop computer and a smartphone. The beacons 31, 32, and 33 are mounted on a stand at a known height above the stage floor, preferably in the range of 1.5 and 2 m. The middle of the front edge of the stage is the centre point (0, 0, 0) and there is a requirement that the beacons don't all share the same plane, and so the beacon 30 is at floor level whereas the others are elevated. For example, for a stage 10 meters across the positions may be as follows:

[0145] Beacon 30: (0, 0, 0). Measure 0.5 m towards the back of the stage.

[0146] Beacon 33: (0.0, 2.0, 2.0). Measure 2 m towards the back of the stage, then put the anchor 2 m above that point.

[0147] Beacon 31 : (4.0, 1.5, 0). Measure 4 m to the right of the centre, then up 1.5 m.

[0148] Beacon 32: (-4.0, 1.5, 0). Measure 4 m to the left of the centre, then up 1.5 m, then up 1.5 m

[0149] The phone is orientated as if to take a picture of each beacon in turn. This is because the UWB antennae are located on the back of the phone. The phone should be held so that the top of the phone isn't covered. This is because the phone’s Bluetooth camera is located here. The phone will automatically look for beacons to connect to. Pressing a particular button on the phone sends a message to the computer to save the current coordinates as a seat in a file called " SeatPositions.j son"

[0150] A lidar scan can be used in unity to visualize the seats and check if any errors in placement have occurred. The lidar scan software positions and rotates the model so that the centre of the front edge is at (0, 0, 0) and the stage itself lies in the +Z plane. Each beacon 30-33 has no internal memory, and its only function is to return the range signal from the Tag (UWB capable installer device that is hand-held). Range is calculated using 'Time of Flight' of the UWB signal as it travels between the anchor and the tag.

[0151] The hardware of the anchors can be different to the point that there will be a noticeable error in the returned distance (typically up to 20 cm). If the distance too long the anchor is not within line- of-sight of the Tag.

[0152] Line-of-sight calibration requires that nothing in the environment can disrupt the signal, especially concrete and metal. Placing the anchors far back on the stage is necessary for this reason, so that the anchors can see the Tag when it's in the front row.

[0153] In many smartphones the UWB antennae are on the back of the device, and so the other hardware in the phone (the battery, motherboard, screen, etc) can disrupt the UWB signal. This problem is addressed by keeping the back of the phone facing towards the beacons during ranging. The action is akin to taking a photo of the anchors with the back camera.

[0154] It will be appreciated that in various examples of the invention a contextualized augmented reality system that has a control computer, user mobile devices each with an inertial measurement unit, an installer mobile device with an Ultra Wide \band (UWB) unit for distance measurement, and UWB beacons on a stage effectively perform a mapping for an auditorium and manage in real time interactivity of user mobile devices in the auditorium during a performance. The mapping is for each auditorium seat, and the control computer receives a seat identifier and records a distance between a mobile device at that seat and each beacon on the stage. For the real time AR interactivity, the control computer records a performance timeline with a series of interactivity windows in which a user should execute an action on their mobile device while it is orientated towards a prop on the stage. Each user mobile device in real time during a performance monitors the user actions and determines if each occurs when the device is orientated towards a certain prop on the stage and during a certain time window. These processor-intensive operations are performed only by the user mobile deice using software of the system and its inertial measurement unit. There is real time uploading of scores, but this requires little bandwidth.

[0155] In use the system operates in the following major steps:

[0156] The auditorium is mapped for each seat position relative to the stage, and for prop positions on the stage. This is done using UWB measurements of an installer mobile device. A 3D map is provided. This is very accurate, and versatile as it accommodates changes in auditorium configurations for different performances.

[0157] A performance is generated with time windows from a performance start time, each window being associated with a prop stage position. Hence the system can be used for multiple performances, provided each is calibrated and recorded.

[0158] At the start of the performance the user mobile device is synchronized by verifying the seat position, on the basis that the user device should capture an image of the stage which is predicted by the 3D map. This achieves very effective initial verification to help ensure accurate management of user interactivity in rea time.

[0159] During the performance the user mobile device automatically tracks the planned performance prop positions using object-oriented technology and determines according to its inertial measurement unit if it has moved to be pointing to a prop during the associated time window to, for example, shoot bats flying from a Pandora’s Box on the stage. The Box is real and is at its pre-planned physical position, and the bats are images generated by the user mobile device App. It will therefore be appreciated that the invention provides a very effective technical platform for any of a number of performances in an auditorium, wit real time management of user interactivity.

[0160] Components of embodiments can be employed in other embodiments in a manner as would be understood by a person of ordinary skill in the art. The invention is not limited to the embodiments described but may be varied in construction and detail.

Claims

Claims1. A contextualized augmented reality system comprising a control computer, a plurality of user mobile devices each with an inertial measurement unit, a least one installer mobile device with a UWB unit for distance measurement, and UWB beacons (31-33), wherein the control computer is configured to perform a mapping process for an auditorium, and to manage in real time interactivity of user mobile devices in the auditorium during a performance, in which: for the mapping process (50) the control computer is configured to, for each of a plurality of auditorium seats: receive (61) a seat identifier, measure (62) a distance between an installer mobile device at that seat location and each beacon on the stage, and according to said measurements generate (62) a 3D map of the stage relative to the seats, for real time performance interactivity, the system is configured to: by the control computer, record a performance comprising 3D positions above the stage of props including a plurality of prop-position pairs, in which each prop-position pair is associated with interactivity windows in which a user should perform an action with their mobile device while it is orientated towards a prop on the stage during that time window, by the control computer, download (82) the 3D map and the performance to each user mobile device performance application which registers with a seat identifier for a performance, and by each user mobile device, in real time (90-94) during a performance, monitoring user actions on their mobile device and determining if required actions occur when the mobile device is orientated towards a prop on the stage and during a time window associated with that prop for that action.

2. A contextualized augmented reality system as claimed in claim 1, wherein the installer mobile device is configured to determine (56) a calibration setting for a beacon and the installer mobile device by comparing (53) each of a plurality (55) of UWB measurements with physically measured distances relative to a beacon; and wherein the installer mobile device is configured to use said calibration settings when determining distances between the installer mobile device and each beacon when at each seat location.

3. A contextualized augmented reality system as claimed in claim 2, wherein there are a plurality of beacons (30-31), and the installer mobile device is configured to determine a calibration setting for each beacon, in which there is at least one front beacon (30) at a (0, 0, 0) position on a stage floor at a front of the stage, and there is another beacon set back from the front beacon by a Z value and at a height Y relative to the stage floor, and there is a beacon (31, 32) on each lateral side of the front beacon.

4. A contextualized augmented reality system as claimed in claim 2 or claim 3, wherein each user mobile device is configured to perform (83) a synchronization before each performance by verifying the seat position.

5. A contextualized augmented reality system as claimed in claim 4, wherein each user mobile device is configured to perform (83) said synchronization by capturing a visual image of the stage from the seat position and comparing the stage image with an expected reference stage image in the 3D map.

6. A contextualized augmented reality system as claimed in any preceding claim, wherein each user mobile device is configured to record a starting device orientation at the start of each performance and to subsequently use its inertial measurement unit to monitor (91) change in orientation from the starting orientation to determine if the device is at a correct orientation during each interactivity window.

7. A contextualized augmented reality system as claimed in claim 6, wherein each user mobile device is configured to store and display video content during a performance, the content being related to the prop positions and interactivity time windows.

8. A contextualized augmented reality system as claimed in claim 7, wherein the content provides video images of a virtual target adjacent the prop, and the user action is pressing a button on the mobile device to simulate shooting of the virtual target and a success score is recorded if the user device is aligned with the location adjacent the prop and the action occurs within the associated time window.

9. A contextualized augmented reality system as claimed in any of claims 4 to 8, wherein each user mobile device is configured to record and upload to the control computer the success or failure of each interactivity action for its associated seat identifier.

10. A contextualized augmented reality system as claimed in claim 9, wherein each user mobile device is configured to, during a performance, only communicate scores with the control computer in real time, the determination of scores being by local processing on the user mobile device.

11. A contextualized augmented reality system as claimed in any preceding claim, wherein each user mobile device is configured to verify a seat position and to verify orientation of the user mobile device relative to a prop position by performing object-oriented processing in which an Augmented Reality AR camera executed n software by the user mobile device is located at the seat position, with a stage object being a child of a seat object and having an offset provided by the 3D map.

12. A contextualized augmented reality system as claimed in claim 11, wherein said offset is in both 3D position terms and in terms of pitch, yaw, and roll values.

13. A method of auditorium mapping and performing contextualized augmented reality interaction in the auditorium, the method being performed by a system of any preceding claim, and the method comprising steps of: performing a mapping process (50) by the control computer for each of a plurality of auditorium seats: receiving (61) a seat identifier, measuring (62) a distance between the installer mobile device at that seat location and each beacon on the stage, and according to said measurements generating (62) a 3D map of the stage relative to the seats, performing real time performance interactivity in which: the control computer records a performance comprising 3D positions above the stage of props including a plurality of prop-position pairs, in which each prop-position pair is associated with interactivity windows in which a user should perform an action with their mobile device while it is orientated towards a prop on the stage during that time window,the control computer downloading (82) the 3D map and the performance to each user mobile device performance application which registers with a seat identifier for a performance, and each user mobile device, in real time (90-94) during a performance, monitoring user actions on their mobile device and determining if required actions occur when the mobile device is orientated towards a prop on the stage and during a time window associated with that prop for that action.

14. A method as claimed in claim 13, wherein the installer mobile device determines (56) a calibration setting for a beacon and the installer mobile device by comparing (53) each of a plurality (55) of UWB measurements with physically measured distances relative to a beacon; and the installer mobile device uses said calibration settings when determining distances between the installer mobile device and each beacon when at each seat location.

15. A method as claimed in claim 14, wherein there are a plurality of beacons (30-31), and the installer mobile device determines a calibration setting for each beacon, in which there is at least one front beacon (30) at a (0, 0, 0) position on a stage floor at a front of the stage, and there is another beacon set back from the front beacon by a Z value and at a height Y relative to the stage floor, and there is a beacon (31, 32) on each lateral side of the front beacon.

16. A method as claimed in any of claims 13 to 15, wherein each user mobile device performs (83) a synchronization before each performance by verifying the seat position.

17. A method as claimed in claim 16, wherein each user mobile device performs (83) said synchronization by capturing a visual image of the stage from the seat position and comparing the stage image with an expected reference stage image in the 3D map.

18. A method as claimed in any of claims 13 to 17, wherein each user mobile device records a starting device orientation at the start of each performance and subsequently uses its inertial measurement unit to monitor (91) change in orientation from the starting orientation to determine if the device is at a correct orientation during each interactivity window.

19. A method as claimed in claim 18, wherein each user mobile device stores and displays video content during a performance, the content being related to the prop positions and interactivity time windows.

20. A method as claimed in claim 19, wherein the content provides video images of a virtual target adjacent the prop, and the user action is pressing a button on the mobile device to simulate shooting of the virtual target and a success score is recorded if the user device is aligned with the location adjacent the prop and the action occurs within the associated time window.

21. A method as claimed in any of claims 13 to 20, wherein each user mobile device records and uploads to the control computer the success or failure of each interactivity action for its associated seat identifier.

22. A method as claimed in claim 21, wherein each user mobile device, during a performance, only communicate scores with the control computer in real time, the determination of scores being by local processing on the user mobile device.

23. A method as claimed in any of claims 13 to 22, wherein each user mobile device verifies a seat position and verifies orientation of the user mobile device relative to a prop position by performing object-oriented processing in which an Augmented Reality AR camera executed in software by the user mobile device is located at the seat position, with a stage object being a child of a seat object and having an offset provided by the 3D map.

24. A method as claimed in any of claims 13 to 23, wherein said offset is in both 3D position terms and in terms of pitch, yaw, and roll values.

Images