Methods and apparatus for projector camera systems

Abstract

A method of transforming data transmitted through a physical environment comprises the following steps An emitted signal is emitted into the physical environment. A detected signal is then detected from detection of the emitted signal after interaction with the physical environment. The detected signal is then transformed to modify a further emitted signal. These steps are repeated until a predetermined objective is met. A system suitable for performing such a method is also described. In particular embodiments, the emitted signal is emitted by a projector, and the detected signal is detected by a camera.

Description

[0001] METHODS AND APPARATUS FOR PROJECTOR CAMERA SYSTEMS

[0002] Field of Invention

[0003] The present invention relates to methods and apparatus for projector camera systems. While embodiments described in detail are particularly relevant to systems including both a projector and a camera, the principles described are applicable to other real-world systems.

[0004] Background to Invention

[0005] Emitted and detected signals - particularly projected images, and images captured from the environment - are typically processed using classical analytical techniques. It would be desirable to achieve new capabilities by using more modern analytical and data processing techniques, particularly in using additional capabilities provided by machine learning and artificial intelligence generally.

[0006] Summary of Invention

[0007] In a first aspect, the invention provides a method of transforming data transmitted through a physical environment, the method comprising: emitting an emitted signal into the physical environment; detecting a detected signal from detection of the emitted signal after interaction with the physical environment; transforming the detected signal to modify a further emitted signal; and repeating the emitting, detecting, and transforming steps until a predetermined objective is met.

[0008] Using iterative processing loops of this kind for emitted and detected signals - as discussed below, this approach is particularly effective with images, but can also be used with signals of other types (such as sound) - can be used to provide a number of new capabilities in image processing and information security. Starting typically with a null signal initially, use of such loops can achieve wide range of desired results. In embodiments, this approach can even be used to develop new types of computing engine.

[0009] In one embodiment, the emitted signal is an image emitted by a projector, such as a video projector (in which case the image is part of a video stream), and the detected signal is an image captured by a camera. Image capture may be synchronized with image projection - for a typical digital video projector the refresh rate may involve refresh approximately every 16ms. The transforming steps can be made at this rate for the most effective response - this involves significant data processing in the case of a typical 1920x1080 pixel projected colour image with a 5360x4600 detected colour image (these are exemplary values for typical systems) - however the loop could also be carried out at some multiple of the refresh rate.

[0010] In embodiments, the transforming step is performed using a generator model adapted to predict an emitted signal from a detected signal. This generator model may be a trained machine learning model. A suitable machine learning model may be an artificial neural network - this may be a temporal model, and could be implemented as a convolutional neural network.

[0011] In certain embodiments, the machine learning model is trained to blend a predetermined image into the further emitted image. In certain cases, this predetermined image is predetermined text, though it could be any other targeted loss function.

[0012] In embodiments, the method may further comprise modifying the generator model in real time - this may in principle be with each detected signal, though in practice it may occur less frequently.

[0013] In embodiments, the transforming step may comprise applying a predetermined style to the further emitted signal.

[0014] In embodiments, the transforming step may comprise operating on the detected signal using a one-way function. This one-way function may comprise a structured noise generator function. The emitting, detecting and transforming steps may be repeated a predetermined number of times to achieve the predetermined objective. In other embodiments, this process may run continuously. This predetermined objective may comprise providing a cryptographic hash, but may involve any other one-way objective. The output of such a method may be a secure data recording.

[0015] In embodiments, the method may be performed in parallel by a plurality of emitter-detector pairs. Here, the plurality of emitter-detector pairs may be configured to act as an adversarial autoencoder, with at least one of the plurality of emitter-detector pairs adversarial to the other emitter-detector pairs. There are various ways in which this structure can be used. In one, multiple agents achieve an outcome (optionally a one-way function) by shining on a shared scene - this is exemplified by encryption of a “real” scene. Another is emission-recording pair verification via autoencoder. In generation of a cryptographic hash as described above, such an adversarial autoencoder may be adapted to verify integrity of the cryptographic hash.

[0016] In further embodiments, an output of the method may be production of a 3D image. This may be achieved by Neural Radiance Fields (NeRF), or by other techniques such as point clouds or depth maps.

[0017] In further embodiments, the predetermined objective may be completion of a computation process.

[0018] In a second aspect, the invention provides a system for transformation of data transmitted through a physical environment, the system comprising: at least one emitter for emitting an emitted signal into the physical environment; at least one detector for detecting a detected signal from detection of the emitted signal after interaction with the environment; and computing means, wherein the computing means is adapted to transform the detected signal to modify a further emitted signal.

[0019] In embodiments, the emitter is a projector and the detector is a camera, and the computing means is a suitably programmed processor. The projector may be a video projector. In such a case, image capture by the camera may be matched to a refresh rate of the video projector.

[0020] In embodiments, the system is adapted to perform the method of one or more of the embodiments of the first aspect.

[0021] Figures

[0022] The following figures are provided with this specification for illustrative purposes.

[0023] FIGURE 1

[0024] 1 .01 : Projector

[0025] 1.02: Camera

[0026] 1 .03: Agent

[0027] FIGURE 2

[0028] 2.04: Speaker 2.05: Microphone

[0029] 2.06: Agent

[0030] FIGURE S

[0031] 3.07: Stereo Projector

[0032] 3.08: Right Channel Filtered Camera

[0033] 3.09: Left Channel Filtered Camera

[0034] 3.10: Agent

[0035] 3.11 : Microphone

[0036] 3.12: Speaker

[0037] FIGURE 4

[0038] 4.13: Right Channel Filtered Camera

[0039] 4.14: Left Channel Filtered Camera

[0040] 4.15: Stereo Projector

[0041] FIGURE 5

[0042] 5.16: Right Channel Unfiltered Camera

[0043] 5.17: Left Channel Unfiltered Camera

[0044] 5.18: Projector

[0045] FIGURE 6

[0046] 6.19: Right Channel Filtered Camera

[0047] 6.20: Left Channel Filtered Camera

[0048] FIGURE 7

[0049] 7.21 : Projector

[0050] 7.22: Camera

[0051] 7.23: Agent

[0052] FIGURE S

[0053] 8.24: Emitter, e.g. Projector or Speaker

[0054] 8.25: Reactor, e.g. Non-Linear Optical or Acoustic Medium

[0055] 8.26: Recorder, e.g. Camera or Microphone

[0056] FIGURE 9

[0057] 9.27: Emitter, e.g. Projector or Speaker

[0058] 9.28: Reactor, e.g. Non-Linear Optical or Acoustic Medium

[0059] 9.29: Recorder, e.g. Camera or Microphone

[0060] 9.30: Emitter, e.g. Projector or Speaker

[0061] FIGURE 10 10.31 : Recorder, e.g. Camera or Microphone

[0062] 10.32: Reactor, e.g. Non-Linear Optical or Acoustic Medium

[0063] 10.33: Emitter, e.g. Projector or Speaker

[0064] FIGURE 11

[0065] 11 .34: Fibonacci Spiral

[0066] 11 .35: High-Dimensional Recorder Array e.g. Cameras or Microphones

[0067] FIGURE 12

[0068] 12.36: High-Dimensional Recorder Array e.g. Cameras or Microphones

[0069] 12.37: Smooth Mating Between Recorders and Reactor, Optional Signal Damping e.g. Black

[0070] Paint or Acoustic Foam

[0071] 12.38: Reactor, e.g. Mirrored Cavity or Acoustic Cavity

[0072] 12.39: High-Dimensional Emitter Array e.g. Projectors or Speakers

[0073] FIGURE 13

[0074] 13.40: Low-Dimensional Recorder Array e.g. Camera or Mirophone

[0075] 13.41 : Dimensional Reduction Reactor e.g. Mirrored Cavity or Acoustic Cavity

[0076] 13.42: Optionally Smooth Mating Between Emitters and Reactor, Optional Signal Damping e.g.

[0077] Black Paint or Acoustic Foam

[0078] 13.43: High-Dimensional Emitter Array e.g. Projectors or Speakers

[0079] FIGURE 14

[0080] 14.44: Low-Dimensional Emitter Array e.g. Projector or Speaker

[0081] 14.45: Dimensional Expansion Reactor e.g. Mirrored Cavity or Acoustic Cavity

[0082] 14.46: Optionally Smooth Mating Between Recorders and Reactor, Optional Signal Damping e.g.

[0083] Black Paint or Acoustic Foam

[0084] 14.47: High-Dimensional Recorder Array e.g. Cameras or Mirophone

[0085] FIGURE 15

[0086] 15.48: Matrix of Agents, e.g. Array of Projector-Scene-Camera-Agent Loops

[0087] 15.49: Scene, e.g. Matte White-Coloured Mannequin

[0088] 15.50: Addressed Subset of Agent elements at i=1

[0089] 15.51 : Lower Bound of Addressed Agent elements at i=1

[0090] 15.52: Upper Bound of Addressed Agent elements at i=1

[0091] 15.53: Area of Overlap between Addressed Agent elements between i=1 and i=2

[0092] 15.54: Novel Area of Addressed Agents at i=2

[0093] 15.55: Area of overlap between i=3 and i=2

[0094] 15.56: Novel Area of Addressed Agents at i=3 Specific Description - Introduction

[0095] Embodiments of the invention will now be described by way of example with reference to the accompanying figures.

[0096] The description takes the following structure:

[0097] • Description of the general operating principles of the invention and its embodiments, including discussion of various processing types;

[0098] • Provision and explanation of pseudocode for specific processing types; and

[0099] • Detailed description and mathematical explanation for specific aspects of processing according to embodiments of the invention.

[0100] General Operating Principles

[0101] The general approach described is for repeated loops of emission into and detection from an environment of a signal. In most embodiments described below, this is a visual signal, but as is indicated in certain embodiments, it may be a more complex signal (for example, comprising image and sound) or it may be a non-visual signal. Between detection and further emission there is processing of the signal. This may involve use of a machine learning system, such as an artificial neural network (particularly a convolutional neural network). This may be thought of as a series of filters applied to an image (“image” will be used for convenience for the discussion that follows - from the statements above, it is clear that this discussion is not limited to images). These filters may distort, resize or recolour the image - how they do this may be adjusted during a training process.

[0102] Training involves manipulating these filters until they provide the desired output for a given input. One approach involves training of these filters to predict a resulting image. In essence, this involves treating the external world as equivalent to one of these filters (or sets of filter). This general approach provides a system that is readily applicable to a wide range of real-world problems.

[0103] A glossary of terms and key concepts will be provided below. This is followed by basic implementations of a variety of key concepts, with discussion of applications that become available using this approach. More detailed implementations are then described for specific examples, followed by an extended description with mathematical underpinning. Glossary of Terms and Key Concepts

[0104] Projection Signal De vice wctem cf image emitted ty the ro ecto'- ?n:c re sce^e

[0105] Captur ed Image The ivage tax- 'ec hy :-e ca-fte. vr ch n : udes the rc ecio-- 5 g-a and the ■■f ue‘-ce cd re :cene.

[0106] Generator Model: A nrch ”e eerrg nrde that p-’&d c:t pirjecc c" : aye : for capcLoed rages.

[0107] Comparative Loss: A o:c hrcti c-n red to reatire re d ffc 'e'Tf betwee1- p’ec cred era ect □■•’ sig nal a ”d the actus project on s-gne .

[0108] Style Function: ~iu"cd?n that apolie; e t oe: nt style c rensfcTr et c“ to an IT age cr :ig"el

[0109] Text Prompt’ A text. e 'put rat gu de: re ge'fatc-' iTvde to t e ’d vs-uel crpac wir tenua ' ■’fcivat cn.

[0110] One-Way Hash Function: A cyotog ’ao’' ■: ■'irction that r aps nout date co a f > ec-; ze ’•ar

[0111] ■,-a ue, vr'ch corrutad t-na' y Me a si ole to ’"evese

[0112] Str uctured Noise Generator A ■'motion that gene-ate; xi-ce petter r- ceted tn e g *en ha-r va □£. red to crate ir que o ’ eject cr : g”f c.

[0113] Adver sarial Autoencoder: A ne. 'a net / ver- arh tectire rat ccrro net- fut:en:;d 'g wir e civera da train ng to ear mtata ’eo 'et-emat on: e r detect anerra es

[0114] Domain Discriminator A fira' rtw ?x ccnrcr'&nt re: t'ctrgu fee oerwee - -ea c \ ge_erated date

[0115] Comparative Discriminator: A 'tree on that co ‘-ice rec etc nout; e ”d c -rpc.tr- a e ri e ’ ty score. fed 'O"’ oct t ciip^tic c"

[0116] Cryptographic Loop A zequen te Afe’e eec" tree :e;ei-d; ?- ;-e :ry teegra t" c cutout cf the tree cu: step. ens. ■ y= data ntegr ?y an : cecudy

[0117] Hash Cham A sequente cf haf vc ue; prev cue ■'3th, trectrg a the nteg ’ ry cf a sequence cf date

[0118] Anomaly Detection The trctec: z* demity 'g rzie z ette ■’*: o - :ata DO ■'t; that deviatei,'ori fe ‘'O‘T>'

[0119] Backpropagatlcm A -lefod fed ' ra 'i-g ‘'eioel "etvvoft; •A*re’e g ’adie'-ts fe confuted s -d trcoegetec ba:o,va'd to uodate mocel oere r.eters.

[0120] Opcmilzer An e go-' dvn that adyf t; re mode cere ITS ter to IT HIT :e the os: 'irctit-n

[0121] Fine-tuning --e pmcest- cf*u ‘rer :re n f e p-'e-tre ne: r'ode c" new seta to adept it to s oec fc cats: >r er- o ^'ierc. Modification - First of all, various types of modification process will now be described, with reference to pseudocode. All of these processes are performed with an emitter-detector pair -that is, for an image, a projector-camera pair. In the case of later processes, multiple emitter-detector pairs may be used.

[0122] 1. Dataset Recording with Projection Signals

[0123] This enables the collection of synchronized datasets of projection signal and captured images.

[0124] This may be employed, for example, with a travelling system - the datasets may represent images as the projector-camera system travels through space. Here, the projector emits known signals onto the scene, and the camera captures images as influenced by the scene and the projection signals. Only images and projection signals are captured in this example - no position data or inertial measurements are recorded (though this is in principle also possible.

[0125] 2. Training Generator to Predict Emissions from Captured

[0126] Images

[0127] This describes the training of a generator model to predict projection signals - emissions - from the captured images. The generator here takes a captured image as input, and predicts the corresponding projection signal. A loss value is computed by comparison of the predicted emission with the actual projection signal. Model parameters are then updated to minimize this loss using backpropagation and optimization techniques.

[0128] 3. Continuous Processing and Emitting of Styled Outputs

[0129] From bererator Import P’’edict_EFiissc.or>

[0130] Rrtm Sty le_Function Impo rt App ly_Style

[0131] Fran Proj ector Irr.oort Emit_P,’oj ection_Signal

[0132] From Camera Import Captu re_Image j / IftilH c Jrueyrrycr-y y

[0133] This generator function continuously processes captured images, applies a style to the predicted emissions, and projects them back onto the scene. Captured images are used to predict emissions, and these are then modified by a style function. These styled outputs are projected onto the scene. A feedback loop results.

[0134] 4. Training a Model to Blend Emission with Text Prompt Initialize count = G

[0135] While court < leogth [ Captu red_Iriace_Dataset ) :

[0136] =t Get the current captured inage

[0137] Gaptured_.IfT.age = Captu red_Inage_Dataset [count]

[0138] = ij.s e rhe generate' to predict a blended cutout using the captured text prompt ,, and weight

[0139] Blenced_Output = ° •-edict_'With_“evt_Pronrpt [ Gene''ato<'_Wodel, Capture Target_Text_Pron,pt, Weight )

[0140] = Ca lcu late the loss based or the weighted Diend of te rt and emiss comparisons te <.t '_c>ss = Text_Ta<'get_Lcss f fa rget_T Output ;

[0141] Emit t ionloss = Corrparative_ios ? ( Proj t [count ] . Blended_Dutput ) ots = Weight ' TextLcss + f l - Weignt ) ’ EmissionLoss

[0142] Bad prooagste the loss to update tne generate-' model parameters enerator_Ciptimize r . ze -'d_g racf ) adpropagate l Loss . bene-'ator_Mocel I enerator_Optiniizer . step( 1 ount = count ■* 1

[0143] Using the approach, the model can be trained to blend the predicted emission with a target text prompt- thus combining visual and textual information. The generator here uses the captured image and the text prompt to predict a blended output. A loss function is computed as a weighted sum of text similarity and emission similarity. In this way, the model learns to incorporate the text prompt into the predictions effectively.

[0144] 5. Fine-tuning the Model o^n Real-Time Recordings

[0145] Continuously

[0146] F-’om Generator Import InitiaLize_ModelrPredict_With_Tex t_Pronpt F-'o-n Loss_Functions Import Text_Tarcet__oss

[0147] F-'om Proj ector Import Emit_Proj ection_Signal

[0148] F rom Camera Import Captu re_Image

[0149] F -'OTI Optimizer Import Backpropagate, Optimizer

[0150] DefineTarget_Text_Pronpt

[0151] Define Weight - A zeAue between 0 anc 1 to Oalance tex t lost and any otner jfflosslsslfljlf dhiddrddllf jc d

[0152] “ Initialize the generato r mode l with text prorrot conditioning Gene rator-Mode l = Initialize_Mocel(

[0153] This approach enables the model to be fine-tuned in real time using new recordings, with the predictions adapted according to the new data. Here, the model updates its parameters on the fly as it receives new images and projection signals, and the feedback loop allows the system to adapt to changes in the environment, or to any other desired output.

[0154] Secure Functions - A further step beyond these uses of a generator as a modifying function is the use of the generator to provide secure output. This may for example use a oneway function to provide a hash output. fi« One-Way Hash Function Loop for Projection Emission

[0155] Using this approach, a cryptographic loop is created. One-way functions - hash functions - are used to secure the projector / camera system, ensuring privacy- preserving image capture and emission cycles. All data is here saved in a common directory- in this case, named after the initialisation vector for easy traceability.

[0156] The loop starts by establishing an initialisation vector (IV) that seeds a structured noise generator and initializes the hash. In each iteration, the emission is derived using this structured noise generator- this may be, for example, a Perlin Noise generator. This structured noise generator is used on the current hash and camera image, which is then emitted, and captured by the camera with the output used to derive a new current hash.

[0157] This loop continues for a defined number of iterations, with the final hash shared with an interlocutor to verify the timing of the whole process. Emissions, recordings, and hashes are stored in respective datasets, which are saved in a directory named after the IV for further analysis or verification.

[0158] 7. Training an Adversarial Autoencoder for Anomaly Detection

[0159] Recording = Recordings_Dataset [ij

[0160] A Gonsatenate the emission and recording Recording )

[0161] S Training loop for adversarial autoencoder jEpoOsf~)30€lfZTf ) if

[0162] For epoch in range) Epochs ) :

[0163] For data in Concatenated_Dara :

[0164] ® Fo>'Ward pass through the encoder and generator to reconstruct the input Latert_Representation = Emcoder( data)

[0165] Recorstri>cted_Data = Generator) Latent_Representation )

[0166] # Compute reconstruction loss forencoder and generator

[0167] Recon

[0168] Using this approach, an adversarial encoder can be trained to detect anomalies in the recording and projection process by learning a normal pattern of emission and recording pairs.

[0169] To do this, emission and recording datasets are concatenated to create training data representing both projected signals and captured responses. The adversarial encoder consists of an encoder, a generator (a decoder) and a domain discriminator. The encoder and generator together learn how to reconstruct the input data, while the domain discriminator is trained adversarially to distinguish between real and reconstructed data. The training loop optimizes the encoder, generator, and domain discriminator so as to minimize reconstruction loss and so maximize the discriminator’s ability to differentiate real from reconstructed data. 8. Verifying Hash Integrity and Analyzing Emission-Recording Pairs Using this approach, the integrity of the hash chain can be verified. It is determined whether the structured noise corresponds to the recorded emissions, and the domain discriminator is used to evaluate the authenticity of emission-recording pairs.

[0170] The first step is hash chain verification - this involves a check to ensure that each hash in the chain is correctly computed from the previous hash and the associated captured image. The next step is structured noise verification - this involves confirmation that the emissions match the structured noise generated from their respective hashes using a comparative discriminator and a predefined threshold. The following step is domain discriminator evaluation - this uses the trained domain discriminator to assess the authenticity of each emission-recording pair, detecting any anomalies or inconsistencies.

[0171] Applications of these techniques include the following:

[0172] Artistic Translation and Modification

[0173] • Prediction of the camera image from the projector image (e.g. for a fixed scene) allows evaluation of potential projections to assess visual appeal.

[0174] • Prediction of the projector image from the camera image may also be used o This can be used to improve coloration (by resonance or similar approaches). o A style function can be imposed on the image to provide a desired result.

[0175] Analytical Techniques

[0176] • The filters can transform the image into a label, such as a name or 3D model of a subject.

[0177] • Labels may be supervised (and explicit, such as a name) or unsupervised, such as Al generated content (e.g. estimated depth maps of each pixel from the camera).

[0178] Verification Techniques

[0179] • Use of the techniques set out above may be more effective than high-resolution images of electromagnetic field interactions (which are costly to simulate) or machine learning models to detect tempering (which are slow).

[0180] • The approach set out here should be relatively resistant to deep fakes. One-Way Function

[0181] • Information security is an obvious application - asymmetries can be detected and used advantageously.

[0182] • Adversaries can be detected and inhibited.

[0183] • A secure communication route can be provided.

[0184] • Physical world images can be effectively “watermarked” in recording with forgery-resistant timecodes.

[0185] • This approach should be relatively resistant to gradient-based attacks.

[0186] Computation

[0187] • The modification process can be used as part of a computation process. o For example, the process may provide a complex shape acting as if it were multiple layers of a neural network. o This could act as a one-way function, and would be versatile as it would be possible to add additional “layers”. o When paired with a light-field camera, this system is capable of complex calculation.

[0188] ■ A grid of projector-camera systems could represent large amounts of data.

[0189] ■ Sparse data - such as a very large Large Language Model - could be represented.

[0190] More detailed code will now be described for specific embodiments. Certain of these embodiments provide more detailed versions of examples provided above.

[0191] Example 1 - Projector- Camera System with Text-Based Training

[0192] This is described stage-by-stage

[0193] 1. Dataset Recording with Projection Signals

[0194] The projector emits a series of known signals onto the scene, the camera captures images influenced by both the scene and the projected signals, with the captured images and their corresponding projections signals stored in synchronized datasets - these can then provide training material.

[0195] 2. Training Generator to Predict Emissions from Captured Images

[0196] Here, the generator is trained to predict the original projection signals based on the captured images. The comparative loss measures the difference between the predicted emission and the actual projection signal. In the training process, the generator’s parameters are updated to minimise the loss and so improve the predictive capacity of the generator.

[0197] 3. Continuous Processing and Emitting of Styled Outputs

[0198] Here, the system continuously captures and processes images. A styling output is applied to the predicted emissions to achieve a desired visual effect. The styled outputs are projected back onto the scene. This creates a dynamic interaction between the environment and the system.

[0199] 4. Training a Model to Blend Emission with Text Prompt

[0200]

[0201] Using this approach, the optical flow loss is computed between the current blended output and the previous blended output within the same text prompt iteration. This supports temporal consistency and smooth transition between iterations.

[0202] 5. Fine-Tuning the Model on Real-Time Recordings Continuously

[0203]

[0204] This allows for continuous fine tuning usingthis approach.

[0205] Example 2 - Secure Recording

[0206] 1. Secure Recording with One-Way Hash Function Loop

[0207] Sa' / eTo[iireCtorv( DIFECTDRi'-’JANE, ENISSIOIIS— DATASET, "emit s lor s " I saeMTnnirprrnr ;i DIFEi TORI'- l^UE, FEi uPniHGS_DOTA>ET, ’ rer r,| dmgu" ) SaveToUirector', (■[HFECTORC-’IANE, HASHES-DATASET. ’iashei j

[0208] END Se, it oRei 11 i 111 g The hash chain ensures any tamperingwith the data can be detected, ensuring data integrity. Emissions are linked to the hash chain via structured noise generation. Emissions, recordings, and hashes are stored forverification.

[0209] 2. Training the Adversarial Autoencoder with Emphasis on the Discriminator

[0210] INITIALIZE Dlncrlninator WITH inputshape = l^atasnape = Input is concatenated

[0211] Here, the discriminator is the main focus - this operates on the data space (concatenated emissions and recordings). It is trained to distinguish between real data and reconstructed (fake) data. Training uses discriminator loss to drive the discriminator to correctly classify real and fake data. Generator loss is used so that the encoder and decoder (generator) aim to reconstruct data that fools the discriminator (making it classify reconstructed data as real). Reconstruction loss is used to ensure that the autoencoder reconstructs the input accurately. In the training process, there is alternation of training between the discriminator and the generator (encoder and decoder). The discriminator's performance directly influences the generator's training, emphasizing its importance.

[0212] 3. Verifying Hash Integrity, Emission-Recording Consistency, and Using the Discriminator for Anomaly Detection

[0213] The discriminator evaluates reconstructed data. A lower output from the discriminator suggests the data is anomalous. The discriminator's output is the primary indicator of anomalies. Reconstruction error is used as supplementary information.

[0214] In other embodiments, this architecture can be used but developed by replacing or augmenting the discriminator with a pixelwise version.

[0215] More generally, the discriminator is crucial for guiding the generator to produce realistic reconstructions. By focusing on fooling the discriminator, the generator improves the quality of reconstructed data. In the verification phase, the discriminator serves as the primary tool for anomaly detection. Its output reflects the likelihood of data being normal or anomalous.

[0216] In using both discriminator and reconstruction loss, the discriminator output provides a direct measure of how 'real' the reconstructed data appears. Anomalies are expected to produce reconstructions that the discriminator flags as fake. Reconstruction error measures how well the autoencoder reconstructs the input data. High errors indicate discrepancies from normal data patterns.

[0217] Using both metrics enhances the robustness of anomaly detection, allowing for capturing anomalies that may not be evident through one metric alone. The separation of components (encoder, decoder and discriminator) in a modular design allows for easy modification and addition. Example 3 - Cryptography

[0218] This process involves shared secret key generation, initial camera image capture, and initial projection and guess generation steps. The outer loop begins, iterating for a specified number of plaintext iterations

[0219] ( NUM_PLAINTEXT_ITERATIONS ).

[0220] In each plaintext iteration, a new plaintext is generated using the GeneratePlaintext() function. The PREVIOUS_PLAINTEXT variable is set to None at the beginning of each plaintext iteration.

[0221] The inner loop begins, iterating for a specified number of iterations

[0222] ( NUM_ITERATIONS_PER_PLAINTEXT ) for each plaintext.

[0223] The inner loop follows these steps:

[0224] The current plaintext is displayed on an e-ink screen.

[0225] Each agent captures an image of the e-ink screen before projecting.

[0226] The agents generate their projections and guesses using their camera images.

[0227] The agents project their signals onto the e-ink screen.

[0228] If there is a previous plaintext available (from the previous iteration of the inner loop), the agents compute their losses based on their decryption of the previous plaintext.

[0229] This process uses an adversarial model. The cooperating agents (Alice, Bob, Charlie, and Diana) aim to minimize their collective loss, which includes their individual losses and Eve's loss subtracted. This encourages them to work together and against Eve. Eve aims to minimize her own loss and maximize the loss of the cooperating agents. She does this by subtracting the sum of the cooperating agents' losses from her own loss.

[0230] The agent models are updated based on their respective losses and optimizers. The cooperating agents update their models using the collective loss, while Eve updates her model using her own modified loss.

[0231] The current plaintext is stored as the previous plaintext for the next iteration of the inner loop. The updated models are used to generate new emissions. After the inner loop completes, the outer loop generates a new plaintext, and the process repeats for the specified number of plaintext iterations. Example 4 -Verification

[0232] 1. Training Script

[0233] This is an embodiment of a reactor for recording verification. Here, the focus is solely on the adversarial loss, without considering the reconstruction loss. The system utilizes an adversarial generator to create fake inner camera images that challenge the verification process.

[0234] During the training phase, the adversarial generator produces fake inner camera images, while the discriminator learns to differentiate between real and fake images. The emission network and recording network are trained to process the emission and recording data, respectively, and their outputs are projected into the passive reactor using the corresponding inner projectors. The inner camera captures the combined projection, resulting in the real inner camera image.

[0235] The training process involves updating the networks based on the adversarial loss:

[0236] The discriminator is updated to minimize the discriminator loss, which measures its ability to correctly classify real and fake images.

[0237] The adversarial generator is updated to minimize the adversarial generator loss, which measures its ability to fool the discriminator by generating realistic fake images.

[0238] The emission network and recording network are updated to minimize the adversarial loss, which measures their ability to produce real inner camera images that the discriminator considers as real.

[0239] By training the networks in this adversarial manner, the system becomes more robust in detecting and rejecting fake inner camera images.

[0240] During the testing phase, new emission-recording pairs are processed through the trained networks. The inner camera captures the combined projection of the processed emission and recording data. The discriminator evaluates the captured inner camera image and determines whether it is real or fake. If the discriminator indicates that the image is real, the emissionrecording pair is considered "Verified." Otherwise, it is marked as "Not Verified."

[0241] By relying solely on the adversarial loss and the discriminator's assessment, the system effectively verifies the authenticity of the emission-recording pairs, rejecting any pairs that involve fake inner camera images.

[0242] This approach enhances the security and reliability of the recording verification process by focusing on the adversarial aspect and eliminating the need for reconstruction loss.

[0243] Example 5 - 3D Image Generation

[0244] The approach described here uses Neural Radiance Fields (NeRF).

[0245] This approach follows the following stages:

[0246] 1. Dataset Loading: and Imtlallzaticn

[0247] • LoadEmlssit-nRecoKhngPahs Lead: a datase; ms ni"g tc r:- to e T ttiz- ■ : g’a : e ’d ccresocnd "g eccrd ng: Gestured ty re camera.

[0248] • DATA5ET_SIZE foe ’•uiv te- ■ z* e T st-i :■ ■-ra::. d "g : = rt- " zhe dataset

[0249] • GerlmajeDlmensut ns Petr dataset

[0250] 2. Camera Parameters initialization

[0251] • l'nlt«?lzeFoca1Length: Ser r e in : ef:>:al e"gr of re ce "veira

[0252] • BulldCameralintrlnsIcsMatrlx: Constructs the camera intrinsics matrix based on the focal length and image dimensions.

[0253] • I'nltiailzeCanieiaExtHnsics: In : e zee re err "s •: oara Teris -ctad :■ a fo re r ez or for ea r deze sort, rer e:e"z ng r a tairera':- p:s zr c"d CT e- ratio" n r e : ;e "e

[0254] 3. NeRF Model Initialization

[0255] • ImltiatlzeNeRFMndel: Sets up re Neure Radiance -iel : l.Ne RF; i"cdel with specified input cranel: i.5D zorr’etes- err ss signal : Tie":- z ri. cur-b: era ’neb rGE color + dens tyi ’tin' ze' layer, : ‘d clden . " rs

[0256] 4. Optimizers I mtlallzatic n

[0257] • hiltiailzeAttam Create: Adsrr opt T :ers for re heRF rz-cle paic"ieter:-. zanra-a i" zdnsi and extrinsics with a specified learning rate.

[0258] 5. Training Loop

[0259] • NUM-EPOCHS: “he IT rt'e" cf : nei-trtn: 'e datasez : rc-cessed.

[0260] • I'TERATIONS-PER.EPOCH foe "Lffoer c* :rc:o"refo'Tr i\i:" ' eec ' ez-cc’’, we'e a r3" :,.;IT daze

[0261] • TOTALJ-OSS: Accumulates the toss over all iterations in an epoch to compute the awrage loss.

[0262] Fci eazh term-

[0263] « Random Data Point Sampling Pandz-T y :e ecz:- an amiss cn- ■eccir ng pa •'fmiT the -dataset

[0264] • Cameia Parameters Retrieval _.:e: re ;ar'e-a in:rnz-ic: e 'd re e.-:rinsir seen f c zo re current data point.

[0265] • GenerateRays Gaertes ryt fr ear- p »e cfo'e ivege dated o” zhe ce r-ere se rrreter o Ct Tt-uzes t -el toz- d "ete:- " ■'o-rna set devize cccrdraze:. DC; space oT,r ^sfci "T p »e czcid nate: i’civ \ZiC :paze zo wci d : zace -s •§ the ce T.ei e intrinsics and extrinsics. o Cz Tt-uzes 'ey c 'ectio": cy z ■■■"■a i:i"-g r-e :i*el tor : rzes rr 'lc soeze. o Set: ray or g ns as r>e ca'^era c-c: tiz " errazted for1the e.-trnsic iveZ’i-' RenclerlmageFramNeRF Gerereresa1' nag? hy irey TrereMig re-’OugMhe :rere re ire NeRF ivcdel, me ere micr s gre , an: me gerereted ire;,-: o Ferfm'n’s ',.o „ ire 're-de ' "g :c accu P. ate cm MS ?"d densir e: :, Io" g me ire> to compute the final pitxel color.

[0266] • ComputeMeani «|uare>IEi >oi Csculsre: me n A a- ; quared error between the rendered image and the actual recording.

[0267] • Backpropagatlon and Optimization: C :-‘o :■ rec g-reefe ’ t: of me lose / >■ re respect to m^ ivcdel an: camere pare peter:- ant ..sdates their ng tre A:c‘"> t-’.tP" sec

[0268] 6, Monlcc-rlng framing Progress

[0269] After t. 'tcecs ’g all teredos ^ r^eprm the 3-,-eirege tss : tcip^.e: e^d cinte:

[0270] 7. Verification Phase

[0271] • ACCURACT..METRIC5: A st to store the F'eak i g'-re '-to-h cite Fa: o <PSVJP' iret'it fit ' earn data point.

[0272] • For each data point o Geneirete: ire-,: _>: ’z the . Mateo ere rm ire sc reiverem- o Renders an image "g the tre -red heR- made I aM the ze "re reted rap. o fit-PMte: \R tet-Aree' tre -e'dered page and tre a tua reccr: nz. o it :■ re: e 'd p-int: me F'ihR ‘or ee th sere sort

[0273] • Compute Aver age Ce re a.es me evereze Pi! , R a: the overell atre.irety ‘"retr c

[0274] Key aspects of this approach are as follows:

[0275] 1. RayTracing

[0276] GenerateRays: Generates rays for each pixel of the image based on the camera parameters.

[0277] Computes pixel coordinates in NDC space and transforms them to world space.

[0278] Computes ray directions by normalizing the pixel coordinates in world space.

[0279] Sets ray origins as the camera position.

[0280] SamplePointsAlongRay: Samples points along each ray between specified near and far bounds.

[0281] 2. NeRF Model Querying

[0282] QueryNeRFModel: Queries the NeRF model for color and density at each sampled point along the rays.

[0283] The NeRF model takes the sampled points, ray directions, and emission signal as inputs. The model outputs the predicted color and density at each point.

[0284] 3. Volume Rendering

[0285] VolumeRendering: Performs volume rendering to accumulate colors and densities along each ray to compute the final pixel color.

[0286] Applies the rendering equation to integrate the colors and densities along the ray.

[0287] Uses techniques like alpha compositing and numerical integration to approximate the integral.

[0288] 4. Loss Computation and Optimization

[0289] ComputeMeanSquaredError: Calculates the mean squared error between the rendered image and the actual recording.

[0290] Backpropagation and Optimization: Computes gradients of the loss with respect to the NeRF model parameters and camera parameters, and updates them using Adam optimizers.

[0291] 5. Model Evaluation

[0292] Verification Phase: Assesses the quality of the trained NeRF model by rendering images for each data point and comparing them with the actual recordings.

[0293] Peak Signal-to-Noise Ratio (PSNR): Used as an accuracy metric to measure the similarity between the rendered images and the recordings.

[0294] Overall Accuracy: Computed as the average PSNR across all data points to provide an aggregate measure of the model's performance.

[0295] The advantages of this approach include the following:

[0296] Detailed Scene Representation: NeRF allows for a continuous and high-resolution representation of the scene, capturing fine details and enabling smooth renderings.

[0297] Implicit Modeling of Emissions: The emission signals are incorporated into the NeRF model, enabling it to learn the effect of emissions on the scene appearance without explicit modeling.

[0298] Joint Optimization: The NeRF model parameters and camera parameters are jointly optimized, allowing for self-calibration and adaptation to the specific scene and camera setup. Example 6 - Computation

[0299] This describes a computational system that utilizes projector-camera setups for advanced image processing tasks. The system is flexible, allowing for various configurations involving multiple projectors and cameras. It can perform operations such as downsampling, convolution, and multiplexing by projecting images into an environment (including an optical resonance cavity) and capturing the results with cameras. The following sections detail each component and process within the system.

[0300] Components

[0301] 1. Projectors

[0302] • Function: Output images onto an environment.

[0303] • Types: Can vary in resolution (e.g., 2160p, 1080pj and scanning methods.

[0304] • Configurations: Single or multiple pro ectuis can be used simultaneously.

[0305] 2. Cameras

[0306] • Function: Capture projected images for processing.

[0307] • Types r dude tollng shutter tameras that capture images line by I ne parti tularly su table for pairing with scanning projectors. A global shutter camera can be paired with scanning or bistable projectors for simultaneous full image processing.

[0308] • Pairing: Can be paired with projectors for synchronized operations.

[0309] 3. Optical Resonance Cavity

[0310] Function: An environment where light can resonate, enhancing optical interactions.

[0311] Usage: Multiple pt o ect ■ r: ran pm ect i ito the cav ty and multi.ile tame'as ran capture the resulting patterns.

[0312] System Configurations

[0313] 1. Paired Resolutions a. Scanning Projector with Rolling Shutter Camera

[0314] • Mechanism: A scam ng projector projects images sequent al y,, while a roll'ng snutter camera captm’es the mage h a syncironized manner.

[0315] • Benefit, ^rec se replication o* the projected image due to syncmmnized t canning and capturing. b. Single Camera to Single Projector

[0316] • Mechanism: Cne camera captures the outout mom one pm ector.

[0317] • Benefit: Direct repl'cation of the rrm ected mage without interference. c. Multiple Projectors Adding Images

[0318] • Mechanism: Two or more pm ect-ors project mages that cvedap,

[0319] • Result. The images from the projectors are added togetner in the captured image, al owing for image addition or blending effects,

[0320] 2. Projection into Optical Resonance Cavity

[0321] • Mechanism; Multiple projector project images nto the cavity, and multip e cameras capture the r esultmg light p arret ns

[0322] • Benefit: Enhanced ordeal nter actions arc cample* image processing canabil t es due to tne resonant properties of the cavity.

[0323] Downsampling Process

[0324] Downsampling reduces the resolution of an image by combining multiple pixels into one. The system employs rasterization techniques to achieve this.

[0325] Example: Downsampling from 2160p to 1080p

[0326] • Goal: Reduce the image •’esolutlcn by a factor co two h noth hor izonta and ve-nical dimensions,

[0327] • Method: Gm-uo 2x2 b ocks o' pi>.el; ar.c man mem onto single p xe s in the ower-r eso utioa image.

[0328] Pixel Mapping Table

[0329] Process Explanation

[0330] 1 . Grouping Pixels: o Each group ito ’tour pixels in the 2‘ tOo maoe corresoo-ids to one p ,< e in the 1080a image,

[0331] 2. Combining Values: o The values of the four pixels are combined (e.g., averaged) to produce the value of the corresponcing pixel in the cownsampled image.

[0332] 3. Projection: o The dnwnsamplec image is pr ojectec using the projector, effect vely cis playing a lowe-- resolution version of the original image.

[0333] Convolution Process

[0334] Convolution involves applying a kernel (filter) over an image to produce a new image emphasizing certain features.

[0335] Parameters

[0336] • Original Image Resolution: l OBOo

[0337] • Kernel Size: 4x4

[0338] • Stride: 1 (the kernel moves one pixel at a time)

[0339] Process Steps

[0340] 1 . Image Padding o Purpose: To ensure the convolution kernel) can be applied to the edges of the image.

[0341] O’ Methods:

[0342] ■ Zero-Padding: Adding zeros around the border of the Image.

[0343] ■ Wrapping: Toe image edges wrap around to the roop usite sice

[0344] 2. Projector Alignment o Multiple Projectors Used: Four projectors. o Starting Positions:

[0345] ) Projector 1 : (0,0) ! i Projector s : (1 0) i o Explanation: Each projector starts projecting the convolved from differ ent offsets to cover all positions where the kernel is applied.

[0346] 3. Projection of Convolved Image

[0347] O’ Each projector outputs the convolved image starting from its assigned position. o The projections overlap appropriately to form the complete convolved image.

[0348] 4. Image Capture Camer.is capti re the combined projections. o The result is a fully convolved mage, a: if the kernel had been applied across the entire image. Multiplexing Embodiments

[0349] 1. Time Division Multiplexing (TDM)

[0350] • Mechanism: D "e-’ert image: are pm ectec and 'aptu ’ei: ir rapic si«cce::ir:n, ead’ during its own time slot.

[0351] • Benefit: Efficient use of a single projector-camera pair for multiple images.

[0352] 2. Frequency Division Multiplexing (FDM)

[0353] • Mechanism: images am projected using ciffomnt light mequenc es (colors), and cameras filter and capture specifi : freijuer :ies.

[0354] • Benefit: Simultaneous project on and captire of multiole images without t me nelays.

[0355] 3. Partial Spatial Multiplexing

[0356] • Mechanism: The projection volume c divided rto sections, eaci rrojecting a c fogrent image.

[0357] • Benefit: Parallel pr ocessing jf multi, jle images in differ ent inter a ttirg spatial ' egiur s.

[0358] Applying DSP techniques like FFT, DCT, wavelet transform, SVD, and DWT in the projectorcamera setup can significantly enhance image processing capabilities. By projecting the transformed image onto the optical resonance cavity, these techniques enable efficient and effective manipulation of visual data.

[0359] The Fast Fourier Transform (FFT) and Discrete Cosine Transform (DCT) convert images into the frequency domain, allowing for frequency-based analysis and manipulation. In the projectorcamera system, the transformed image can be projected onto the cavity, where specific frequency components can be filtered, suppressed, or amplified by modulating the cavity's response. This enables tasks such as noise reduction, image compression, and selective feature extraction. For example, by comparing images in the frequency domain, similarities and differences between patterns or textures can be easily identified.

[0360] Wavelet transforms, including the Discrete Wavelet Transform (DWT), decompose images into multi-scale representations, capturing both spatial and frequency information. By projecting wavelet-transformed images onto the cavity, the system can perform multi-resolution analysis and processing. This is particularly useful for tasks like image denoising, where noise can be effectively removed by thresholding wavelet coefficients. Additionally, wavelet-based compression can be achieved by selectively preserving important wavelet coefficients and discarding less significant ones. Singular Value Decomposition (SVD) factorizes an image matrix into singular values and vectors, providing a compact representation of the image's structure. In the projector-camera setup, SVD-based techniques can be employed for image compression, denoising, and low- rank approximation. By projecting the SVD-truncated image onto the cavity, the system can efficiently process and reconstruct the image using a reduced set of singular values and vectors. This is beneficial for applications like background subtraction or matrix completion, where the essential information of the image can be captured and manipulated using a lowerdimensional representation.

[0361] Incorporating these DSP techniques in the projector-camera system opens up a wide range of possibilities for advanced image processing. By leveraging the optical resonance cavity, the transformed images can be efficiently manipulated and analyzed in ways that are computationally challenging or infeasible with traditional digital methods. This hybrid approach combines the power of DSP with the parallelism and speed of optical computing, enabling realtime processing of complex visual data for various applications, including computer vision, pattern recognition, and image enhancement.

[0362] Combining DSP techniques can further enhance the capabilities of the projector-camera system, particularly for operations like convolution. One such approach involves pairing a lower-resolution image with a higher-resolution projector to perform convolution through clever image tiling and multiplexing.

[0363] Consider the case of convolving a 1080p image using a 2160p projector. The process begins by copying the 1080p image into each quadrant of the 2160p projector's frame, effectively creating a 2x2 tiled arrangement. To create translated images, zero-padding is applied to the edges of each quadrant. The padding scheme depends on the quadrant's position, as illustrated in the following table:

[0364] Quadrant Position Zero-Padding Scheme

[0365] Top Left No padding required

[0366] Top Right Zero-padded on the top

[0367] Bottom Left Zero-padded on the left

[0368] Bottom Right Zero-padded on the left and top After tiling and zero-padding, the image is rasterized in a multiplexed manner, interleaving 4 pixels from each quadrant in the projection time correspondent to a single pixel on the 1080p camera e.g.

[0369] { Image [0,0] (0,0) , Image [0,1] (0,0) , Image [1,0] (0,0) ,

[0370] Image [1,1] (0,0) }, { Image [0,0] (0,1) , Image [0,1] (0,1) , Image [1,0] (0,1) , Image [1,1] (0,1) } ,

[0371] The cavity's response to the projected image effectively performs the convolution operation.

[0372] To capture the convolved result, a synchronized 1080p camera is used. The camera records the optical response of the cavity to the projected multiplexed image. Due to the carefully designed tiling, zero-padding, and multiplexing scheme, the captured 1080p image represents the convolved output of the original image.

[0373] A detailed discussion of implementation of the invention, including mathematical underpinnings, will now be provided.

[0374] Setup

[0375] Setup: Data

[0376] The invention concerns itself with Emitter-Scene-Recorder Loops.

[0377] An example projector-Scene-camera Loop is illustrated in Figure 1. An example speaker-Scene- microphone Loop is illustrated in Figure 2. An example ‘stereo projector’ -speaker-Scene- ‘stereo camera’-microphone-Agent Loop is illustrated in Figure 3. A head-mounted ‘stereo projector’- Scene-‘stereo camera’ Loop is illustrated in Figure 4, and a head-mounted projector-Scene-camera Loop is illustrated in Figure 5.

[0378] Within the context of the invention, Recorder refers to a mechanism for the production of a signal in response to an external system. These include video cameras and sound microphones. A Recorder’s signal data, whether live or stored, optionally including metadata, at time t , is referenced as a Recording, or l?(t), including but not limited to an array of real numbers of dimensions comprising the Recorders’, c , channels, h , height, w , width, or cRXhRXwR, R (t)elRCsXhsXWs.

[0379] Within the context of the invention, Emitter refers to a mechanism for the production of an emanation in response to a signal. These include video projectors or screens, and sound speakers. An Emitter’s signal data, whether live or stored, optionally including metadata, at time t , is referenced as an Emission, or E (t), including but not limited to an array of real numbers of dimensions comprising the Emitters’, c , channels, h , height, w , width, or cEXhEXwE,

[0380] Within the context of the invention, a sample at point i, e.g. a global shutter camera image or sound sample over an interval, is labelled and its corresponding Emission E^.

[0381] Industiy standard differentiable methods for the depiction of metadata exist [1]. Within the context of the invention, when referenced explicitly, metadata in these or any format including but not limited to that related to Recordings or Emissions, e.g. timestamps, or intrinsic or extrinsic matrices, are labelled R(t) or E (t).

[0382] Within the context of the invention, a Scene refers to the medium or environment with which an Emitter interacts, and from which a Recorder derives its data.

[0383] Within the context of the invention, stored or live Emission and / or Recording data corresponding to the interval between time and t2are labelled { E (t)|t t2] }, respectively. Within the context of the invention, collections of discrete samples of Emissions and / or Recordings are labelled

[0384] Within the context of the invention, a Loop, x , comprises any number of Emitters, labelledxE(t), and Recorders, labelledxR(t), with the signal state of the Emitter and / or Emitters derived from that of the Recorder and / or Recorders.

[0385] Within the context of the invention, stored or live sets of data with an optional temporal offset T are labelled { (XE (t) ,XR (t + T))| tG [t1512] } with discrete data labelled { (xE[l1,xR[,+al)|z G { 1 , ... , n } } , e.g. a = 0. Sets of data exist as discrete elements or are combined, e.g. via appropriate resizing and concatenation along any channel, e.g. for a projector-Scene-camera system hEand hRare resized to h, wEand wRare resized to w. As RGB (Red-Green-Blue) images, c = cE=cR= 3 naturally. These data are concatenated forming data of dimensions, (cX hX2(w)), or (cX2(h)Xw), or (6XhXw) in the case of a projector-Scene-camera Loop.

[0386] Within the context of the invention, when data from Emitters and Recorders external any Loop are used, these Emitters and Recorders are referenced asyE (t) andyR (t), e.g. a projector-Scene- camera Loop, x, labelled (xE(t) ,xR (t + T )) and an optionally synchronised RecorderyR(t) and EmitteryE (t).

[0387] In some embodiments, multichannel Emitters and / or Recorders are used, e.g. stereo projectors and / or multichannel speakers as Emitters, and / or cameras optionally using filters corresponding to projector channels, and / or multichannel microphones as Recorders.

[0388] In some embodiments, camera shutters are synchronised with the corresponding projector’s frame refresh synchronisation signal.

[0389] In some embodiments, cameras with a global shutter are used, optionally with an exposure time tuned to the projector’s frame time.

[0390] Setup: Dataset Recording and Initial Analysis

[0391] In some embodiments, representative datasets are created by the selection and Emission of a dataset,XE, and the storing of Recordings of the resultant interaction,XR, with a Scene, e.g. any dataset, including but not limited to “VIL: Synthetic Video Texture Dataset” [2] in the visual domain or “Syntex: parametric audio texture datasets” [3] in the audio domain, is Emitted and the resultant Recordings stored.

[0392] Within the context of the invention, Output labels are optionally derived via analysis of Recordings and / or Emissions via any industry standard mechanism. Output label signals are labelled O.

[0393] Methods for the creation of Output labels from the Recording component of Emission-Recording data include but are not limited to the application of 3D reconstruction from multiple images (e.g. COLMAP [4]) to Recordings, i.e. the creation of depth maps from a set of camera images in a recorded dataset.

[0394] Industiy standard techniques for Recording analysis include 3D reconstruction from multiple images models [5], [6], [7]. In some embodiments, these or any other Recording analysis techniques are applied to Emission-Recording Loop signals, e.g. the application of a 3D cost volume minimisation model to Recordings and / or Emission signals including but not limited the training of a Deep Structure from Motion Models (e.g. [8]) G { 1 ,... , n } } or

[0395] Industry standard methods for the creation of Output labels include the calculated creation of Emissions and analytic processing of their corresponding Recordings, including but not limited to Structured Light Emission, i.e. the selection of any known Emission pattern, e.g. a Color M-Array dataset, the storage of Recordings resultant from of the Emission of these patterns, and the application of any industry standard Structured Light analysis including but not limited to those described in [9], and the creation of depth maps corresponding to Emissions and the respective Recordings. In some embodiments, Structured Light Emission-Scene-Recording signals with optical flow properties are created via any industry standard data augmentation method(s) applied to industiy standard Structured Light Emissions.

[0396] Industry standard methods for the unsupervised or self-supervised creation of classification labels include the training and application of Unsupervised or Self-Supervised Video Object Segmentation models

[0397] In some embodiments, Output labels are derived via any industry standard method, e.g. the capturing of depth maps using RGB-D cameras (including but not limited to the use of the Microsoft (TM) Kinect (TM) Azure (TM)), or the manual application of classification labels.

[0398] Setup: Trained Functions

[0399] Within the context of the invention, a Comparative Discriminator, C, references a function which attempts to distinguish two input data via any industry standard method including but not limited L 1 loss, VGG loss, or the training of a discriminator in a Generative Adversarial Network

[0012] . In other embodiments, any comparative function is used.

[0400] Within the context of the invention, a Generator, G , comprises a trained model capable of mapping from Recordings, Emissions and / or Output labels to Recordings, Emissions and / or Output labels, including but not limited to a Generator model created during the training of a Generative Adversarial Network (GAN)

[0013] . In some embodiments, these models are recursive, and / or comprise any regression and / or classification models, attention models, or transformers, or comprise any industry standard trainable function.

[0401] Within the context of the invention, an Agent references a mapping via Generator, and / or One-Way Function between Recorders or a Recorder and Emitters or an Emitter, and any optional Output labels. Agents operating on Loop x are labelledXG.

[0402] Example external elements, y , are illustrated in Figure 6, i.e. external synchronised cameras used in the training ofXG but not in the provision of data toXG .

[0403] Within the context of the invention, the use of Emitter-Recorder signals, { (xE(t),xH(t+r))|t G [ti,t2] }, in the training of an AgentXG mappingxl?(t) with an optional time offset T to a generated Emission PredictionxE(t), with the parameters ofXG optimised via back propagation, according to the output of a Comparative Discriminator, C , providedxE(t) and real Emission data,xE(t), via industry standard GAN methods or any other is described via: or any subset thereof, or equivalent thereto.

[0404] Such specifications are read from the bottom up, specifying signals available to the models being trained, their outputs, the networks laid out, and the objective functions being used to update their network parameters, with up-arrows indicating that a function is being trained, and down arrows that it is being used as an objective function in the training.

[0405] Within the context on the invention, an objective function, L, comprises an arbitrary loss function or reward function which maps an input to a value representing an evaluation of the input to be minimised or maximised. In some embodiments, any industry standard objective function is used including but not limited to a style loss function

[0014] .

[0406] Setup: Multiple Channels and Domains

[0407] In some embodiments, crossover occurs between Recorder and Emitter channels i.e. in the case of a stereo projector, E(t), comprising channel I,xEj(t), and the other channel II,xEn(t), with a camera filtered to view channel I, xR^t), and a camera filtered to view channel II,xRa(t), channel I is used for a Recorder, and channel II used for the Emitter.

[0408] In some embodiments, crossover occurs between domains. Figure 3 illustrates a configuration of stereo projector,xEz(t),xEn(t), stereo cameras filtered to view these channels, (t + Tfl), speakerxEra(t), and microphonexRni(t + Tin). In some embodiments Record from any permutation of domains and Emit any in permutation of domains. Examples include but are not limited to speaker-Scene-camera Loop (e.g. for the generation of cymatics using a speaker-diaphragm-particulate system) or a ‘mechanical display’-Scene-microphone Loop.

[0409] Setup: Direct Training

[0410] Within the context of the invention, Reinforcement Learning (RL) references an area of machine learning concerned with how intelligent agents ought to take actions in an environment in order to maximize the notion of cumulative reward.

[0411] In some embodiments, Reinforcement Learning is applied to Emitter-Scene-Recorder Loops, i.e. a Reinforcement Learning framework, including but not limited to

[0015] is used to apply an Objective Function in the training via back propagation of an Agent. Industry standard RL techniques include the training of Generators as Agents against objective functions applied to Recordings e.g. camera images or microphones. Industry standard RL techniques include the output of discrete or continuous signals by Agents. In some embodiments, Emitters are treated as robotic actuators for the purpose of RL training e.g. an Agent is configured with a continuous output of dimension cEXhEXwE, or any continuous output e.g. of dimensions cX hX w, followed by a Visualisation function applied to the Agent’s output. In some embodiments, any reinforcement learning techniques are applied in the training of Agents and / or any Generator and / or Discriminator or any other models within the scope of the invention (e.g. Output label generation), e.g.

[0016] ,

[0017] .

[0412] Within the context of the invention, Physics Aware Training comprise any industiy standard method for the training of models physical systems via back propagation

[0018] ,

[0019] optionally according to an objective function

[0020] . In some embodiments, these methods are applied to projector-Scene- camera systems e.g. using projectors, E(t) or E^, as output and cameras, l?(t) or R^l\ used as input.

[0413] Within the context of the invention, Agents trained via Reinforcement Learning, Physics Aware Training, or any industry standard method with an Objective Function applied to Recordings resultant from the Generator’s Emissions are described via: or any subset thereof, or equivalent thereto.

[0414] Setup: Domain Discriminators

[0415] Within the context of the invention, a Domain Discriminator, here labelled D, refers to a classifier model which is trained to determine whether a Loop elements provided to it is are authentic components of the training set, e.g. the discriminator trained as part of the training of an autoencoder, including but not limited to

[0021] ,

[0022] is described as t D, i.e. ®(xE(t),xR(t+T„))(xE M ,xR (t+ Tn))

[0416] With f QE jtj^^^+Tjjltett^ G] } : or any subset thereof, or equivalent thereto.

[0417] When addressed specifically, the encoder used in the training of D is here described as GD, i.e. or any subset thereof, or equivalent thereto.

[0418] If left unspecified, Domain Discriminators are trained directly on data or on embeddings within a latent space, or via any industiy standard manner.

[0419] Setup: Domain Generators

[0420] Within the context of the invention, a Domain Generator model comprises a model trained to reproduce data within a domain, optionally according to a conditional function (e.g. an objective function applied to any subset of Generated Elements), including but not limited variational autoencoders (VAEs), generative adversarial networks (GANs), and auto-regressive models

[0021] ,

[0022] .

[0421] E.g. a Domain Generator trained to receive data from Loop Alice, xA, and to generate data evaluated according to Domain Discriminators evaluating membership of xA, comparing Generated output with input, or a Domain Discriminator evaluating membership of the Generated output in Loop Bob, xB. is described via: or any subset thereof, or equivalent thereto.

[0422] Setup: Alternate Reality Prediction

[0423] In some embodiments, Generator models are trained to predict Scenes given alternative Emission states and are labelledXGR, i.e. multiple sets of Emission-Scene-Recording data with an equivalence are created, including but not limited to via multichannel Emission and multichannel Recording e.g. a stereo projector,xE(t), comprising channel I, xE^t), and the other,xEn(t), with a camera filtered to view channel I, and a camera filtered to view channel II,xRn(t). or any subset thereof, or equivalent thereto. or any subset thereof, or equivalent thereto.

[0424] In some embodiments, the simulated Recordings resultant from the outputs of an Agent are used in the offline training of models via RL.

[0425] Setup: Multi- Agent Systems

[0426] Within the context of the invention, when Agents are trained in mult- Agents systems, they are distinguished via the industry standard naming scheme of Alice, zA, Bob, zB, and Eve, zE.

[0427] A projector-Scene-camera- Agent Loop mounted as a robotic actuator for use in training multiAgent systems is illustrated in Figure 7.

[0428] Industiy standard methods for the training of cryptographic Generative Adversarial Networks include the representation of plaintext, U , ciyptographic keys, K , and training methods here described as Oppositional

[0023] , in which one Agent, e.g. Agent Alice, zA, trains in Opposition to the Objective Function of another Agent, e.g. Agent Eve, zE.

[0429] Within the context of the invention, these Objective Functions are described as Oppositional Objective Functions, and a Comparative Discriminator trained Oppositionally is labelled G , a Domain Discriminator trained Oppositionally labelled © , and an arbitraiy Objective Function trained Oppositionally is labelled t .

[0430] Creation of Emitter-Scene-Recorder Loops with One-Way Properties

[0431] Within the context of the invention, a Physical Unclonable Function (PUF) references a mechanism for modifying a function via a material method incorporating a manufactured random component. An example in the visual domain includes, but is not limited to, a speckle pattern used to modify the output of a projector and / or the input to a camera. An example in the auditory domain includes, but is not limited to, the application of a muffler to a speaker sound output and / or a microphone input. An example in the electronic signal domain includes, but is not limited to, an analog PUF

[0024] .

[0432] Within the context of the invention, a One-Way Function, labelled H, references any mapping between data more easily performed in one direction than another, e.g. the application of a PUF to by Alice to her Recordings,xR (t), is labelledxH (t)=xH (xR (t + T)), or the output of a ciyptographic function including but limited to a hash function applied tox is labelled

[0433] Within the context of the invention, a Visualisation function, V, references a mapping between data including but not limited to the output of a One-Way Function and Emitter, e.g. in the case of a PUF applied to each channel an identity, and the Emission is equal to the output of the PUF. In some embodiments, any industry standard remappings are used. An example of a visualisation function in the discrete case, includes but is not limited to the the application of a variable output length hash function toxH^, +a^ (e.g. the hashing of the 256 bitxH^, +a^ via blake 3(xH^+a\ output_size=768)), the reshaping of this output to cHXhHXwH(e.g. 3 X16X 16), and the resizing of this output to cEXhEXwE. In some embodiments, other functions are used, e.g. blake 3 , output. _size=1536) is mapped to a complex 3 X16X 16 array which is resized or modified via any number of rounds of Inverse Fast Fourier Transforms and / or any other matrix transform to an output size of cEXhEXwE. As an industry standard ungermane practice in machine learning, such remappings are not listed explicitly if trivial. n Emission derived from the application of a One-Way-Function is labelledxE(t), i.e. x E(t)=xH(xR(t+T)) , and the resultant Recordings labelledxR(t).

[0434] Within the context of the invention a One-Way Function Payload comprises arbitraiy information, including but limited to timestamps, to be input to the One-Way Function, or derived from it and labelled Q. In some embodiments, comprises any signal or timestamp derived from a ledger, or interlocutor, or widely available signals or signals derived via non-interactive method e.g. a Nothing-Up-My-Sleeve number including but not limited to Phi, i.e. or any replicable signal e.g. Q(t)=sin(t).

[0435] In some embodiments, payload data comprises the output of the One-Way Function of another Loop

[0436] In some embodiments, the output of the One-Way Function is provided to be used as a One-Way Function Payload by an external Loop xB, i.e. e.g. a = — 1, or x A^ is transmitted to a ledger or interlocutor, or broadcast.

[0437] In some embodiments, H is applied recursively e.g.xH (t)=xH(xH (t+T)) or x H[i]=xH (xH[i+a]), e.g. a = — 1.

[0438] In some embodiments, the invention comprises the Emission of In some embodiments, data including but not limited to

[0439] (^ = (^=0 represent the secured system.

[0440] Validation of Stored Data Resultant from the Emission of One-Way Function

[0441] I some embodiments, secured element correspondence is validated via any industry standard analysis method including but not limited to C (t+Tffl))) , any subset thereof, or e.g. a = 0 .

[0442] Within the context of the invention, Domain Discriminators, trained on data from Loop Alice data, xA, to predict the authenticity of novel data originating in Loop Bob, xB, i.e. or any subset thereof, or equivalent thereto.

[0443] Direct One-Way Function Training

[0444] In some embodiments, One-Way function / Discriminator pairs are trained directly on Loops, i.e. or any subset thereof, or equivalent thereto.

[0445] Reality Encryption

[0446] In some embodiments, Agent Alice, xA, uses Neutral Recording Generator, GN, models trained to retarget Emissions resultant from One-Way Functions and their resultant Recordings to neutral Emission EN(e.g. a grey signal) and their resultant Recordings RNderived via channel separation, Generative Means or any other.

[0447] 9RN(t)=GN(R(t + TIV),E(t+Tv)) E (t) = V (K(t + Tj)

[0448] With {(E (tj , or any subset thereof, or equivalent thereto.

[0449] In some embodiments, multiple Agents are configured to cooperate or interfere with each other’s ability to reconstruct a neutral image e.g. stereo projector, stereo camera systems Alice, zA, Bob, zB, and Eve, z£, are trained using channel II as a neutral channel i.e.xE^ =xE^+a‘^ and the resultant neutral Recordings on channel II ,xR^ =xR^ja^ .

[0450] or any subset thereof of equivalent thereto, including absent Keys.

[0451] In some embodiments, any other cryptographic configuration is deployed.

[0452] In other embodiments, another other configuration is used in training, e.g. RGB cameras combined with single channel Emitters and Recorders, or results are based on Generated data, or other Objective or Oppositional Objective Functions are used.

[0453] In some embodiments, these Agents are deployed without channel separation.

[0454] Cryptographic Agents

[0455] In some embodiments, Agents Alice, Bob, and Eve are trained according to any ciyptographic scheme including but not limited to or any subset thereof of equivalent thereto, including absent Keys. Modification of the State of and Emitter-Scene-Recorder-Agent Loop According to an Objective Function

[0456] Agent Distortion

[0457] In some embodiments, Agents are trained with an Objective Function applied to generated Emissions, i.e. or any subset thereof, or equivalent thereto.

[0458] External Transform

[0459] Within the context of the invention, a Transform function, F, comprises a fast style transform

[0025] , a Generator

[0021] , any image-to-image, video-to-video

[0026] ,

[0027] , audio-to-audio or other information transform function including selection of a dataset within a target domain or via human feedback i.e. the application of a Transform, F, toXE is described as F(XE). The transformed output is labelledx3, and the resultant Recordings,XK .

[0460] In some embodiments, paired sets of Transformed and Untransformed data pairs are created by the Emission of paired Transformed and Untransformed Emissions on an Scene.

[0461] In some embodiments, an Agent comprises a the application of a Transform function to the output of a model trained to predict Emissions from Recordings i.e. or any subset thereof, or equivalent thereto.

[0462] In some embodiments, Agents are trained to map from any combination of Transformed or Untransformed Recordings to Transformed or Untransformed Emissions e.g. or any subset thereof, or equivalent thereto.

[0463] Generation of Synthetic Loop Element Data in a Styled Domain

[0464] In some embodiments, Domain Generators are used to Generate Emitter-Scene-Recorder Loop data and are trained against Objective functions including but not limited to similarity of Generated data with input Emitter-Recorder Loop data, or Domain Discriminators trained on Emitter-Recorder Loop data, or Domain Discriminators trained on or Transformed Emitter-Recorder Loop data, Output label model correspondence, any other Objective function applied to any subset of Generated data, or any other industry standard generative technique i.e. or any subset thereof, or equivalent thereto.

[0465] In some embodiments, this data is Emitted directly or Agents are trained on Generated data i.e. or any subset thereof, or equivalent thereto. External Elements

[0466] In some embodiments, Loop data is Generated with an objective function applied to external elements or any subset thereof, or equivalent thereto, and this data emitted directly or used in the training of Agents, optionally provided any industiy standard metadata, i.e.yR(t) oryE (t), i.e. or any subset thereof. Or equivalent thereto. In some embodiments, any methods are used in the prediction of external elements e.g. with known external Emissions, E , andyeE (e.g. multiple known songs or films for projection), or external Recorder,yR , Agent x is trained to predict these separate known Emissions, or predict or modify these external Recording or any subset thereof, or equivalent thereto.

[0467] In some embodiments, Agents are provided estimated metadata regarding an intended viewer in deployment, absent the presence of element y which was present for training e.g. the position of a human head derived via any industiy standard method e.g. any industry standard head-tracking mechanism.

[0468] Multi-Agent Modification

[0469] In some embodiments, Agents are trained directly as Agent models in multi-Agent systems including but not limited to the training of Agent xAto predict or modify any component of Agent xBand vice versa while they train to oppose Eve, i.e.

[0470] or any subset thereof, or equivalent thereto. In some embodiments, Scenes are treated as untrainable layers in a neural network or any machine learning network via any industiy standard technique.

[0471] Use of Emitter-Reactor-Recorder Loops In Computation

[0472] Within the context of the invention, an Information Reactor comprises any volume capable of passing a signal from an Emitter to a Recorder e.g. a optical or acoustic lens, mirror, resonance cavity, non-linear medium, orb, air gap, diffraction grating, vacuum, or any medium.

[0473] In some embodiments, the invention comprises any number of Emitters, Reactors, and Recorders, with Emissions originating via Agents, or a Visualisation functions applied to a One-Way Function applied to any permutation of Recorder inputs. Within the context of the invention, these Agents are labelledZG .

[0474] An Emitter-Reactor-Recorder network is illustrated in Figure 8. A two-input Emitter-Reactor- Recorder network is illustrated in Figure 9. In some embodiments, Reactor Agents are recursive

[0475] Multiple Emitter-Reactor-Recorder Architecture

[0476] Figure 10 illustrates a basic structure for any number of Emitters and Recorders configured as a fully connected network with a Reactor in the projector-Reactor-camera network comprising a mirrored resonance cavity.

[0477] Within the context of the invention, an Array comprises a larger number of Emitters or Recorders. In some embodiments, any pattern of Emitters and / or Recorders is used as an array, including but not limited to a spiral pattern, as illustrated in Figure 11.

[0478] A fully connected layer of Emitters mated to a Reactor mated to and array of Recorders is illustrated in Figure 12. In some embodiments, the coupling is smooth, without sharp edges. In some embodiments, a signal damper (e.g. matte black paint or sound-insulating material) is applied at any point, including but not limited to at the mounting plate and Emitter-Recorder mounting points.

[0479] Basic Function

[0480] In some embodiments, Loop z is trained to act as a function including but not limited to in the modification of Loop x :

[0481] or any subset thereof, or equivalent thereto.

[0482] Reactor as One-Way Function

[0483] In some embodiments, Emitter-Reactor-Recorder Loops are trained to act as One-Way Functions for external systems or for the training of Discriminators, i.e. or any subset thereof, or equivalent thereto.

[0484] Reactor for Direct Modification

[0485] In some embodiments, Emitter-Reactor-Recorder- Agent systems are deployed in the director modification of Emitter-Scene-Recorder systems, i.e.

[0486] or any subset thereof, or equivalent thereto.

[0487] Emitter-Reactor-Recorder-Agent based Dimensional Reduction

[0488] In some embodiments dimensional reduction is accomplished via the mating of a Reactor with a larger dimension of Emitter than of Recorder. An example system is illustrated in Figure 13 in which a mirrored Fibonacci Ram’s Horn is used in the coupling of a single Recorder to multiple Emitters.

[0489] Within the context of the invention, the set of sets of Emitter and / or Recorder signals across multiple Agents from Agent xtto Agent X|x| are labelled via (xE^\xR^+a')=(xE^ |E| ,XRJ+A'|R|) , the set of signals resultant from the application of a Transform to Emissions and the resultant Recording are labelled via , and Generated signals

[0490] Dimension reduction is used in the creation of a lower-dimensional representation of a higher dimensional signal, used for any industiy standard application including but not limited to to training of Output models, including but not limited a Discriminator with respect to data input via high-dimensional Emitters,ZEDon their resultant Recordings,ZRD, with input data including but not limited to an input of a set of Loop elements, i.e. or any subset thereof, or equivalent thereto. In some embodiments, Discriminators are trained on Transformed or Generated Loop elements, i.e. ,Da, or .

[0491] In some embodiments, dimension expansion is accomplished with with the mating of a Reactor with a larger dimension of Recorders than Emitters. An example Loop is illustrated in Figure 14 where a single Emitter is coupled to multiple Recorders.

[0492] In some embodiments, dimensional reduction and expansion are paired e.g. in an Emitter-Reactor- Recorder Loop comprising Recorders and Emitters, with lower dimensional Emitters and Recorders labelledZESandZRPrespectively, with the signal for the former optionally being derived via the application ofZGC, or via a Visualisation Function to the latter, and higher dimensional Emitters labelledZEP, and and Recorders labelled Rsrespectively.

[0493] In some embodiments, dimensional reduction / expansion Loops are used in the training of encoder,ZGP, and decoder,ZGS, Generators as encoder / decoder Loops.

[0494] In some embodiments, Encoder / Decoder Loops are coupled with Discriminators, creating a Emitter-Reactor-Recorder based Generative Adversarial Network, e.g. or any subset thereof, or equivalent thereto. In some embodiments, models are trained via multiplexing techniques. Figure 15 illustrates an array of Emitter-Scene-Recorder Loops being addressed via multiplexing e.g. a sliding window. In some embodiments, a Comparative Discriminator comparing a subset of the outputs of an Emitter- Reactor-Recorder- Agent network at t with earlier outputs is used in training.

[0495] Emitter-Reactor-Recorder-Agent based One-Way Functions / Discriminator Training

[0496] In some embodiments, Emitter-Reactor-Recorder-Agent Loops are trained to act as One-Way Functions or for the training of Discriminators, i.e. or any subset thereof, or equivalent thereto.

[0497] In some embodiments, Discriminators are trained on Transformed or Generated elements i.e.

[0498] 4-ZD^D, or TzD^ respectively.

[0499] Emitter-Reactor-Record-Agent Based Modification According to an Objective Function

[0500] In some embodiments, Emitter-Reactor-Recorder- Agent systems are trained to modify Loop elements e.g. according to trained Discriminators, i.e. or any subset thereof, or equivalent thereto.

[0501] External Elements

[0502] In some embodiments, Agents are trained with Objective Functions applied to external systems, including but not limited external Emitters and / or Recorders, i.e. 4- L(£E (t + T )) , or In some embodiments, any permutation of Emitter-Reactor-Recorder Systems are provided payloads, private keys, public keys, ciphertext, stegotext, or any other ciyptographic input and any differentiable cryptographic scheme implemented and trained on Emitter-Reactor-Recorder Loops.

[0503] An example symmetric cipher comprises the provision of plaintext U , private key K to encoderdecoder network, Alice - zA, the acquisition of an output from Alice of ciphertext, S , and the provision of this cipher text to encoder-decoder-discriminator network, Bob - zB, alongside the key. Encoder-decoder-discriminator network, Eve - zE, is provided the ciphertext with no key.

[0504] Eve trains to reconstruct Alice’s plaintext against Comparative and Domain discriminators comparing her output with the plaintext content or Domain.

[0505] Bob trains to reconstruct Alice’s plaintext, against Comparative and Domain discriminators comparing his output with the plaintext content or Domain, and trains in opposition to Alice.

[0506] Alice trains in Bob’s favour and in Eve’s disfavour i.e. or any subset thereof, or equivalent thereto.

[0507] A steganographic scheme is implemented by providing Alice with a Payload text, stegotext, W , and key K , the training of Alice against the similarity of the ciphertext to the stegotext

[0508] In some embodiments, Emitter-Reactor-Recorder systems are used in key generation, e.g.

[0509] In some embodiments, any arrangement of similar dimensions of Emitters and Recorders coupled with a Reactor are treated as an untrainable fully connected layer of a neural network or other machine learning network. In some embodiments, any machine controllable deformation or alteration to the Reactor is treated as a trainable function parameter. REFERENCES - the teaching of the references provided below is incorporated in this disclosure to the extent permitted by applicable law.

[0510] [1] Mallya, A., Wang, T.-C., Sapra, K. and Liu, M.-Y. (2020). World-Consistent Video-to- Video Synthesis, [online] arXiv.org. Available at: https: / / arxiv.org / abs / 2007.08509.

[0511] [2] Ma, D. (Creator), Katsenou, A. (Creator), Bull, D. (Creator) (26 Oct 2018). VIL: Synthetic Video Texture Dataset - SynTex. University of Bristol. 10.5523 / bris.24imj6d9s27me2n211cf6jxkio

[0512] [3] Wyse, L. and Ravikumar, P.T. (2022). Syntex: parametric audio texture datasets for conditional training of instrumental interfaces. International Conference on New Interfaces for Musical Expression, [online] doi:https: / / doi.org / 10.21428 / 92fbeb44.0fe70450.

[0513] [4] Schonberger, J., & Frahm, J.M. (2016). Structure-from-Motion Revisited. In Conference on Computer Vision and Pattern Recognition (CVPR)

[0514] [5] Schonberger, J., Zheng, E., Pollefeys, M., & Frahm, J.M. (2016). Pixelwise View Selection for Unstructured Multi- View Stereo. In European Conference on Computer Vision (ECCV)

[0515] [6] Schonberger, J., Price, T., Sattler, T., Frahm, J.M., & Pollefeys, M. (2016). A Vote-and- Verify Strategy for Fast Spatial Verification in Image Retrieval. In Asian Conference on Computer Vision (ACCV)

[0516] [8] Wei, X., Zhang, Y, Li, Z., Fu, Y. and Xue, X. (2020). DeepSFM: Structure From Motion Via Deep Bundle Adjustment, [online] arXiv.org. doi:https: / / doi.org / 10.48550 / arXiv.l912.09697.

[0517] [9] Pribanic, T. (2010). A state of the art in structured light patterns for surface profilometry. Pattern Recognition.

[0518]

[0010] Araslanov, N., Simone Schaub-Mayer, and Roth, S. 2021. Dense Unsupervised Learning for Video Segmentation. In Advances in Neural Information Processing Systems (NeurlPS).

[0519]

[0011] Lu, X., Wang, W., Ma, C., Shen, J., Shao, L., and Porikli, F. 2019. See More, Know More: Unsupervised Video Object Segmentation With Co-Attention Siamese Networks. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[0520]

[0012] Wang, T., Zhu, J.-Y, Tao, A., Kautz, J. and Catanzaro, B. (2017). High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs. doi:https: / / doi.org / 10.48550 / arxiv.1711.11585.

[0521]

[0013] Wang, T.-C., Liu, M.-Y, Zhu, J.-Y, Liu, G., Tao, A., Kautz, J. and Catanzaro, B. (2018). Video-to- Video Synthesis, [online] Neural Information Processing Systems. Available at: https: / / papers.nips.cc / paper_files / paper / 2018 / hash / d86ea612dec96096c5e0fcc8dd42ab6d- Abstracthtml [Accessed 16 Aug. 2023].

[0522]

[0014] Gatys, L.A., Ecker, A.S. and Bethge, M. (2015). A Neural Algorithm of Artistic Style, [online] arXiv.org. Available at: https: / / arxiv.org / abs / 1508.06576.

[0523]

[0015] Brockman, G., Cheung, V, Pettersson, L., Schneider, J., Schulman, J., Tang, J. and Zaremba, W. (2016). OpenAI Gym. [online] arXiv.org. Available at: https: / / arxiv.org / abs / 1606.01540.

[0016] Moerland, T.M., Broekens, J., Plaat, A. and Jonker, C.M. (2022). A Unifying Framework for Reinforcement Learning and Planning. Frontiers in Artificial Intelligence, 5. doi:https: / / doi.org / 10.3389 / frai.2022.908353.

[0524]

[0017] Pretorius, A., Tessera, K., Smit, A.P., Formanek, C., Grimbly, S.J., Eloff, K., Danisa, S., Francis, L., Shock, J., Kamper, H., Brink, W., Engelbrecht, H., Laterre, A. and Beguir, K. (2021). Mava: a research framework for distributed multi-agent reinforcement learning, [online] arXiv.org. doi:https: / / doi.org / 10.48550 / arXiv.2107.01460.

[0525]

[0018] Wright, L.G., Onodera, T., Stein, M.M., Wang, T., Schachter, D.T., Hu, Z. and McMahon, P.L. (2022). Deep physical neural networks trained with backpropagation. Nature, [online] 601(7894), pp.549-555. doi:https: / / doi.org / 10.1038 / s41586-021-04223-6.

[0526]

[0019] Thuerey, N., Holl, P., Mueller, M., Schnell, P., Trost, F. and Um, K. (2022). Physics-based Deep Learning, [online] arXiv.org. doi:https: / / doi.org / 10.48550 / arXiv.2109.05237.

[0527]

[0020] AbinavRavi (2019). Physics-Aware-GAN / Physics Aware GAN.ipynb at master •

[0528] AbinavRavi / Physics-Aware-GAN. [online] GitHub. Available at: https: / / github.com / AbinavRavi / Physics-Aware-GAN / blob / master / Physics%20Aware %20GAN.ipynb [Accessed 16 Aug. 2023].

[0529]

[0021] Skorokhodov, I., Tulyakov, S. and Elhoseiny, M. (2022). StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2. [online] arXiv.org. doi:https: / / doi.org / 10.48550 / arXiv.2112.14683.

[0530]

[0022] Yu, S., Tack, J., Mo, S., Kim, H., Kim, J., Ha, J.-W. and Shin, J. (2022). Generating Videos with Dynamics-aware Implicit Generative Adversarial Networks, [online] arXiv.org. doi:https: / / doi.org / 10.48550 / arXiv.2202.10571.

[0531]

[0023] Abadi, M. and Andersen, D.G. (2016). Learning to Protect Communications with Adversarial Neural Glyptography. arXiv: 1610.06918 [cs]. [online] Available at: https: / / arxiv.org / abs / 1610.06918.

[0532]

[0024] Csaba, G., Ju, X., Chen, Q., Porod, W., Schmidhuber, J., Schlichtmann, U., Lugli, P. and Riihrmair, U. (2009). On-Chip Electric Waves: An Analog Circuit Approach to Physical Uncloneable Functions, [online] ePrint IACR. Available at: https: / / eprint.iacr.org / 2009 / 246 [Accessed 16 Aug. 2023].

[0533]

[0025] Huang, H., Wang, H., Luo, W., Ma, L., Jiang, W., Zhu, X., Li, Z. and Liu, W. (2017). Real- Time Neural Style Transfer for Videos, [online] openaccess.thecvf.com. Available at: https: / / openaccess.thecvf.com / content__cvpr__2017 / html / Huang__Real- Time_Neural_Style_CVPR_2017_paper.html [Accessed 16 Aug. 2023]

[0534]

[0026] Bansal, A., Ma, S., Ramanan, D. and Sheikh, Y. (n.d.). Recycle-GAN: Unsupervised Video Retargeting, [online] Available at: https: / / openaccess.thecvf.eom / content__ECCV__2018 / papers / Aayush__Bansal__Recycle- GAN__Unsupervised__Video__ECCV__2018_paper.pdf [Accessed 16 Aug. 2023].

[0535]

[0027] Liu, K., Gu, S., Romero, A. and Timofte, R. (2020). Unsupervised Multimodal Video-to- Video Translation via Self-Supervised Learning, [online] arXiv.org. doi:https: / / doi.org / 10.48550 / arXiv.2004.06502.

Claims

Claims1 . A method of transforming data transmitted through a physical environment, the method comprising: emitting an emitted signal into the physical environment; detecting a detected signal from detection of the emitted signal after interaction with the physical environment; transforming the detected signal to modify a further emitted signal; and repeating the emitting, detecting, and transforming steps until a predetermined objective is met.

2. The method of claim 1 , wherein the emitted signal is an image emitted by a projector and the detected signal is an image captured by a camera.

3. The method of claim 2, wherein the projector is a video projector.

4. The method of claim 3, wherein image capture by the camera is matched to a refresh rate of the video projector.

5. The method of any preceding claim, wherein the transforming step is performed using a generator model adapted to predict an emitted signal from a detected signal.

6. The method of claim 5, wherein the generator model is a trained machine learning model.

7. The method of claim 6, wherein the machine learning model is an artificial neural network.

8. The method of claim 6 or claim 7, wherein the machine learning model is trained to blend a predetermined image into the further emitted image.

9. The method of claim 8, wherein the predetermined image is predetermined text or other loss function.

10. The method of any of claims 6 to 9, further comprising modifying the generator model in real time.11 . The method of any preceding claim, wherein the transforming step comprises applying a predetermined style to the further emitted signal.

12. The method of any preceding claim, wherein the transforming step comprises operating on the detected signal using a one-way function.

13. The method of claim 12, wherein the one-way function comprises a structured noise generator function.

14. The method of claim 12 or claim 13, wherein the emitting, detecting and transforming steps are repeated a predetermined number of times to achieve the predetermined objective, the predetermined objective comprising a one-way function.

15. The method of any of claims 12 to 14, wherein an output of the method is a secure data recording.

16. The method of any preceding claim, wherein the method is performed in parallel by a plurality of emitter-detector pairs.

17. The method of claim 16, wherein the plurality of emitter-detector pairs is configured to act oppositionally, with at least one of the plurality of emitter-detector pairs adversarial to the other emitter-detector pairs.

18. The method of claim 17 where dependent on claim 14, wherein an adversarial autoencoder or any anomaly detector is adapted to verify integrity of the emission-recording pairs.

19. The method of claim 16 where dependent on claim 6 or claim 7, wherein an output of the method is production of a 3D image or other analysis.

20. The method of any of claims 1 to 7, wherein the predetermined objective is completion of a computation process.21 . A system for transformation of data transmitted through a physical environment, the system comprising: at least one emitter for emitting an emitted signal into the environment; at least one detector for detecting a detected signal from detection of the emitted signal after interaction with the environment; and computing means, wherein the computing means is adapted to transform the detected signal to modify the emitted signal.

22. The system of claim 22, wherein the emitter is a projector and the detector is a camera, and wherein the computing means is a suitably programmed processor.

23. The system of claim 22, wherein the projector is a video projector.

24. The system of claim 23, wherein image capture by the camera is matched to a refresh rate of the video projector.

25. The system of any of claims 21 to 25, wherein the system is adapted to perform the method of one or more of claims 5 to 20.

Images