User interface and targeting system

Abstract

A targeting system uses data streamed from multiple user interfaces (10). The targeting system comprising a digital representation (56) of a real-world environment (14) in which the multiple user interfaces (10) are being used by users (12). Data is obtained from the multiple user interfaces (12), wherein the data comprises, for each interface: location data giving the location of the user interface (10), orientation data giving a vector of a target axis (26) of the user interface, image data from a camera device (30) pointing in a direction that extends along the target axis (26), and an activation state of a trigger device (32) of the user interface (10). The digital representation (56) is kept continuously updated with the locations of the multiple user interfaces (10) and the vectors of the target axes 26 as the user interfaces (10) are moved by their users (12). The image data is used to determine the placement of real-world target objects including the placement of users (12) of the user interfaces (10) and the placement of intervening objects in the real-world. A simulated shot (21, 22, 23, 24) is fired by a first user (12) by activation of the trigger device (32) of their user interface (10). The targeting system determines if a path (52) of the simulated shot (21, 22, 23, 24) has passed within a certain distance of a second user (12) of the second user interface (10). This is done based on at least two of: the image data from the first user interface (10), the vector of the target axis (26) of the first user interface (10), and a relative location of the first and second user interfaces (10). The targeting system also simulates the effect of any intervening objects (16), (18) on the path (52) of the simulated shot (21, 22, 23, 24) based on penetration ability for the simulated shot (21, 22, 23, 24) and on determination of the properties of the real-world intervening objects (16, 18). The second user (12) receives feedback, e.g. audible feedback, if the path (52) of the simulated shot (21, 22, 23, 24) is determined to have passed within a certain distance of the second user (12).

Description

[0001] USER INTERFACE AND TARGETING SYSTEM

[0002] The present invention relates to a user interface, a targeting system using multiple user interfaces, and a method of operating the targeting system. Computer programme products are provided for implementing the method. The targeting system uses data from the multiple user interfaces, which may each be a simulated weapon and / or may be a part of a weapons training system.

[0003] It is known to use simulated weapons or weapons training system for various purposes, such as for leisure activities or for training / assessment purposes. This may involve training / assessment in the use of weapons or training in strategy via “wargames”. User interfaces with similar capabilities may be used in leisure games such as laser tag. It is commonplace for such user interfaces to use line of sight targeting systems like lasers or infrared beams. This then requires a target device such as a reflector or receiver of some sort in order that it can be determined if a targeted object has been hit.

[0004] Various types of simulated weapon have been developed alongside the increased usage of smart devices (internet-of-things) and smartphones. These simulated weapons involve a gun-like body that is provided with a smartphone cradle and they are used with a game that operates on the smartphone. An example of this type of toy gun is disclosed in CN203483841U. Such systems can make use of the smartphone camera along with augmented reality to provide virtual targets overlaid on a real-world image and / or to identify real -world objects to be the targets within the game. The user can aim at the target by pointing the gun, with its orientation being detected using motion sensors on the gun or on the smartphone. A normal smartphone GPS can be used to identify location in order to allow for location specific game features or tracking of the user. It has also been proposed to link such systems with other devices in order to provide an improved gaming experience, such as by means of Bluetooth connected headgear or clothing as disclosed in CN109126118A.

[0005] An example of a weapons training system for military use is described in US20150354922A1. This discloses a laser targeting system using modified firearms shooting blanks rather than live rounds. A vest is worn with several target devices that can emit light and sound when hit. Via the use of TENS (Transcutaneous Electrical Nerve Stimulation) units in the vests the system can give more realistic conditions in terms of stress and the reaction to stress.

[0006] Another example of a known targeting system is found in US 2014 / 178841 Al, which describes a laser-less targeting system where a hit / miss is determined making use of the orientation / position of the weapon used alongside image data from a camera that is aligned with the target axis of the weapon. The path of the bullet is calculated using ballistics formulae and the image data enables an assessment of if the target is visible at the point where the bullet will strike.

[0007] It will however be appreciated that a need remains for improvements in user interfaces and targeting systems for such purposes. It would be an advantage to provide a system that could better simulate a real-world scenario and / or to provide hardware elements for such a system.

[0008] Viewed from a first aspect, the present invention provides a method of operating a targeting system using data streamed from multiple user interfaces, the targeting system comprising a digital representation of a real-world environment in which the multiple user interfaces are being used; wherein the method includes: receiving data from the multiple user interfaces, wherein the data comprises, for each interface: location data giving the location of the user interface, orientation data giving a vector of a target axis of the user interface, image data from a camera device pointing in a direction that extends along the target axis, and an activation state of a trigger device of the user interface; updating the digital representation with the locations of the multiple user interfaces and the vectors of the target axes as the user interfaces are moved by respective users thereof; using the image data to determine the placement of real -world target objects including the users of the user interfaces as well as using the image data and / or the digital representation to determine the placement of intervening objects in the real -world; registering an activation of a trigger device by a first user of a first user interface, wherein activation of the trigger device indicates that a simulated shot has been fired; determining if a path of the simulated shot has passed within a certain distance of a second user of the second user interface based on at least two of: the image data from the first user interface, the vector of the target axis of the first user interface, and a relative location of the first and second user interfaces based on the location data; simulating the effect of any intervening objects on the path of the simulated shot based on an assessment of penetration ability for the simulated shot and based on determination of the properties of the real -world intervening objects; and providing feedback to the second user if the path of the simulated shot is determined to have passed within a certain distance of the second user.

[0009] With the use of this method it becomes possible to more accurately replicate real- world scenarios involving weapons or similar for simulated shooting of targets objections including the users of the user interfaces. The user interfaces may for example be simulated weapons, such as guns. By using data including images, location, and orientation from each user interface then it is possible to reliably determine not only when a target object has been struck, but also when there has been a “near miss” or other event that may be detectable in the real-world. Moreover, by use of a “digital twin” via the digital representation of the real-world environment then the targeting system can react realistically to objects in between the user interface and the target object(s), so that users may take cover as in the real-world, but simulated shots are able to penetrate that cover and still hit a user even when they are not visible in a direct line of site from the user interface that fires the simulated shot. Thus, the feedback to the second user of the second user interface includes an indication of when a path of the simulated shot passes within the certain distance of the second user without hitting the second user, e.g. close enough for them to hear and / or see the effects of the simulated shot as well as an indication of a hit, with this feedback to the second user occurring even when there is an intervening object. This is to be contrasted to known systems where the user is only notified if they have been hit by the simulated shot and / or where any intervening object is deemed to have provided cover for the target object / user irrespective of the physical properties of that object. The users get a more accurate real-world experience due to the added feedback as users will know when they are in danger of being struck by a shot and / or when they are in a zone of active fire, as well as when they are exposed due to inadequate cover from the surrounding environment, even when there are intervening objects that block line of sight. The effect of all of the simulated shots thus becomes more realistic and allows for a high performance “train as you fight” simulation.

[0010] The invention extends to a targeting system that is adapted for use with the method. Thus, viewed from a second aspect, the present invention may provide a targeting system configured to use data streamed from a plurality of user interfaces, wherein the targeting system is configured to carry out a method as set out above. This may be done by the use of one or more computer devices.

[0011] An example targeting system is configured to receive and use data streamed from multiple user interfaces, wherein the data comprises, for each interface: location data giving the location of the user interface, orientation data giving a vector of a target axis of the user interface, image data from a camera device pointing in a direction that extends along the target axis, and an activation state of a trigger device of the user interface; the targeting system comprises: a digital representation of a real-world environment in which the multiple user interfaces are being used; a digital twin sub-system for updating the digital representation with the locations of the multiple user interfaces and the vectors of the target axes as the user interfaces are moved by respective users thereof; an object tracking sub-system configured to use the image data to determine the placement of real -world target objects including the users of the user interfaces as well as to use the image data and / or the digital representation to determine the placement of intervening objects in the real -world; a simulated shot sub-system configured to: register an activation of a trigger device by a first user of a first user interface, wherein activation of the trigger device indicates that a simulated shot has been fired; determine if a path of the simulated shot has passed within a certain distance of a second user of the second user interface based on at least two of: the image data from the first user interface, the vector of the target axis of the first user interface, and a relative location of the first and second user interfaces based on the location data; and simulate the effect of any intervening objects on the path of the simulated shot based on an assessment of penetration ability for the simulated shot and based on determination of the properties of the real -world intervening objects; and a user feedback sub-system for providing feedback to the second user if the path of the simulated shot is determined to have passed within the certain distance of the second user.

[0012] It will be understood that the various sub-systems as well as the targeting system as a whole may be implemented via software and / or hardware elements that are distributed in different locations, such as at the user interface and / or at a remote server, which may be in the cloud. The sub-systems may use shared or overlapping software and / or hardware elements in order to perform the required functions. All the sub-systems may be combined together, or they may be distributed in different parts. For example, the user feedback subsystem may comprise hardware at the user interface for giving feedback to the user as well as one or more computer implemented system(s) for determining the nature of the feedback based on the data from the user interface, where the computer implemented system(s) could include parts at a remote server and / or at the user interface. The digital twin sub-system, the object tracking sub-system and / or the simulated shot sub-system may be implemented within a single common software structure running on a single computer system, or some functions for the sub-systems may be separated to be provided via separated modules using multiple distributed computer systems, in the cloud or otherwise.

[0013] With the above described method and system the targeting system is advantageously an environmentally aware targeting system, and thus the user interface may be configured for such an environmentally aware system. The targeting system may be environmentally aware in the sense that it receives information about the real-world environment in which the user interface is used (e.g. in a simulated fire-fight) as well as providing feedback to the user based on this information. Thus, the feedback provided via the user feedback subsystem will differ depending on the surrounding environment, with changes to that feedback that are independent of the location and position of the user interfaces. The same gunshot with the weapons and users in the same positions can thus result in different outcomes due to differing environments.

[0014] The targeting system may comprise the multiple user interfaces, optionally along with other remote software or hardware components, such as a remote server as mentioned above. Thus, the targeting system may take the form of a network of user interfaces each providing data as discussed above, with this data being streamed from each of the user interfaces to one or more servers. The targeting system may comprise other hardware devices as well, such as: add-ons for an improved user experience as discussed below, remote controlled devices to provide added data (e.g. remote cameras) or to enhance a simulated scenario (e.g. drones with simulated weapons), and / or location devices that interact with a positioning system of the user interfaces.

[0015] It will be appreciated that the capabilities of the user interface are important for the beneficial effects of the targeting system and in fact they share similar core technical features in relation to the data that is required and the ability to provide feedback to the user. Thus, viewed from a third aspect, the present invention provides a user interface for a targeting system, the user interface comprising: a target axis; an orientation sensing system for determining a vector of the target axis; a positioning system for determining the location of the user interface; an image handling system for receiving image data from a camera device pointing in a direction that extends along the target axis, wherein the image data allows for determination of placement of real -world target objects including the users of other user interfaces; a trigger device for activation to indicate when the user of the user interface intends to fire a simulated shot; and a user feedback device for providing feedback to the user if a path of a simulated shot fired via another user interface is determined to have passed within a certain distance of the user.

[0016] This user interface allows for implementation of the key feature of simulation of a “near miss” as well as other potentially detectable simulated shots that would not strike the user. Since the user interface provides all three of image data, location, and orientation it becomes possible to accurately determine placement of the real -world target objects relative to the simulated shots whether or not the simulated shot will actually strike the user. Advantageously, there is no need for any kind of receiver or reflector device as with known laser gun systems and the like, since the proposed targeting system does not rely on the user detecting a hit. Instead the user interface provides data that allows the targeting system to accurately map the interaction of multiple user interfaces in the real-world environment.

[0017] The user interface may be configured for use with the method or system of the first or second aspects. In some embodiments multiple such user interfaces may be comprised within the targeting system. The various systems and devices of the user interface may be configured to interact with the related sub-system of the targeting system and / or may be considered as a part of the relevant sub-system when the targeting system comprises the user interfaces. Thus, for example, the user feedback device of the user interface may be a part of the user feedback sub-system, where a remote part of the user feedback sub-system (e.g. in a remote server) determines the nature of the feedback and provides a signal to the user interface allowing the user feedback device to provide the feedback (e.g. audible feedback) to the user.

[0018] To further enhance the method, targeting system and / or the user interface they may be include one or more additional features as discussed below.

[0019] The step of determining if the simulated shot has passed within a certain distance of a user may also include input from a physics engine that calculates a form for the path of the simulated shot. In a very simple implementation, e.g. for short ranges using simulated handguns, the path of the simulated shot may be a straight line from the user interface along the vector of the target axis. However, a better simulation of real-world ballistic paths, and / or paths of simulated shots from non-ballistic weapons, may be introduced by means of a more sophisticated physics engine to determine a suitable path, e.g. a path having curvature due to gravity and / or wind conditions. Thus, the start of the path may be the vector of the target axis, but the subsequent path may vary from this due to curvature or other effects introduced in accordance with the physics engine. It will be appreciated that the proposed targeting system may use different forms of physics engine and that in some cases the physics engine may use physics models of known form, but that even with known physic models there are improvements for the proposed system due to the use of better input data reflecting accurate position and orientation of the user interface. By using the physics engine with the data about the target axis of the user interface and added information from machine processing (e.g. Al) of the image data the targeting system can calculate the simulated path along with ricochets, penetration capabilities, damage estimates and hence suitable user feedback (e.g. sound effects).

[0020] Determining if the path of the simulated shot has passed within a certain distance of a user is based on at least two of: the image data, the vector of the target axis of the first user interface, and the relative location of the first and second user interfaces. It will be appreciated that all three of these may be taken into account and the method / system may thus include using all three, e.g. when each of the three is considered accurate and will thus contribute to the realism of the simulation. In many cases the image data and location data are sufficient to determine if a simulated shot has an effect on another user without the use of the orientation data. However, there is an advantage from a redundancy allowing for two out of three types of data to be used, since in some cases there can be sources of error or inaccuracy, such as when there is particularly fast movement of the user interface, unexpected (e.g. external) delay in the transmission or processing of the data, or intervening objects preventing a clear image from the camera device. Fast movement may result in blurred images and intervening objects may block the view of the target object, leading to the image data being inaccurate. Optionally, the quality of the image data is assessed before the step of determining if a simulated shot has passed within a certain distance of a user. In the event of potential inaccuracies in the image data (e.g. if the target user is not visible, or if there is blurring) then the location of a path of the simulated shot relative to the target object (e.g. the second user) may be determined based on the target axis vector and the relative location, without use of the image data.

[0021] If it is determined that the path of the simulated shot will pass through an intervening object, such that the second user is not visible, or is partly obscured, in image data from the first user interface, then the location of the second user may be determined based on the relative location (and optionally also the orientation) of the first and second user interfaces as well as image data from the second user interface. Thus, the accuracy of determination of the location of the second user may be improved by using the image data from the second user interface (as well as location / position information) when assessing the effect of simulated shots aimed at the second user. This added image data may for example give information about the proximity of the second user to the intervening object and / or the positioning of the body of the second user. Optionally the method / system may also use image data from further user interfaces in which the second user is visible, e.g. image data from a third user interface and / or a fourth user interface (or additional user interfaces) that show further detail of the second user’s location and / or body position. Since the object tracking sub-system (and related method steps) can advantageously use the image data from and or all of the user interfaces, as well as other information from the digital representation, such as location and / or orientation of the user interfaces, then it becomes possible to accurately determine the outcome of a simulated shot even when the second user is totally hidden from the viewpoint of the first user interface. Another option is for image data from remote cameras to be used.

[0022] The orientation sensing system may comprise a rotation sensor and / or a motion sensor, or combinations thereof. The orientation sensing system may comprise a gyroscope and / or an accelerometer. In some examples the orientation sensing system is provided via components of an inertial navigation system, such as an Inertial Measurement Unit (IMU). The orientation sensing system may include a sensor or sensor package providing sensing in six degrees of freedom, i.e. it may be a so-called 6-DOF system. An example implementation may include a 3-axis gyroscope, a 3-axis accelerometer, and a 3-axis magnetometer. This provides a full set of data for both of orientation and motion tracking. When the orientation system has the capabilities of an inertial navigation system and / or a 6-DOF system then the same sensors / sensor package may provide positioning / location data as well. This is particularly beneficial when GPS signals are poor and / or if the user interface is indoors.

[0023] In some examples the orientation data and the location data may be provided by a single system and / or via shared sensors, e.g. a combined GPS and inertial navigation system where the motion sensors contribute to the location data. Thus, in some examples of the user interface the orientation sensing system and the positioning system may share sensors and / or the two systems may be sub-systems / separate functionalities of a single navigation system.

[0024] The orientation data may comprise a direction (i.e. forward / backward direction) and an angular position of the target axis. Thus, the vector may include information sufficient to define the direction and angular orientation of the target axis. Other information may be added to this vector, such as an indication of weapon or projectile type that may be expressed as a form of magnitude of the vector. As discussed further below this may influence the feedback to the user. The angular position may for example be expressed in terms of a polar angle and an azimuthal angle, as used in a spherical coordinate system.

[0025] The location of the user interface is advantageously known to an accuracy of 20 cm or better, such as a location accuracy of 10 cm or better. This allows for accurate determination of the placement of the target object relative to a path of the simulated shot. Increased accuracy of the location data allows for the relative locations of the user interfaces together with the orientation of the target axis of the user interface that fires the simulated shot to be used to more reliably determine information about a hit or near-miss even when the image data is not useful, e.g. due to blurring or as a result of intervening objects. The location data may include altitude data and the positioning system may hence include an altitude sensor or the ability to calculate altitude.

[0026] The positioning system of the user interface may be capable of providing location data having an accuracy as above. This positioning system may comprise a Global Positioning System (GPS) device with added accuracy enhancements, such as GNSS enhancements like Real-time Kinematic Positioning (RTK) or Carrier Phase GPS. Alternatively, or additionally, the user interface may make use of inertial positioning and / or a localized positioning system using external reference points. In some examples inertial positioning data may be obtained via sensors of the orientation sensing system. A localized positioning system may comprise one or more fixed beacons for wireless connection to the user interfaces. An array of multiple fixed beacons may be used to provide accurate positioning absent GPS data. A single beacon, or multiple beacons, may be used to enhance the accuracy of GPS positioning by removing sources of error, e.g. as done with the RTK system. A virtual base station may be used, e.g. for a GPS RTK positioning system, or there may be one or more physical base stations, e.g. mounted on tripods placed in suitable locations relative to the real-world environment in which the user interfaces are used.

[0027] The feedback to the second user if the simulated shot is determined to have passed within a certain distance of the second user may include audible feedback, such as feedback provided via a speaker on the user interface or via a peripheral device such as a headset or earphones. The feedback may include other elements as discussed below, e.g. visible feedback, tactile or otherwise, but advantageously it always includes audible feedback. Visible feedback may include lights on the user interface or visual feedback given via a display as discussed in more detail below.

[0028] In example embodiments the content of the audible feedback is determined based on one or more of the simulated weapon and / or projectile type, the distance of the path of the simulated shot away from the user, the location of the path of the simulated shot relative to the user, and / or features of the surrounding environment such as any intervening objects. The image data may be used to determine relevant features of the surrounding environment, as well as aiding in assessing the relative location of the user and the path of the simulated shot. Counter-intuitively, the image data can have greater importance for enhancing the realism of the user experience by virtue of its influence on the audible feedback rather than its impact on visual feedback or other factors (e.g. augmented reality use, as discussed below).

[0029] The simulated weapon and / or projectile type may influence the audible feedback in relation to its content (e.g. the differing sound of different weapons or of penetration of different intervening objects) and / or the volume. The distance of the path of the simulated shot away from the user may influence the audible feedback in relation to its volume. The location of a path of the simulated shot relative to the user may influence directionality of the sound as well as Doppler effects, e.g. in relation to the use of stereo sound for the audible feedback so that a near miss to the right sounds different to a near miss to the left. In that regard the image data from the user interface of the (first) user firing the shot may be used to determine the direction in which the target (second) user is facing. This can further enhance the realism of the experience.

[0030] The features of the surrounding environment (including, for example, intervening o objects) may be determined based on the image data alone or may take into account data about the real-world environment from sources external to the user interface(s), such as terrain data from maps or environmental conditions based on weather reports. The features of the surrounding environment may influence the content of the audible feedback and / or the volume. For example, if the image data indicates that a simulated shot will pass by the user without striking anything then the sound content will be different to a simulated shot that will strike an object and / or pass through an intervening object. Additionally, a simulated shot that will strike (or pass through) different types of objects will give rise to differing audible feedback.

[0031] Simulation of the effect of any intervening objects on the path of the simulated shot is done based on an assessment of penetration ability for the simulated shot and based on determination of the properties of the real -world intervening objects. The characteristics of the simulated weapon and / or a selected ammunition type may be used to determine penetration ability for the simulated shot. As with other features of the surrounding environment the determination of the properties of the real -world intervening objects may be done based on one or more sources of information including the image data and / or data about the real-world environment from sources external to the user interface(s), such as terrain data from maps and / or predetermined categorisation of intervening objects, e.g. data entered in a set-up phase in which potential sources of cover are assessed and categorised. When the image data is used then computer implemented image recognition may be in place to allow for different types of objects (e.g. trees, vehicles, walls, fences, buildings / building features and so on) to be automatically identified.

[0032] The user interface and / or the user feedback sub-system may be provided with a library of sounds relating to different objects being struck and / or to noises made by different weapon or projectile types. The method may include obtaining samples of sounds from real-world environments and the system may comprise a library of sound samples. Suitable software may be used for sound design, such as with the sound being simulated in the cloud before being communicated to the user by means of the user feedback sub-system and instructions provided to the user feedback device at the user interface. For example, the user feedback sub-system may include software such as Wwise (Wave Works Interactive Sound Engine) supplied by Audiokinetic, Inc. of Canada, or the FMOD system developed by Firelight Technologies Pty Ltd of Australia.

[0033] The user feedback device of the user interface may be configured to provide audible feedback for simulated shots that are not aimed at the user but that strike objects within an audible range. The user may also be presented with sounds relating to impacts of their own simulated shots as well as impacts of the simulated shots of other users. The user feedback sub-system may be configured to interact with the user interface in order to provide such audible feedback, e.g. by sending one or more signal(s) indicating a sound or set of sounds to present to each user.

[0034] The trigger device at the user interface may include a sensor element for providing an electrical signal when the trigger device is activated. Registering an activation of a trigger device may include registering a timestamp for the activation, with this timestamp then being used to identify relevant data (i.e. image data etc.) for determining if a path of the simulated shot has passed within a certain distance of a user. The activation of the trigger device may be done via actuation of a physical trigger similar to a gun, e.g. a lever. In some examples this can be a trigger of a real gun, e.g. with a modified trigger guard piece that includes a switch or other sensor for detecting motion of the gun trigger. Alternatively, it may be a sensor that measures the firing of blank cartridges, e.g. a sensor in the magazine or in the barrel of the gun. The trigger device may be configured to provide tactile feedback similar to a real-world weapon, such as by means of an internal mechanism that provides a pattern of reaction force during movement of a trigger lever. The user interface may provide other feedback relating to activation of the trigger device, such as simulated recoil and / or audible feedback. The audible feedback may include sounds simulating the firing of a weapon, falling of a shell casing and / or other sounds associated with a simulated shot. This audible feedback may be provided by the same user feedback device and / or via the same user feedback sub-system that provides an indication of whether or not a simulated shot from another user has passed within the certain distance of the user. The user interface may include a recoil simulation device such as an electromagnetic linear actuator, e.g. a voice coil actuator.

[0035] There is feedback to the user if a simulated shot from another user is determined to have passed within a certain distance of them. If the distance from the user to the path of a simulated shot exceeds the certain distance, then there may be no feedback given. In this way the user may be deemed to be unaware of a faraway event. The length of the certain distance may be determined based on simulated shots that would be detectable events in the real-world, e.g. audible events. The length of the certain distance thus may vary depending on the nature of the simulated weapon. For example, a larger calibre weapon or a weapon using explosive projectiles may give rise to a need for feedback to the user at a larger distance than a smaller calibre weapon or a non-explosive projectile. Information about the simulated weapon may be provided in the data sent from the user interface, for example as an addition to the vector or as an added separate data structure. The data sent by the user interface may include data identifying the user interface including details of the simulated weapon type.

[0036] The length of the certain distance may vary depending on the environment surrounding the user. For example, if there is a wall in between the user and the path of the simulated shot then the audible feedback may differ compared to the same relative locations absent the wall. Similarly, if the path of the simulated shot is determined to have struck and / or penetrated an object (e.g. an intervening object) that is within the certain distance of the user then the audible feedback may take this into account. Thus, a shot passing through a (closed) window or door, or penetrating and being deflected by a vehicle, may result in different audible feedback compared to a path of the simulated shot that does not pass through any objects. A near-miss that passes by the user without striking any object may result in different feedback to a near-miss that strikes an object in a way that could cause an injury to the user, e.g. due to an explosive projectile or due to shrapnel. Features of the environment surrounding the user can be determined with reference to image data from the user’s user interface and from other user interfaces. Features of the surrounding environment may be mapped in the digital representation of the real-world environment, with the content of the digital representation being built up based on some or all of: past image data, mapping or surveying data, and / or user input.

[0037] The targeting system may include simulated shots or other events from other simulated weapons systems, e.g. to include the impact of artillery, ballistic missiles, non- ballistic missiles, airborne weapons, minefields or other external sources when simulating an event in the real-world environment. For example, the system may include audible feedback concerned with the effect of artillery or air support on an infantry fight.

[0038] If a simulated shot is determined to have struck the user then this may trigger additional feedback mechanisms such as a specific visual alert or alarm sound that indicates a hit. This may be used in a scoring or monitoring system to determine a degree of success of different users, or different groups of users, in a simulated event such as a weapons training event.

[0039] Tactile feedback or other forms of physical stimulation may be included in some implementations, preferably in addition to audible feedback. For example, the user feedback device may include an electrical shock system such as a Transcutaneous Electrical Nerve Stimulation (TENS) system, which may be triggered via the user feedback sub-system when the user is hit. This can enhance the realism and allow for increased physical and mental stress that can help simulate a real-world event. The electrical shock system may be triggered at a lower level of intensity when a simulated shot is a near miss. In different intensity settings the effect of an electrical nerve stimulating pulse from a TENS system can range from a low setting that provides a numbing sensation to a high setting that can temporarily incapacitate a muscle group. This may use a wristband (or a pair of wristbands) or other peripheral devices attached to the body, such as a vest. A possible source of tactile feedback is a motion or vibration of the user interface, e.g. to indicate a simulated shot deemed to be a hit on the user or to simulate a simulated shot deemed to be a hit on the user interface. A recoil simulation device of the user interface can be used to provide motion as feedback for a simulated shot from another user.

[0040] The image data is used to determine the placement of real -world target objects in the field of view of the camera device. The image data may include images based on visible light and / or images making use of infrared and / or UV imaging to give enhanced capabilities compared to human sight. When determining the placement of target objects in the form of the users of other user interfaces then this placement may include not only location but also form, e.g. the position of head, limbs and torso, in order that a determination of a simulated shot making a hit on a user can include detail of the location of the hit on the body. This may influence the feedback to the user and / or future action of the user. For example, if the user is struck in a leg there may be a different form of feedback to if the user is struck in the torso or the head. The feedback to the user may include an indication of a hit that is determined to partially or fully disable the user, e.g. a simulated shot may be deemed to be a kill shot. Where the user interface is connected to wearable peripherals then feedback may be provided to differing parts of the user’s body by wearable elements on differing locations of the body in order to better simulate a hit on a specific body part.

[0041] The image data may be used for identification of different users. Thus, images of the users may be registered with the targeting system so that the image data provided from the user interfaces can be used with image recognition systems to identify the users that are targeted by the simulated shots. This has advantages over systems that rely on identification of users via identification of the user interfaces since it allows for the user interfaces to be exchanged freely, e.g. by taking a user interface off another user, replacing a depleted simulated weapon with a fully loaded simulated weapon, or switching user interfaces to change between weapons types. The user interface may also provide identification data to allow for identification of the simulated weapon type and / or to supplement systems for identifying the users by allowing association of a user with a particular user interface.

[0042] The digital representation of the real-world environment is continually updated with the locations of the multiple user interfaces and the vectors of the target axes as the user interfaces are moved by respective users thereof. It may also be updated with information from the image data, e.g. in relation to environmental conditions and / or objects (including the intervening objects) within the real -world environment that may impact on the effect of a simulated shot and / or on feedback to be given to a user. The digital representation may also be described as a digital twin for the real-world environment. It may provide a simplified representation of real-time events in the real-world environment, including the locations of users and / or user interfaces as well as the paths of simulated shots. The digital representation may exist in the cloud, such as within one or more server of the targeting system, i.e. a server remote from the user interfaces. The digital representation and / or the digital twin sub-system may be used for predictive calculations, e.g. to assess the future paths of simulated shots for slower moving projectiles such as artillery fire, mortars or RPGs. The digital representation may advantageously contribute to additional aspects of hit detection and / or realism, such as by assessing the outcome in the case of multiple objects in the image at different distances or by enhancing sound simulation by identifying characteristics of regions through which the simulated shot passes as well as the locality of the user interface (i.e. the source of the ‘gunshot’ sound) and / or the locality in which the simulated shot strikes an object. The digital representation (digital twin) can receive a range of data from the user interface including the image data and location / orientation data as well as potentially other inputs, e.g. from additional sensors where present, and this can be used for training Al elements of the digital twin. This rich data input will provide an incredibly good situational understanding, which is crucial for creating an Al trainer that can assist in real-time with advice and assistance, as well as for post-action analysis and simulation of outcomes based on changes in combat history. Digital and physical players or participants can meet on the same training field. For instance, pilots can fly an aircraft such as a fighter jet in a simulator, and soldiers on the training field can see the virtual aircraft in augmented reality (AR). The aircraft can engage real troops, and real soldiers can simulate shooting down the virtual aircraft. The digital twin can also help with positioning, enabling the camera to better recognize the surroundings based on expected features of that part of the real-world environment.

[0043] The remote server(s) for the digital representation may also include software for the other functions / sub-systems of the targeting system, or these may be distributed over several remote locations. The step of receiving data streamed from the multiple user interfaces may comprise receiving data at the remote server(s).

[0044] The user interface may have a gun-like form and thus in some embodiments it may be referred to as a gun. Alternatively, the user interface may be a module that is configured for attachment to a pre-existing device like a weapon or weapons system, either real or simulated, in order to allow for such a pre-existing device to be used as a part of the targeting system. For example, the user interface may be adapted to be attached to the barrel of a weapon, e.g. via existing rails, and / or may be configured to replace or be attached to a sight of a weapon or vehicle (such as a military vehicle including an armed vehicle), e.g. rifles, handguns, vehicle mounted guns, grenade launchers, artillery, land vehicles including tanks and armoured cars, boats, helicopters, drones, or fixed wing aircraft as well as other aircraft. If this form of user interface module is used with real- world devices such as a real-world gun then it enhances the performance of the “train as you fight” targeting system.

[0045] For a user interface module configured to connect with a gun the invention may comprise a modified gun including the module.

[0046] In such a modified gun the trigger device for the user interface may be provided via an electronic system. This may for example rely on external input such as a sensor connected to the trigger of the weapon itself, along with an electronic circuit activates when the trigger on the weapon is pressed to thereby activate the electronic trigger device and indicate that the user has fired a simulated shot. The sensor for the external input may be connected to the trigger by joining it to the trigger guard or by replacing the trigger guard with a trigger sensor device. In one example this is a switch that is configured to be activated by movement of the trigger of the gun.

[0047] The modified gun may be configured for use with blank ammunition or alternatively it may be modified by use of an electronic magazine allowing for digital ammo. Where blanks are used then the gun can give accurate recoil effects. If blanks are not used then the modified gun may include a recoil simulator device that may be similar to the recoil simulator device mentioned elsewhere herein for the user interface. A recoil simulator device for a modified gun may for example be provided within a replacement buttstock for the gun, e.g. a buttstock that fits to existing buttstock connectors of the gun.

[0048] The user interface comprises the orientation sensing system and the positioning system, which are advantageously rigidly coupled together, e.g. held in a single rigid housing, in order that the orientation and position system are physically linked together. Other parts of the user interface, such as for example the trigger device, the user feedback device or a recoil simulator may be provided outside of this housing, or within it. There may also be software / hardware elements located remotely and / or with a combination of elements inside the housing and outside (e.g. remotely located) such as software for the orientation sensing system and or the positioning system; or software / hardware of the image handling system.

[0049] The target axis may include a physical targeting element, such as an element of elongate shape and / or including a sight as found in a gun. The target axis may for example be aligned with a barrel element of the user interface, or with a structure configured for attachment to the barrel of a weapon.

[0050] The image data is received from a camera device pointing in a direction extending along the target axis, preferably in a direction that is fully aligned with the target axis. The image data will typically be video image data. An imaging direction of the camera device may be aligned with the vector (i.e. the direction and angular orientation) of the target axis. The image handling system of the user interface can be integrated with the camera device. The camera device, e.g. an optical device including a digital image sensor, may be a part of the user interface. In one example the camera device is integrated into a main part of the user interface, e.g. located in a common housing with the orientation sensing system and the positioning system. Alternatively, the camera device may be provided by a secondary device attached at an outer location of the user interface, such as a smartphone attached to a cradle. Thus, the user interface may include the cradle as well as making use of features of the smartphone.

[0051] A smartphone may be used to provide the camera device and / or for other purposes. Thus, the method and operating system may include using the smartphone for purposes such as data transmission, data processing, and / or feedback to the user. The smartphone may hence provide a data transmission device for transmission of some or all of the image data, orientation data and location data, along with optionally pre-processing of the data before transmission e.g. compacting of data. The smartphone may provide for other wireless communications, such as for wireless connection to peripheral devices (e.g. via Bluetooth) that may be used to provide feedback to the user.

[0052] The targeting system relies on data transmitted from the multiple user interfaces, which may be provided with a data transmission system via a dedicated on-board device or via a smartphone. The communication between the user interface and other parts of the targeting system, e.g. a remote server that receives the streamed data from the multiple user interfaces, may be done via a low latency streaming technology advantageously having a latency of less than 100ms. Possible streaming technologies include open source video transport protocols such as Secure Reliable Transport (SRT), application-level network protocols such as Real Time Streaming Protocol (RTSP), and Web Real-Time Communication (WebRTC). In an example embodiment a suitable low latency streaming technology is used to transmit the image data to the remote server (e.g. cloud server) in the form of video feed as an underlying layer. The orientation data and position data are also sent via the low latency streaming technology. Reductions to latency may allow for a higher resolution of data (e.g. higher resolution image data, higher frequency of position and orientation updates) can be sent so that the effect of a simulated shot can be determined more accurately. Using low latency systems also allows for the processing of data to be focussed at a remote server (e.g. in the cloud) and minimises the processing power / software requirements for the user interface. The user interfaces can be kept relatively simple and standardised, whilst the remote processing of data can be continually refined and enhanced with added processing power as needed, e.g. for enhanced forms of Al processing of images and so on, without any penalty in terms of the overall speed of the targeting system. When processing of data is focussed in the cloud then the user interface becomes less susceptible to unwanted phenomenon such as feature creep or bloating and therefore it may have a longer service life even as the overall targeting system is upgraded.

[0053] The camera device may use visible light as is typically known for smartphone cameras and other similar digital imaging devices. Alternatively, or additionally, the camera device may use different imaging techniques such as thermal imaging or night vision devices (image intensification) in order to provide the targeting system with increased information about the real-world environment and more accurate simulation even in low lighting and / or where vision is obscured by objects or by weather conditions.

[0054] The user interface may include a display for providing information to the user, including optionally for providing visual feedback in relation to simulated shots. A touchscreen display may be used in order that the user can also input information or access control / set-up functions. It will be appreciated that when a smartphone is incorporated into the user interface then the smartphone display can be used as a display for the user interface. A display may alternatively or additionally be provided via a head-up display (HUD) such as via a helmet, glasses, or other headgear.

[0055] When a display is present then this may be used to provide information to users about their status and / or status for the user interface. For example, in context of a wargame the status may include information about ammunition, health, shields / armour, and other relevant details. This ensures an engaging and immersive gameplay experience. Where the display is used to provide visual feedback in relation to simulated shots then this may comprise text information or imagery depicting the path of the simulated shot. The visual feedback may also provide information about simulated impact of a hit on the user, such as likely injury and so on.

[0056] A display may optionally be used for an augmented reality system, i.e. to overlay information and / or imagery onto a display of the real-world environment. The augmented reality elements may include features such as the paths of simulated shots, status, or tactical information, e.g. relating to other users that are in view, simulated target objects or other simulated features such as geotagged locations. The geotagged locations provide added functionality linked to a gaming or training scenario, such as by providing checkpoints, added resources (e.g. ammunition, health, amour), bonus features, boosts or information about a mission. Augmented reality graphics could use a smartphone display or other separate display of the user interface or a connected peripheral device, such as via a scope or augmented reality headwear.

[0057] Other add-ons may be provided in addition to those discussed above, such as a telephoto lens for providing extended range and capturing more precise details of a target area. It will be appreciated that by enhancing the information content of the image data then the targeting system can provide more accurate information about the simulated shots and the placement of the target object (e.g. a target user).

[0058] In yet another aspect the invention provides a computer programme product comprising instructions that, when executed, will configure a computer system to operate in accordance with the method of the first aspect and optionally also other features thereof as discussed above. The computer programme product may be a software product for providing and then a targeting system as in the second aspect, when the software is installed on a suitable computer system. In some examples the computer programme product may comprise a part that provides an app for configuring a mobile computer device such as a smartphone to interact with the user interface and with a remote server of the targeting system.

[0059] Certain embodiments of the present invention will now be described by way of example only and with reference to the accompanying drawings in which:

[0060] Figure 1 shows an example weapons training scenario using a targeting system and multiple user interfaces;

[0061] Figure 2 is a perspective view of an example user interface in the form of a simulated weapon;

[0062] Figures 3 and 4 are side and perspective views of an example user interface in a modular form that is for attachment to a pre-existing weapon such as a gun or military vehicle;

[0063] Figure 5 shows the modular user interface of Figures 3 and 4 mounted to a tank;

[0064] Figures 6 and 7 show the modular user interface of Figures 3 and 4 mounted to first and second variants of a modified assault rifle;

[0065] Figures 8 and 9 show variations of the user interface of Figure 2;

[0066] Figure 10 shows a user interface in the form of a military style training weapon incorporating similar functionality to the user interface of Figure 2;

[0067] Figure 11 shows an example of optional use of a visual display; and

[0068] Figure 12 is a schematic diagram of multiple user interfaces interacting with other parts of the system in the cloud.

[0069] An example of targeting system is shown in use as a weapons training system in Figure 1. This includes computer systems operating in the cloud 8 and in communication with multiple user interfaces 10, in this case each with an associated user 12. The user interfaces 10 may for example be as shown in any of Figures 2 to 9. The users 12 are identifiable as Al, A2, A3, A4 and Bl, B2, B3, B4, B5 in this case in a scenario where the A-team and the B-team are aiming to achieve competing objectives. The A-team includes infantry supported by artillery and by an Armoured Personnel Carrier (APC) or tank 13, which can optionally include a module providing the APC with a user interface 10 as shown in Figure 5 below. The B team includes infantry as well as a drone 15. The tank 13 and drone 15 can be provided with a user interface 10 either integrated into the design (e.g. a bespoke drone 15 for training use) or attached to pre-existing hardware (e.g. a modular user interface 10 attached to a real tank 13 as in Figure 5). The users 12 and user interfaces 10 are within a real-world environment 14 that includes physical features relevant to operation of the targeting system such as trees 16, rock 17 and a vehicle 18. The targeting system is configured to provide feedback to the users 12 in relation to simulated shots 21, 22, 23, 24 based on data from the user interfaces 10 as described in more detail below.

[0070] In this example the two sets of users 12 are spaced apart by about 300 metres. The targeting system is aware of the distance between the users 12 as a result of positioning data from the user interfaces 10. The targeting system is also aware of the simulated shots 21, 22, 23, 24 and can determine the path of the simulated shots relative to the users 12 based on the data from the various user interfaces 10. This allows the targeting system to determine what feedback is appropriate for the various users 12, with suitable feedback, e.g. audible feedback, being given to the individual users 12 through features of the user interfaces 10, also described in more detail below.

[0071] In this scenario, for a time period where the users 12 are located as shown and where four simulated shots 21, 22, 23, 24 occur, the feedback occurs as follows:

[0072] • For the user Al a simulated artillery shell 21 passes overhead, being fired at user Bl. Al will hear distant thunder and the artillery shell passing overhead before it hits the target 300 metres away. Al may also hear gunshots from other simulated shots 22, 23, 24, but the audible feedback relating to their impact will be at a lower volume as they are further away.

[0073] • For the user A2 audible feedback and a simulated recoil will be provided when they fire the simulated gunshot 22. The shot 22 is aimed at B2 through a vehicle 18. The targeting system uses the data from the user interface 10 along with Al and a physics engine to determine that the shot 22 will hit B2. A2 will hear the noise of the bullet’s impact on metal / glass as well as an audible indication of the hit on B2. In addition, augmented reality systems or other visual feedback mechanisms may be used to indicate a hit. The user A2 may also hear noises associated with the simulated artillery shell 21 as well as the other simulated gunshots 23, 24.

[0074] • For the user A3 audible feedback and a simulated recoil will be provided when they fire the simulated gunshot 23. The shot 23 is aimed at B3. In this case the targeting system determines that the shot 23 will not hit B3 but will pass close by. The Al determines that the shot 23 hits a tree 16 based on modelling of the surrounding environment 14. A3 will hear the noise of the bullet’s impact on the tree. In addition, augmented reality systems or other visual feedback mechanisms may be used to indicate a miss, such as by showing the bullet trajectory. The user A3 may also hear noises associated with the simulated artillery shell 21 as well as noises associated with the other simulated gunshots 22, 24.

[0075] • The APC 13 (user A4) may include a user interface 10 attached to it, or alternatively the system may assign handheld user interfaces 10 of occupants of the APC as a user interface for the APC 13 for feedback purposes. User B4 has fired a simulated shot 24 that hits the APC 13. A4, e.g. the occupants of the APC 13 will hear the noise of the impact of a shot on the APC 13. A4 may also hear the artillery shell 21 and gunshots 22, 23, but with the sound suitably modified to simulate the effect of being inside the APC 13.

[0076] • If there is a physical drone 15 as user B5 then A4 can see the physical drone 15, or in the alternative A4 (and the other users) may see a virtual drone 15 (or other aircraft) using augmented reality. The virtual unit B5 can attack the real unit A4 and vice versa.

[0077] • The user Bl has taken cover behind rock 17. The targeting system knows that simulated shots from rifles may not be able to penetrate the rock 17 so the user Bl has protected themselves from Al, A2 and A3.

[0078] • The user Bl hears the approaching artillery shell 21 as well as the noise of an explosion upon impact. The targeting system determines a damage radius taking account of the surrounding environment, e.g. the shielding effect of rock 17 and / or the ability of the artillery shell 21 to pass over rock 17 and injure the user Bl. A visual display system and / or augmented reality system may be used to show Bl an explosion and smoke. Bl may also hear gunshots and / or impacts from the simulated shots 22, 23, 24, but the audible feedback relating to their impact will be at a lower volume as they are further away.

[0079] • The user B2 hears the simulated shot 22 and the impact of the bullet on the vehicle 18. The targeting system determines there is a hit but that there is less damage than an equivalent direct hit since the bullet loses kinetic energy by penetrating the vehicle 18. The user B2 may also hear noises associated with the simulated artillery shell 21 and its explosion as well as the other simulated gunshots 23, 24. The targeting system will determine a different outcome for user B2, who has taken cover behind vehicle 18, compared to user Bl who is sheltered by rock 17.

[0080] • The user B3 hears shots fired from 300 metres away along with sonic booms (cracks) and the bullet of simulated shot 23 narrowly missing. There is the sound of bullet impact on the surrounding environment, i.e. the trees 16. The simulated shot 23 fired by user B4 is also audible to user B3. They may also hear the other gunshot 22 and / or noises from the simulated artillery shell 21, but the audible feedback relating to their impact will be at a lower volume as they are further away.

[0081] • The user B4 fires at A4, the APC 13, and hits. However, B4 fires with a user interface 10 simulating a regular rifle calibre, and the bullets bounce off the APC 13 as the armour is too thick. The bullet ricochets and may hit nearby surroundings. B4 hears their gunshot 24 and the ricochet at B4, and they may also hear sounds relating to the artillery shell 21 and the other gunshots 22, 23.

[0082] • The user B5 is a physical or virtual drone 15 with simulated weaponry, e.g. explosives. A physical drone can provide overhead image data to the digital representation to aid determination of positioning of other users. Virtual drones (piloted by real users) can navigate using the digital representation (digital twin) and thus know the exact location of a target such as the tank 13. Both physical and virtual units can converge in a training field where the digital twin augments the real environment.

[0083] It will be appreciated that providing audible feedback of missed shots and other shots happening in the same simulated scenario then there is enhanced realism for all of the users 12. The B-team become aware of enemy fire even if they are not hit. The effect of the hit on B2 takes account of the fact that this user 12 has found some protective cover behind the vehicle 18. This is done without the need for hit sensing systems or reliance on line-of-sight principles as with lasers and the like. The system described herein can simulate penetration ability based on real objects and surfaces so that the terrain and surroundings can be actively and strategically used in games / training, much like in a real gunfight.

[0084] For example, if B4 had fired with an anti-tank weapon, the damage would be simulated based on what B4 is firing and where it hits the APC 13 (A4). The audible feedback to the various users 12 would be changed accordingly. No sensors are necessary on the APC 13 to achieve this, i.e. no laser sensor or IR sensor. We achieve a highly realistic simulation even without any modification to the APC 13. This simulation can be further enhanced for users 12 within the APC 13 by means of either a user interface 10 carried by such a user 12, or by fitting the APC 13 with a module that provides it with a dedicated user interface 10. Such a module may be similar to that described below with reference to Figure 3. It can provide user feedback as well as allowing for simulated shots via the weapon(s) of the APC 13, if it is set up with appropriate alignment for a target axis 26.

[0085] Figure 2 shows an example of a user interface 10 for the targeting system. This user interface 10 takes the form of a gun 10 and it is used as a simulated weapon, such as for games or for training purposes. The user interface 10 has a target axis 26 passing along a barrel of the gun form, which in this case extends along a lens system 28 of a camera device 30. The lens system 28 includes an optical zoom system with a zoom control 29, which in this case is a control wheel. The lens 28, zoom control 29 and camera device 30 can be part of a removable module allowing for different forms of camera to be fitted and / or allowing for the camera to be omitted (e.g. with use of a smartphone camera instead, as discussed below). The camera device 30 is aligned to capture images facing along the direction of the target axis 26, and it may also include an image handling system for receiving image data from the camera device 30 (e.g. from an imaging chip thereof) and sending image data to other parts of the targeting system. A trigger device 32 is included in a similar location to a gun. This trigger device 32 is activated by the user 12 in order to fire a simulated shot along the target axis 26.

[0086] The main body 34 of the user interface 10 of this example holds a power source such as batteries (not shown), an orientation sensing system 36, a positioning system 38, and a recoil simulation device 40. A user feedback device 42 is provided for audible and optionally visual feedback. The user feedback device 42 can also take the form of a peripheral device such as a headset or earphones, which may be in wireless communication with the user interface, such as via Bluetooth. The main body 34 can also hold a communications system 44 for network communications with other parts of the targeting system in the cloud 8, such as communications via a cellular network.

[0087] The orientation sensing system 36 is for determining a vector of the target axis 26, i.e. to determine its angle in space and the direction that it is pointing. The orientation sensing system 36 may use a gyroscope sensor or similar sensing arrangement. It is rigidly connected to the main body 34 of the user interface 10 so that it is in a fixed alignment with the target axis 26. Orientation data comprising the vector is passed via the communication system 44 to remote parts of the targeting system for use in determining the effect of simulated shots.

[0088] The positioning system 38 should have a suitably high accuracy, ideally being able to determine location to an accuracy of 10 cm or better. In this example it is a GPS RTK system.

[0089] The recoil simulation device 40 can be of known type, such as a device using an electromagnetic actuator or a pressurised gas to simulate recoil of a weapon firing a projectile. The recoil simulation device 40 may be activated by activation of the trigger device 32 when the user 12 wishes to fire a simulated shot. At the same time the user feedback device 42 may emit the noise of a gun firing as well as optionally other sounds, such as the noise of a shell casing landing on the floor. The nature of these noises may be controlled via input from an Al of the targeting system, which may determine a suitable noise based on assessment of the image data to determine relevant factors of the surrounding environment. For example, where the image data indicates that the user 12 is stood on a forest floor surrounded by sound absorbing features such as bushes and trees then the sounds of the simulated shot will differ compared to when the image data indicates that the user is stood in an enclosed building with a hard floor.

[0090] The user feedback device 42 may include speakers and optionally lights on the main body 34. Alternatively, or additionally, it may include a peripheral device as noted above. The use of stereo sound, e.g. provided via ear phones, is beneficial for audible feedback since it allows for simulation of the location of the sound source. A combination of sound sources, e.g. at the main body 34 as well as via headwear, may be used to provide an added dimension to the sound, giving the opportunity for immersive sound. With stereo or immersive sound, the user 12 then has a more immersive experience in which they can hear if a simulated shot is striking an object to the left or to the right. The user feedback device 42 can provide audible feedback as discussed above in relation to the example of Figure 1. The user feedback device 42 may be provided with a library of suitable sounds.

[0091] In the example of Figure 2 the user interface 10 also includes a mounting cradle 46 for a smartphone 48. The mounting cradle 46 connects to rails 50, which may alternatively be used to connect other devices such as a scope, laser pointer, night-vision camera, or other accessory. With the use of a smartphone 48 then the camera of the smartphone may be used in place of the camera device 30 shown in Figure 2. The smartphone 48 may also provide the connectivity needed for joining the user interface 10 to other parts of the targeting system, and thus may replace the communication system 44. However, it is not typically possible for the smartphone 48 positioning system to provide the positioning system 38 of the user interface 10 since typical smartphone positioning systems will not provide sufficiency accuracy. Also, even if the smartphone 48 includes a suitably accurate gyroscope for the orientation sensing system 36 it is preferred to use an integrated orientation sensing system 36 in order to ensure accuracy and rigidity of the alignment of this with the target axis 26.

[0092] The smartphone 48 includes a display, typically a touch screen display, and this may be used for visual feedback to the user 12, such as for an alert indicating a hit or a near miss, or information about the nature of a simulated impact of a hit on the user 12, such as likely injury and so on. The display may also be used to give information to users about their status and / or status for the user interface or to provide an augmented reality view. Other display systems may be used for those purposes in place of the smartphone display or in addition to it. For example, the user interface 10 may be include wirelessly connected peripherals for augmented reality systems such as a headset, helmet or glasses.

[0093] Figure 3 shows an alternative form of user interface 10, which is a module for attachment to an existing weapon such as a gun or an armed vehicle. Figure 4 shows a perspective view of the same user interface 10. Figure 5 illustrates a similar user interface 10 mounted to a tank 13. This module form of the user interface 10 might be for attachment to a hand weapon or it may be for a weapon that is installed on other hardware (e.g. a military vehicle or tank such as APC 13). It will be appreciated that different adaptations can be made for mounting in different ways, e.g. for mounting on the rail system or the barrel of a pre-existing weapon 100. Examples are shown in Figures 6 and 7, where the user interface 10 is a module attached to a Heckler & Koch HK416, which is an assault rifle 100 as used by the Norwegian armed forces.

[0094] In Figure 6 the assault rifle 100 is modified by mounting the main module form of the user interface 10 on to the barrel 102 of the rifle, e.g. via rails 50, and by mounting a trigger device 32 to the gun in a modified form of a trigger guard 106. The trigger device 32 includes a switch that is activated by movement of the existing trigger of the rifle 100. This modified gun 100 may be used with blank ammunition in magazine 108 in order to also provide for gunshot sounds and recoil effects through buttstock 104, whilst the user interface 10 and the other features of the targeting system determine the path of simulated shots and assess the feedback to give to the user based on hits or near misses.

[0095] In Figure 7 the rifle is further modified by use of an e-magazine 109, i.e. effectively digital ammo, along with a modified buttstock 105 that includes a recoil simulation device 40 to replicate the recoil effect from discharge of ammunition. With this configuration the user interface 10 can use the user feedback device (e.g. speakers, tactile feedback, headset etc.) to provide aural feedback for both the gunshots from the user’s gun as well as for hits / near misses. It is also possible to use the e-magazine 109 without the modified buttstock 105.

[0096] As with the user interface 10 of Figure 2 the user interface 10 of Figures 3 to 7 has a target axis 26 along a lens system 28 of a camera device 30. The lens system 28 includes an optical zoom system with a zoom control 29. The camera device 30 is aligned to capture images facing along the direction of the target axis 26, which may for example be aligned with (or parallel with and at a known relationship to) the barrel of the rifle of Figures 6 and 7.

[0097] The main body 34 of the user interface 10 holds a power source such as batteries (not shown), an orientation sensing system 36, a positioning system 38, a user feedback device 42, and a communications system 44 for network communications with other parts of the targeting system in the cloud 8. These components can be similar to those described above in relation to Figure 2. A computer device and eSIM can be included so that the user interface 10 can operate without use of a smartphone 48 or other external communication device. The main body 34 may optionally include a recoil simulation system (not shown), or a recoil simulation system may be provided as an add on to be mounted at another location on the weapon, e.g. as seen in Figure 7. A trigger device 32 can be provided as a part of a modified weapon, such as in Figures 6 and 7.

[0098] With this user interface 10 there is no mechanical trigger device 32 provided on the main body 34. Instead, the trigger device 32 is provided by an electronic system (e.g. as seen in Figures 6 and 7), which may for example rely on external input such as a sensor connected to the trigger of the weapon itself, along with an electronic circuit activates when the trigger on the weapon is pressed to thereby activate the electronic trigger device 32 and indicate that the user 12 has fired a simulated shot.

[0099] The module of Figures 3 and 4 may be a module for providing a standalone simulated gun of the type shown in Figure 2. Thus, the same module can be a user interface 10 for mounting to an existing gun, or it may form a removable part of a larger user interface with a main body 34 shaped like a gun as in Figure 2.

[0100] Further variations of the user interface 10 are shown in Figures 8, 9 and 10. Figure 8 shows a device similar to that of Figure 2, with the same reference numbers indicating corresponding parts. In place of the smartphone 48 and cradle 46 this example includes a scope 49 mounted to the rail 50 of the main body 34. In Figure 9 another similar user interface 10 is shown, in this case with a red dot sight 51 mounted to the rail 50. Figure 10 depicts a user interface 10 with a military design for more realistic training of armed forces. This user interface 10 can have similar features and functionality to that of Figure 8 or 9, but with a form and weight more closely matched to real weapons and thus having a realistic shape / weight including for the e-magazine 109 and buttstock 105 with recoil simulation device 40. Unlike the examples of Figures 6 and 7 there are no modifications or assembly needed to adapt a real gun 100, so this variation can provide realistic training quickly and effectively.

[0101] Figure 11 illustrates a screenshot from an example using a display of the smartphone 48, which depicts the image data for their user interface at the time of a simulated shot. This shows another user 12 who is being targeted by the first-person user. An augmented reality visual element is added to depict the path 52 of a simulated shot that is a near miss, including an indication of proximity of the path 52 to the user 12 at its closest point 54. In this instance the Al of the targeting system can be able to differentiate between the target user’s clothing and their torso, in order to determine that they have not been hit. The Al can also use the image data to assess the surroundings, in this case a forest, and to adapt the audible feedback accordingly for both users. The visual display on the smartphone 48 can also include other augmented reality imagery and / or further information including scoring, network performance, number of players, status bars and menu links and so on.

[0102] The Al of the targeting system is found in the cloud 8 in this example, and Figure 12 depicts a possible configuration. There are multiple user interfaces 10, which may have features as discussed above, and the primary features that interact with the cloud are: the camera device 30, which provides the image data; the orientation sensing system 36, which provides the orientation data; the positioning system 38, which provides the location data; and the user feedback device 42, which receives information from the cloud 8 and provides suitable feedback to the user 12.

[0103] In the cloud 8, such as in one or more servers remote from the real-world environment 14, various sub-systems of the targeting system interact with one another and with the user interfaces 10. As noted above these sub-systems can be in shared, separate, or overlapping computer systems, i.e. implemented via shared, separate, or overlapping software and / or hardware elements. The cloud-based elements include a digital representation 56 of the real-world environment 14; a digital twin sub-system 58 for updating the digital representation 56; an object tracking sub-system 60; a simulated shot sub-system 62; and a user feedback sub-system 64. Each of these systems may be integrated into an Al system, or a system with combinations of Al and traditional computer processing. The Al may be used for image recognition and for other aspects of processing of the image data, such as for updating the digital representation. The various subsystems can interact with a physics engine 66 that is configured to provide modelling for the simulated shots.

[0104] The various sub-systems along with the Al and the physics engine 66 can make predictions, based on the collected data, to estimate what one hits, who one hits, the distance to an object that is hit, what the object is, and so on. The digital representation 56 can contain predefined elements in a map (photogrammetry), such as a 3D model of a tree or a rock, for example. This can help make hit detections more accurate and improve physics calculations.

[0105] The digital representation 56 of the real-world environment 14 is a simplified copy of the real-world environment 14 that reflects real-time paths and events, e.g. for movement of users 12 and paths 52 of simulated shots. The digital twin sub-system 58 continually updates the digital representation with the locations of the multiple user interfaces 10 and the vectors of the target axes 26. The digital representation 56 is also continually updated via the Al with information derived from the image data and other sensor inputs as well as external information (e.g. mapping, weather reports).

[0106] The object tracking sub-system 60 uses the image data for each of the user interfaces 10 to determine the placement of real -world target objects including the users 12 of the user interfaces 10. This can include determining location and form of a person in the image data, as depicted in Figure 11. The object tracking sub-system 60 and / or the digital twin sub-system 58 can also be used to identify other relevant objects and features of the surrounding environment by use of machine learning / Al routines for image recognition. The targeting system can hence identify the nature of the real-world environment 14 (e.g. indoors / outdoors, urban, field, forest) as well as the locations of objects such as trees 16 and vehicles 18.

[0107] The simulated shot sub-system 62 registers an activation of a trigger device 32 of a user interface 10 and thereafter determines a path 52 of the resulting simulated shot. This is done with input from the physics engine 66. The simulated shot sub-system 62 determines if a path 52 of a simulated shot from a first user 12 using a first user interface 10 has passed within a certain distance of any other user, e.g. a second user of a second user interface 10. This is done based on at least two of the image data from the first user interface 10, the vector of the target axis 26 of the first user interface 10, and a relative location of the first and second user interfaces 10. It is important to realise that for most cases only two of these three are needed, which enables the targeting system to provide realistic and accurate determinations of the effect of simulated shots even when one data source is not available, e.g. due to blurred focus, poor lighting, or where the line of sight of the camera device is blocked by an intervening object. Inclusion of the image data adds other benefits including the ability to recognise users from their images as well as to ensure that the correct feedback is given for the features of the real -world environment 14 that are in the vicinity of the users 12 who experience the simulated shot.

[0108] The output of the simulated shot sub-system 62 is used along with optionally other data, in particular from the digital twin sub-system 58 and the physics engine 66, in order to determine what feedback is needed for each user 12. The user feedback sub-system 64 is used for providing feedback to users 12 if a path of a simulated shot is determined to have passed within the certain distance of the respective user 12. This may be done via a signal (e.g. instructions and / or a sound file) that is sent back to the respective user interface 10 in order to prompt audible feedback to the user 12, as well as optionally other forms of feedback. Examples of user feedback are described above, including with reference to Figure 1.

[0109] Thus, it will be appreciated that the targeting system described herein can provide an advanced gaming and / or weapons training system giving unique and engaging simulations of real-world experiences. Highly realistic simulations are created by leveraging the users’ surroundings 14, 16, 18 as an integral part of the user feedback. The targeting system determines not only if a user 12 has been hit, but also how a user 12 will experience simulated shots that have not hit them. This includes feedback to the user based on the proximity of the simulated shot, the nature of the simulated weapon, and the surrounding environment include objects around the user 12. That includes simulating the effect of intervening object (e.g. a tree 16 or vehicle 18) on the path of a projectile, allowing for users 12 to take cover as in the real -world, but also allowing for shots to penetrate that cover and still hit a user 12 even when they are not visible in a direct line of site from the user interface 10 that fires the simulated shot. The simulation can include determining the nature of the feedback to the user such as variations in the sound created by impacts on different objects. This may also include predictive calculations such as assessment of the future paths of simulated shots for slower moving projectiles such as artillery fire, mortars or RPGs. This can then allow for the effects of such shots to be determined with reference to movements of the target objects (e.g. users 12) and / or movements of intervening objects, e.g. a moving vehicle 18 that is used as cover by a user 12. It will be appreciated that it may only necessary to make predictive calculations for the simulated shot and not for the users or for other objects since at the future time when the shot hits (or misses) its target then the actual location of the real -world objects can be used to determine the effects of the slow-moving shot.

[0110] This innovative approach can be implemented via a user interface 10 that is used in conjunction with a mobile phone 48 as shown in Figure 2. In that case a main body 34 serves as a central unit that integrates with the mobile phone 48, with the phone's camera, screen, and data transmission capabilities being used to enhance the gaming experience. Alternatively, the user interface 10 may be provided as a stand-alone version where the mobile phone 48 is not needed as shown in Figures 3 to 10. These versions includes added integrated features such as a camera 30, optional display, and computational power. Other forms of user interface 10 are also possible, such as a module having different designs of attachment to mount it to existing weapons of various types or to other equipment such as a vehicle. It is an advantage to allow for some scenarios to use authentic equipment, e.g. for military training purposes. The option for a user interface 10 in the form of a module for attachment to existing military equipment can allow for all the advantages of the proposed training system with added realism in relation to the equipment that is used, such as weight, size, and so on. Before use in a training scenario the targeting system may be configured with the correct weapon characteristics to be associated with each user interface 10.

[0111] The nature of the system means that augmented reality (AR) can be used to add further “gaming” elements and / or interactive information displays. AR graphics can be shown via a scope or AR glasses that may be provide as add-ons / peripherals for connection to the main user interface 10. This can allow for virtual elements to be added to the real- world environment for reasons of gameplay and / or to enhance immersion. AR may be used as visual user feedback such as to show fire, smoke, sparks, shrapnel or other simulated outcomes in the event of a simulated shot that affects a user using AR devices.

[0112] Data collected from various sensors, including the camera 30, gyroscopes 34, GPS RTK 36 (and optionally UWB location sensing), and Al software, is transmitted via a low latency protocol and processed in the cloud server 8. This allows for user identification, hit detection, and simulation of realistic game / training scenarios. Advanced technologies such as machine vision (Al), physics calculations, centimetre-precise GPS, gyroscopes, and altitude data are used to achieve precise hit detection. This enables accurate tracking of hits on targets and enhances the realism of the gameplay. Real-time positioning is an important part of the system. In the example embodiments an accurate system such as GPS RTK technology and / or UWB location sensing technology ensures centimetre-accurate real-time positioning of all players, facilitating precise gameplay interactions and strategic decisionmaking. In some examples the targeting system and related methods may be configured for a transition between different sensing systems as users 12 move from outdoor to indoor environments, for example switching to UWB for indoor navigation. This can provide a seamless transition with suitably accurate determination of location (and orientation) as users 12 change locations within the real -world environment, which is paired with location updates in the digital representation.

[0113] By means of the user feedback sub-system 64 and / or user feedback device 42 it is possible to provide a rich audio experience, including the sound of projectiles, explosions, and other environmental cues. Visual effects, such as AR graphics and / or on-screen indicators, provide real-time feedback and enhance the visual appeal of the game / training system. The system incorporates real-world environmental factors, such as weather conditions and terrain, into the gameplay simulation. This adds an additional layer of complexity and strategic elements for players to consider. Through a HUD (Heads-Up Display) on the screen, users 12 can receive vital information about their game status, ammunition, health, shields, and other relevant details. This allows for an engaging and immersive gameplay / training experience.

Claims

CLAIMS:

1. A method of operating a targeting system using data streamed from multiple user interfaces, the targeting system comprising a digital representation of a real-world environment in which the multiple user interfaces are being used; wherein the method includes: receiving data from the multiple user interfaces, wherein the data comprises, for each interface: location data giving the location of the user interface, orientation data giving a vector of a target axis of the user interface, image data from a camera device pointing in a direction that extends along the target axis, and an activation state of a trigger device of the user interface; updating the digital representation with the locations of the multiple user interfaces and the vectors of the target axes as the user interfaces are moved by respective users thereof; using the image data to determine the placement of real -world target objects including the users of the user interfaces as well as using the image data and / or the digital representation to determine the placement of intervening objects in the real- world; registering an activation of a trigger device by a first user of a first user interface, wherein activation of the trigger device indicates that a simulated shot has been fired; determining if a path of the simulated shot has passed within a certain distance of a second user of the second user interface based on at least two of: the image data from the first user interface, the vector of the target axis of the first user interface, and a relative location of the first and second user interfaces based on the location data; simulating the effect of any intervening objects on the path of the simulated shot based on an assessment of penetration ability for the simulated shot and based on determination of the properties of the real -world intervening objects; and providing feedback to the second user if the path of the simulated shot is determined to have passed within a certain distance of the second user.

2. A targeting system configured to receive and use data streamed from multiple user interfaces, wherein the data comprises, for each interface: location data giving the location of the user interface, orientation data giving a vector of a target axis of the user interface, image data from a camera device pointing in a direction that extends along the target axis, and an activation state of a trigger device of the user interface; wherein the targeting system comprises: a digital representation of a real-world environment in which the multiple user interfaces are being used;a digital twin sub-system for updating the digital representation with the locations of the multiple user interfaces and the vectors of the target axes as the user interfaces are moved by respective users thereof; an object tracking sub-system configured to use the image data to determine the placement of real -world target objects including the users of the user interfaces as well as to use the image data and / or the digital representation to determine the placement of intervening objects in the real -world; a simulated shot sub-system configured to: register an activation of a trigger device by a first user of a first user interface, wherein activation of the trigger device indicates that a simulated shot has been fired; determine if a path of the simulated shot has passed within a certain distance of a second user of the second user interface based on at least two of the image data from the first user interface, the vector of the target axis of the first user interface, and a relative location of the first and second user interfaces based on the location data; and simulate the effect of any intervening objects on the path of the simulated shot based on an assessment of penetration ability for the simulated shot and based on determination of the properties of the real -world intervening objects; and a user feedback sub-system for providing feedback to the second user if the path of the simulated shot is determined to have passed within the certain distance of the second user.

3. A method or system as claimed in claim 1 or 2, wherein determining if the path of the simulated shot has passed within a certain distance of a user is based on all of the image data, the vector of the target axis of the first user interface, and the relative location of the first and second user interfaces.

4. A method or system as claimed in claim 1, 2 or 3, wherein if it is determined that the path of the simulated shot will pass through an intervening object such that the second user is not visible in image data from the first user interface then the location of the second user is determined based on the relative location of the first and second user interfaces, image data from the second user interface, and optionally image data from further user interfaces in which the second user is visible.

5. A method or system as claimed in any preceding claim, wherein the quality of the image data is assessed before determining if a simulated shot has passed within a certain distance of a user, and wherein in the event of potential inaccuracies in the image data then the location of the path of the simulated shot relative to the second user is determined based on the target axis vector and the relative location without use of the image data.

6. A method or system as claimed in any preceding claim, wherein the location data allows the location of the user interface to be known to an accuracy of 20 cm or better.

7. A method or system as claimed in any preceding claim, wherein the feedback to the second user includes audible feedback, and wherein the content of the audible feedback is determined based on one or more of: the simulated weapon and / or projectile type, the distance of the path of the simulated shot away from the user, the location of the path of the simulated shot relative to the user, and / or features of the surrounding environment.

8. A method or system as claimed in any preceding claim, wherein each user interface provides audible feedback for simulated shots that are not aimed at the respective user but that strike objects within an audible range, including sounds relating to impacts of simulated shots fired by the respective user from their own user interface.

9. A method or system as claimed in any preceding claim, wherein registering an activation of a trigger device includes registering a timestamp for the activation, with this timestamp then being used to identify relevant data for determining if the path of the simulated shot has passed within a certain distance of the second user.

10. A method or system as claimed in any preceding claim, wherein the length of the certain distance is determined based on simulated shots that would be detectable events in the real-world, and wherein the length of the certain distance varies depending on the nature of the simulated weapon and / or on the environment surrounding the user.

11. A method or system as claimed in any preceding claim, wherein the image data is used to determine relevant features of the surrounding environment and / or for identification of the users of the user interfaces.

12. A method or system as claimed in any preceding claim, wherein the image data from all of the user interfaces is used to determine the placement of real-world target objects in the field of view of the camera device, and wherein the digital representation of the real-world environment is used for mapping of the real-world target objects as well as other features of the surrounding environment.

13. A method or system as claimed in any preceding claim, wherein the determination of the placement of target objects that are users of other user interfaces includes not only location but also the form of the users of other user interfaces in the image data in order that a determination of a simulated shot making a hit on such a user can include detail of the location of the hit on that user’s body.

14. A method or system as claimed in any preceding claim, wherein in addition to visible light the camera device uses thermal imaging.

15. A method or system as claimed in any preceding claim, wherein the user interface includes a smartphone that is used to provide the camera device and / or for data transmission.

16. A computer programme product comprising instructions that, when executed, will configure a computer system to operate in accordance with the method of any of claims 1 or 3 to 15.

17. A user interface for a targeting system as claimed in any of claims 2 to 15, the user interface comprising: a target axis; an orientation sensing system for determining a vector of the target axis; a positioning system for determining the location of the user interface; an image handling system for receiving image data from a camera device pointing in a direction that extends along the target axis, wherein the image data allows for determination of placement of real -world target objects including the placement of users of other user interfaces and the placement of intervening objects in the real -world; a trigger device for activation to indicate when the user of the user interface intends to fire a simulated shot; and a user feedback device for providing feedback to the user if a path of a simulated shot fired via another user interface is determined to have passed within a certain distance of the user.

18. A user interface as claimed in claim 17, wherein the positioning system comprises a Global Positioning System, GPS, device with Real-time Kinematic Positioning, RTK.

19. A user interface as claimed in claim 17 or 18, wherein the positioning system uses inertial positioning data obtained via sensors of the orientation sensing system.

20. A targeting system as claimed in any of claims 2 to 15, the targeting system comprising multiple user interfaces as claimed in any of claims 17 to 19.

Images