System for displaying an interactive training scenario and determining the position of relevant objects in a training area

Abstract

A system for displaying an interactive training scenario and determining the position of relevant objects (8) in a training area (10) comprises a screen (1) consisting of panels (2) containing lighting units (3) emitting light in the visible light spectrum. The positioning patterns (6) are generated on an IR module (4) that comprises IR lighting elements (5) arranged in an array having grid distribution. The grid distribution of IR lighting units (5) substantially coincides with the array of pixels on the panel (2). The IR modules (4) are controlled by the IR control software module which, using an algorithm, switches individual IR lighting units (5) inside a respective IR module (4) on and off, so that at least one positioning pattern (6) is generated on one IR module (4), which is captured by the positioning camera (7). The IR lighting units (5) may be integrated in the panel (2) such that respective lighting units (3) of the panel (2) in a certain pixel (2a) are either replaced by IR lighting units (5) or IR lighting units (5) are added to a certain pixel (2a) next to a lighting unit (3). An IR module (4) can be positioned on the panel (2) in a way that the IR lighting units (5) are connected to an IR module (4) on a transparent circuit board (4b) that is arranged on the front side of the panel (2).

Description

[0001] System for displaying an interactive training scenario and determining the position of relevant objects in a training area

[0002] The present invention relates to a system for displaying an interactive training scenario, where the training scenario is displayed on a display and trainees participate in the training scenario and where, to assess training performance, the position of a trainee or other training-relevant object needs to be determined in real time. So, the system is suitable both for military simulators intended for instance for a military training with digital replica weapons and for civilian simulators intended for instance for sports activities such as biathlon, where the trainees aim the replica weapon at a virtual target displayed in the training scenario and shoot. The system must therefore determine the position and / or orientation of a trainee or other training-relevant object, such as a training replica weapon, in real time to assess whether a trainee has hit the target with the replica weapon, for instance.

[0003] Within the context of this application, the expression “determining the position” meaningfully relates to “determining the position and / or orientation and / or movement” of a relevant object.

[0004] The invention is based on known training systems in which the position and / or orientation of an object, such as a trainee or a replica weapon, within a training area is determined by analysing the images captured by a positioning camera mounted on an object. During training, a positioning camera captures images of positioning patterns emitting EM waves of a specific wavelength. The positioning patterns are statically positioned with respect to the main screen, on which a training scenario is displayed; during training, a positioning camera captures at least two, optionally more, positioning patterns, which allows determining the position and orientation of an object with respect to the main screen and with respect to the interactive training scenario displayed on the main screen. The data on the position and / or orientation of an object comprise some or all data points that describe a body in a 3D space, in systems having six degrees of freedom, for instance, three data points represent the position (X, Y, Z) and three data points describe orientation: yaw, pitch and roll; in systems having three degrees of freedom, the three data points describe orientation: yaw, pitch and roll. The calculation needed to determine the position and orientation of an object from the images of the positioning patterns captured by the positioning cameras are performed by a computer using an adequate software module.

[0005] For instance, a similar system is disclosed in WO 2018 / 088968. The positioning patterns are statically positioned with respect to the main screen, on which a training scenario is displayed, and consist of positional fields and additional positional fields. A positional field comprises at least 4 EM wave point sources which are arranged within the field in a non-coplanar way, all positioning fields sharing an identical architecture. Since all the positional fields are identical, a camera on a moving object identifies the position and orientation of an object. To identify the absolute position of an object the system is provided with an additional positional field configured as an additional EM wave point source within or in direct vicinity of each positional field. Said system does not allow for a flexible generation of different types of positioning patterns depending on the training / scenario or the possibility to change positioning patterns during the training.

[0006] In the system of the invention, the screen, on which a training scenario is displayed, consists of at least one panel that comprises lighting units, a respective lighting unit comprising one or several LEDs emitting light in the visible light spectrum, the positioning patterns being generated on an IR module comprising IR lighting units emitting light in the infrared (IR) spectrum, the IR module being substantially aligned with the screen, more precisely the screen panel, so that both the screen and the IR module emit light in the same direction, i.e. in the direction of the space in front of the screen. An array of IR lighting units in the IR module is positioned on the screen panel to substantially coincide with the face of the lighting units on the panel.

[0007] As will be explained in more detail in the following, the IR lighting units may be integrated in the screen, more precisely the screen panel, and respective lighting units of the screen in a certain pixel are either replaced by IR lighting units or IR lighting units are added to a respective pixel next to a lighting unit. The IR module may be positioned on the screen differently, namely the IR lighting units may be integrated in a transparent circuit board that is mounted on the front side of the screen, more precisely the screen panel.

[0008] The invention will be described in more detail hereinbelow and illustrated on the figures which show:

[0009] Figure 1 shows an embodiment of a system of the invention - schematic view

[0010] Figure 2 shows a detail from Figure 1 , a portion of the screen, more precisely the panel, with a portion of an IR module, where the IR lighting units are arranged in x and y axes equally and equidistantly and the IR lighting units are connected to the IR module on a transparent circuit board - schematic view

[0011] Figure 3A shows an example of an IR module on the panel, in which one positioning pattern is illuminated, and Figure 3B shows the same IR module, on which a different positioning pattern is illuminated - schematic view

[0012] Figure 4 shows an embodiment of arrangement of an IR module on the screen panel, in which the IR lighting units are connected to the IR module on the transparent circuit board which is mountable on the screen panel (either with an adhesive or magnets) - schematic view

[0013] Figure 5 shows a magnified schematic view of a few examples of the pixel of the screen panel of the system of the invention for an embodiment in which the IR lighting units are integrated in the screen panel, wherein figures A and B show a pixel, when a lighting unit is replaced by an IR lighting unit, figure C shows a pixel when an IR element is added to the pixel next to the lighting unit, wherein the lighting unit in figure A includes one LED, while the lighting unit in figures B and C includes three LEDs.

[0014] The system of the invention comprises: a screen 1 , on which a training scenario is displayed, the screen 1 consisting of at least one panel 2 comprising lighting units 3 to emit visible light, at least one positioning pattern 6 generated in the IR light spectrum on at least one IR module 4, at least one positioning camera 7 mounted on an object 8, the position of which needs to be determined in order to perform the training of the scenario, for instance on a weapon 8a or a trainee 8b, the positioning camera 7 being configured to detect light in the IR spectrum and thus to detect the positioning patterns 6, and a computer system 9 having processing and memory capabilities and provided with connecting means for connection to at least the screen 1 and the positioning camera(s) 7.

[0015] The computer system 9 is configured to: run the positioning software module used to determine a position, which, based on the data on the positioning patterns 6 received from the positioning camera 7, determines the position and movement of said camera, which represents the position and movement of the object 8, 8a, 8b and is available in an adequate format for further processing, run the IR control software module to generate positioning patterns 6, and run the software training module to start and run training scenarios, which includes a display of an interactive training scenario on the screen 1 , integrates the interactive training scenario on one side and the positions) and orientation(s) of the adequate object(s) 8, 8a, 8b on the other, optionally also additional inputs from additional input devices, preferably a trigger device, on the adequate object 8, resulting in an interaction of the trainee 8b with the interactive training scenario.

[0016] The lighting units 3 of a respective panel 2 comprised in the screen 1 are arranged on the panel 2 in a certain array of pixels 2a. Within the context of the present application, the term pixel 2a is used as a geometric surface to which a respective lighting unit 3 is assigned, the pixels 2a completely filling the panel 2 of the screen 1 . A respective lighting unit 3 comprises one or several LEDs 3a emitting light in the visible light spectrum.

[0017] The panels 2 making up the screen 1 may be provided as flat LED panels and may be of various shapes, for instance square, rectangular, hexagonal or triangular, and sizes. The panels 2 can also be curved with a radius of three meters for instance. The lighting units 3 are distributed on the panel 2 over the pixels 2a normally having a size between 1 and 3 mm. The selected size of pixels 2a depends on the desired image resolution and the desired accuracy of the image displayed based on the anticipated distance of a trainee 8b from the panels 2. The number of lighting units 3 that a respective panel 2 comprises depends on the size of the panel 2 and the size of a respective pixel 2a. If the panel 2 has a size of 320 x 160 mm and the pixel size is 1 mm, the panel has 51 ,200 lighting units, if the pixel size is 1 .86 mm, the panel has 14,796 lighting units, if the pixel size is 3 mm, the panel has 5,689 lighting units. A respective prior art lighting unit 3 that is assigned to one pixel 2a normally consists of several different LEDs 3a.

[0018] Depending on how the panels 2 are laid, the screen 1 as a whole can be configured as a flat surface (linear), a curved surface or a combination of flat surfaces (linear in parts) and / or curved surfaces. The panels 2 may be laid in a plane, which means that each subsequent panel 2 is in the same plane as the previous panel 2, or each subsequent panel 2 may be laid at a certain angle with respect to the adjacent panel 2. In this way, the screen 1 may be configured as a flat surface, as a surface in the form of a cylindric section, for instance 210 degrees, a hemisphere, a hexagon and the like.

[0019] The screen 1 preferably consists of flat LED panels 2. In one of the embodiments, the screen 1 consists of a grid of 12 x 12 flat LED panels 2 having a size of 320 mm x 160 mm. A training scenario may be displayed on the entire screen 1 or only on one portion of the screen 1 .

[0020] Individual training scenarios are implemented in the computer system 9 as software training modules of the scenario inside the coordinate system of the training area 10. The scenarios can be selected and can be interactive, such that the activity of a trainee 8b (for instance a position or a shoot) influences the course of the scenario.

[0021] The positioning patterns 6 are generated on the IR module 4. A respective IR module 4 comprises IR lighting units 5 which are arranged on the front side of the IR module 4 into an array 4a in a grid distribution, the array 4a of IR lighting units 5 substantially coinciding with the face of the lighting units 3 on the panel 2, and the lighting units 3 and the IR lighting units 5 being oriented in the same direction to radiate substantially in the same direction.

[0022] One IR lighting unit 5 consists of one or several IR diodes.

[0023] In one of the embodiments, the grid distribution of IR lighting units 5 substantially coincides with the grid of pixels or the distribution of pixels 2a on the panel 2 of the screen 1 , which means that each individual IR lighting unit 5 considerably coincides with the associated pixel 2a on the panel 2 of the screen 1. It needs to be stressed that said coincidence does not necessarily mean that a respective pixel 2a of the panel 2 is associated with a respective IR lighting unit 5.

[0024] A distance between respective IR lighting units 5 in the array 4a within a respective IR module 4 in the x and y axes can be different or equal. For instance, a mutual distance between the IR lighting units 5 in the array 4a in one dimension is 1 cm to 5 cm, which provides for good resolution of the positioning pattern 6 at a range of 2 m to 10 m, although the positioning pattern 6 comprises two adjacent IR lighting units 5, which in consequence enables the positioning camera 7 to well identify the positioning pattern 6 at said range, which represents typical dimensions of the training area 10 in front of the screen 1 . A distance between the adjacent IR units 5 in the array 4a within a respective IR module 4 in the x and y axes is preferably equal or equidistant.

[0025] The array 4a of IR lighting units 5 is preferably rectangular in shape, for example square-shaped, for example of dimensions 10 x 10 cm, or rectangle-shaped, for example of dimensions 5 cm x 20 cm, or the array 4a corresponds to a dimension of a respective panel 2 making up the screen 1 .

[0026] In one of the embodiments, the array 4a of IR lighting units 5 is square-shaped and comprises 10x10 IR lighting units 5 equally and equidistantly arranged in the x and y axes, wherein a certain number of IR lighting units 5 will be illuminated for a specific positioning pattern 6.

[0027] In some embodiments, the positioning of the IR module 4 on the panel 2 making up the screen 1 is performed by integrating the IR lighting units 5 into the panel 2 of the screen 1 in two ways.

[0028] In the first way, the IR lighting units 5 in certain pixels 2a replace a lighting unit 3, so a single pixel 2a of the panel 2 comprises merely an IR lighting unit 5 or merely a lighting unit 3. In this way, the density of the IR lighting units 5 compared to the density of the lighting units 3 is sufficiently lower to prevent inconvenient distortion of the visible part of the image on the screen 1 (scenario) due to a lack of lighting elements 3 in the pixels 2a, where an IR lighting unit 5 is present.

[0029] In a second way, an IR lighting unit 5 is added to a lighting unit 3 in a certain pixel 2a, so the pixel 2a having a lighting unit 3 also comprises an IR lighting unit 5. In this way, the visible part of the image is not distorted regardless of the density of the IR lighting units 5, because the density, more particularly the number of lighting units 3 is not changed regardless of the density or number of the IR lighting units 5.

[0030] In preferred embodiments of the two described ways of integration of IR lighting units 5 into the panel 2, the number of pixels 2a which comprise an IR lighting unit 5 is lower than the number of pixels 2a which only comprise a lighting unit 3.

[0031] When the IR lighting units 5 are integrated in the panel 2 of the screen 1 , the most practical way to proceed is to integrate the lighting units 3 and the IR lighting units 5 into the same circuit board, this is why adequate integration of IR lighting units 5 should be performed during the production of panels 2. IR lighting units 5 need to be integrated to adequate spots, i.e. pixels 2a, either next to a lighting unit 3 in the associated pixel 2a or without a lighting unit 3 in the associated pixel 2a. If the panel 2 with the IR lighting units 5 and lighting units 3 is already manufactured, but the density of the IR lighting units 5 is too high or the units are distributed over a larger portion of the panel 2 than would be sufficient for our needs, only one part of the IR lighting units 5 can be included in the control of the IR module 4, those which are necessary for the specific performance of the system.

[0032] In a further embodiment, the positioning of an IR module 4 on the panel 2 is carried out in a way that the IR lighting units 5 are connected 4 on the transparent circuit board 4b which is predominantly transparent for the visible light spectrum (transparent circuit board on a flexible, bendable surface, so- called flexible electronics or a stretchable surface, so-called stretchable electronics). The transparent circuit board 4b can be positioned subsequently, preferably glued or attached by a magnet to the front side of the panel 2 of the screen 1 . Also, in this embodiment are the IR lighting units 5, once positioned, oriented in a way to emit light substantially in the same direction as the lighting units 3, namely to the training area 10 in front of the screen 1 .

[0033] In a preferred embodiment of the IR modules 4 with the transparent circuit board 4b, the grid distribution of IR lighting units 5, once positioned on the panel 2, substantially coincides with the array of pixels or the distribution of pixels 2a of the lighting elements 3 of the panel 2 of the screen 1.

[0034] In another embodiment of the IR modules 4 with the transparent circuit board 4b, the grid distribution of IR lighting units 5, once positioned on the panel 2, does not coincide with the array of pixels or the distribution of pixels 2a of the lighting elements 3 of the panel 2 and the IR lighting units 5 are offset with respect to the array of pixels 2a. In this case, the system needs to be pre-calibrated prior to system operation to determine the position of IR lighting units 5 with respect to the associated panel 2 of the screen 1 or the screen 1.

[0035] In preferred variants of the two embodiments of positioning IR modules 5 on the panel 2 of the screen

[0036] 1 described above, the number of IR lighting units 5 is lower than the number of lighting units 3.

[0037] The visible light emitted by the lighting units 3 on the panel 2 travels through the transparent or substantially visible light transparent circuit board 4b of the IR lighting units 5, this is why the display of the scenario on the screen 1 will not be distorted or only minimally distorted. Due to their preferred sparse arrangement, the IR lighting units 5 on the transparent circuit board 4b will also not cause distortion of the display of the scenario on the screen 1 . When installing the transparent circuit board 4b with IR lighting units 5 on the front side of the panel 2 of the screen 1 , care should be taken to install it on a pre-defined spot with respect to the location of the lighting units 3 on the panel 2, for instance the edges of the transparent circuit board 4b and the edges of the panel 2 of the screen 1 should be aligned, otherwise the inaccuracy (for instance the angle between the edges) would have a negative impact on determination of position of the object 8 or the positioning camera 7 with respect to the screen 1 or an extra calibration process would be needed prior to training.

[0038] The advantage of providing the IR module 4 using a transparent circuit board 4b is that prior art panels

[0039] 2 may be used to make up the screen 1 , which only comprise lighting units 3 and are commercially available in many different forms and are also cheaper than the production of panels 2 with integrated IR lighting units 5 since the IR module 4 is installed onto the panels 2 subsequently.

[0040] Several positioning patterns 6 may be generated on one IR module 4, preferably one positioning pattern 6 is generated on one IR module 4.

[0041] IR lighting units 5 are controlled by the IR control software module which, using an algorithm, switches individual IR lighting units 5 inside a respective IR module 4 on and off, so that positioning patterns 6 are generated on the screen 1 that may be captured by the positioning camera 7. The IR control software module is operably linked to the positioning software module, since the latter must receive information on the generated positioning patterns 6 from the IR control module, so it can determine the position of the positioning camera 7 in combination with the absolute position of the generated positioning patterns 6 with respect to the screen 1 .

[0042] The positioning software module operates according to known algorithms used to determine the position and / or orientation and / or motion of the positional camera 7 with respect to the positioning pattern 6 as captured by the positioning camera 7, such as of visual simultaneous localization and mapping (SLAM) algorithms, SLAM marker algorithms, extended or nonlinear Kalman filter algorithms.

[0043] In one embodiment, when the screen 1 is configured as a flat surface for instance, generating one positioning pattern 6 on one IR module 4 is enough to determine the position of an object 8, the positioning pattern 6 being generated by at least four IR lighting units 5 which are sufficiently spaced from each other to be detected and distinguished by the positioning camera 7.

[0044] In another embodiment, when the screen is configured as a curved surface for instance, several positioning patterns 6 on several IR modules 4 need to be generated to determine the position of an object 8, their number depending on the curvature shape.

[0045] To accurately determine the position of an object 8, it is desirable that the positioning camera 7 in each moment detects at least three positioning patterns 6 generated on at least two IR modules 4, a respective positioning pattern 6 being generated by at least six IR lighting units 5.

[0046] In some embodiments, the system may include further position determining devices (not shown in figures), such as gyroscopes and / or accelerometers, arranged on adequate objects 8 and positioning cameras 7. The output data of these devices are used by the positioning software module to calculate the position and / or orientation of the positioning cameras 7. Using known methods, such as sensor data fusion, the position and / or orientation of the positioning cameras 7 can be calculated more accurately or with a frequency higher than the image capture frequency of the positioning cameras 7. The image capture frequency of the positioning cameras 7 must namely be high enough to allow for a sufficient frequency of the calculated positions and / or orientations of the relevant objects 8 necessary for a smooth interaction of the trainees 8b with the interactive training scenario. The image capture frequency is normally 30 images per second (30 fps or Hz) and is equal to or higher than the positioning data frequency, for instance15 Hz. In these embodiments, the image capture frequency of the positioning cameras 7 is not necessarily equal to or higher than the positioning and / or orientation data frequency of the positioning cameras 7 using the positioning software module.

[0047] In one embodiment, a positioning camera 7 is part of the SAS module mounted on an object 8, the position and movement of which needs to be determined, for instance on a digital replica weapon 8a for training. Normally, in addition to a positioning camera 7 (image refreshing at least every 30 ms) a SAS module also has an accelerometer (data refreshing every 1 -10 ms), a gyroscope (data refreshing every 1-10 ms) and a part of the computer system (computer subsystem), on which part of the positioning software module is run, which is relevant for determining the position of this SAS module. Using the data obtained from these three sources, the position of an object 8 in space can be determined more accurately and with less delay by using known methods, such as sensor data fusion.

[0048] If additional devices are used for determining position in combination with the positioning camera 7, for instance an accelerometer, it is enough that the positioning pattern 6 is generated by two or three IR lighting units 5 since an additional component used to calculate the position of the object 8 is acceleration of gravity.

[0049] The IR modules 4 may be distributed with respect to the screen 1 in various ways. They may be positioned at the edges of the screen 1 or in the interior of the screen 1 or across the entire screen 1 . The aim of the distribution is that the surface of the screen 1 is the most reasonably covered by IR modules 4 so that adequate positioning patterns 6 can be displayed depending on training scenarios and training modes.

[0050] The distribution of IR modules 4 depends on the size of the training area 10, the distance of the training area 10 from the screen 1 on which a training scenario is displayed, the field of view of the positioning camera 7, the contents of the training scenario, and the required accuracy, with which the position of an object 8, such as a trainee 8b or a replica weapon 8a, is to be determined.

[0051] To determine the position of an object 8 as accurately as possible, it is desirable for the positioning camera 7 to capture at least three positioning patterns 6 when the screen 1 is configured as a flat surface. When the screen 1 is not configured as a flat surface, it is necessary for the positioning camera

[0052] 7 to capture more positioning patterns 6 to determine the position of an object 8 as accurately as possible. For instance, when SLAM algorithms are used, the position of an object 8 is determined more accurately if the positioning camera 7 simultaneously captures several positioning patterns 6 and also the convergence of the algorithm is faster, which means that the position and / or orientation of an object

[0053] 8 is determined more rapidly but with increased computational complexity.

[0054] A distribution of IR modules 4 at the bottom and / or top edge of the screen 1 allows for the positioning camera 7 to capture an adequate number of positioning patterns 6 regardless of the position, orientation and rotation of an object 8.

[0055] A distribution of IR modules 4 inside the screen 1 over the central portion allows for using positioning cameras 7 having a smaller capture angle, for instance 70 degrees.

[0056] The positioning camera 7 namely has a limited image capture angle which is preferably in a range of 70 to 120 degrees. Lenses with a larger image capture angle could be used, however, this would increase the distortion of image capture which negatively affects correct detection of the positioning patterns 6 and also the accuracy of determining the position of an object 8, or the lens on the positioning camera 7 is bigger in this case and might hinder mounting into the SAS module.

[0057] As the proposed system is suitable for tactical warfare training, and the movement of trainees 8b during the execution of the scenario is anticipated, the system of the invention allows to dynamically change the individual positioning patterns 6 and / or the density of the positioning patterns 6, thus allowing to capture positions at places where there are more trainees 8b. For example, if three trainees 8b are moving more to the left in front of the screen 1 , then the density of positioning patterns 6 there can be increased. The density of positioning patterns 6 can also be increased on the other side of the screen 1 if trainees 8b are aiming from the left side to the right side of the screen 1 , if the scenario so requires. This results in a shooting accuracy that is independent on the position of a trainee 8b, the orientation of weapon 8a relative to the screen 1 and even when trainees 8b are in a group.

[0058] The IR modules 4 may be controlled in various ways.

[0059] In the first - static - control mode, the positioning patterns 6 change or adapt according to the type of scenario or training, but remain static during the training, i.e. during the execution of one scenario. This is a universal solution and the individual positioning patterns 6 are generated to meet the required accuracy (e.g. 2 mm) and dispersion (e.g. 6 mm radius) requirements regardless of the position of the trainee and the contents of the scenario. An appropriate selection of positioning patterns 6 is made upon the system start-up, and is automatically selected when any scenario is selected.

[0060] In the second - dynamic control mode, the positioning patterns 6 change or adapt during the training, i.e. during the execution of one scenario, according to the positioning accuracy needs of an object 8 (for example a weapon 8a), which further depends on the type of weapon 8a, the position, orientation and rotation of an object 8 (for example a weapon 8a), and the contents of the displayed scenario at a given moment in terms of where the relevant targets are located on the screen 1 . For example, a different type of weapon 8a that is prescribed to a trainee 8b by the contents of the scenario has such an effect that one type of weapon 8a requires greater accuracy, for example with a sniper rifle, than a different type of weapon 8a, for example with a hand grenade launcher. When higher accuracy is required, more positioning patterns 6 will be generated within the field of view of the positioning camera 7. The position, orientation and rotation of an object 8 (for example a weapon 8a) have such an effect that more positioning patterns 6 need to be generated within the field of view of the positioning camera 7 than outside the field of view. The scenario shown affects the generation of positioning patterns 6 in such a way that more positioning patterns 6 are generated in the portion of the screen 1 where relevant targets (for example, enemies) are displayed than in the portion of the screen 1 with no relevant targets.

[0061] For example, for a sniper, a maximum resolution and accuracy are required. Based on the detected position and orientation of the positioning camera 7 on an object 8, the system allows the IR modules 4 to be controlled in such a way that an appropriate positioning pattern 6 and / or an appropriate density of positioning patterns 6 will be generated on the screen 1 in the area of the relevant target in that portion of the screen 1 , where the sniper will be focused, for example in the sniper's field of view, in order to enable the positioning software module to more accurately detect the position of the weapon 8a from the image provided by the positioning camera 7.

[0062] A combination of static and dynamic controls is also possible, where part of the positioning patterns 6 is static during the execution of the scenario, while part of the positioning patterns 6 changes during the execution of the scenario.

[0063] In one embodiment, to reduce the time delay in generating the dynamically changing positioning patterns 6, the system of the invention comprises the use of eye-tracking systems of the trainee 8a, such as eyetracking goggles for monitoring the orientation of the sight of the eye. When a trainee 8a is wearing such goggles and the positioning camera 7 (SAS module) is mounted on an object 8, for example on a replica weapon 8a or helmet of a trainee 8b, the trainee 8b instinctively moves his eyes first before physically moving the replica weapon 8a or helmet in the desired direction. The eye movement is detected by the eye-tracking system of the trainee 8a, which transmits the movement information, i.e. where the line between the eye and the screen 1 intersects the screen 1 at any given moment, to the computer system 9 or the IR control software module, which determines the new position of the positioning pattern 6 and / or the shape of the positioning pattern 6 on the basis of the received information. Thus, during the actual physical movement of the object 8 to a new position, the new positioning pattern 6 and / or the correspondingly adjusted density of positioning patterns 6 is already generated in the direction, or in the portion of the screen 1 , which will provide the best detection, i.e. the least noise and the least measurement scatter.

[0064] The computer system 9 is implemented in known ways and is connected in known ways to the positioning cameras 7, the screen 1 and the IR modules 4.

[0065] The computer system 9 may be configured centrally, for instance as a laptop or a stationary computer having one or several processors. On the computer system runs the positioning software module for all positioning cameras 7. The IR control software module and the software training module are also run on such a central computer system.

[0066] The computer system 9 may also be implemented in a distributed way consisting of several computer subsystems that perform the described software and connection functions. For example, a respective positioning camera 7 may be connected to an associated computer subsystem (for example a graphics card) running a positioning software module that determines the position and / or orientation and / or movement of a respective positioning camera 7 or an associated object 8. The computer subsystems are suitably connected to the computer system 9 in known ways to perform all of the functions described.

[0067] The connections between the computer system 9 on one hand and the positioning cameras 7, the screen 1 and the IR modules 4 on the other are implemented in known ways, for instance via wire or wireless connections, the wireless connections being preferred. When making connections, digital transmission protocols are used, for instance, and video transmission interfaces, e.g. HDMI, DP. Ethernet, Thunderbolt, USB3 or wireless WiFi or 5G, 6G can also be used.

[0068] Optionally, an IR module 4 may also be adapted to generate calibration patterns, for example in the case where the screen 1 is not perfectly flat and the calibration process is required to determine the curvature of the screen 1 and its absolute orientation and position in space, which is necessary for the accurate operation of the positioning software module, namely to determine the position and / or orientation and / or movement of an object 8 in the training area 10. In this case, the system of the invention may further comprise a calibration camera which must capture the entire screen 1 , i.e. all the calibration patterns displayed on the screen 1 . The IR software module is in this case adapted to further control the IR modules 4 to display the calibration patterns during the calibration process, either before training or after a new setup. Known Aruco calibration patterns may be used for calibration, which are generated on the IR modules 4, including known associated calibration software modules. Since the matrix or array 4a of the IR lighting units 5 is larger than the matrix of the Aruco calibration pattern, the Aruco calibration patterns can be generated on the IR modules 4. Since the Aruco patterns may be less complex than the positioning patterns 6, less processing power is typically used for the generation of calibration patterns and for the calibration process than is used by the positioning software module.

[0069] Also, different calibration patterns may be generated on the IR modules 4, which may be simpler than the Aruco calibration patterns.

[0070] Since the array 4a of the IR lighting units 5 is substantially coincident with the plane of the lighting units 3 on the panel 2, the advantage of the system of the invention is that there is no need for additional space adjacent to the screen 1 to install the devices for generating / projecting the positioning patterns 6 at the edge of the screen 1 , thus allowing the system to be installed in smaller spaces, for example in standard size shipping containers. Since the array 4a of the IR lighting units 5 is essentially coincident with the surface of the screen 1 or panel 2 and with the pixels 2a of the screen, due to its design, it is an additional advantage for the position calculation algorithms and there is no need for calibration and less computationally demanding calibration is required.

[0071] The system of the invention allows to generate different types of positioning patterns 6 depending on a specific training / scenario, or even to modify the positioning patterns 6 within a training scenario, thus significantly improving the flexibility and performance of the system. For example, for a sniper, positioning patterns 6 are needed in the smaller portion of the screen 1 where the sniper will be focused (in the sniper's field of view), as the sniper is usually more static. In other words, on the screen 1 , the number of positioning patterns 6 may be increased in the sniper’s field of view, while there may be a smaller number of positioning patterns 6 in the remaining portions of the screen 1. The system allows the individual IR modules 4 to be controlled in a way to generate an appropriate positioning pattern 6 such that its maximum resolution, i.e. the best detectability of a single positioning pattern 6 and the detectability of multiple positioning patterns 6 by the positioning camera 7, is within the narrow field of view of the sniper, and the other IR modules 4 are controlled in a way to generate the positioning patterns 6 for tactical training.

[0072] As already mentioned, the system of the invention makes it possible to change the positioning patterns during training. The positioning patterns for a certain scenario can be set prior to training depending on the scenario type, the weapon type to be used for training, and on the number of trainees. When a scenario is run, the IR control software module generates the positioning patterns 6 according to the algorithm over a wired or wireless connection, which are displayed on the IR modules 4. The positioning patterns 6 may also be generated / changed during the execution of a training scenario. For example, given the detected movement and orientation of a given positioning camera 7 in space, positioning patterns 6 are generated so as to provide, for given positions of the positioning cameras 7, the highest resolution, i.e. the highest accuracy and the lowest scatter of the measurements of a given position of the object 8 based on the images from the positioning camera 7, and to minimise occlusions by other trainees in the training area 10 in front of the screen 1 .

Claims

Claims1 . Asystem for displaying an interactive training scenario and determining the position of relevant objects (8) in a training area (10), the system comprising a screen (1), on which a training scenario is displayed, the screen (1) consisting of at least one panel (2) comprising lighting units (2) arranged on the panel (2) in a certain array of pixels (2a), a respective lighting unit (3) associated with one pixel (2a) and comprising one or several LEDs (3a) emitting light in the visible light spectrum; at least one positioning pattern (6) generated in the IR light spectrum; at least one positioning camera (7) mounted on an object (8), the position of which needs to be determined in order to perform the training of the scenario, the positioning camera (7) being configured to detect light in the IR spectrum and thus to detect the positioning patterns (6), and a computer system (9) having processing and memory capabilities and provided with connecting means for connection to at least the screen (1) and at least one positioning camera (7), the computer system (9) being adapted at least to run the positioning software module used to determine a position, which, based on the data on the positioning patterns (6) received from the positioning camera (7), determines the position and movement of said camera, which represents the position and movement of the object (8) and is available in an adequate format forfurther processing, and to run the software training module to start and run training scenarios, which includes a display of an interactive training scenario on the screen (1), integrates the interactive training scenario on one side and the position of at least one object (8) on the other, characterized in that the system further comprises at least one IR module (4) for generating at least one positioning pattern (6), a respective IR module (4) comprising IR lighting units (5) which are arranged on the front side of the IR module (4) into an array (4a) in a grid distribution, the array (4a) of IR lighting units (5) substantially coinciding with the face of the lighting units (3) on the panel (2), and the lighting units (3) and the IR lighting units (5) being oriented in the same direction to radiate substantially in the same direction, the IR modules (4) being controlled by the IR control software module which, using an algorithm, switches individual IR lighting units (5) inside a respective IR module (4) on and off, so that at least one positioning pattern (6) is generated on one IR module (4), which is captured by the positioning camera (7), the IR control software module being run on the computer system (9) which is further adapted to run the IR control software module, and the IR control software module being operably linked to the positioning software module to transmit the information on the generated positioning patterns (6), such that the positioning software moduledetermines the position of the positioning camera (7) in combination with the absolute position of the generated positioning patterns (6) with respect to the screen (1).

2. The system according to claim 1 , characterized in that the grid distribution of IR lighting units (5) substantially coincides with the grid of pixels (2a) on the panel (2) of the screen (1), which means that each individual IR lighting unit (5) considerably coincides with the associated pixel (2a) on the panel (2) of the screen (1), and wherein one IR lighting unit (5) comprises one or several IR diodes.

3. The system according to claims 1 and 2, characterized in that the grid distribution of IR lighting units (5) is configured in a way that a distance between respective IR lighting units (5) in the array (4a) within a respective IR module (4) in the x and y axes is different or equal, preferably equal.

4. The system according to claims 1 to 3, characterized in that the array (4a) is square-shaped and comprises 10x10 IR lighting units (5) equally and equidistantly arranged in the x and y axes, wherein a certain number of IR lighting units (5) is illuminated for a specific positioning pattern (6).

5. The system according to claims 1 to 4, characterized in that the positioning of the IR module (4) on the panel (2) making up the screen (1) is performed by integrating the IR lighting units (5) into the panel (2) of the screen (1), preferably by integrating into the same circuit board, wherein the IR lighting units (5) in certain pixels (2a) replace a lighting unit (3), so a single pixel (2a) of the panel (2) comprises merely an IR lighting unit (5) or merely a lighting unit (3), or an IR lighting unit (5) is added to a lighting unit (3) in a certain pixel (2a), so the pixel (2a) having a lighting unit (3) also comprises an IR lighting unit (5).

6. The system according to claim 5, characterized in that the number of pixels (2a) which comprise an IR lighting unit (5) is lower than the number of pixels (2a) which comprise merely a lighting unit (3).

7. The system according to claims 1 to 4, characterized in that the positioning of the IR module (4) on the panel (2) making up the screen (1) is performed by connecting the IR lighting units (5) on the transparent circuit board (4b) which is mostly transparent to the visible light spectrum, the transparent circuit board (4b) being arranged on the front side of the panel (2) of the screen (1) subsequently, the grid distribution of IR lighting units (5) substantially coinciding with the grid of pixels (2a) on the panel (2) of the screen (1), which means that each individual IR lighting unit (5) considerably coincides with the associated pixel (2a) on the panel (2) of the screen (1).

8. The system according to claims 1 , 3 and 4, characterized in that the positioning of the IR module (4) on the panel (2) making up the screen (1) is performed by connecting the IR lighting units (5) on the transparent circuit board (4b) which is mostly transparent to the visible light spectrum, the transparent circuit board (4b) being arranged on the front side of the panel (2) of the screen (1) subsequently, the grid distribution of IR lighting units (5) not coinciding with the array of pixels (2a) on the panel (2) of the screen and the IR lighting units (5) being offset with respect to the array of pixels (2a).

9. The system according to claims 7 or 8, characterized in that the number of IR lighting units (5) is lower than the number lighting units (3).

10. The system according to claims 1 to 9, characterized in that the positioning pattern (6) is generated by at least four IR lighting units (5) which are sufficiently spaced from each other to be detected by the positioning camera (7).1 1. The system according to claims 1 to 10, characterized in that three positioning patterns (6) are generated on the screen, which are generated on at least two IR modules (4), a respective positioning pattern (6) being generated by at least six IR lighting units (5).

12. The system according to claims 1 to 11 , characterized in that the IR modules (4) are positioned at the edges of the screen (1) or in the interior of the screen (1), the distribution of IR modules (4) with respect to the screen (1) depending on the size of the training area (10), the distance of the training area (10) from the screen (1) on which a training scenario is displayed, the field of view of the positioning camera (7), the contents of the training scenario, and the required accuracy, with which the position of an object (8) is to be determined.

13. The system according to claims 1 to 12, characterized in that the IR control software module is adapted to control the IR modules (4) in a way that the positioning patterns (6) are adapted depending on the contents of the scenario, but remain static during the execution of one scenario.

14. The system according to claims 1 to 13, characterized in that the IR control software module is adapted to control the IR modules (4) in a way that the positioning patterns (6) are adapted during the execution of a scenario with respect to the needs for accuracy of position determination of an object (8) depending on the type of weapon (8a) assigned to an object (8) by the scenario, the position of the object (8) and the contents of the scenario displayed in a certain point of time, in the sense of where the relevant targets are positioned on the screen (1).

15. The system according to claims 1 to 14, characterized in that the IR module (4) may be adapted to generate calibration patterns, too, in this case the system further comprising a calibration camera which captures all the calibration patterns displayed on the screen (1), and the IR software module is adapted to further control the IR modules (4) to display the calibration patterns during the calibration process, either before training or after a new setup.

16. The system according to claims 1 to 15, characterized in that the system further includes an eyetracking system of the trainee (8a), which transmits the eye movement information to the computer system (9) which determines a new position of the positioning pattern (6) and / or the shape of the positioning pattern (6) on the basis of the received information, such that during the actual physical movement of the object (8) to a new position, the new positioning pattern (6) and / or the correspondingly adjusted density of positioning patterns (6) is already generated in the direction, or in the portion of thescreen (1), which will provide the best detection, which provides for a reduced time delay in determining the positions of the dynamically changing positioning patterns (6).

Images