Simulation binoculars and simulation system and method

Description

TECHNICAL FIELD

[0001] The present invention relates to the simulation on real terrain of virtual military situations using mixed reality sighting glasses. STATE OF PRIOR ART

[0002] Call for Fire procedures are military operations that allow for the precise use of firepower, in a timely manner and in synchronization with ground teams, to support troops close to the enemy and avoid friendly fire. During these operations, officers of the armed forces, deployed in the theater of operations, are responsible for coordinating any type of strike, for example air strikes, by providing, among other things, information on a target to be hit, its position and its immediate environment.We then speak of forward air controllers (FAC) for an old NATO (North Atlantic Treaty Organization) terminology exclusively designating air fire support, artillery observers (JFO) for the coordination of artillery fire, or JTAC (Joint Terminal Attack Controller) controllers according to the American terminology and the new NATO terminology, capable of coordinating any type of air, land or naval fire support. In the remainder of the document, we will speak of fire support controller (or "Fire Support Controller" in English) to designate indifferently a forward air controller (FAC), a JFO artillery observer or a JTAC controller.

[0003] To enable fire support controllers to be trained without risk, systems for reconstructing operational battlefield environments using a virtual environment have been developed. One example is the system described in patent document FR 3 007 161 A1.

[0004] These are simulation systems that are installed indoors in dedicated infrastructures. These simulation systems require substantial computer equipment to generate and display virtual scenes. Their display also typically requires large screens. These simulation systems therefore do not allow fire support controllers to train in real conditions of light, weather, and terrain configuration. Other augmented reality systems are described in US 2013 / 050432 A1 and ES 2 300 204 B1.

[0005] It is desirable to completely or partially overcome the drawback of the prior art mentioned above. There is therefore a need for a solution aimed at providing a simulation environment that allows fire support controllers to train in more realistic conditions. There is also a need for a solution whose portability makes it possible to deploy easily and quickly to take into account the configuration of a new training ground. There is also a need to provide a solution allowing realistic training to be carried out in interaction and collaboration mode between different actors, such as for example for fire support training scenarios. There is also a need for a solution that allows this realism to be maintained under zoom conditions. STATEMENT OF THE INVENTION

[0006] For this purpose, simulation sighting binoculars are provided comprising a camera, a screen, a pair of lenses arranged to face the screen, and electronic circuitry configured to: obtain a video frame from the camera, the video frame being intended to be displayed by the simulation sighting binoculars in an E3 projection window; obtain geolocation information from the simulation sighting binoculars and orientation information from the camera; transmit the geolocation and orientation information to a simulation platform; receive simulation graphics elements and positioning information in the space of three-dimensional virtual objects corresponding to the simulation graphics elements;performing a two-dimensional rendering of the virtual objects according to the positioning information, in order to display the virtual objects in an E2 projection window, and when a magnification factor is to be applied, keeping the dimensions of the E2 projection window fixed and adjusting the field of view of the two-dimensional rendering according to the magnification factor;superimposing, in a virtual scene local to the simulation binoculars, the two-dimensional rendering of the virtual objects in the projection window E2 and the video frames in the projection window E3, the projection window E3 being placed in a plane further from the simulation binoculars than the one where the projection window E2 is placed in the field of view of the simulation binoculars, the simulation binoculars having a line of sight which passes through the center of the projection windows E2 and E3, and when a magnification factor is to be applied, adjusting the dimensions of the projection window E3 according to the magnification factor; obtaining a mixed reality stereoscopic image from the superposition of the planes using a pair of virtual cameras reproducing binocular vision adapted to the pair of lenses;and perform, on the screen, a right eye and left eye display of the resulting mixed reality stereoscopic image. Thus, thanks to the real scene captured by the camera, the fire support controllers are immersed in a more realistic simulation environment. Thanks to the advantageous processing of the fields of vision and the projection windows in the event of a magnification factor to be applied, the immersion realism is preserved in the event of zooming. And the remote processing of the graphic elements (relating to the virtual objects linked to the training scenario) in the simulation platform makes it possible to limit the internal processing resource requirements of the simulation sighting binoculars, which facilitates the integration of the simulation sighting binoculars and their adaptability to new training grounds.;

[0007] According to a particular embodiment, the electronic circuitry is configured to further superimpose, in the local virtual scene on the simulation sighting binoculars, a reticle in a projection window E1 placed in a closer plane of the simulation sighting binoculars than that where the projection window E2 is placed in the field of vision of the simulation sighting binoculars, the sighting axis also passing through the center of the projection window E1, and when a magnification factor is to be applied, adjust the dimensions of the projection window E1 according to the magnification factor. Thus, the realism of the simulation sighting binoculars is enhanced.

[0008] According to a particular embodiment, the electronic circuitry is configured to further superimpose, in the virtual scene local to the simulation sighting binoculars, additional information in a projection window E0 placed in a closer plane of the simulation sighting binoculars than that where the projection window E1 is placed in the field of view of the simulation sighting binoculars, the projection window E0 being insensitive to the magnification factor. Thus, the simulation sighting binoculars can display information consistent with real sighting binocular electronic displays, with a realistic stereoscopic effect, independently of any magnification factor that may be applied.

[0009] According to a particular embodiment, the screen, the camera and the electronic circuitry are provided by a smart mobile phone assembled to a housing comprising the pair of lenses, which are arranged to face the screen of the smart mobile phone. Thus, the deployment of the equipment necessary for the simulation is easy.

[0010] According to a particular embodiment, the simulation sighting binoculars further comprise an inertial measurement unit type sensor, and the electronic circuitry implements a SLAM (Simultaneous Localization and Mapping) algorithm, and the camera orientation information is provided by the SLAM (Simultaneous Localization and Mapping) algorithm, except when the SLAM (Simultaneous Localization and Mapping) algorithm stalls, in which case the camera orientation information is provided by the inertial measurement unit type sensor. Thus, the sighting binoculars use the orientation information from the SLAM algorithm, considered to be more accurate, as soon as possible, and use the orientation information from the inertial measurement unit type sensor, in degraded mode.

[0011] According to a particular embodiment, the geolocation information of the simulation sighting binoculars is provided by the SLAM Simultaneous Localization and Mapping algorithm, except when the SLAM Simultaneous Localization and Mapping algorithm stalls, in which case the geolocation information of the simulation sighting binoculars is provided by a GPS receiver ("Global Positioning System" in English) integrated into the simulation sighting binoculars. Thus, the sighting binoculars use the geolocation information of the SLAM algorithm, considered to be more precise, as soon as possible, and use the geolocation information of the GPS receiver, in degraded mode.

[0012] According to a particular embodiment, the SLAM Simultaneous Localization and Mapping algorithm provides a confidence index associated with the camera orientation information, for each video frame captured by the camera, and the electronic circuitry is configured to determine whether the SLAM Simultaneous Localization and Mapping algorithm has stalled according to the value of the confidence index. Thus, it is easy to know whether the information provided by the SLAM algorithm is reliable or not.

[0013] According to a particular embodiment, when the SLAM Simultaneous Localization and Mapping algorithm has stalled, the electronic circuitry corrects the updated orientation information from the IMU inertial measurement unit type sensor as follows: Q abs t = Q abs t0 * Q imu − 1 t0 * Q imu t Or : Q abs (t) represents the corrected orientation information; Q abs (t0) represents the absolute orientation information, i.e., that from the SLAM algorithm, before stalling; Q imu (t) represents the updated orientation information from the IMU inertial measurement unit type sensor; and Q imu (t0) represents the orientation information from the IMU inertial measurement unit type sensor before stalling from the SLAM algorithm. Thus, the aforementioned degraded mode is easily implemented.

[0014] According to a particular embodiment, the electronic circuitry corrects the orientation information provided by the SLAM Simultaneous Localization and Mapping algorithm according to deviations determined during calibration operations. Thus, it is possible to receive information from the SLAM algorithm in a relative manner.

[0015] According to a particular embodiment, the electronic circuitry is configured to, during calibration operations: receive simulation graphic elements corresponding to a virtual calibration object and positioning information in the space of the virtual calibration object; perform, on the screen, a right-eye and left-eye stereoscopic display of a mixed reality stereoscopic image including the virtual calibration object; obtain first orientation information of the camera when a first press on a calibration button of the simulation sighting binoculars is made; obtain second orientation information of the camera when a second press on the calibration button of the simulation sighting binoculars is made; and compare the first orientation information and the second orientation information and deduce therefrom corresponding orientation and altitude information deviations.Thus, calibration operations are easily carried out.

[0016] Also provided is a simulation system comprising a simulation platform, as well as simulation sighting binoculars according to any one of the embodiments presented above, the simulation platform comprising electronic circuitry configured to place, according to a training scenario, virtual objects in a digital terrain model modeling a training ground where the simulation sighting binoculars are intended to be used, and to transmit or cause to be transmitted to the simulation sighting binoculars the simulation graphic elements according to the geolocation information of the simulation sighting binoculars and the camera orientation information which are transmitted by the simulation sighting binoculars.Thus, the remote processing of graphic elements (relating to virtual objects and the digital terrain model linked to the training scenario) in the simulation platform makes it possible to limit the internal processing resource requirements of the simulation sighting binoculars, which facilitates the integration of the simulation sighting binoculars and their adaptability to new training grounds.

[0017] According to a particular embodiment, the simulation platform transmits or causes to be transmitted to the simulation sighting binoculars the simulation graphic elements asynchronously with respect to the geolocation information of the simulation sighting binoculars and to the camera orientation information, the simulation sighting binoculars perform positioning interpolations of the virtual objects and a first processing unit of the simulation system which is remote with respect to the simulation sighting binoculars performs, on behalf of the simulation sighting binoculars, a positioning recalibration using the digital terrain model. Thus, the remote processing linked to the positioning recalibration using the digital terrain model, potentially costly in terms of processing resources, makes it possible to limit the processing resource requirements internal to the simulation sighting binoculars.

[0018] According to a particular embodiment, when the simulation sighting binoculars activate a range reading, a second processing unit of the simulation system which is remote from the simulation sighting binoculars performs a distance calculation with respect to a target targeted by the simulation sighting binoculars using a ray tracing method in a virtual scene including said virtual objects and the digital terrain model. Note that the second processing unit can also be the first processing unit. Thus, the remote processing linked to the range reading, potentially costly in terms of processing resources, makes it possible to limit the processing resource requirements internal to the simulation sighting binoculars.

[0019] There is also provided a simulation method, implemented by an electronic circuitry of simulation sighting binoculars further comprising a camera, a screen and a pair of lenses arranged to face the screen, the method comprising the following steps: obtaining a video frame from the camera, the video frame being intended to be displayed by the simulation sighting binoculars in a projection window E3; obtaining geolocation information of the simulation sighting binoculars and orientation information of the camera; transmitting the geolocation and orientation information to a simulation platform; receiving simulation graphic elements and positioning information in the space of three-dimensional virtual objects corresponding to the simulation graphic elements; performing a two-dimensional rendering of the virtual objects according to the positioning information, in order to display the virtual objects in a projection window E2, and when a magnification factor is to be applied, keeping the dimensions of the projection window E2 fixed and adjusting the field of view of the two-dimensional rendering according to the magnification factor;superimposing, in a virtual scene local to the simulation binoculars, the two-dimensional rendering of the virtual objects in the projection window E2 and the video frames in the projection window E3, the projection window E3 being placed in a plane further from the simulation binoculars than the one where the projection window E2 is placed in the field of view of the simulation binoculars, the simulation binoculars having a line of sight which passes through the center of the projection windows E2 and E3, and when a magnification factor is to be applied, adjusting the dimensions of the projection window E3 according to the magnification factor; obtaining a mixed reality stereoscopic image from the superposition of the planes using a pair of virtual cameras reproducing binocular vision adapted to the pair of lenses;and performing, on the screen, a right eye and left eye display of the obtained mixed reality stereoscopic image.;

[0020] Also provided is a computer program product, comprising instructions causing a processor to execute the above-mentioned method, in any of its embodiments, when said instructions are executed by the processor. Also provided is an information storage medium on which such a computer program product is stored.

[0021] A simulation method is also proposed, implemented by a simulation system including simulation sighting binoculars implementing the simulation method defined above in any of its embodiments and a simulation platform comprising electronic circuitry implementing the following steps: placing, according to a training scenario, virtual objects in a three-dimensional digital terrain model modeling a training ground where the simulation sighting binoculars are used, and transmitting or having transmitted to the simulation sighting binoculars graphic elements corresponding to the virtual objects according to the geolocation information of the simulation sighting binoculars and the orientation information of their camera which are transmitted by the simulation sighting binoculars. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The above-mentioned features of the invention, as well as others, will appear more clearly on reading the following description of at least one exemplary embodiment, said description being made in relation to the attached drawings, among which: [ Fig. 1 ] schematically illustrates a field training simulation system intended for fire support controllers; [ Fig. 2A ] schematically illustrates a binocular sight housing of the simulation system of the Fig. 1 ; [ Fig. 2B ] schematically illustrates a coupling between the binocular sight housing of the Fig. 2A and a smart cell phone, to make binoculars; [ Fig. 2C ] schematically illustrates the binoculars housing and the smart mobile phone after coupling according to the Fig. 2B ; [ Fig. 2D ] schematically illustrates the definition of Euler angles in relation to sighting binoculars; [ Fig. 3 ] schematically illustrates an example of hardware architecture of at least one device of the simulation system of the Fig. 1 ; [ Fig. 4 ] schematically illustrates a display management algorithm via the viewing binoculars; [ Fig. 5 ] schematically illustrates an algorithm for obtaining orientation information used in display management via sighting binoculars; [ Fig. 6 ] schematically illustrates offsets to be taken into consideration for calibration of the display via the sighting binoculars; [ Fig. 7 ] schematically illustrates an algorithm for calibrating the display via the sighting binoculars; [ Fig. 8A ] schematically illustrates an arrangement of projection windows for constituting mixed reality video frames, without magnification factor; and [ Fig. 8B ] schematically illustrates an arrangement of projection windows to constitute mixed reality video frames, with magnification factor. DETAILED PRESENTATION OF IMPLEMENTATION METHODS

[0023] There Fig. 1 schematically illustrates a field training simulation system for combatants. In one embodiment, the simulation system allows field training of combatants and various combined arms and joint combat participants, in order to train them, directly in the field, to request and control fire support from different armies.

[0024] The simulation system comprises aiming binoculars, in the form of a binocular HMD (Head Mounted Display) 120. The HMD aiming binoculars 120 preferably simulate models of aiming binoculars as used by the armed forces. The HMD aiming binoculars 120 are mixed reality equipment that allows a fire support controller in simulation, called the “main trainee”, to interact with virtual actors on the real ground. The simulation system immerses the main trainee in a training scenario, by superimposing a virtual scene on video images captured by a camera (“video camera” in English) of the HMD aiming binoculars 120. The simulation system may comprise a plurality of such aiming binoculars, so as to equip a teammate and / or a supervisor, in order to allow them to immerse themselves in the simulation independently of the main trainee.

[0025] As detailed below in relation to the Figs. 2A , 2B et 2C , the HMD 120 sighting binoculars preferably comprise a sighting binocular housing 1 coupled with a smart mobile phone (“smartphone” in English) 2.

[0026] The simulation system also preferably comprises a FSIM (fire simulator) 160. The FSIM 160 fire simulator makes it possible to prepare, control and trigger a fire simulation according to a simulated operational context: for example, an air-to-ground fire simulator as part of an air fire support exercise, a ground-to-ground fire simulator as part of an infantry fire support exercise, a sea-to-ground fire simulator as part of a fire support exercise from maritime platforms, etc. The FSIM 160 fire simulator makes it possible to take control, for example using a joystick, of a simulated vehicle in charge of a strike towards a target (combat aircraft, combat helicopter, drone, artillery, naval vessel). It is an autonomous simulation station in which a simulated geographical view of the training ground in 3D (three dimensions) is displayed.The FSIM 160 shooting simulator is preferably installed on a laptop.

[0027] The simulation system further comprises a simulation platform SIMP (simulation platform) 130. The simulation platform SIMP 130 is connected to a tactical situation server TSS (tactical situation server) 170, to a geographical server GS (geographical server) 180 and to a simulation database SDB (simulation database) 190. The tactical situation server TSS 170 and / or the geographical server GS 180 and / or the simulation database SDB 190 can be connected to the simulation system or integrated into the simulation system, for example within the simulation platform SIMP 130.

[0028] The simulation system preferably further comprises a ROVER (Remotely Operated Video Enhanced Receiver) type system simulator 140, namely an assembly at least formed of an electro-optical sensor simulator and a simulator for remote reception of video streams originating from these sensors. ROVER type systems are typically used in combat situations to receive on the ground an on-board view, in real time (with a slight time lag), from an aircraft or drone in operation, so as to facilitate the execution of the tasks incumbent on fire support controllers on the ground. In a fire support request training exercise, an observer in charge of carrying out strike guidance towards a target, as fire support controllers do in combat, generally has simulated video feedback from a weapon system used in the training scenario and controlled by means of the FSIM firing simulator 160.The ROVER system simulator thus provides a video stream display of a simulated view taken from the weapon system in question. This view simulation is based on 3D terrain modeling, using a digital terrain model which is then textured from satellite imagery, weather modeling and a 3D rendering of the target. The ROVER system simulator is for example integrated into a tablet, a smartphone or a laptop, the simulated video stream then being preferably displayed via a dedicated application or via an HTML page (HyperText Markup Language) in a browser.

[0029] The SIMP 130 simulation platform is configured to create a 3D (three-dimensional) simulation environment comprising a 3D virtualization of the terrain of the training area, a virtualization of weapon systems implemented in the FSIM 160 shooting simulator and their associated kinematics, a virtualization of targets to be reached and their associated kinematics, and kinematics of projectiles towards these targets. The SIMP 130 simulation platform is configured to execute a training scenario comprising definitions of different events to be applied in the simulation and to generate a rendering of at least one view used by the FSIM 160 shooting simulator, a rendering of at least one video return view for the ROVER 140 type system simulator, and the provision of different simulation elements, as described below, to allow a mixed reality representation in the HMD 120 sighting binoculars.Different virtual scenes, synchronized with each other, then exist in the simulation system, as described below.

[0030] The SIMP 130 simulation platform may include a training scenario editor. The training scenario editor is configured to provide an interface for configuring various parameters and entities of the simulation, such as: a list of real actors (trainer, supervisor, etc.), a list of synthetic entities (weapon systems, strike force and operator of the shooting simulator, command post, etc.), a geographical delimitation of the training area (theater of operations), an initial situation (position of the synthetic entities, date and time), a list of dated milestones for each synthetic entity involved in the simulation. The training scenario is then used as a reference when conducting the simulation, then analyzing its progress.The SIMP 130 simulation platform may include a training scenario recorder, making it possible to record the training scenario thus edited in a database. The SIMP 130 simulation platform may also obtain the training scenario to be applied in the SDB 190 simulation database, intended to provide sets of training scenarios applicable in accordance with the terrain where the simulation is practiced.

[0031] The simulation system may further include a COM 150 communication terminal, in order to increase the realism of the simulation by placing the fire support controller as close as possible to his actual field conditions. In this case, the FSIM 160 firing simulator is preferably also equipped with such a communication terminal, comprising for example an audio headset and a voice microphone, in order to allow an operator of the FSIM 160 firing simulator to communicate vocally with the main trainee to receive guidance information in real time. The COM 150 communication terminal may also allow the main trainee to communicate vocally with any other observer or actor in the training scenario also equipped with such a communication terminal. The COM 150 communication terminal preferably uses VoIP (“Voice over IP” in English) type technology.The COM 150 communication terminal may further include tactical messaging communication means (SICS, L16, VMF for example). In a particular embodiment, the COM 150 communication terminal is integrated into the smart mobile phone 2 which is used to partly produce the HMD 120 sighting binoculars, for example by connecting an audio headset and a voice microphone. The COM 150 communication terminal may alternatively be integrated into the ROVER type system simulator.

[0032] The simulation system further comprises a communication bus, or a NET 100 communication network, making it possible to connect all of the aforementioned elements of the simulation system. For example, the NET 100 communication network complies with all or part of the IEEE 802.11 family of standards and an AP access point 110 manages the implementation of the NET 100 communication network. The AP access point 110 can integrate a router or bridge function, for example with a cellular network of the LTE type (Long-Term Evolution), in order to remote elements of the simulation system, such as for example the FSIM 160 shooting simulator, or to involve additional remote systems and / or simulators that can take part in the training scenario (command post, databases, etc.).

[0033] The GS 180 geographic server provides the simulation system with a digital 3D terrain model, preferably textured with satellite or aerial images for a realistic 3D rendering of the training area. The texture is used to increase the realism of the simulation for the FSIM 160 shooting simulator and for the ROVER type system simulator. The GS 180 geographic server provides information on the inter-visibility and distance of synthetic entities in the virtual scene (e.g., the presence of terrain elements masking a synthetic entity from an observation point). The GS 180 geographic server provides meteorological data used for weapon system modeling: influence on projectile trajectories of ambient temperature, atmospheric pressure, air humidity, visibility conditions, possible precipitation (rain, snow), wind strength and direction, etc.

[0034] The TSS 170 tactical situation server provides the simulation system with simulation elements useful for representing in the virtual scene weapon systems, targets and objects involved in the training scenario, as well as their behavior during the exercise (kinematics, etc.).

[0035] The SMIP 130 simulation platform can act as a relay, possibly with data reformatting, between the tactical situation server TSS 170 and / or the geographic server GS 180, on the one hand, and the other elements of the simulation system that need to receive their data, on the other hand. The SMIP 130 simulation platform can also program the tactical situation server TSS 170 and / or the geographic server GS 180 to transmit this data to the rest of the simulation system. The transmission of data from the tactical situation server TSS 170 and / or the geographic server GS 180 can, for example, be carried out using all or part of the IEEE 1278 family of standards relating to the DIS protocol (Distributed Interactive Simulation).

[0036] There Fig. 2A schematically illustrates an embodiment of a housing 1 of the HMD 120 sighting binoculars. The housing 1 of the HMD 120 sighting binoculars comprises a housing intended to receive and hold fixed a smart mobile phone 2. The housing 1 of the HMD 120 sighting binoculars comprises a pair of lenses 3 arranged to face the screen of the smart mobile phone 2 when the latter is fixed on the housing 1 of the HMD 120 sighting binoculars. The main trainee is supposed to place his eyes on the other side of the lenses 3, so as to have a binocular vision on the screen of the smart mobile phone 2 through the lenses 3.

[0037] The housing 1 of the HMD 120 sighting binoculars preferably comprises a button 4 for activating a calibration mode of the HMD 120 sighting binoculars. For example, the main trainer activates the calibration mode by pressing the button 4. Particular embodiments of the calibration mode are described below in relation to the Fig. 7 .

[0038] The housing 1 of the HMD 120 sighting binoculars comprises a magnification factor adjustment control member, such as a pair of buttons 5. By pressing a first button of the pair of buttons 5, the main drive controls a zoom in, and by pressing the second button of the pair of buttons 5, the main drive controls a zoom out. The pair of buttons 5 can be replaced by an electromechanical wheel, a zoom in being controlled by action of the electromechanical wheel in one direction of rotation or a zoom out being controlled by action of the electromechanical wheel in the opposite direction of rotation.

[0039] The housing 1 of the HMD 120 sighting binoculars preferably comprises a button 6 for triggering telemetry reading. The telemetry reading is triggered by pressing the button 6, and the telemetry reading is stopped by subsequently pressing the button 6 and / or after expiry of a predefined time. Alternatively, the telemetry reading is maintained as long as the button 6 is pressed. The telemetry reading consists of displaying in the HMD 120 sighting binoculars distance information which separates the position of the trainee from that of a target pointed at (which may be a virtual target, or another virtual object, or a terrain element) by the line of sight of the HMD 120 sighting binoculars. This distance is measured using a method called ray tracing in a virtual scene synchronized with the real scene.The virtual scene used for the ranging calculation, called detailed virtual scene, is distinct from the virtual scene used for rendering objects in mixed reality in the HMD 120 sighting binoculars and includes a 3D representation of the terrain, the objects of the scenario (target, forces present, etc.) and objects representative of the surface (tree, building, infrastructure, etc.). Ray tracing methods are well known in the state of the art and are part of the standard functionalities offered by commercial 3D rendering engines. In the context of the present description, the distance measurement is determined by a ray trace in the direction of sight of the HMD 120 sighting binoculars, namely along the axis passing through the position of the observer and the center of the field of view FOV of the HMD 120 sighting binoculars. The more precise the digital model of the terrain modeling, the more precise the distance measurement for the ranging reading.The more complete the virtual scene used is in terms of virtualization of objects present in the real scene (on-ground), the more the distance measurements made are in line with reality. The counterpart to this richness and precision of the virtualization of the training ground and the objects that compose it is the volume of calculations and the memory occupation necessary to carry out the calculation of such a scene in 3D essential to the ray tracing methods implemented for the distance measurement. The calculation of this virtual scene in 3D is therefore deported outside the HMD 120 aiming binoculars, so that the HMD 120 aiming binoculars do not have to receive, process and manipulate data from the terrain modeling.Thus, when the telemetry reading is triggered, a request for measuring the distance between the main trainee and a pointed target, comprising, among other things, the geolocation of the HMD 120 aiming binoculars and a direction vector representative of the aiming direction (for example a vector), is transmitted to a remote processing unit, such as for example the processing unit (processor, memory, etc.) of the ROVER type system simulator or the SIMP 130 simulation platform. The remote processing unit performs the measurement by ray tracing in the detailed virtual scene, using the geolocation of the HMD 120 aiming binoculars and the direction vector transmitted in the request. This measurement is returned to the HMD 120 aiming binoculars to then be preferentially displayed in a plane representing an aiming reticle according to display characteristics of the simulated aiming binocular model.

[0040] According to a particular embodiment, the distance measurement between the main train and a pointed target is then refreshed at a predefined frequency by the remote processing unit without any further request from the HMD sighting binoculars 120, and the distance measurement is obtained and transmitted by the remote processing unit to the HMD sighting binoculars 120 as soon as the value of the measurement differs from the previous one. In this particular embodiment, the HMD sighting binoculars 120 inform the remote processing unit of any change in their geolocation and / or their orientation (i.e., their direction of sight), so that the remote processing unit can take this into account to calculate the required distance.This means that the HMD 120 binoculars always have up-to-date distance information regardless of the movements of the HMD 120 binoculars and the position of the target being aimed at (moving or not), and is ready to be displayed when the rangefinder reading mode is engaged.

[0041] The housing 1 of the HMD 120 sighting binoculars preferably includes a button 7 for triggering magnetic compass reading. The housing 1 of the HMD 120 sighting binoculars may include additional control elements, such as buttons or dials. All or part of the button controls of a simulated sighting binocular model may thus be reproduced.

[0042] When the HMD 120 sighting binoculars are based on an architecture using the smart mobile phone 2, the control members of the housing 1 are connected to an integrated wireless transmission component compatible with a radio technology implemented by the smart mobile phone 2, such as for example Bluetooth (registered trademark). The manipulation of the control members is then transformed into respective control messages, transmitted to the smart mobile phone 2 via the integrated wireless transmission component. Alternatively, the control of the smart mobile phone 2 can be carried out via a wired connection interface, such as for example a “Lightning” type connection interface for iPhone (registered trademark) smart mobile phones from Apple (registered trademark).

[0043] There Fig. 2B schematically illustrates that the housing 1 of the HMD 120 sighting binoculars is adapted to be mechanically coupled to the smart mobile phone 2. The smart mobile phone 2 typically comprises a camera (video camera) 8, a speaker 9 and a set of integrated components 10 including an IMU (Inertial Measurement Unit) type sensor and preferably a GPS (Global Positioning System) receiver. When the HMD 120 sighting binoculars do not require a smart mobile phone to operate, the camera 8 and the set of integrated components 10 including at least the IMU (Inertial Measurement Unit) type sensor are directly incorporated into the housing of the HMD 120 sighting binoculars, as is the screen.

[0044] There Fig. 2C schematically illustrates the housing 1 of the HMD 120 sighting binoculars and the smart mobile phone 2 after mechanical coupling. The mechanical coupling is such that the smart mobile phone 2 is secured to the housing 1 of the HMD 120 sighting binoculars. The mechanical coupling is such that, when the main trainee places the HMD 120 sighting binoculars, the main trainee sees the screen of the smart mobile phone 2 through the lenses 3, the camera 8 then being located on the opposite side of the HMD 120 sighting binoculars from that where the main trainee is placed.

[0045] The HMD 120 viewing binoculars are configured, thanks in particular to a configuration of the smart mobile phone 2 and the arrangement of the smart mobile phone 2 with the housing 1, to reproduce a see-through display (referred to as a “video-see-through” type device). To do this, an ARCore library (registered trademark) from the company Google (registered trademark), or an ARKit library (registered trademark) from the company Apple (registered trademark), can be used. Thus, when using the HMD 120 viewing binoculars, the main trainee sees through the lenses 3, in mixed reality, a real-time video captured by the camera 8 enriched by virtual objects. In order to reproduce the binocular effect of observation binoculars, the smart mobile phone 2 is configured to display on its screen the captured video in the form of a right-eye view and a left-eye view.

[0046] Thus, a virtual target present in the virtual scene of the training scenario which is placed in the field of vision of the observer defined by the camera 8 of the smart mobile phone 2 is displayed superimposed on the real view captured by the camera 8. In addition to the virtual scene resulting from the training scenario, the display on the screen of the smart mobile phone 2 can be enriched with additional information (azimuth, telemetry, aiming target, magnification applied, etc.).

[0047] For reference, the Fig. 2D schematically illustrates the definition of Euler angles relative to the HMD 120 sighting binoculars, in a direct orthonormal frame (X, Y, Z). These Euler angles typically serve as orientation information for the HMD 120 sighting binoculars, thus defining a sighting direction vector. The Euler angles are thus a vertical inclination or pitch angle θ (“pitch” in English) around the Y axis, an orientation angle relative to magnetic North or yaw angle ψ (“heading” in English) around the Z axis, and a roll angle ϕ (“roll” in English) around the X axis.

[0048] There Fig. 3 schematically illustrates an example of hardware architecture of a device of the simulation system. The example of hardware architecture is applicable to a control and processing unit of the HMD 120 sighting binoculars, in particular to the smart mobile phone used to partly constitute the HMD 120 sighting binoculars. The example of hardware architecture is also applicable to a control and processing unit of the SIMP 130 simulation platform. The example of hardware architecture is also applicable to a control and processing unit of the ROVER 140 type system simulator. The example of hardware architecture is also applicable to a control and processing unit of the FSIM 160 shooting simulator. The example of hardware architecture is also applicable to a control and processing unit of the TSS 170 tactical situation server and / or the GS 180 geographic server and / or the SDB 190 simulation database.The hardware architecture example is also applicable to a control and processing unit of the COM 150 communication terminal.

[0049] The device then comprises, connected by a communication bus 210: one or more processors or CPUs (Central Processing Unit) 301; a RAM (Read-Only Memory) 302; a ROM (Read Only Memory) 303, for example of the EEPROM (Electrically-Erasable Programmable ROM) type or a Flash memory; a storage unit 304, such as a hard disk drive (HDD), or a storage media reader, such as an SD (Secure Digital) card reader; and a COM communication interface manager 305.

[0050] The COM 305 communication interface manager allows the device to communicate with other devices in the communication system, more particularly via the NET 100 communication network.

[0051] The processor 301 is capable of executing instructions loaded into the RAM 302 from the ROM 303, an external memory, a storage medium (such as an SD card), or a communication network (potentially separate from the NET 100 communication network). When the device is powered on, the processor 301 is capable of reading instructions from the RAM 302 and executing them. These instructions form a computer program causing the processor 301 to implement some or all of the steps, operations, and behaviors described herein in relation to the device in question.

[0052] All or part of the steps, operations and behaviors described here in relation to the device in question can thus be implemented in software form by executing a set of instructions by a programmable machine, for example a DSP (Digital Signal Processor) type processor or a microprocessor, or be implemented in hardware form by a machine or a dedicated component, for example an FPGA (Field Programmable Gate Array) or ASIC (Application Specific Integrated Circuit) component. Generally speaking, the device in question comprises electronic circuitry adapted and configured to implement, in software and / or hardware form, the steps, operations and behaviors described here in relation to the device in question.

[0053] There Fig. 4 schematically illustrates a display management algorithm via the HMD 120 sighting binoculars.

[0054] In a step 401, calibration operations are preferably performed. The calibration comprises operations making it possible to obtain and correct a lateral deviation (“heading deviation” in English) ΔH, a tilt deviation (“pitch deviation” in English) Δθ and a vertical deviation (“vertical deviation” in English) ΔV between the real view of the terrain as captured by the camera 8 and the terrain modeling used by the SIMP simulation platform 130 to create the virtual scene. The deviations ΔH and ΔV are represented on the Fig. 6 . The deviation Δθ corresponds to the difference between the inclination angles θ1 and θ2 on the Fig. 6 This deviation Δθ is determined during calibration operations to estimate the vertical deviation ΔV.

[0055] The values ​​of the deviations in question can be obtained by prior configuration. The values ​​of the deviations in question are preferably obtained through field calibration operations, as described in particular below in relation to the Fig. 7 .

[0056] In a step 402, the HMD sighting binoculars 120 obtain a real-time capture video frame. More particularly, the smart mobile phone 2 receives a capture video frame generated by the camera 8. Each video frame provided by the camera 8 is preferably accompanied by 6-DoF (“6 Degrees of Freedom” in English, or 6 degrees of freedom in French) geographic tracking metadata. For an advantageous application of a magnification factor by the HMD sighting binoculars 120, the video frames are captured in the native FOV (“Field of View” in English) of the camera 8, that is to say without application of a magnification factor by the camera 8. The majority of smart mobile phones on the market have a camera, such as the camera 8, having a FOV field of view between 65° and 80° and a horizontal dimension to vertical dimension ratio of the shot of 4 / 3.

[0057] In a step 403, the HMD sighting binoculars 120 obtain in real time geolocation information and orientation information in the space of the HMD sighting binoculars 120. For the orientation information, the HMD sighting binoculars 120 implement a SLAM algorithm (Simultaneous Localization And Mapping). Many SLAM algorithms are known, for example based on an extended Kalman filter or on a particle filter (also known as the sequential Monte Carlo method), and are currently used in the majority of commercially available smart mobile phones. For the geolocation information, the HMD sighting binoculars 120 also preferentially use the SLAM algorithm. Alternatively, the HMD sighting binoculars 120 use the GPS receiver.However, the SLAM algorithm provides more accurate geographic position information (translation information in the case of relative position information) than the GPS receiver. In this case, the GPS receiver is only used to determine the initial position of the SLAM algorithm, or in the event of a temporary SLAM outage. Thus, the HMD 120 sighting binoculars can use the GPS receiver in degraded mode, when the SLAM algorithm has outage.

[0058] The SLAM algorithm determines orientation information, typically Euler angles, and position information, typically translation information, by recognizing objects located in the field of view of camera 8, at the refresh rate of camera 8 (e.g., 30 or 60Hz). The orientation and position information provided by the SLAM algorithm does not undergo any drift. However, the accuracy of the SLAM algorithm is dependent on the objects located in the field of view of camera 8. When the filmed scene does not have enough usable objects, the SLAM algorithm does not work correctly. Thus, if the main trainee is looking at the sky and not at the terrain in front of him, the SLAM algorithm may not have enough physical landmarks and stalls.This situation can also occur, for example, when capturing a video of the terrain offers little relief or contrast, particularly in winter conditions with the presence of snow or fog, or, conversely, in conditions of saturation of the video sensor in the event of refraction of the sun on wet ground, for example.

[0059] As detailed below in relation to the Fig. 5 , when the SLAM algorithm is operational, the HMD 120 sighting binoculars use the orientation information, typically the Euler angles, as well as preferentially the position information, typically the translation information, provided by the SLAM algorithm. And when the SLAM algorithm stalls, the HMD 120 sighting binoculars use the orientation information, typically the Euler angles, provided by the IMU inertial measurement unit type sensor equipping the HMD 120 sighting binoculars (more particularly equipping the smart mobile phone 2) and preferentially position information from geolocation measurements provided by the GPS receiver. Preferably, the HMD 120 sighting binoculars use the last known absolute position before stalling, as long as the SLAM algorithm is inoperative.When the HMD 120 aiming binoculars use the orientation information provided by the IMU type sensor, the HMD 120 aiming binoculars correct them by compensating for a measurement difference between the orientation information provided by the IMU type sensor and the orientation information provided by the SLAM algorithm at the time of the SLAM algorithm stall. The HMD 120 aiming binoculars further correct the orientation information used, when it comes from the SLAM algorithm, in accordance with the deviations revealed by the calibration operations. Thus, the main trainee can follow a virtualized aircraft in the sky within the training scenario, without losing the orientation of the camera despite the SLAM algorithm stalling.

[0060] In a step 404, the HMD sighting binoculars 120 obtain in real time simulation graphic elements relating to virtual objects of the training scenario, that is to say elements relating to virtual scene objects to be displayed in mixed reality. The simulation graphic elements are provided to the HMD sighting binoculars 120 by the tactical situation server TSS 170, preferably via the simulation platform SIMP 130. The simulation graphic elements can be the graphic objects themselves, in three dimensions. However, considering that the HMD sighting binoculars 120 can have internally a library of graphic objects previously installed and configured, the simulation graphic elements provided by the tactical situation server TSS 170 are then indications of types of graphic objects in the virtual scene and their kinematics.The simulation graphics are for example provided using the DIS protocol. Instead of receiving the complete virtual scene incorporating the terrain modeling, the HMD 120 sighting binoculars only receive information relating to the virtual objects of the scenario (i.e. without the 3D model of the training ground and preferably without the 3D model of the objects). The processing to be carried out by the HMD 120 sighting binoculars to achieve the mixed reality display is thus reduced, which facilitates a synchronized superposition in real time of the virtual scene on the real scene in the HMD 120 sighting binoculars. This approach is particularly well suited to a mixed reality display achieved using a smart mobile phone (which typically has much lower processing performance than a computer or a server), while respecting the video frame rate imposed by the capture of video frames by the camera 8.

[0061] Thus, the SIMP simulation platform 130 uses 3D terrain modeling to define the scenario elements in the applicable field of view FOV (virtual objects that are not hidden by the terrain relief, kinematics, etc.), but only the scenario elements visible in the field of view FOV with the exception of the terrain representation are transmitted to the HMD sighting binoculars 120. So that the SIMP simulation platform 130 can know the applicable field of view FOV, the HMD sighting binoculars 120 transmit, at the same frequency as the video capture from the camera 8 (eg 30 or 60 Hz), the geolocation information provided by the SLAM algorithm or the GPS receiver, and the orientation information from step 403.The SIMP 130 simulation platform is capable of determining the altitude to be applied for determining the field of view FOV, thanks to the geolocation of the HMD 120 sighting binoculars and to elevation information from the digital terrain model, corrected by the calibration operations, as well as preferentially the position information (eg, translation) provided by the SLAM algorithm (thus making it possible, for example, to detect a transition of the main trainee from the “crouching” position to the “standing” position). The HMD 120 sighting binoculars obtain, at the same rate, the virtual objects at the same rate, so as to be able to carry out a refresh of the virtual objects which is synchronized with that of the images captured by the camera 8.The HMD 120 sighting binoculars can receive the simulation graphics elements relating to the virtual objects to be displayed at a lower rate, and the HMD 120 sighting binoculars then perform interpolation operations in order to be able to achieve a refresh of the virtual objects which is synchronized with that of the images captured by the camera 8. Thus, a perfect synchronization of the real scene and the virtual scene can be obtained in mixed reality in the HMD 120 sighting binoculars at the frequency of the video capture of the camera (30 or 60 Hz).

[0062] The HMD sighting binoculars 120 can receive simulation graphic elements in response to each sending of the geolocation information and the orientation information, according to a request-response mechanism. A preferred approach, however, consists of transmitting the simulation graphic elements asynchronously with respect to the sending of the geolocation information and the orientation information. A re-alignment of the simulation graphic elements (or of the corresponding virtual objects) is then preferable, to prevent them from giving an impression of “floating” in mixed reality. Thus, in a particular embodiment, the simulation graphic elements obtained in step 404 are, prior to their display by the HMD sighting binoculars 120, re-aligned on the terrain model. This re-alignment consists of calculating the altitude on the terrain model of the virtual object in question, at its current geographical position.Indeed, training scenarios use trajectories of moving objects defined by a starting position, an arrival position and possible intermediate positions, as well as a movement speed and a direction of movement. From this information, a simulation engine animates the virtual object in question, so that it moves along its trajectory, according to the given direction and speed of movement. Between two known positions of the scenario, the simulation engine interpolates the positions of the virtual object in order to stick as closely as possible to the trajectory predefined in the training scenario. This interpolation does not take into account the underlying terrain reality. A discrepancy between the interpolated altitude and the altitude of the terrain model under the virtual object may exist.This offset should be corrected to prevent it from affecting the representation of the moving object in the virtual scene of the HMD 120 viewing binoculars and causing jumping effects of the virtual object when rendering the virtual scene in the HMD 120 viewing binoculars. In a particular embodiment, the recalibration of the altitude of each virtual object on the digital terrain model is requested by the simulation engine at each change of position of said virtual object in the scenario, by reading in the digital terrain model available on the geographic server GS 180 the altitude value at the position of the virtual object considered.

[0063] In another embodiment, the altitude value is extracted from the detailed virtual scene generated by the already mentioned remote processing entity. This enriched virtual scene, also used for calculating the telemetry information of the sighting binocular, consists in particular of a 3D representation of the most precise terrain model available on the geographical training area. The altitude value of the terrain at the position of the virtual object considered can therefore also be obtained by calculating the vertical distance separating the base of the virtual object and the underlying terrain.

[0064] Regardless of the two particular embodiments previously mentioned, the recalibration of the simulation graphic elements (or virtual objects) on the digital terrain model can also use information representative of the slope of the terrain at the position of the virtual object, to recalibrate the orientation of said virtual object in the scene. Indeed, the virtual object moving on its trajectory, according to the parameters given by the training scenario, may find itself on a change in slope of the terrain. In this configuration, the recalibration of the altitude may not be sufficient to correctly carry out the recalibration on the terrain. It is then necessary to also adjust its orientation relative to the slope of the terrain. This adjustment consists of adjusting the orientation of the virtual object considered relative to the slope of the terrain represented for example by a vector corresponding to the normal to the slope of the terrain at the geographical position considered.For this recalibration of the orientation of the virtual object in question, the normal to the terrain slope can be obtained by calculation from the altitude information extracted from the terrain model in the vicinity of the geographical position considered, or by an average calculation of the normals of the faces of the terrain in 3D located in the vicinity of the geographical position considered in the detailed virtual scene.

[0065] In a particular embodiment, the simulation graphic elements are accompanied by simulation audio elements, so as to increase the realism of cinematic effects (explosion of a virtual target, noise of an approaching vehicle, etc.). These simulation audio elements are intended to be played by the HMD 120 sighting binoculars, via the speaker 9, synchronously with the simulation graphic elements that they accompany.

[0066] In a step 405, the HMD viewing binoculars 120 perform a 2D rendering (projection) of the 3D virtual objects (i.e., the elements of the virtual scene) whose corresponding graphic elements were received in step 404. For example, the HMD viewing binoculars 120, and more particularly the smart mobile phone 2, use for this purpose the Unity simulation engine (registered trademark) developed by the company Unity Technologies. The 2D rendering is intended to be displayed in a projection window (“viewport” in English) E2 (see Fig. 8A ). The background of the 2D rendering is transparent. The size of the E2 projection window is fixed, with a constant image resolution, even when a magnification factor (zoom) must be applied. When a magnification factor is to be applied, the HMD 120 sighting binoculars receive the graphic elements for simulating the virtual objects to be displayed according to a field of view FOV which is adjusted according to the magnification factor (horizontal dimension and vertical dimension of the field of view FOV of the virtual camera used by the HMD 120 sighting binoculars divided by "2" when the magnification factor is "2" (x2 zoom)). Thus, the sizes of the virtual objects to be represented in the adjusted field of view FOV are twice as large as their dimensions without magnification factor in the E2 projection window.So that the SIMP 130 simulation platform is aware of the adjustment according to the magnification factor to be applied to the applicable field of view FOV, when the main trainee commands a zoom in or a zoom out on the HMD 120 viewing binoculars (e.g., respectively by pressing a ZOOM+ or a ZOOM- button), the HMD 120 viewing binoculars inform at least the SIMP 130 simulation platform. Thus, the SIMP 130 simulation platform adjusts the dimensions of the FOV field of view of the virtual camera accordingly. For example, without a magnification factor, the FOV field of view of the virtual camera is equal to 110° vertically and the same horizontally. This FOV value and this 1 / 1 ratio between its horizontal and vertical value are constrained by the binocular device (see lenses 3) used in the HMD 120 sighting binoculars, the latter reproducing the binocular stereoscopic field of vision in humans.The FOV field of view of the virtual camera and by construction the projection window E2 are thus square in shape, facilitating the display of a right eye / left eye stereoscopic image. By applying a magnification factor equal to "2", the FOV field of view of the virtual camera becomes equal to 55° vertically and the same horizontally.

[0067] In a step 406, when a magnification factor is to be applied, the HMD sighting binoculars 120 redefine the dimensions of a projection window of the capture video frame, and preferably of a projection window of a reticle.

[0068] Indeed, by default, without magnification factor, the capture video frame is projected onto an E3 projection window whose dimensions conform to the horizontal FOV field of vision of 77° of camera 8 and according to a horizontal dimension to vertical dimension ratio of 4 / 3 (see Fig. 8A ). Thus, by construction, given the difference between the FOV of the camera 8 and that of the virtual camera of the sighting binocular, the dimensions of the projection windows E3 and E2 are not identical and, although they are both centered, do not completely overlap. An opaque mask (e.g., black) is then applied by the sighting binoculars HMD 120 around the projection window E3, so as to mask the difference in surface area between the projection window E3 and the projection window E2. The reticle in the present description is an image on a transparent background of square or rectangular dimensions, which may represent, for example, a single or multiple graduation, crossed or not, and allowing the main trainee to carry out precise sightings. The reticle is projected onto a projection window E1 (see Fig. 8A ) whose dimensions conform to a horizontal field of view FOV corresponding to the angular distance between the left end and the right end of the reticle graduations (expressed for example in NATO mils: 6400 mils = 2 PI radians). It is therefore sufficient to count the horizontal graduations of the reticle to obtain the horizontal dimensions of the field of view FOV corresponding to the reticle in NATO mils (any other angular measurement unit can be used depending on the simulated binocular model and its reticle). The same principle applies to the vertical dimensions of the field of view FOV corresponding to the reticle, in the event that the latter has vertical graduations. Otherwise, the vertical dimensions of the field of view FOV corresponding to the reticle are defined so as to precisely respect the horizontal and vertical ratio of the reticle image.

[0069] When a magnification factor is applied, the dimensions of the E3 projection window are increased (vertically and horizontally, compared to the E3 projection window without magnification factor) by the same proportion as the magnification factor to be applied. In other words, the horizontal dimension and the vertical dimension of the E3 projection window are doubled when the magnification factor is equal to "2". An adaptation of the size of the E3 projection window according to a magnification factor to be applied is schematically shown in the Fig. 8B . Note that, following the resizing of the E3 projection window according to the magnification factor to be applied, the E3 projection window may become larger than the E2 projection window. When a reticle is to be displayed, the HMD 120 sighting binoculars adjust the dimensions of the E1 projection window in the same way as they adjust the dimensions of the E3 projection window (i.e., doubled when the magnification factor is equal to "2").

[0070] In a step 407, the HMD sighting binoculars 120 superimpose, in a virtual scene local to the HMD sighting binoculars 120, the 2D rendering of the virtual objects with the capture video frame, and possibly the reticle.

[0071] The projection window E2 is placed on a plane, orthogonal to the line of sight, at a distance D2 from the position of the virtual stereoscopic camera which coincides with the position of the observation point. The line of sight passes through the center of the projection window E2. For example, the distance D2 is set to 10 meters.

[0072] The projection window E3 is placed on a plane, orthogonal to the line of sight, at a distance D3 from the position of the main trainee (observation point). The line of sight passes through the center of the projection window E3. The distance D3 is greater than the distance D2 to ensure that the virtual objects in the training scenario are superimposed on the capture video frame in mixed reality. For example, the distance D3 is set to approximately 10 meters plus 1 centimeter.

[0073] When a reticle (on a transparent background) is to be displayed, the projection window E1 is placed on a plane, orthogonal to the line of sight, at a distance D1 from the position of the main trainee (observation point). The line of sight passes through the center of the projection window E1. The distance D1 is less than the distance D2 to ensure that the reticle is superimposed on the virtual objects of the training scenario and on the capture video frame in mixed reality. For example, the distance D1 is set at approximately 1 meter (a value that can comfortably accommodate the vast majority of users).

[0074] Thus, when a magnification factor is to be applied, no magnification modification is made at the level of camera 8, which remains in the native field of view FOV, which limits the dropouts of the SLAM algorithm.

[0075] In an optional step 408, the HMD sighting binoculars 120 insert, superimposed in the foreground, additional information, such as telemetry information (distance to a target, aiming azimuth, etc.) and / or the possible magnification factor applied in step 406. This additional information is typically information of a textual nature, but can also be information of a graphic nature (icon, logo, etc.). This possible additional information is displayed in a projection window E0 in the foreground, for example at 1 meter minus 1 centimeter. The projection window E0 is not affected by any possible magnification factor, because the latter must have no impact on the display size of this additional information. In other words, the projection window E0 is insensitive to the magnification factor.

[0076] In a step 409, the HMD viewing binoculars 120 obtain a mixed reality stereoscopic image from the superposition of the different planes explained above. The stereoscopic image (in reality a pair of images) is obtained using a pair of virtual cameras 801, 802. The field of view FOV (angular) of the virtual camera used for the 2D rendering in step 405 without applying a magnification factor is identical to the field of view FOV (angular) of each of the virtual cameras 801, 802, for example 110°. The virtual cameras 801, 802 are placed on either side, equidistant, from the viewing axis of the HMD viewing binoculars 120, and have the same spacing between them (“d” on the Figs. 8A And 8B ) than the lenses 3, generally called the interpupillary distance of the binocular device (see lenses 3). Their respective viewing directions are parallel to the viewing axis of the HMD 120 binoculars.

[0077] In a step 410, the HMD viewing binoculars 120 display the mixed reality right eye / left eye stereoscopic image as obtained by executing the preceding steps of the algorithm of the Fig. 4 on the screen of the smart mobile phone 2 or on the screen of the HMD viewing binoculars 120 if they have their own integrated screen. The screen is separated into two equal parts. The stereoscopic image corresponding to the virtual camera 801 (left eye) is displayed in the left part of the screen, the stereoscopic image corresponding to the virtual camera 802 (right eye) is displayed in the right part of the screen. The HMD viewing binoculars 120 can apply a deformation of the right eye / left eye stereoscopic image obtained in step 409, so as to compensate for optical deformations induced by the lenses 3. In the case where the simulation graphic elements relating to the virtual objects displayed in this mixed reality display step were accompanied by simulation audio elements, the HMD viewing binoculars 120 play, simultaneously with the mixed reality display, said simulation audio elements.

[0078] The algorithm of the Fig. 4 is thus successively iterated for each video frame captured by camera 8.

[0079] There Fig. 5 schematically illustrates an algorithm for obtaining orientation information used in display management via the HMD 120 aiming binoculars. The algorithm of the Fig. 5 is a particular embodiment of step 403 with regard to the recovery, for each video frame captured by the camera 8, of the applicable orientation and possibly position information.

[0080] In a step 501, the HMD sighting binoculars 120 carry out monitoring of the SLAM algorithm. As already indicated, the SLAM algorithm provides orientation and (geographic) position information of the camera 8, and therefore more generally of the line of sight of the HMD sighting binoculars 120. The SLAM algorithm also provides information representative of whether or not the SLAM algorithm has stalled. According to one example, the SLAM algorithm manages a flag indicating whether or not the SLAM algorithm has stalled (e.g., flag with the value “1” or the Boolean value “yes”) or not (e.g., flag with the value “0” or the Boolean value “no”). According to another example, the SLAM algorithm provides a confidence index associated with the orientation and position information. The value of this confidence index depends on the number of references that the SLAM algorithm has managed to detect in the video frames captured by camera 8 to locate itself in space.When the confidence index is below a predefined threshold, the SLAM algorithm indicates that it has dropped out; otherwise, the orientation and position information from the SLAM algorithm is considered reliable. The HMD 120 binoculars can also compare the confidence index with a pair of thresholds (high and low) to implement a hysteresis principle.

[0081] In a step 502, the HMD sighting binoculars 120 checks whether the SLAM algorithm has stalled or not. When the SLAM algorithm has stalled, a step 505 is performed; otherwise, a step 503 is performed.

[0082] In step 503, the HMD sighting binoculars 120 obtain the orientation information from the SLAM algorithm.

[0083] Then, in a step 504, the HMD sighting binoculars 120 correct the orientation information of the SLAM algorithm in accordance with the result of the calibration operations. As described in connection with the Fig. 4 , this corrected orientation information is used to request from the SIMP 130 simulation platform the elements to be displayed in the virtual scene. The algorithm of the Fig. 5 .

[0084] In step 505, the HMD sighting binoculars 120 obtain the orientation information of the SLAM algorithm before stalling. Typically, the HMD sighting binoculars 120 store in memory the orientation information of the SLAM algorithm at each video frame captured by the camera 8, overwriting the previously stored orientation information, as long as the SLAM algorithm has not stalled. When the SLAM algorithm stalls, the HMD sighting binoculars 120 retrieve from memory the last orientation information thus stored, in order to obtain the orientation information of the SLAM algorithm before stalling.

[0085] Then, in a step 506, the HMD sighting binoculars 120 obtain the orientation information of the IMU inertial measurement unit type sensor before the SLAM algorithm is switched off. Typically, the HMD sighting binoculars 120 store in memory the orientation information of the IMU inertial measurement unit type sensor for each video frame captured by the camera 8, overwriting the previously stored orientation information. When the SLAM algorithm is switched off, the HMD sighting binoculars 120 retrieve from memory the last orientation information thus stored, in order to obtain the orientation information of the IMU inertial measurement unit type sensor before the SLAM algorithm is switched off.

[0086] In a step 507, the HMD sighting binoculars 120 obtain the updated orientation information from the inertial measurement unit (IMU) type sensor.

[0087] In a step 504, the HMD sighting binoculars 120 correct the updated orientation information of the inertial measurement unit type IMU sensor in accordance with the result of the calibration operations, and by adjusting them by the difference between the orientation information of the inertial measurement unit type IMU sensor before stalling of the SLAM algorithm and the orientation information of the SLAM algorithm before stalling.

[0088] In a particular embodiment, the HMD 120 aiming binoculars correct the updated orientation information from the inertial measurement unit (IMU) type sensor as follows: Q abs t = Q abs t0 * Q imu − 1 t0 * Q imu t Or : Q abs (t) represents the corrected orientation information; Q abs (t0) represents the absolute orientation information, i.e., that from the SLAM algorithm, before stalling; Q imu (t) represents the updated orientation information from the IMU inertial measurement unit type sensor; and Q imu (t0) represents the orientation information from the IMU inertial measurement unit type sensor before stalling from the SLAM algorithm, and where the orientation information is represented in the form of quaternions representative of the Euler angles after correction, for the SLAM algorithm, of the deviations revealed by the calibration operations.

[0089] You can refer to the instructions provided at the following link to convert Euler angles into quaternions: https: / / en.wikipedia.org / wiki / Conversion _ between _ quaternions _ and _ Euler_angles

[0090] This corrected orientation information is then used to request from the SIMP 130 simulation platform the elements to be displayed in the virtual scene. The algorithm of the Fig. 5 .

[0091] The Euler angles provided by the SLAM algorithm may be provided in a relative manner, i.e. by difference from a previous value (e.g. by difference from the first video frame used by the SLAM algorithm). The position information provided by the SLAM algorithm may also be provided in a relative manner (which provides translation information). It may therefore be necessary to carry out preliminary calibration operations, as described below, to accurately determine absolute references for the position and orientation of the camera 8 before applying the relative orientation and translation information provided by the SLAM algorithm.

[0092] In a particular embodiment, when the SLAM algorithm has stalled, the HMD sighting binoculars 120 rely on the GPS receiver to obtain position information (geolocation) of the HMD sighting binoculars 120, as previously described; otherwise, the HMD sighting binoculars 120 rely on the SLAM algorithm to further obtain said position information of the HMD sighting binoculars 120 (whether relative to a previous value or absolute).

[0093] There Fig. 6 schematically illustrates offsets to be taken into consideration for display calibration via the HMD 120 sighting binoculars. The Fig. 6 shows a representation of terrain 600 in side view, where two cameras 601 and 602 are placed, where camera 601 corresponds to camera 8 and camera 602 corresponds to the virtual camera. The geographical positions of cameras 601 and 602 coincide, but camera 602 may have a lateral deviation ("heading deviation" in English) ΔH, a tilt deviation ("pitch deviation" in English) Δθ and a vertical deviation ("vertical deviation" in English) ΔV or altitude with respect to camera 601. The right part of the Fig. 6 shows a two-dimensional rendering 620 of the superposition of the video frames of the cameras 601 and 602, by projection onto a plane 630. The same object 610, geographically located at a distance D relative to the geographical position of the cameras 601 and 602, is represented there by an object 611 in the real scene and by an object 612 in the virtual scene, respectively obtained by projection of the object 610 onto a camera projection plane 631 and onto a camera projection plane 632. When the position and orientation of the virtual camera are perfectly synchronized and merged with the camera 8, the objects 611 and 612 are merged.

[0094] The referencing deviations between the real scene, as captured by the camera 8, and the virtual scene, as generated by the HMD 120 viewing binoculars and the SIMP 130 simulation platform can thus be determined by trigonometry, in order to be corrected, as described below, during the training simulation.

[0095] There Fig. 7 schematically illustrates a display calibration algorithm via the HMD 120 sighting binoculars.

[0096] In a step 701, the HMD sighting binoculars 120 activate a calibration mode. The calibration mode is for example activated by pressing the button 4. According to another example, the calibration mode is activated by receiving a message from the SIMP simulation platform 130 or the ROVER type system simulator 140. The message may alternatively come from any other device external to the HMD sighting binoculars 120 and capable of communicating with the HMD sighting binoculars 120.

[0097] In a step 702, the HMD sighting binoculars 120 request from the SIMP simulation platform 130 to receive data allowing the display of a virtual object which serves as a calibration target. This virtual object is placed in the virtual scenario in a notable position of the real scene (in front of a tree or a pole for example). Initially, since the calibration has not yet been carried out, the virtual object is displayed in the wrong place on the real image of the camera 8.

[0098] In a step 703, adjustments are made to perform the calibration. To do this, the main trainee aims with the HMD sighting binoculars 120 at this virtual object, then presses a button (calibration button) of the HMD sighting binoculars 120. Aiming at the virtual object amounts to placing the virtual object on the line of sight of the HMD sighting binoculars 120, using in particular the simulated reticles. The HMD sighting binoculars 120 then obtains first orientation information from the camera 8 when a first press on the calibration button of the HMD sighting binoculars 120 is made. The HMD sighting binoculars 120 then memorizes the orientation of this sight not yet corrected by the calibration. Then, the main trainee aims with the HMD sighting binoculars 120 in the real scene at the desired remarkable position for this virtual object (defined in the virtual scenario), then presses a button of the HMD sighting binoculars 120.The HMD 120 sighting binoculars then obtain second orientation information from the camera 8 when a second press on the calibration button of the HMD 120 sighting binoculars is made. The HMD 120 sighting binoculars then memorize the orientation of this sight, which corresponds to the orientation after application of the calibration correction. The HMD 120 sighting binoculars can then compare the first orientation information and the second orientation information and deduce corresponding deviations in orientation and altitude information.

[0099] In an alternative embodiment, the adjustments are made by progressively compensating for the deviations (e.g. angular) observed between the position of the virtual target and its actual position visible in the view captured by the camera 8. The progressive compensation is carried out using controls of the HMD 120 sighting binoculars, for example by rotating electromechanical knobs. For example, a first electromechanical knob makes it possible to vary the lateral deviation ΔH and a second electromechanical knob makes it possible to vary the vertical deviation ΔV, which indirectly has an impact on the inclination deviation Δθ. When the virtual calibration target coincides with the actual calibration target, the calibration adjustments are made.

[0100] In a step 704, the HMD 120 sighting binoculars memorize the settings made in step 703.

[0101] In a step 705, the HMD sighting binoculars 120 perform a conversion of the adjustments into corresponding orientation and altitude information offsets. For example, the HMD sighting binoculars 120 determine the lateral deviation ΔH to be applied by the difference between the yaw angle (heading) extracted from the first orientation information and the yaw angle (heading) extracted from the second orientation information successively obtained in step 703.

[0102] According to another example, the HMD 120 sighting binoculars determine the vertical deviation ΔV to be applied using a trigonometric formula using the pitch angle extracted from the first orientation information and the pitch angle extracted from the second orientation information obtained successively in step 703. Indeed, by geometric construction as presented in the Fig. 6, the pitch angle extracted from the first orientation information is the angle θ2 and the pitch angle extracted from the second orientation information is the angle θ1. The distance to the target D being determined by a telemetry calculation as detailed previously, or fixed by configuration in the system, we can deduce the value of the difference between the real altitude of the camera 8 and the altitude of the camera in the virtual scene by the following formula: Δ V = tan θ 1 − tan θ 2 * D

[0103] In a step 706, the HMD sighting binoculars 120 deactivate the calibration mode. Only the vertical deviation ΔV and the lateral deviation ΔH are then taken into account to absolutely determine the position and orientation of the camera 8 and the virtual camera of the HMD sighting binoculars 120, so that they are merged, and thus allow the generated images of the virtual scene to be perfectly synchronized and superimposed with the video frames of the real scene filmed by the camera 8.

Claims

1. Simulation sighting binoculars comprising a camera, a screen, a pair of lenses which are arranged to face the screen, and electronic circuitry which is configured to: - obtain a video frame from the camera, the video frame being intended to be displayed by the simulation sighting binoculars in a projection window E3; - obtain geolocation information for the simulation sighting binoculars and orientation information for the camera; - transmit the geolocation and orientation information to a simulation platform; - receive simulation graphical elements and spatial positioning information for three-dimensional virtual objects corresponding to the simulation graphical elements; - carry out a two-dimensional rendering of the virtual objects depending on the positioning information, in order to display the virtual objects in a projection window E2, and, when a magnification factor is to be applied, keep the dimensions of the projection window E2 fixed and adjust the field of view of the two-dimensional rendering according to the magnification factor; - superimpose, in a virtual scene which is local to the simulation sighting binoculars, the two-dimensional rendering of the virtual objects in the projection window E2 and the video frames in the projection window E3, the projection window E3 being placed in a plane which is farther from the simulation sighting binoculars than the one where the projection window E2 is placed in the field of view of the simulation sighting binoculars, the simulation sighting binoculars having a line of sight which passes through the centre of the projection windows E2 and E3, and, when the magnification factor is to be applied, adjust the dimensions of the projection window E3 according to the magnification factor; - obtain a mixed-reality stereoscopic image from superimposing the planes using a pair of virtual cameras reproducing binocular vision; and - carry out, on the screen, a right-eye and left-eye display of the obtained mixed-reality stereoscopic image.

2. Simulation sighting binoculars according to Claim 1, wherein the electronic circuitry is configured to: - further superimpose, in the virtual scene which is local to the simulation sighting binoculars, a reticle in a projection window E1 placed in a plane which is closer to the simulation sighting binoculars than the one where the projection window E2 is placed in the field of view of the simulation sighting binoculars, the line of sight also passing through the centre of the projection window E1, and, when a magnification factor is to be applied, adjust the dimensions of the projection window E1 according to the magnification factor.

3. Simulation sighting binoculars according to Claim 1 or 2, wherein the electronic circuitry is configured to: - further superimpose, in the virtual scene which is local to the simulation sighting binoculars, additional information in a projection window E0 placed in a plane which is closer to the simulation sighting binoculars than the one where the projection window E1 is placed in the field of view of the simulation sighting binoculars, the projection window E0 not being sensitive to the magnification factor.

4. Simulation sighting binoculars according to any one of Claims 1 to 3, wherein the screen, the camera and the electronic circuitry are provided by a smart mobile telephone joined to a housing comprising the pair of lenses, which are arranged to face the screen of the smart mobile telephone.

5. Simulation sighting binoculars according to any one of Claims 1 to 4, further comprising a sensor of inertial measurement unit type, and wherein the electronic circuitry implements a Simultaneous Localization and Mapping, SLAM, algorithm, and the orientation information for the camera is provided by the Simultaneous Localization and Mapping, SLAM, algorithm, except when the Simultaneous Localization and Mapping, SLAM, algorithm stalls, in which case the orientation information for the camera is provided by the sensor of inertial measurement unit type.

6. Simulation sighting binoculars according to Claim 5, wherein the geolocation information for the simulation sighting binoculars is provided by the Simultaneous Localization and Mapping, SLAM, algorithm, except when the Simultaneous Localization and Mapping, SLAM, algorithm stalls, in which case the geolocation information for the simulation sighting binoculars is provided by a GPS receiver integrated into the simulation sighting binoculars.

7. Simulation sighting binoculars according to Claim 5 or 6, wherein the Simultaneous Localization and Mapping, SLAM, algorithm provides a confidence index associated with the orientation information for the camera, for each video frame captured by the camera, and the electronic circuitry is configured to determine whether the Simultaneous Localization and Mapping, SLAM, algorithm has stalled according to the value of the confidence index.

8. Simulation sighting binoculars according to Claim 5 or 6, wherein, when the Simultaneous Localization and Mapping, SLAM, algorithm has stalled, the electronic circuitry corrects the updated orientation information from the sensor of inertial measurement unit, IMU, type in the following way: Q abs t = Q abs t0 * Q imu − 1 t0 * Q imu t where: - Qabs(t) represents the corrected orientation information; - Qabs(t0) represents the absolute orientation information, that is to say that originating from the SLAM algorithm, before stalling; - Qimu(t) represents the updated orientation information from the sensor of inertial measurement unit, IMU, type; and - Qimu(t0) represents the orientation information from the sensor of inertial measurement unit, IMU, type before stalling of the SLAM algorithm.

9. Simulation sighting binoculars according to any one of Claims 5 to 8, wherein the electronic circuitry corrects the orientation information provided by the Simultaneous Localization and Mapping, SLAM, algorithm according to deviations determined during calibration operations.

10. Simulation sighting binoculars according to Claim 9, wherein the electronic circuitry is configured to, during the calibration operations: - receive simulation graphical elements corresponding to a calibration virtual object and spatial positioning information for the calibration virtual object; - carry out, on the screen, a right-eye and left-eye stereoscopic display of a mixed-reality stereoscopic image including the calibration virtual object; - obtain first orientation information for the camera when a first press on a calibration button of the simulation sighting binoculars is carried out; - obtain second orientation information for the camera when a second press on the calibration button of the simulation sighting binoculars is carried out; and - compare the first orientation information and the second orientation information and deduce therefrom corresponding orientation and altitude information deviations.

11. Simulation system comprising a simulation platform, as well as simulation sighting binoculars according to any one of Claims 1 to 10, the simulation platform comprising electronic circuitry configured to place, according to a training scenario, virtual objects in a digital terrain model modelling a training ground where the simulation sighting binoculars are intended to be used, and to transmit or cause to be transmitted, to the simulation sighting binoculars, the simulation graphical elements according to the geolocation information for the simulation sighting binoculars and the orientation information for the camera which is transmitted by the simulation sighting binoculars.

12. Simulation system according to Claim 11, wherein the simulation platform transmits or causes to be transmitted, to the simulation sighting binoculars, the simulation graphical elements asynchronously with respect to the geolocation information for the simulation sighting binoculars and to the orientation information for the camera, the simulation sighting binoculars carry out interpolations of the positioning of the virtual objects and a first processing unit of the simulation system, which is remote from the simulation sighting binoculars, carries out, on behalf of the simulation sighting binoculars, a positioning adjustment by virtue of the digital terrain model.

13. Simulation system according to Claim 11 or 12, wherein, when the simulation sighting binoculars activate a telemetry reading, a second processing unit of the simulation system, which is remote from the simulation sighting binoculars, carries out a distance calculation with respect to a target sighted by the simulation sighting binoculars using a ray tracing method in a virtual scene including said virtual objects and the digital terrain model.

14. Simulation method, implemented by electronic circuitry of simulation sighting binoculars further comprising a camera, a screen and a pair of lenses which are arranged to face the screen, the method comprising the following steps: - obtaining a video frame from the camera, the video frame being intended to be displayed by the simulation sighting binoculars in a projection window E3; - obtaining geolocation information for the simulation sighting binoculars and orientation information for the camera; - transmitting the geolocation and orientation information to a simulation platform; - receiving simulation graphical elements and spatial positioning information for three-dimensional virtual objects corresponding to the simulation graphical elements; - carrying out a two-dimensional rendering of the virtual objects depending on the positioning information, in order to display the virtual objects in a projection window E2, and, when a magnification factor is to be applied, keeping the dimensions of the projection window E2 fixed and adjusting the field of view of the two-dimensional rendering according to the magnification factor; - superimposing, in a virtual scene which is local to the simulation sighting binoculars, the two-dimensional rendering of the virtual objects in the projection window E2 and the video frames in the projection window E3, the projection window E3 being placed in a plane which is farther from the simulation sighting binoculars than the one where the projection window E2 is placed in the field of view of the simulation sighting binoculars, the simulation sighting binoculars having a line of sight which passes through the centre of the projection windows E2 and E3, and, when the magnification factor is to be applied, adjusting the dimensions of the projection window E3 according to the magnification factor; - obtaining a mixed-reality stereoscopic image from superimposing the planes using a pair of virtual cameras reproducing binocular vision which is adapted to the pair of lenses; and - carrying out, on the screen, a right-eye and left-eye display of the obtained mixed-reality stereoscopic image.

15. Simulation method, implemented by a simulation system including simulation sighting binoculars implementing the simulation method according to Claim 14 and a simulation platform comprising electronic circuitry, implementing the following steps: - placing, according to a training scenario, virtual objects in a three-dimensional digital terrain model modelling a training ground where the simulation sighting binoculars are used; and - transmitting or causing to be transmitted, to the simulation sighting binoculars, graphical elements corresponding to the virtual objects according to the geolocation information for the simulation sighting binoculars and the orientation information for their camera which is transmitted by the simulation sighting binoculars.

16. Computer program product, comprising instructions causing simulation sighting binoculars according to Claim 1 to execute the method according to Claim 14 when said instructions are executed by a processor.