Augmented reality device

Abstract

Augmented reality device The present invention relates to an augmented reality device (1) comprising a real environment (31) and a display module (33) capable of projecting a virtual scenario superimposed on a field of vision of a user, in which the display module (33) comprises a substantially transparent viewing means onto which the display module is capable of projecting said virtual scenario, the display module (33) being configured to model and define the virtual scenario within a frame, characterized in that the augmented reality device (1) comprises at least three real markers (2, 3, 4) arranged within the real environment (31) and at least three virtual markers associated with the frame defining the virtual scenario, each of the three virtual markers being intended to coincide with one of the three real markers (2, 3, 4). Figure for abstract: figure 1

Description

Title of the invention: Augmented reality device

[0001] The present invention relates to the field of devices allowing a virtual display and more particularly concerns an augmented reality device.

[0002] Augmented reality technology makes it possible to model a virtual scenario, that is to say a set of virtual objects, and to display it superimposed on a real environment. Such technology is particularly useful for having an overview of the rendering of a product before a large-scale purchase, such as the purchase of furniture or a vehicle. An augmented reality device comprises in particular a display module which may be a screen or a pair of glasses, as well as appropriate calculation means, the calculation means making it possible to calculate the virtual scenario to be modeled so that it can be superimposed on the real environment. The display module then allows the user to view the rendering of this superimposition of the virtual scenario on the real environment.

[0003] However, it happens that the use of an augmented reality device presents some imperfections in the display due in particular to the morphology of the person using this device. In other words, the modeling of the virtual scenario can give for a first user a very coherent rendering, which fits well into the real environment, and subsequently give, for a second user, who presents for example a display size significantly different from that of the first user, a less coherent rendering.Such imperfections may in particular consist of poor orientation of elements of the virtual scenario, relative to the physical display context, which may complicate the user's perception of the product in the real environment, it being understood that an orientation error, even minimal, of a modeled element of the virtual scenario is amplified depending on the distance between the user and the real environment on which this modeled element is displayed.

[0004] In order to circumvent these problems of display consistency from one user to another, the augmented reality device can be configured to allow a calibration operation before each new use of said device, in order to adapt the virtual scenario to the real environment in a manner specific to each user. In particular, there are calibration techniques for correcting orientation errors of elements forming part of the virtual scenario so that their display in the real environment is consistent from the user's point of view. Each of these orientation calibration operations proves to be complex, requiring on the one hand robust calculation means and on the other hand a significant amount of time to be considered before using the augmented reality device.

[0005] An augmented reality device according to the invention comprises a display means which may be a screen or a pair of glasses, on which a display module is capable of broadcasting a virtual scenario superimposed on a real environment in the user's field of vision. The augmented reality device may also comprise means for capturing objects in said field of vision, comprising for example a camera, associated with calculation means making it possible to calculate and move the frame of the virtual scenario so that it can be superimposed on the real environment while reducing display errors. The display module then allows the user to view, via the display means, the rendering of this superposition by displaying the virtual scenario on the real environment.

[0006] The present invention makes it possible to optimize the visualization of virtual models in augmented reality, by improving the position and / or angular orientation calibration operations associated with said visualization.

[0007] More particularly, the invention proposes an augmented reality device comprising a real environment and a display module for viewing a virtual scenario superimposed on a field of vision of a user, in which the display module comprises a substantially transparent viewing means on which the display module is capable of viewing said virtual scenario, the display module being configured to model and define the virtual scenario within a frame, characterized in that the augmented reality device comprises at least three real markers arranged within the real environment and at least three virtual markers associated with the frame defining the virtual scenario, each of the three virtual markers being intended to coincide with one of the three real markers.

[0008] According to an optional characteristic of the invention, the augmented reality device comprises capture means comprising a camera directed towards said field of vision of the user. In particular, the camera is embedded on the display module.

[0009] The display module allows the visualization of the virtual scenario integrated into a real environment present in the field of vision of a user through a transparent element.

[0010] The user can, for example, look at the real environment through a transparent screen of a portable device, on which the virtual scenario is superimposed, or wear a pair of glasses whose lenses form the transparent support on which the virtual scenario is superimposed. Said pair of glasses then integrates cameras filming the real environment in order to be able to project the associated virtual scenario in superposition and conforming to the real environment. The use of the display module is essential to make the virtual scenario visible.

[0011] The real environment is an area materially present from the user's point of view. The real environment may be a simple wall such as a floor or a wall, or may be arranged in a more elaborate manner, for example a model presented in a form adapted to what the augmented reality device allows to be visualized. The augmented reality device therefore makes it possible to superimpose the virtual scenario on the real environment, the virtual elements appearing within the virtual scenario having the objective of being visualized in a coherent manner. The real environment includes a target element for which maximum precision in visualization is desired.

[0012] A virtual model of said target element is defined digitally and stored in the memory of a control and calculation unit, which may in particular be embedded in the display module, or which may be remote and which then includes means of communication with the display module. The virtual model is in particular defined by a frame inside which is inserted a digitally defined study element that it is desired to superimpose on the target element. In other words, the virtual model can be considered as a frame in which the study elements are inserted. The virtual model, or the frame of the virtual scenario to be projected superimposed on the real environment, may be formed by a rough digital representation of the real environment obtained by scanning for example and on which the finely digitized study elements are precisely arranged.Thus, the virtual model does not require a complete, detailed and costly digital definition of the real environment. It only forms the virtual framework in which the digital study element(s) are arranged.

[0013] Furthermore, the virtual model includes the three virtual markers, which make it possible to define a reference frame for the position and angular orientation of the frame of the virtual scenario and which subsequently make it possible to ensure the correct position and orientation of the frame of the virtual scenario in relation to the real environment, and therefore the correct position and orientation of the study elements in relation to the corresponding target elements.

[0014] According to one example, the virtual scenario may consist of a piece of furniture placed within the real environment, which may be a room in which one wishes to place said piece of furniture, in order to be able to project oneself. According to another example, the real environment may be a model of a vehicle interior, the virtual scenario allowing the display of elements usually found within the interior of a vehicle, for example a steering wheel or a dashboard.

[0015] The three real markers are arranged on the real environment, for example by collage. The real markers are landmarks of the target element. The augmented reality device uses the real markers to define the target element on which displays the virtual scenario superimposed on the virtual model of the target element with the study elements integrated into said virtual model. The real markers can, for example, be QR barcodes.

[0016] Preferably, the modeling of the virtual scenario begins with a step of scanning the real environment and the target element, with the real markers which are already in position in this real environment. The augmented reality device is capable of thus entering reference orientations and positions of the virtual markers in the frame delimiting the virtual scenario, these reference orientations and positions of the markers subsequently defining the orientation and position of the virtual model on which virtual objects, forming the study elements, can then be added.

[0017] The augmented reality device according to the invention is capable of carrying out a position calibration and / or an angular orientation calibration, to adjust the position and orientation of this virtual scenario and of the study elements, depending on the user of this augmented reality device. During this calibration, in a first step the augmented reality device compares the arrangement of the real markers as they are perceived by the user, in particular via the capture means associated with the display module, with the arrangement of the virtual markers as they are provided in the virtual scenario to be projected, then in a second step it modifies the arrangement of the virtual scenario frame in which the virtual markers are inscribed in order to adapt the arrangement of these virtual markers in relation to that of the real markers.

[0018] Thus, during the calibration operation, the user must first look at the real markers so that the augmented reality device, via the capture means and the associated calculation means, can determine whether the orientation and the reference position of the virtual scenario makes it possible to guarantee the optimal operation of the augmented reality device, with the closest possible superposition of the virtual study elements on the real target elements.

[0019] According to a characteristic of the invention, the augmented reality device is capable of carrying out a position calibration of the frame of the virtual scenario in which the three virtual markers are inscribed as a function of a position of the three real markers, the augmented reality device being capable, during this position calibration, of determining an offset between the position of each virtual marker and the position of the real marker corresponding to said virtual marker, and modifying the position of the frame of the virtual scenario in which the virtual markers are inscribed as a function of said offset.

[0020] The positioning calibration operation consists of adapting the positioning of the virtual scenario in relation to the positioning measurements of the real markers arranged in the real environment. The positioning takes into account the posi tions of the three real markers as a whole with the positioning errors, or position offset, between each virtual marker and its corresponding real marker. The position of the target of the virtual scenario which is superimposed on the real environment can be determined simply and quickly with the measurement of the positions of the three virtual markers.

[0021] The position calibration can be done prior to the orientation calibration, in order to adjust the position of the frame of the virtual scenario relative to the real environment, with the virtual markers defining the position and orientation of this frame which are arranged as faithfully as possible like the real markers.

[0022] For this, the capture means of the augmented reality device acquire the position of the real markers by selecting a real reference marker. Said real reference marker can be chosen arbitrarily or according to the user's point of view.

[0023] Once the real reference marker is selected, the augmented reality device maps the position of the virtual reference marker, i.e. the virtual marker corresponding to the real reference marker, onto the position of this real reference marker, then determines the position offset between the other two virtual markers relative to their corresponding real marker. Each of the positions of the real markers and the virtual markers can be defined as a position relative respectively to the real reference marker and to the virtual reference marker. The offset can for example correspond to the association of a vertical component and a horizontal component separating a virtual marker and its associated real marker.

[0024] This being done, if the determined position offsets are not harmed, the augmented reality device proceeds to a correction of the position of the virtual model in relation to the real target element, by moving the frame of the virtual scenario as a whole, by a compensation translation, the value of which is the inverse of the offset determined between the target element in the real environment and the study element in the virtual scenario, via the analysis of the markers as mentioned previously.

[0025] The position of the virtual model can in particular be modified by translation of a value corresponding to an average value of all the measured offsets.

[0026] More particularly, according to a characteristic of the invention, the augmented reality device may be able to modify the position of the frame of the virtual scenario according to an average offset measured from each of the position offsets determined between the virtual markers and their corresponding real marker. The average offset is calculated by summing all the offsets, where appropriate by distinguishing the sum of all the vertical components of all the offsets and the sum of the set of horizontal components of all the offsets, divided by the number of pairs of markers formed by a virtual marker and its corresponding real marker. Once this average offset has been calculated, it is applied to the frame of the virtual scenario in which the virtual markers are registered, including the reference virtual marker.

[0027] Thus the positions of the three virtual markers are modified according to an average offset value according to, for example, the vertical and horizontal components. This results in an optimal superposition of the virtual model on the real target element. The position calibration thus makes it possible to adapt the position of the virtual scenario in relation to the real environment, and this according to the profile and the point of view of the user.

[0028] According to a characteristic of the invention, the augmented reality device is capable of carrying out an angular orientation calibration of the virtual frame in which the three virtual markers are inscribed as a function of an orientation of the three real markers, the orientation calibration being carried out from the creation of a real reference point having as its origin one of the three real markers from which at least two intersecting real axes extend to each of the other two real markers, the augmented reality device being configured to correct an orientation of a virtual reference point as a function of the formed real reference point, said virtual reference point having as its origin one of the three virtual markers from which at least two intersecting virtual axes extend to each of the other two virtual markers.

[0029] Orientation calibration consists of adapting the orientation of the frame of the virtual scenario and the corresponding virtual markers and therefore the orientation of each of the elements forming part of the virtual scenario according to the orientation of the virtual markers, so that the modeled elements forming part of the virtual scenario are oriented in a consistent manner with respect to a user's point of view.

[0030] Thus, the angular orientation calibration consists of mapping as efficiently as possible the angular orientation of the frame in which the virtual markers are inscribed on the angular orientation of the target element present in the real environment and defined by the orientation of the real markers.

[0031] The augmented reality device has in memory three virtual markers each with their previous position and orientation that these markers had during the previous use, or at the origin of the modeling of the model of the virtual scenario. The virtual scenario modeled by the augmented reality device, in its orientation and position defined by the virtual markers, is then projected onto the visualization means superimposed on the real environment, the augmented reality device being configured to adapt the display of the virtual scenario according to the movements of the user and / or his field of vision. The calibration operation allows the initial position and orientation of the virtual marker to be modified, by matching virtual markers and real markers, so that when the augmented reality device is used again, the virtual scenario can be optimally superimposed regardless of the user's movements. The display of the virtual scenario in the display module, superimposed on the real environment, is then carried out on the basis of the modeling of the virtual scenario.

[0032] It should be noted, as previously mentioned, that the virtual scenario is formed from virtual elements, or study elements, added to the model produced by scanning the real environment or the target element. The virtual model thus has no graphic representation. It simply ensures consistency in the superposition of the virtual scenario on the real environment in visualization.

[0033] A reference frame is defined comprising two axes for both the three reference markers and the three virtual markers. The two axes have as their origin one of the three markers which is then identified as the origin marker, the two axes being formed by the straight lines joining the origin marker to each of the two other corresponding markers. The two axes forming the reference frame, whether they are two virtual axes or two real axes, are intersecting with each other.

[0034] With respect to the real reference mark, the augmented reality device selects from among the three real markers one of the real markers as being the origin real marker. The origin real marker can be chosen arbitrarily or according to the user's point of view. The device then creates a real reference mark from the three real markers, the origin of the real reference mark being the origin real marker and the axes of the real reference mark extending, in a secant manner, from the origin real marker to each of the other two real markers.

[0035] A virtual reference frame is also formed, in the same way as for the real reference frame but this time considering the virtual markers inscribed in the framework of the virtual scenario as it is positioned and oriented at this instant in the display module. The virtual reference frame is formed by choosing in particular as the virtual origin marker the virtual marker associated with the chosen real origin marker. In other words, one of the virtual markers, and more particularly the virtual marker associated with the chosen real origin marker, serves as the origin of the virtual reference frame and is therefore selected as being the virtual origin marker, and each virtual axis of the reference frame extends, in a secant manner, from the virtual origin marker to one of the other virtual markers.

[0036] Once the two reference points have been determined, the virtual reference point is compared with the real reference point. More particularly, the angles formed by the pairs of associated axes are compared, i.e. on the one hand the angle formed between the first measured real axis and the first virtual axis as it is present in the context of the virtual scenario modeled at this stage in the display module, and on the other hand the angle formed between the second measured real axis and the second virtual axis. The virtual reference frame is then adjusted to present an orientation identical or at least as similar as possible to the real reference frame. The correction of the virtual reference frame is made by taking for example an average of the angular deviations, so as to compensate for this value representative of the angular deviations, by rotation of the frame of the virtual scenario corresponding to this virtual reference frame.

[0037] In other words, the angular orientation calibration makes it possible to modify the orientation of the virtual markers as a whole, and therefore the orientation of the corresponding virtual frame and consequently the orientation of the virtual scenario and the associated study elements, relative to the real markers and the user's vision of the real environment. This is how the virtual scenario is superimposed on the real environment in the most coherent and realistic way possible in the eyes of the user.

[0038] Thanks to the augmented reality device whose display module makes it possible to ensure such orientation calibration, the display of the virtual scenario is done more quickly and more precisely. In particular, the orientation calibration is not done point by point, by comparing the arrangement of each virtual marker with respect to its corresponding real marker and by taking a neutral reference frame, but by considering reference points formed by the real and virtual markers, without other reference frames, which thus facilitates the calculations carried out by the augmented reality device during the calibration operation.

[0039] According to a feature of the invention, the orientation calibration and / or the position calibration may be configured to be initiated by manual action of the user. The latter may for example initiate the calibration by pressing a button arranged at a display module. The user may for example focus on one of the real markers before launching the calibration.

[0040] According to a characteristic of the invention, the orientation calibration and / or the position calibration can be configured to be restarted automatically at regular intervals. The calibration can, for example, be done automatically when the augmented reality device is started.

[0041] The invention also covers an augmented reality calibration method implemented by an augmented reality device as described previously, comprising

[0042] - a first step of acquiring an orientation of the virtual markers by report to the actual markers by the display module,

[0043] - a second step of creating the real reference point from the three real markers,

[0044] - a third step of creating the virtual marker from the three markers virtual,

[0045] - a fourth step of comparing the virtual reference point and the real reference point,

[0046] - a fifth step of correcting the orientation of the virtual reference point.

[0047] The calibration method thus makes it possible to calibrate the orientation of the virtual scenario to be projected by the augmented reality device as described previously.

[0048] All of these steps, more particularly the second step and the following ones, may be preceded by a position calibration operation as described previously.

[0049] According to a characteristic of the invention, the method comprises a first intermediate step, prior to the second step, of acquiring the position of the three virtual markers relative to the real markers.

[0050] According to a characteristic of the invention, the method comprises a second intermediate step, subsequent to the first intermediate step and prior to the second step, of calculating a position offset between each of the virtual markers and their respective real marker.

[0051] According to a characteristic of the invention, the method comprises a third intermediate step, subsequent to the second intermediate step and prior to the second step, of calculating an average offset between the virtual markers and their respective real marker, said average offset being applied to modify the position of the frame of the virtual scenario in which the virtual markers are included.

[0052] Other characteristics and advantages of the invention will become apparent from the following description on the one hand, and from several exemplary embodiments given for informational and non-limiting purposes with reference to the appended schematic drawings on the other hand, in which:

[0053] [Fig-1] is a schematic representation of a user using a device augmented reality according to the invention,

[0054] [Fig.2] is a schematic representation of a real environment of the device of augmented reality according to the invention,

[0055] [Fig.3] is a schematic representation of a virtual scenario modeled by a display module of the augmented reality device and superimposed on the real environment,

[0056] [Fig.4] is a representation of a step of acquiring the position of markers real in the real environment for the purpose of carrying out a position calibration operation by comparing the position of these real markers with virtual markers,

[0057] [Fig.5] is a representation of a step of adjusting the position of a frame of the virtual scenario in which the virtual markers are placed in relation to the position of the real markers in the real environment,

[0058] [Fig.6] is a representation of a step of acquiring the angular orientation of the real markers for the purpose of carrying out an orientation calibration by comparing the angular orientation of these real markers with the virtual markers,

[0059] [Fig.7] is a representation of a step of correcting an orientation of the frame of the virtual scenario in which the virtual markers are located relative to the orientation of the real markers in the real environment.

[0060] [Fig. 1] schematically represents an augmented reality device 1 according to the invention. The augmented reality device 1 is, according to the example illustrated here, used in a test context related to the automotive field. For example, a user 30 can use the augmented reality device in order to have a virtual preview of a vehicle interior and for example to ensure that the various components intended to equip the interior are correctly positioned and are well suited to his desires. Subsequently, the augmented reality device will be described in relation to this example, but it should be noted that without departing from the context of the invention, the augmented reality device 1 can however be used in any context allowing a concrete and functional application of said augmented reality device 1.

[0061] In order to implement the augmented reality device 1, the user 30 may for example be installed on a seat 32 simulating a car seat, arranged in front of a real environment 31. The real environment 31 is an area on which the augmented reality device 1 is able to superimpose a virtual scenario. The real environment 31 comprises at least one target element on which maximum precision is desired in visualization. The real environment 31 may for example be a model made of various materials, more or less faithful to a cockpit equipping a passenger compartment of the vehicle according to the example stated previously, and the target element(s) may consist of equipment of this cockpit. It could also be envisaged without departing from the context of the invention that the entire real environment is considered as a target element.

[0062] A virtual model of the real environment or said target element is defined digitally and stored in the memory of a control and calculation unit of the augmented reality device. This virtual model comprises study elements defined digitally, with a finer definition than that with which the rest of the model is digitized. In other words, the virtual model can be considered as a frame in which the study elements are inserted. The virtual model, or the frame of the virtual scenario to be projected superimposed on the real environment, can be formed by a rough digital representation of the real environment obtained by scanning for example and on which the finely digitized study elements are precisely arranged. Thus the virtual model does not require a complete digital definition of the real environment, detailed and expensive. It only forms the virtual framework in which the digital study elements are arranged.

[0063] According to the invention, the virtual model, that is to say the frame which delimits it, is represented by three virtual reference markers. Advantageously, the three markers make it possible to define the position and orientation of the frame of the virtual model. In order for the augmented reality device to be able to superimpose the virtual frame on the real environment 31, the device according to the invention comprises real reference points or markers positioned on the real environment and on which the virtual markers of the virtual model coincide.

[0064] The object of the invention is to optimally superimpose said study elements on the real target element(s) of the real environment by optimally superimposing the virtual model of the target element on the real environment. On the virtual model, virtual objects can then be added to form the virtual scenario which will be superimposed on the user's field of vision. In other words, by virtual scenario, we understand a frame in which virtual objects, or study elements, are inscribed, and whose angular orientation and position are defined by the angular orientation and position of the virtual markers associated with the frame.

[0065] Thanks to the augmented reality device 1, from the observation of the real environment 31, the user 30 can thus see for example in a virtual manner an arrangement for a vehicle interior. For this purpose, the augmented reality device comprises at least one display module 33 and capture means, such as a camera 34, the interaction of which allows the display of a virtual scenario superimposed on the real environment.

[0066] The display module 33 comprises a substantially transparent viewing means through which the user 30 is able to see the real environment and on which the user will see virtual objects appear, modeled by calculation means of the augmented reality device 1 and which will be superimposed on the real environment.

[0067] The display module 33 may for example correspond to a pair of glasses worn by the user 30, the optical surface of the glasses forming the viewing means of the display module. It is only through the display module 33 that the user 30 can observe the augmented reality generated by the augmented reality device 1. The display module may integrate the camera 34 as well as a device for locating the user's eyes, making it possible at least to determine in which direction the user is looking.

[0068] The camera 34 makes it possible to film objects in the user's field of vision, and the eye location device makes it possible to identify eye movements and to deduce the user's field of vision therefrom.

[0069] As a reminder, a user's field of vision is the part of the space that the user's eyes can perceive around them.

[0070] In the augmented reality device, the display module, and for example the glasses worn by the user, make it possible to modify the field of vision of this user by adding information which is, according to the invention, virtual objects.

[0071] The display module may also comprise an accelerometer and / or a motion detector to improve the display quality of the virtual scenario, i.e. the modeled images, by the augmented reality device 1, and therefore to improve the impression of immersion felt by the user 30. The accelerometer makes it possible in particular to recover data on the movement of the user's head and adapt the display of the virtual image to this movement.

[0072] In order to ensure the most coherent superposition possible of the virtual scenario on the real environment 31, the augmented reality device 1 comprises calibration means including a plurality of markers, both real markers arranged on the real environment and virtual markers arranged within the framework of the virtual scenario.

[0073] The augmented reality device 1 is configured to perform at least one specific calibration operation as will be described below. The calibration means comprise in particular virtual markers associated with the virtual image as it is modeled in the display module, as well as a plurality of real markers which is arranged on the real environment 31, for example here three markers among which a first real marker 2, a second real marker 3 and a third real marker 4. By real marker is meant a material element directly arranged on the real environment 31, for example here stuck on a model of a cockpit forming the real environment.

[0074] The camera of the augmented reality device, here embedded on the display module, worn by the user associated with a calculation and control unit not shown, is capable of locating the real markers arranged in the user's field of vision, that is to say in the real environment, and of estimating or determining the positions and orientations of each of said real markers.

[0075] The augmented reality device must comprise at least three real markers arranged on the real environment, but it will be understood that their number may be greater than the number of three in the illustrated example. The real markers may be of various types, as long as they are capable of participating in the implementation of the principle of augmented reality. For example, the real markers may for example be a QR type barcode. The real markers are for example stuck on the real environment 31.

[0076] Within the display module 33, the camera 34 and the control and calculation unit can communicate with each other wired or wirelessly, for example via Bluetooth®.

[0077] The calibration operation of the augmented reality device 1 as previously mentioned is carried out via the camera and the control and calculation unit of the display module from the three real markers arranged on the real environment 31 and from the associated virtual markers, and the calibration consists, during the calibration operation, in matching a position and an orientation of the virtual frame, in which each of these virtual markers are inscribed, with respect to the corresponding real markers.

[0078] [Fig. 2] is another view of the real environment 31 in the absence of the user. Thus, it is possible to see that the first real marker 2 can for example be arranged on the real environment 31, facing the seat 32. The second real marker 3 is centered relative to the real environment 31 and is therefore placed to the right of the first real marker 2. The third real marker 4 is arranged on the real environment 31, to the right of the second real marker 3 and at an opposite end of the first real marker 2.

[0079] Furthermore, the real environment 31 has a plurality of target elements for which maximum precision is desired in visualization. These target elements may for example consist of dummy elements, for example made of cardboard or of the same material as the real environment 31. By way of example, and as shown in [Fig. 2], the real environment 31 comprises a target steering wheel 41 and a target dashboard 42. The real environment 31 also comprises a console 35 placed to the right of the seat 32 and substantially at the same level as a seat of the latter. The console 35 comprises a target gear lever 43. The real environment 31 may also comprise a target central rearview mirror 44 overlooking the real environment 31. These target elements are listed here in a non-exhaustive manner and they make it possible to give a physical dimension to the virtual study elements which will be projected in superposition via the augmented reality device.Of course, the number of target elements is also not limiting of the invention and the real environment could consist of a single target element.

[0080] The real environment 31 as represented in [Fig.2] corresponds to the environment observed by the user before he is equipped with the display module of the augmented reality device.

[0081] When the user equips himself with the display module and observes the real environment 31, the virtual scenario 45 mentioned previously is superimposed on the real environment, the virtual scenario 45 being schematically illustrated in [Fig.3].

[0082] In the example illustrated, a virtual steering wheel 37 is superimposed on the target steering wheel illustrated in [Fig.2] when the user sits on the seat 32 while being equipped with the module display. The same applies to a virtual dashboard 38 which is superimposed on the target dashboard. It is a virtual gear lever 39 which is superimposed on the target gear lever, the latter being arranged on the support 35 near the seat 32 as shown in [Fig.2]. Finally, a virtual rearview mirror 40 is superimposed on the target rearview mirror. These are four non-exhaustive examples, the augmented reality device being able to be configured with any modeling of target element present within the real environment, here the passenger compartment of a vehicle. In order to optimize the performance of the augmented reality device and not to tire the user's vision, only the target element is modeled in the virtual scenario superimposed on the real environment, the other elements appearing as they are through the display module.

[0083] In order to be able to superimpose the virtual scenario 45 on the real environment in an appropriate manner, that is to say with the modeled elements of the virtual scenario which are correctly arranged on the corresponding target element, the augmented reality device takes into account the modeling of the virtual scenario originally made or modified following the previous calibration operation, and chooses the shape of the virtual scenario to be projected as a function of the position of the user detected by the camera and / or as a function of the detection of the orientation of the user's gaze.

[0084] It should be noted that the virtual scenario may have been modified, moved relative to the target element during a previous use, in accordance with the reality of the viewing angle of a previous user.

[0085] In order to make the virtual scenario 45 appear as cleanly as possible, and therefore to ensure that the modeled elements forming the virtual scenario are projected into the display module in accordance with the reality of the viewing angle of a given user, the augmented reality device is configured to allow a calibration operation, during which a position and / or an orientation of the virtual scenario is adjusted by comparing the position and / or the orientation of the virtual markers of the virtual scenario with the real markers fixed on the real environment. This calibration operation makes it possible to carry out a calculation modifying the position and / or the orientation of the virtual scenario and of the modeled elements that this scenario contains, so that the superposition of the virtual scenario on the real environment which is subsequently carried out is as coherent and as realistic as possible.

[0086] The details concerning the calibration of the augmented reality device will be explained later, with reference to an example in which the augmented reality device is able to compare the position of three virtual markers with respect to the three real markers arranged on the real environment 31 and previously described, with a first virtual marker 5, a second virtual marker 6 and a third virtual marker 7 which it is sought to superimpose respectively on the first marker real, to the second real marker and to the third real marker.

[0087] It is understood from what has just been described that such an augmented reality device has several applications in the automotive field. As described previously, it can, for example, allow a buyer to select a vehicle model without the latter being available. The buyer can also choose from customizable elements within said vehicle. Such an augmented reality device can also be used by engineers working in the automotive field, for the purpose of verifying the positioning of the various elements in a passenger compartment during the design of a vehicle, for example.

[0088] In light of what has been mentioned, the calibration operation takes on its full meaning insofar as the augmented reality device must adapt the configuration of the virtual scenario to be projected according to the users of said device, for example two successive users who are not the same size and / or who sit in such a way as to present a position at a distance from the steering wheel that is different from one user to the other. For a perfect superposition of the virtual scenario on the real environment, the virtual markers must have a relative orientation and position between them that are exactly the same as those between the real reference markers. The calibration operation aims to optimize this similarity of positioning and orientation of the sets of markers, by modifying when necessary the position and orientation of the virtual frame associated with the virtual markers.It should be noted that in the event of adjustment during the calibration operation, the frame in which the virtual markers are inscribed is not structurally modified, but it can be moved in translation or rotation so that all the study elements modeled finely in this frame of the virtual scenario are moved accordingly.

[0089] Therefore, in order to calibrate the virtual scenario 45 so that it can be adapted to each of the users, the augmented reality device according to the invention is configured for a specific angular position and orientation calibration, said angular orientation calibration being particular in that it involves a comparison of axis markers formed respectively from the real markers and the virtual markers.

[0090] In order to carry out the calibration operation, the user observes the real environment while being equipped with the display module. The user can optionally focus on one of the real markers of the real environment. The calibration operation takes place when the augmented reality device is started, that is to say before any display of the virtual scenario superimposed on the real environment. The calibration operation can for example be initiated by manually pressing a button by the user, or automatically when the augmented reality device is started.

[0091] We will now describe in more detail the calibration operation of the augmented reality device.

[0092] Figures 4 and 5 are schematic representations of two successive steps of a position calibration operation. It is possible to observe in these two figures a positioning of the three real markers of the augmented reality device as they were mentioned previously, namely incorporated for example by gluing in the real environment, and of three virtual markers forming the frame of the virtual scenario to be superimposed on said real markers of the real environment. In both [Fig.4] and [Fig.5], the real markers are represented in solid lines while the virtual markers are represented in dotted lines.

[0093] According to an arbitrary arrangement and independent of what is illustrated in the preceding figures, the three real markers are arranged so that the first real marker 2 is interposed between the second real marker 3 and the third real marker 4. More particularly, in [Fig.4], the second real marker 3 is to the left of the first real marker 2 while the third real marker 4 is to the right of the first real marker 2.

[0094] During the calibration operation, the augmented reality device makes it possible in particular to correct a position defect which may occur, for example, at the level of the second virtual marker 6 relative to the second real marker 3 and / or the third virtual marker 7 relative to the third real marker 4 when the user focuses his gaze particularly on the first real marker 2. In other words, the position calibration consists of a verification of the position of each of the virtual markers relative to its corresponding real marker and an adjustment of the position of the frame of the virtual scenario to be projected formed by all of the virtual markers, so that when using the augmented reality device after calibration, the position of the frame formed by the virtual markers is as consistent as possible with respect to the real markers and the target element.

[0095] [Fig. 4] here represents a step of the position calibration during which the position of each real marker and each virtual marker is acquired when the virtual scenario is superimposed on the real environment. In [Fig. 4], the real markers are represented as they appear in the user's field of vision, and therefore as they are detected by the capture means of the display module directed towards this field of vision. The second real marker 3 and the third real marker 4 are thus inclined in order to take into account the deformation of the objects present on the periphery of the user's field of vision when the latter looks directly at the first real marker 2.

[0096] The position calibration aimed at adjusting the position of the frame forming the virtual scenario is done first of all by choosing one of the real markers as a reference, know here the first real marker 2 according to the example of [Fig.4]. The position calibration can nevertheless be carried out advantageously by taking any of the markers as a position reference, in particular depending on which marker is in the user's field of vision. The reference marker can also be chosen arbitrarily by the augmented reality device.

[0097] The position calibration continues by determining the position offset for each real marker - virtual marker pair. According to an operating mode of the invention, this determination of the offset is carried out after having superimposed one of the virtual markers on the corresponding real marker. In the example illustrated in [Fig.4], the first real marker 2 being chosen as the position reference, the position of the first real marker 2 and the position of the first virtual marker 5 are merged.

[0098] From the position of the first real marker 2 and therefore of the corresponding first virtual marker 5, the augmented reality device determines a position of the second virtual marker 6 and a position of the third virtual marker 7 relative respectively to the second real marker 3 and to the third real marker 4.

[0099] If the augmented reality device determines that the superposition of each virtual marker on the corresponding real marker is perfect, the control and calculation unit does not make any modification to the position and orientation of the virtual frame.

[0100] Otherwise, the calibration operation comprises a modification of the position and orientation of the virtual frame. In the example illustrated in [Fig.4], the positions of the second virtual marker 6 and the third virtual marker 7 are respectively offset relative to the position of the second real marker 3 and the third real marker 4. The role of the position calibration is therefore to correct, as much as possible, as explained below, this offset between the frame of the virtual image and the real environment on which the virtual image must be superimposed.

[0101] The calculation of the position offsets between a virtual marker and the corresponding real marker is done from a precise point of each of the markers, for example the center of the markers or one of the corners of the markers, in particular when these markers are in the form of a QR code, each corner having to be the same for each marker.

[0102] According to the example of [Fig.4], it is possible to see that the second virtual marker 6 is slightly offset relative to the second real marker 3 according to a first vertical component VI and a first horizontal component HL. The third virtual marker 7 has a significant offset relative to the third real marker 4, according to a second vertical component V2 and a second horizontal component H2.

[0103] The position calibration continues during a second intermediate step, as shown in [Fig. 5], during which, in accordance with the objective of the position calibration by the augmented reality device, the position of all the virtual markers, i.e. the frame of the virtual scenario, is modified according to the previously determined offsets, in order to obtain a virtual scenario that is consistent with the positioning of the user using the augmented reality device.

[0104] Preferably, the average of the offsets of the three markers is taken and all three markers, i.e. the frame of the virtual scenario, is moved to compensate for this average offset.

[0105] The average offset between the virtual frame in which the virtual markers are inscribed and the set formed by the real markers can be calculated by taking an average of all the offsets determined previously, by successively considering the horizontal and vertical components of the offsets. The sum of the vertical components of all the offsets is carried out then divided by the number of real markers involved in the calculation in order to find the vertical component of the average offset. And the sum of the horizontal components of all the offsets is carried out then divided by the number of real markers involved in the calculation in order to find the horizontal component of the average offset.

[0106] Applied to the example of [Fig.5], the average offset is calculated and applied inversely to compensate for the poor positioning of the virtual scenario relative to the real environment. The frame of the virtual scenario, and therefore all the virtual markers, is thus offset relative to the real markers according to a single vertical component of value V0, equal to the inverse of the average offset calculated on the basis of the vertical components VI and V2, and according to a single horizontal component H0, equal to the inverse of the average offset calculated on the basis of the horizontal components H1 and H2. At the end of the position calibration operation, all the virtual markers are offset, in one piece, relative to all the real markers.

[0107] We will now describe the specific angular orientation calibration operation that the augmented reality device according to the invention is able to implement, in particular with reference to Figures 6 and 7. The angular orientation calibration can be carried out simultaneously with the position calibration or after it. The objective of the angular orientation calibration is to correct the orientation of the frame of the virtual scenario, and therefore of the study element(s) whose orientation is defined by that of the frame, relative to the real environment and the field of vision of the user of the augmented reality device.

[0108] To return to the example previously described, a new user will start the augmented reality device from a position which may be different from that of the previous user, so that by fixing an area of ​​the real environment, the user will perceive the target element(s) on the periphery of his field of vision with a certain inclination which will be specific to him and which will differ from the perception that the previous user had, for example due to his morphology and the distance at which he is from these target elements. For an optimal augmented reality experience, the study elements modeled within the virtual scenario should have a relative inclination which is as close as possible to that of the target elements on which they are to be superimposed.

[0109] Once the angular orientation calibration has been carried out, and where appropriate the position calibration carried out, the augmented reality device is configured to consider the positioning of the user in the real environment, via the capture means in particular, and to consider the viewing angle defined by the orientation of the user's eyes through the display module, to determine a good orientation of each of the virtual elements to be projected onto the real environment via the display module.

[0110] In accordance with what has been described previously, [Fig. 6] here represents a step of the orientation calibration during which the orientation of each real marker and each virtual marker is acquired when the virtual scenario is superimposed on the real environment. In [Fig. 4], the real markers are represented as they appear in the user's field of vision, and therefore as they are detected by the capture means of the display module directed towards this field of vision. The second real marker 3 and the third real marker 4 are thus inclined in order to take into account the deformation of the objects present on the periphery of the user's field of vision when the user looks directly at the first real marker 2.

[0111] The angular orientation calibration according to the invention is carried out by comparing a virtual reference frame 20 defined by the three virtual markers and a real reference frame 10 defined by the three real markers described previously. Such a technique is more precise and faster than carrying out a calculation of comparison of virtual and real reference frames for each of the pairs of markers formed by a reference marker and a virtual marker.

[0112] For this, just as during the position calibration, one of the real markers is used as the original real marker to serve as the origin of the real reference frame 10. The acquisition of the orientation of the markers for the angular orientation calibration can be done simultaneously with the acquisition of the positions of the markers for the position calibration. In [Fig.6], it is the first real marker 2 which serves as the origin for the real reference frame 10. As was previously described for the position calibration, the real marker serving as the origin can for example be the one on which the user's gaze is focused, or can be chosen arbitrarily.

[0113] The augmented reality device is configured to define the real reference point 10 with a first real axis 11 and a second real axis 12 originating respectively from the origin real marker, forming the origin of the real reference frame. The first real axis 11 is formed between the origin real marker, here the first real marker 2, and one of the two other real markers, here for example the second real marker 3. The second real axis 12 is formed between the origin real marker, here the first real marker 2, and the third real marker 4.

[0114] Preferably, the two real axes forming the real reference frame are intersecting or form intersecting directions.

[0115] The second virtual marker 6 and the third virtual marker 7 associated with the virtual scenario as configured before calibration are visible in [Fig.6].

[0116] As stated previously, the orientation of the second virtual marker 6, when starting the augmented reality device, corresponds to the orientation of the second real marker 3 when the user fixes the first real marker 2, such that the virtual image was calculated during a previous use of the augmented reality device. When changing users of the augmented reality device, the visual perception of the new user may differ from that of the previous user and this influences the orientation of the real markers located on the periphery of the field of vision of the new user. Therefore, the orientation of the frame of the virtual scenario in which the virtual markers are located must be adjusted so that the virtual scenario presents an overall orientation consistent with the visual perception of the user.Thus, due to the user profile, there may be a difference in orientation between the second real marker 3 and the second virtual marker 6, when the orientation of the latter is not yet modified by the calibration operation. The reasoning is the same for the third virtual marker 7 compared to the third real marker 4.

[0117] After having determined the real reference point 10, as a function of the visual perception of the real markers by the user, and therefore as a function of the acquisition of the position of the real markers by the camera and the calculation of the vectors formed between the real markers by the control and calculation unit, the augmented reality device is configured to take into account a virtual reference point 20, then to compare the real reference point 10 and the virtual reference point 20.

[0118] The virtual reference mark 20 corresponds to a reference mark consisting of a first virtual axis 21 and a second virtual axis 22. The first virtual axis 21 is formed between the first virtual marker 5, which, just as during the position calibration step, is merged with the first real marker 2 and forms the virtual origin mark, and the second virtual marker 6. The second virtual axis 22, for its part, is formed between the first virtual marker 5 and the third virtual marker 7. Preferably, the two virtual axes forming the virtual reference mark are intersecting or form intersecting directions.

[0119] In the illustrated example, each of the axes, whether real or virtual, extends from the center of a marker to the center of another marker, said markers being able to be real or virtual. Just as was described during the position calibration, it will be possible, without departing from the context of the invention, to choose reference points on the markers which are other than the center of these and for example corners of a QR code, since the same reference point is used to define the position of each marker and the vector of each virtual or real axis determined on the basis of the position of two markers.

[0120] The comparison of the real reference point and the virtual reference point thus determined has the objective of reorienting the virtual reference point 20, and therefore of reorienting all of the virtual markers forming the framework of the virtual scenario in relation to the previous situation, whether this previous situation is initial or following a previous calibration by another user.

[0121] The orientation of the frame of the virtual scenario is adjusted with respect to an angular offset value between the real reference frame 10 and the virtual reference frame 20. Such an angular offset value is manifested by a first angular difference 51 between the first real axis 11 and the first virtual axis 21, and by a second angular difference 52 between the second real axis 12 and the second virtual axis 22, as illustrated in [Fig.6].

[0122] The reality device can then cause the rotation of the virtual reference frame 20, and therefore of the virtual model, that is to say the frame of the virtual scenario, in order to reduce these angular differences and thus to jointly adjust the orientation of the virtual markers so that they are in an orientation consistent with that of their corresponding real marker. Preferably, the angle of rotation of the effective virtual reference frame can be the average of the angular offset values ​​calculated above. The virtual reference frame can then be rotated as a whole by a value inverse to that of the rotation angle mentioned above. As illustrated in [Fig.7], after this adjustment of the orientation of the frame, angular offsets may remain, if necessary with an inverted direction of offset, but the corresponding absolute values ​​are minimal compared to those of the angular offsets present before calibration.Here, an attenuated angular difference 53 between the first real axis 11 and the first virtual axis 21 is equal to the value of the first angular difference 51 reduced by the previously mentioned rotation angle, and an attenuated angular difference 54 between the second real axis 12 and the second virtual axis 22 is equal to the value of the first angular difference 51 reduced by the previously mentioned rotation angle.

[0123] In the case where several virtual markers are adjusted to conform to their respective real marker, for example if there are more than three markers on the environment real, an average rotation angle is calculated from all the angular offset values ​​between each of the corresponding real and virtual reference frames, and therefore from the angular offset values ​​between the real and virtual axes.

[0124] The angular orientation calibration is thus carried out in such a way as to make the study elements of the virtual scenario appear, the framework of which is defined by the virtual markers, according to a position and an orientation as faithful as possible to the position and the orientation of the real markers as they are perceived by the user according to his position in relation to the real environment.

[0125] The modification of the position and / or orientation of the frame of the virtual scenario in which the virtual markers are inscribed can be done automatically when the user marks a pause, for example on a particular real marker. The modification of the arrangement of this frame, and therefore of the virtual scenario and the study elements that it contains, can also be done manually, for example by pressing a button arranged on the display module, once the user is ready. The reference marker can be any of the real markers and each marker can take the role of reference marker during the calibration operation, the relative position of the virtual markers being all the more precise.

[0126] Of course, the invention is not limited to the examples which have just been described and numerous adjustments can be made to these examples without departing from the scope of the invention.

[0127] The invention, as just described, achieves the aim it set itself, and makes it possible to propose an augmented reality device equipped with means for calibrating position and orientation from at least three real markers present in the real environment in a number equivalent to that of the virtual markers associated with the framework of a virtual scenario. Variants not described here could be implemented without departing from the context of the invention, provided that, in accordance with the invention, they comprise an augmented reality device in accordance with the invention.

Claims

Claims

1. Augmented reality device (1) comprising a real environment (31) and a display module (33) for viewing a virtual scenario superimposed on a field of vision of a user, wherein the display module (33) comprises a substantially transparent viewing means on which the display module is capable of viewing said virtual scenario, the display module (33) being configured to model and define the virtual scenario within a frame, characterized in that the augmented reality device (1) comprises at least three real markers (2, 3, 4) arranged within the real environment (31) and at least three virtual markers (5, 6, 7) associated with the frame defining the virtual scenario, each of the three virtual markers (5, 6, 7) being intended to coincide with one of the three real markers (2, 3, 4),the augmented reality device (1) being capable of carrying out an angular orientation calibration of the virtual frame in which the three virtual markers (5, 6, 7) are inscribed as a function of an orientation of the three real markers (2, 3, 4), the orientation calibration being carried out from the creation of a real reference frame (10) having as its origin one of the three real markers (2, 3, 4) from which at least two real axes (11, 12) intersecting extend to each of the two other real markers, the augmented reality device (1) being configured to correct an orientation of a virtual reference frame (20) as a function of the real reference frame (10) formed, said virtual reference frame (20) having as its origin one of the three virtual markers (5, 6, 7) from which at least two virtual axes (21, 22) intersecting extend to each of the other two virtual markers.,

2. Augmented reality device (1) according to claim 1, characterized in that it comprises capture means comprising a camera (34) directed towards said field of vision of the user.

3. Augmented reality device (1) according to one of the preceding claims, characterized in that it is capable of carrying out a position calibration of the frame of the virtual scenario in which the three virtual markers (5, 6, 7) are inscribed as a function of a position of the three real markers (2, 3, 4), the augmented reality device being capable, during this position calibration, of determining an offset between the position of each virtual marker (5, 6, 7) and the position of the real marker (2, 3, 4) corresponding to said virtual marker (5, 6, 7), and modifying the position of the frame of the virtual scenario in which the virtual markers (5, 6, 7) are inscribed as a function of said offset.

4. Augmented reality device (1) according to the preceding claim, characterized in that it is capable of modifying the position of the frame of the virtual scenario according to an average offset measured from each of the position offsets determined between the virtual markers (5, 6, 7) and their corresponding real marker (2, 3, 4).

5. Augmented reality device (1) according to one of claims 3 to 4, wherein the orientation calibration and / or the position calibration are configured to be initiated by manual action of the user (30).

6. Augmented reality calibration method implemented by an augmented reality device (1) according to any one of the preceding claims, comprising: - a first step of acquiring an orientation of the virtual markers (5, 6, 7) relative to the real markers (2, 3, 4) by the display module (33), - a second step of creating the real marker (10) from the three real markers (2, 3, 4), - a third step of creating the virtual marker (20) from the three virtual markers (5, 6, 7), - a fourth step of comparing the virtual marker (20) and the real marker (10), - a fifth step of correcting the orientation of the virtual marker (20).

7. Augmented reality calibration method according to the preceding claim, comprising a first intermediate step, prior to the second step, of acquiring the position of the three virtual markers (5, 6, 7) relative to the real markers (2, 3, 4).

8. Augmented reality calibration method according to the preceding claim, comprising a second intermediate step, subsequent to the first intermediate step and prior to the second step, of determining an offset between each of the virtual markers (5, 6, 7) and their respective real marker (2, 3, 4).

9. Augmented reality calibration method according to the preceding claim, comprising a third intermediate step, subsequent to the second intermediate step and prior to the second step, of calculating an average offset between the virtual markers (5, 6, 7) and their respective real marker (2, 3, 4), said average offset being applied to modify the position of the frame of the virtual scenario in which the virtual markers (5, 6, 7) are located.

Images