Apparatus and method for generating dynamic virtual content in mixed reality

Abstract

Dynamic virtual content to be overlaid to a representation of a real 3D scene conforms to a scenario that is defined before run-time and involves real-world constraints (23). Real-world information is captured in the real 3D scene (22), and the scenario is executed at run-time in the presence of the real-world constraints (14). When real-world constraints are not identified (12) from the real-world information, a transformation of the representation of the real 3D scene to a virtual adapted 3D scene is performed (13) before executing the scenario, such that the virtual adapted 3D scene complies with those constraints, and the scenario is executed in the virtual adapted 3D scene instead of the real 3D scene. Application of mixed reality.

Description

[0001] This application is a divisional application of patent application No. 201780080787.9, filed on December 19, 2017, entitled "Apparatus and Method for Generating Dynamic Virtual Content in Mixed Reality". Technical Field

[0002] This invention relates to the field of mixed reality (MR), in which computer-generated (CG) content is not merely overlaid on top of a real-world view, but truly interacts with the real-world environment, enabling a seamless fusion of digital and physical reality. Background Technology

[0003] Augmented reality (AR) applications refer to a live view of a real-world environment where elements are enhanced by CG content (such as video or graphics). In MR, the CG content also interacts with the real-world environment.

[0004] In AR / MR applications, visual CG content is inserted into a real-world view at a given location. This location is typically defined relative to a predefined visual 2D / 3D pattern (for example, in industrial applications), predetermined by spatial coordinates (for example, in outdoor positioning-based applications) or screen coordinates (for example, in Google Glass applications), or manually selected by the user (for example, in interactive applications).

[0005] However, in advanced applications, virtual scenes may contain pre-animated content. This ranges from animations based on a single plane and with a reduced footprint (e.g., a character jumping in place or a dancer spinning) to complex scenario-built animations based on different planes involving several characters and / or objects with possible interactions and complex trajectories (e.g., a character moves from an initial position to a second position on a plane, grabs an object in the second position, then changes direction to hide that object behind another object, then changes direction again and suddenly falls into a hole).

[0006] For a good user experience, immersion, and realism, such complex virtual animation content should interact with the real environment in space: the plane on which the character walks can be a real flat surface, which should be large enough and have enough free space, such as a table; the occluding objects can be real objects on the table; the holes can correspond to the boundaries of the table, and so on.

[0007] The creation of animated CG content by artists is based on strong assumptions and constraints regarding the geometry of the real environment and the presence or absence of realistic elements. However, without controlling the real environment, effectively presenting such complex MR animation becomes problematic. For example, the horizontal space may be much smaller or larger than expected, and the horizontal plane may have many objects on it, leaving insufficient free space to perform animation without geometric aberrations (such as lack of occlusion). This situation may also lead to questioning the semantics of the animation itself.

[0008] Known solutions include adapting virtual animations to the real environment. This can be done in two steps: upstream when preparing the virtual animation content, by parametrically configuring the animation; and in a later preprocessing step, by scanning the real environment and analyzing its layout to calculate the optimal positioning, orientation, and scale to be applied to the virtual content.

[0009] In Daqri LLC's patent application US2015 / 0187108, a flexible augmented reality solution was developed in this regard, in which augmented reality content is adapted to changes in the real-world spatial geometry of the local environment. More precisely, the device can identify physical changes in the real-world spatial geometry based on the mapped real-world spatial geometry and modify virtual objects in response to those physical changes. Such modifications can relate to the size, shape, positioning, or behavior of the virtual objects.

[0010] However, such existing solutions do not guarantee that the real-world environment contains the minimum requirements sufficient for effective animation rendering. For example, there may be a lack of a horizontal plane for the virtual character to walk on, or real objects may obscure virtual objects that the character is grabbing.

[0011] Furthermore, as proposed in US2015 / 0187108, dynamically adapting virtual content may be detrimental to the desired scenario. Additionally, providing sufficiently flexible scenarios to adapt to a wide range of real-world situations may require significant storage and computing resources. Summary of the Invention

[0012] The purpose of this disclosure is to advantageously enable flexible adaptation of MR scenarios to a wide range of real-world situations without requiring a large amount of additional storage or computing resources.

[0013] In a preferred embodiment, multiple scenarios with various constraints can therefore be executed in real time based on a real environment.

[0014] In this regard, the purpose of this disclosure is particularly a device for generating at least one dynamic virtual content for a representation to be overlaid onto a real 3D scene, the dynamic virtual content conforming to at least one scenario defined prior to the runtime of the scenario and relating to at least one real-world constraint. The device includes:

[0015] - At least one input terminal, adapted to receive real-world information captured in a real 3D scene, and

[0016] - At least one processor is configured to execute a scenario at runtime in the presence of real-world constraints identified from real-world information.

[0017] According to this disclosure, at least one processor is further configured to, when at least one real-world constraint is not identified from real-world information, perform a transformation from a representation of a real 3D scene to a virtual adapted 3D scene before executing the scenario, such that the virtual adapted 3D scene conforms to the constraint, and execute the scenario in the virtual adapted 3D scene instead of executing the scenario in the real 3D scene.

[0018] Therefore, the device is advantageously configured to function as if the information obtained about the virtual adapted 3D scene were information about the actual, real 3D scene, resulting in a kind of deliberate system misdirection. That is, the planned scenario can be advantageously executed normally, conforming to some preliminary virtual patches applied to the representation of the real world.

[0019] It can be noted that MR scenarios may have been designed upstream without any accurate knowledge of the environment, or for a large number of different environments (e.g., for a television set and the table on which it is placed) without much accuracy. Then, it is advantageous to adapt and implement them locally at the operating site to make those scenarios operational.

[0020] In the following text, references will be made to “adapted” or “complete” environments or real-world scenes, or “moving” or “transformed” real-world objects. For convenience and brevity, these expressions should be understood to mean that the real-world 3D scene undergoes relevant transformations in its representation before the scenario is executed through virtual effects. These virtual effects may in particular include additions, reductions, and modifications to color or texture. More generally, they include any alteration in the user’s perception of the real-world 3D scene resulting from actions performed on the representation of reality.

[0021] Such representation also refers to any kind of representation to the user, which can be specifically through transparent or translucent glasses (direct viewing) or through the mediation of a camera that captures images of a real 3D scene that are visible to the user on a screen (indirect viewing). Those techniques will not be developed here, as they are well known to those skilled in the art.

[0022] A scenario is defined as CG content animation pre-designed by the creator.

[0023] Preferably, the representation of the real 3D scene and the representation of the virtual adapted 3D scene are executed in real time.

[0024] Contrary to existing solutions, and to the surprise of those who have adapted virtual worlds to the real world, this device artificially introduces virtual modifications as if applied to the real world to satisfy virtual scenario constraints. While the applied transformations actually remain within the virtual world's domain, they are introduced before executing a given scenario to satisfy constraints typically associated with the real world.

[0025] In this embodiment, a transformation of the representation of the real 3D scene is performed during an initial adjustment phase, prior to any scenario runtime.

[0026] In another embodiment, the transformation of the representation is performed during the runtime of the preceding scenario and anticipates one or more subsequent scenarios. In this regard, it is possible to consider different consecutive portions of a given scenario, comprising two or more scenarios, as different scenarios, provided that each of those consecutive portions is associated with at least one corresponding real-world constraint.

[0027] In the enhanced version, the transition from one part of a scenario to another is triggered by changes in the real 3D scene (e.g., removing a real object from the environment). This even advantageously includes cases where the scenario comprises similar features and real-world constraints in both parts, but at least one of these constraints depends on features of the real 3D scene modified during runtime. This change can then be considered, as previously disclosed, by satisfying the relevant constraints in a virtual adapted 3D scene within the framework of MR processing, by tricking the perception of the real 3D scene. The part of the scenario that follows the change is then considered itself the "scenario".

[0028] Real-world information captured in a real 3D scene advantageously includes point clouds or 3D models obtained from the scanned environment. Furthermore, at least one processor is advantageously configured to determine environmental characteristics, such as, in particular, identifying some or all of the objects present in the real 3D scene and their type, location, size, and shape, as well as free flat regions. It is also advantageously configured to analyze MR application requirements (real-world constraints, which may in particular involve elements or free flat regions) and check whether the environment complies with those requirements, even if only partially.

[0029] In addition, at least one processor is advantageously configured to update the MR application to be executed or currently being executed by utilizing a virtual adapted 3D scene that replaces the real 3D scene.

[0030] In a preferred implementation, at least one real-world constraint is at least two and may be interactive, and at least one processor is also configured to jointly perform transformations against those real-world constraints, such that the virtual-adapted 3D scene conforms to those constraints.

[0031] In some related embodiments, the joint transformation is then iteratively implemented through several successive attempts involving different transformation terms until two or more constraints are satisfied.

[0032] In another implementation, at least one real-world constraint is at least two, possibly interactive, and ranked according to importance. At least one processor is also configured to preferentially perform transformations on constraints having at least one highest priority ranking, and to execute scenarios at runtime in the absence of at least one real-world constraint different from the real-world constraint having the highest priority ranking.

[0033] Therefore, some constraints can be satisfied while others are not, provided that the latter are considered less important in the execution of the relevant scenario.

[0034] According to the preferred embodiment, either alone or in any suitable combination:

[0035] - At least one constraint includes the existence of at least one real-world object, and at least one processor is configured to add a virtual representation of the real-world object in the virtual adaptation 3D scene;

[0036] - At least one constraint relates to at least one size of at least one real-world object, and at least one processor is configured to modify the representation of at least one size of the real-world object in a virtual-adapted 3D scene such that at least one constraint on the size is satisfied.

[0037] - At least one processor is configured to transform the virtual representation of at least one real-world object that is unfavorable to at least one constraint in a virtual-adapted 3D scene, such that at least one constraint is satisfied.

[0038] - At least one processor is configured to perform a transformation by removing, reducing and moving at least one of real-world objects in a virtual-adapted 3D scene;

[0039] - At least one constraint relates to at least one real-world object, and at least one processor is configured to perform a transformation by moving, rotating, translating, scaling, or copying the real-world object such that at least one constraint on the real-world object is satisfied.

[0040] Advantageously, the transformation involves cleaning at least one region in a virtual adapted 3D scene, and at least one processor is also configured to repair at least one region in the virtual adapted 3D scene.

[0041] Then, even if some irregularities are introduced by virtual adaptation (e.g., by virtually removing real objects from a real 3D scene), a sense of continuity can still be provided to the user.

[0042] In some implementations, at least one processor is also configured to allow users to explicitly participate in the execution of the transformation.

[0043] Therefore, users can complete automated processing, which makes it especially possible to execute more user-friendly or customized scenarios.

[0044] Advantageously, at least one input is also adapted to receive the scene, and the device further includes at least one output adapted to output at least a portion of a representation of the virtual adapted 3D scene to the user so that the scene is visible in the virtual adapted 3D scene.

[0045] This disclosure also relates to an apparatus including a device according to this disclosure, which is a video receiver, a mobile phone, a tablet, or an augmented reality head-mounted display system.

[0046] Another object of this disclosure is an apparatus for generating at least one dynamic virtual content for a representation to be overlaid onto a real 3D scene, the dynamic virtual content conforming to at least one scenario defined prior to the runtime of the scenario and involving at least one real-world constraint. The apparatus includes:

[0047] - Components for receiving real-world information captured in real 3D scenes, and

[0048] - A component for executing scenarios at runtime in the presence of real-world constraints identified from real-world information.

[0049] According to this disclosure, the component for execution is adapted to perform a transformation from a representation of a real 3D scene to a virtual adapted 3D scene before the execution scenario, when at least one real-world constraint is not identified from real-world information, such that the virtual adapted 3D scene conforms to the constraint, and to execute the scenario in the virtual adapted 3D scene instead of the real 3D scene.

[0050] This disclosure also relates to a method for generating at least one dynamic virtual content for a representation to be overlaid onto a real 3D scene, the at least one dynamic virtual content conforming to at least one scenario, the at least one scenario being defined prior to the runtime of the scenario and involving at least one real-world constraint. The method includes:

[0051] - Receive real-world information captured in real 3D scenes, and

[0052] - Execute scenarios at runtime when real-world constraints identified from real-world information exist.

[0053] According to this disclosure, the method further includes, when at least one real-world constraint is not identified from real-world information, performing a transformation from a representation of the real 3D scene to a virtual adapted 3D scene before executing the scenario, such that the virtual adapted 3D scene conforms to at least one constraint, and executing the scenario in the virtual adapted 3D scene instead of executing the scenario in the real 3D scene.

[0054] This method can be advantageously performed by a device according to any implementation pattern of this disclosure.

[0055] Furthermore, this disclosure relates to a computer program for generating at least one dynamic virtual content, including software code adapted to perform a method conforming to any of the above execution modes when the program is executed by a processor.

[0056] This disclosure also relates to a non-transitory program storage device that can be read by a computer and tangibly contains a program with computer-executable instructions to perform a method conforming to this disclosure for generating at least one dynamic virtual content.

[0057] Such non-transitory program storage devices can be, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor devices, or any suitable combination thereof. It should be understood that the following, while providing more specific examples, are merely illustrative and not exhaustive lists that will be readily understood by one of ordinary skill in the art: portable computer disks, hard disks, ROM (read-only memory), EPROM (erasable programmable ROM) or flash memory, portable CD-ROM (compressed disc ROM). Attached Figure Description

[0058] This disclosure will be better understood by reading the following description of specific, non-limiting, illustrative embodiments, with reference to the accompanying drawings, in which:

[0059] - Figure 1 This is a schematic block diagram illustrating a device for generating dynamic virtual content in accordance with this disclosure;

[0060] - Figure 2 The diagram illustrates how to utilize Figure 1 Examples of user environments considered by the device;

[0061] - Figure 3 It is shown that Figure 1A flowchart of the continuous mixed reality steps performed by the device;

[0062] - Figure 4 The illustration shows including Figure 1 The MR device of the equipment shown above. Detailed Implementation

[0063] This description illustrates the principles of this disclosure. It will therefore be understood that those skilled in the art will be able to design various arrangements, which, while not explicitly described or shown herein, embody the principles of this disclosure and are included within its spirit and scope.

[0064] All examples and conditional language described herein are intended for educational purposes to help the reader understand the principles of this disclosure and the concepts contributed by the inventors to promote the technology, and are to be interpreted as not being limited to such specific examples and conditions.

[0065] Furthermore, all statements herein recounting the principles, aspects, embodiments, and specific examples thereof are intended to include their structural and functional equivalents. Additionally, such equivalents are intended to include both currently known equivalents and those developed in the future, i.e., any element developed to perform the same function, regardless of its structure.

[0066] Therefore, for example, those skilled in the art will understand that the block diagrams shown herein represent conceptual views of illustrative circuits embodying the principles of this disclosure. Similarly, it will be understood that any flowchart, diagram, etc., representation can substantially represent various processes in a computer-readable medium and thus performed by a computer or processor, whether or not such computer or processor is explicitly shown.

[0067] The terms “adapted” and “configured” are used in this disclosure to broadly include the initial configuration of the device, its later adaptation or supplementation, or similarly any combination thereof, whether implemented by material or software means (including firmware).

[0068] The functionality of the various elements shown in the figure can be provided using dedicated hardware and hardware capable of executing the software, associated with appropriate software. When provided by a processor, the functionality can be provided by a single dedicated processor, a single shared processor, or multiple separate processors (some of which may be shared). Furthermore, the explicit use of the term "processor" should not be construed as exclusively referring to hardware capable of executing software, but rather as referring in a general sense to a processing device, which may include, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device (PLD). Additionally, instructions and / or data capable of performing the associated and / or resulting functions can be stored on any processor-readable medium, such as, for example, an integrated circuit, a hard disk, a CD (compressed disc), an optical disc such as a DVD (Digital Versatile Disc), RAM (Random Access Memory), or ROM memory. In particular, instructions can be stored in hardware, software, firmware, or any combination thereof.

[0069] It should be understood that the elements shown in the figure can be implemented in various forms of hardware, software, or a combination thereof. Preferably, these elements are implemented in a combination of hardware and software on one or more appropriately programmed general-purpose devices, which may include processors, memory, and input / output interfaces.

[0070] References Figure 1 This disclosure is described using a specific functional embodiment of the device 1 for generating dynamic virtual content shown above.

[0071] Device 1 is adapted to generate at least one dynamic virtual content and is related to mixed reality (MR), which is to be superimposed on a representation of a real 3D scene and conforms to at least one scenario.

[0072] Device 1 is advantageously an apparatus or a physical part of an apparatus designed, configured, and / or adapted to perform the mentioned functions and produce the mentioned effects or results. In alternative implementations, device 1 is implemented as a group of apparatuses or physical parts of an apparatus, whether grouped in the same machine or in different, possibly distant, machines.

[0073] In the following text, modules are to be understood as functional entities rather than materially or physically distinct components. Therefore, they can be implemented as grouped together within the same tangible and concrete component, or distributed across several such components. Furthermore, each of those modules may be shared between at least two physical components. Additionally, modules can be implemented in hardware, software, firmware, or any hybrid form thereof. They are preferably implemented within at least one processor of device 1.

[0074] Device 1 includes a module 11 for analyzing, for example, real-world information 22 retrieved from camera captures of a real 3D scene; a module 12 for analyzing and examining the requirements of the MR scene (virtual application 23) based on the analyzed real-world information 22; a module 13 for performing a relevant virtual adaptation applied to the representation of the real 3D scene; and a module 14 for executing the scene (applied to the virtual adapted 3D scene generated from the adaptation). The output of module 13 can potentially be returned to module 12, allowing the adaptation to be iteratively adjusted until a satisfactory adaptation solution is found.

[0075] Information can be entered and retrieved by a user through a user interface 20 that interacts with device 1. The user interface 20 includes any component suitable for entering or retrieving data, information or instructions, especially visual, tactile and / or audio capabilities, and may include any one or more of the following components well known to those skilled in the art: screen, keyboard, trackball, touchpad, touch screen, speaker, voice recognition system.

[0076] Device 1 is also adapted to a virtual tool 21 provided with storage resources 10, specifically designed to perform virtual adaptations to MR scenario requirements. Those storage resources 10 can be obtained from any kind of suitable storage components, in particular RAM or EEPROM (Electrically Erasable Programmable Read-Only Memory), such as flash memory, possibly within an SSD (Solid State Drive).

[0077] The functions of device 1 are described in detail below.

[0078] 5.1 Prerequisites: Scenario Definition

[0079] In the initial steps, the MR application creator defines the scenario. They specifically specify the real-world elements involved in the scenario and how virtual elements interact with them. Examples of consecutive scenario elements are as follows:

[0080] - The first virtual stone falls, bounces off the real cube, and stops on the real table;

[0081] - The second virtual stone strikes a real, empty glass object;

[0082] - The virtual animal has predefined and unmodifiable animations and is located in a free and flat area on a real table;

[0083] - Virtual characters walk on the ground in the free area, climb onto real table legs, walk on the real table, and meet up with virtual animals; their trajectory depends on the free area on the real table.

[0084] To adhere to this scenario, the creators specified some requirements regarding the real-world environment. Below are some examples of requirements for the previous scenario, which takes place indoors:

[0085] - A table with a minimum size of 1.20m × 0.60m;

[0086] - A free, flat area on the table, measuring 0.3m × 0.2m and centered on the center of the table;

[0087] - A cube with a side length of 0.10m, located on the left side of the table;

[0088] - An empty glass object, placed on the table;

[0089] - A free, flat area on the ground, measuring 0.6m x 0.3m, located to the left of the table and including the left front leg of the table.

[0090] Requirements can also impose specific locations or orientations. They may also be subject to a “level of constraint.” That is, some requirements are strong constraints, while others are less stringent. For example, a strong constraint requirement might include a table measuring 1.20m × 0.60m, because if the table exists, it must have these precise dimensions to comply with the requirement. A less stringent constraint requirement might include a cube with sides of 0.10m located to the left of the table, because its precise location is not imposed.

[0091] 5.2 Prerequisites: Required hierarchical sorting

[0092] Preferably, requirements are categorized in a hierarchical manner related to their relationships. Therefore, each requirement is assigned a hierarchical level, with the content creator completing the hierarchical categorization during the content creation phase.

[0093] Considering the requirements above, the most important requirement is the first one ("a table with a minimum size of 1.20m × 0.60m"). The other four requirements depend on the presence of the table but are independent of each other. Therefore, they have equal importance, i.e., the same hierarchical level. This leads to the following classification table:

[0094] One table, with a minimum size of 1.20m × 0.60m.

[0095] 1.1 A free, flat area on the table, measuring 0.3m × 0.2m, centered on the center of the table.

[0096] A 1.2m cube with a side length of 0.10m is located on the left side of the table.

[0097] 1.3 Empty glassware, located on a table

[0098] 1.4 A free, flat area on the ground, measuring 0.6m × 0.3m, located to the left of the table and including the left front leg of the table.

[0099] If additional requirements are included, such as “a cylinder with a radius of 0.1m and a height of 0.3m, located on a cube,” then its class is 1.2.1, because it depends on compliance with requirement 1.2.

[0100] 5.3 Environmental Scanning and Feature Determination

[0101] The next step involves environmental scanning and the identification of relevant features. This involves creating an environmental description by detecting and labeling environmental features, such as objects present in the environment (e.g., a table, glassware on the table, plants) and free, flat areas.

[0102] Therefore, users can scan their environment, which can be achieved in particular by using devices equipped with depth sensors (such as products commercialized under the name "Intel RealSense"), which provide point clouds of the scanned scene.

[0103] Then, the point cloud is analyzed to detect and label environmental features. The technical solution used to achieve this task is specifically described in the following article: Proceedings of the 15th Robot World Cup International Symposium. thRoboCup International Symposium, Istanbul, July 2011: Dirk Holz, Stefan Holzer, Rady Bogdan Rusu, and Sven Behnk, “Real-Time Plane Segmentation using RGB-D Cameras”; Humanoids 2012, pp. 198-203, André Uckermann, Robert Haschke, and Helge J. Ritter, “Real-time 3D segmentation of cluttered scenes for robot grasping”; MultiMedia Modeling 2014, pp. 449-460, Zhe Whang, Hong Liu, Xiangdong Wang, and Yueliang Qian, “Segment and Label Indoor Scene Based on RB-D for the Visually Impaired”; and Robotics and Autonomous Systems 2013. See J. Xiao, J. Zhang, B. Adler, H. Zhang and J. Zhang’s “Three-dimensional point cloud plane segmentation in both structured and unstructured environments”, 2013), 61(12), pp. 1641-1652.

[0104] Environmental analysis can also be achieved using devices such as the commercially available Microsoft Hololens, which outputs a 3D mesh. Some solutions capable of retrieving environmental characteristics from such a 3D mesh include those described in the following articles: “Mesh Based Semantic Modelling for Indoor and Outdoor Scenes” by Julien PCValentin, Sunando Sengupta, Jonathan Warrel, Ali Shahrokni, and Philip HSTorr at CVPR 2013; and “A survey on Mesh Segmentation Techniques” by Ariel Shamir in Computer Graphics forum, Vol. 27, 2008, No. 6, pp. 1539-1556.

[0105] Module 11, which is responsible for analyzing real-world information, can be embedded in the device used to scan the scene or run on a server.

[0106] 5.4 Determination of Virtual Information

[0107] At this level, the description of the environment is known. This step involves analyzing various environment descriptions in conjunction with the scenario requirements to determine the virtual environment that may be needed. This virtual environment is unique to each user and is overlaid on the real environment to provide similar behavior and experience to all users.

[0108] This step consists of two parts: checking the scenario requirements that have been complied with and those that have not been complied with (Module 12), and determining how to compensate for each scenario requirement that has not been fully complied with for each real-world environment (Module 13).

[0109] The following details these two parts based on the previously shown example requirements. Accordingly, the environment must include:

[0110] 1 / Table, with a minimum size of 1.20m × 0.60m;

[0111] 2 / A free, flat area on the table, with dimensions of 0.3m × 0.2m and centered on the center of the table;

[0112] 3 / A cube with a side length of 0.10m, located on the left side of the table;

[0113] 4 / Empty glassware, placed on the table;

[0114] 5 / A free, flat area on the ground, measuring 0.6m x 0.3m, located to the left of the table and including the left front leg of the table.

[0115] exist Figure 2 The above illustrates a realistic environment, in Figure 2 The table 31 has a cube 32 on the left and a vase 33 on the right, while the plant 34 is placed on the right side of the table 31.

[0116] 5.4.1 Checking the Scenario Requirements

[0117] Module 12 checks which scenario requirements are followed and which are not followed, with the total number of requirements denoted as nbReq.

[0118] The corresponding algorithm is as follows:

[0119] - Load the properties derived from environmental scanning and feature determination (Module 11);

[0120] - Create a list of nbReq boolean values, statusList, which stores the status of each requirement (fully complied or not fully complied);

[0121] - Create a list of nbReq data, dataList, which stores real-world features / multiple real-world features if the corresponding scenario requirements are partially or fully met, or stores nothing in other cases;

[0122] - Load the list of scenario requirements.

[0123] - For each scenario requirement, Module 13 will then be responsible for:

[0124] - Check whether the requirements are being followed;

[0125] - If so, set the corresponding element of statusList to True and populate the corresponding element of dataList with the environmental features / multiple environmental features associated with the requirement;

[0126] - If not, set the corresponding element of statusList to False and populate the corresponding element of dataList with the environmental features / multiple environmental features associated with the requirement (e.g., the object exists but is smaller than expected or is in the wrong location), or set the corresponding element of statusList to Empty if there are no features (e.g., the required object does not exist).

[0127] It should be noted that even if the requirements are only partially complied, storage environment characteristics / multiple environment characteristics are of interest due to the reasons for development below.

[0128] Following the example above, statusList has the following values:

[0129] 1 / False (Table 31 does not have the required size)

[0130] 2 / True

[0131] 3 / True (Cube 32)

[0132] 4 / False

[0133] 5 / True

[0134] And dataList has the following values:

[0135] 1 / The table measures 1.0m x 0.6m.

[0136] 2 / The size is 0.5m × 0.6m, with (X tfp ,Y tfp A free, flat area centered on )

[0137] 3 / Having a size of 0.10m, with (X c ,Y c ) centered on a cube 32

[0138] 4 /

[0139] 5 / 2m×1m, with (X gfp ,Y gfp A free, flat area on the ground centered on )

[0140] 5.4.2 Analysis of Scenario Requirements

[0141] Following the analysis of MR application requirements (specifically, elements and free-form flat regions) and the checking of whether the environment complies with these requirements (even partially), module 13 utilizes virtual elements to complete existing portions of the user's (real-world) environment that does not fully comply with the requirements. This adaptation ensures that any environment ultimately includes common features that ensure MR applications are presented in a similar manner.

[0142] Finally, module 13 uses a virtual environment that enhances the real environment to update the MR application.

[0143] More precisely, Module 13 determines how to compensate for each scenario requirement that is not fully complied with, for each specific real-world environment.

[0144] In the first case, features are missing, which may be equivalent to lacking required elements (such as glass artifacts in the example above). This problem is solved by adding virtual objects, which are based on predefined virtual models of the objects.

[0145] In the second case, the features exist in the real environment, but not in an appropriate manner. This particularly covers situations where the desired object exists, but its size, position, and / or orientation do not conform to the context requirements. With the real object labeled, its associated 3D model, and possibly its texture, is known. Then, in a favorable implementation, this 3D model is transformed (moved, scaled, and / or rotated).

[0146] If a 3D model is too small, it can be enlarged; if it is too large, it can be shrunk. In the latter case, parts of real-world objects that are "removed" or "missing" in the real environment can be filled in, which can be done using remediation techniques, such as those developed, for example, in Jan Herling and Wolfgang Broll's "High-Quality Real-Time Video Inpainting with PixMix" in IEEE Transactions on Visualization and Computer Graphics, Vol. 20, 2014, No. 6, pp. 866-879.

[0147] If the desired object exists in the real 3D scene but is in the wrong location, it is advantageous to move or copy / clone it. Then, if the applied solution involves moving the object, the area of ​​that object that is no longer covered by the real object is filled (this can be done through repair). A similar action can be taken if the object is incorrectly oriented.

[0148] In another embodiment, the user is asked to manually move and / or rotate the real object to the correct position and / or orientation. In this regard, a virtual object is displayed to the user to show the object's final position and / or orientation; the geometry of the virtual object is identical to that of the real object. The user moves / rotates the representation of the real object until it overlaps with the virtual object.

[0149] In cases where a free flat region / volume is lacking or not large enough, the two methods previously stated can be applied. By using repair techniques, it becomes possible to "remove" representations of real elements occupying a flat region / volume, and the user can be asked to manually "clear" that region / volume.

[0150] 5.4.3 Joint Management for Multiple Scenario Requirements

[0151] The fact that a requirement for compensation is not fully met may lead to the fact that another requirement is no longer met. For example, by adding missing glassware to table 31 ( Figure 2The free area (a free, flat area on a 0.3m x 0.2m table) in requirement 2 may disappear.

[0152] The following describes the algorithm used to manage this situation. For each current element of statusList that has its value set to False (the current element then has an index named Ifalse), the following steps are performed:

[0153] - Determine a solution that complies with the requirements corresponding to the current element, as described above;

[0154] - Set the current element of stateList with index Ifalse to True;

[0155] - For each other element in statusList that sets its value to True (and has an index named Itrue), check whether the corresponding requirement is still met when the above solution is applied to the current element;

[0156] - If not, then another configuration is allowed for testing the current element, which would still allow the requirement for index Itrue to be followed;

[0157] - If not, set the current element of stateList with index Ifalse to False.

[0158] This enhanced version of the process makes it possible to avoid the uncontrolled practice of satisfying only the first request processed, while requests considered later are left unattended (“first-come, first-served”). Therefore, requests with the highest priority are processed first, and other requests are processed according to a continuously decreasing order (down to the lowest priority).

[0159] In a further improved version, it is examined whether some modifications can be applied to the representation of one or more real elements (especially those that have already been arranged) that have a statusList value of True and have met the requirements, in order to obtain:

[0160] - Maintain full compliance with existing requirements;

[0161] The current requirement of having index Ifalse is also fully complied with.

[0162] In other words, this step attempts to find a combination of modifications that satisfy both objectives, although modifying the representation of the real element may cause modifications to the representation of one or more (potentially many) other elements.

[0163] Similarly, when not all constraints can be satisfied, it is advantageous to use the hierarchy of requirements to give preference to the highest ordering.

[0164] If all elements of statusList are equal to True, a solution that perfectly conforms to the scenario is found. Otherwise, a trivial solution is not found, and one or more constraints with lowest priority ordering (or multiple lowest priority orderings) may not be considered. In a variant implementation, in the absence of any global solution, all requirements are presented to the user, who is then asked to modify the representation of their real-world environment (adding / moving / removing elements) to meet the requirements.

[0165] During execution, such as Figure 3 As shown above, device 1 preferably performs MR operations as follows: During the acquisition phase, features of the real 3D environment (those features associated with corresponding, possibly identical, themes) are loaded (step 41) and a list of scenario requirements is loaded (step 42). It is checked whether all requirements have been processed (step 431). If so, the processing is stopped (step 432). Otherwise, the next requirement is loaded, the corresponding statusList is initialized to False and the corresponding dataList is Empty (step 433).

[0166] Next, check if all environment features have been processed (step 441). If yes, the process loops back to step 431. Otherwise, load the next environment feature (step 451). Then, check if the feature relates to the requirement topic (step 452). If no, the process loops back to step 441. Otherwise, check if the feature meets the considered requirement (step 453). If no, set the corresponding statusList to False, add the feature to the corresponding dataList, and the process loops back to step 441.

[0167] Otherwise, set the corresponding statusList to True, add the feature to the corresponding dataList, and the process loops back to step 441.

[0168] exist Figure 4 The specific device 5 visible above embodies the aforementioned device 1. It may correspond, for example, to a tablet, smartphone, head-mounted display (HMD), game console (such as a dedicated game console that generates and displays live images), or laptop computer. Device 5 is suitable for augmented reality, whether for direct viewing (the user typically views a real 3D scene through glass) or indirect viewing (the user views a real 3D scene displayed on a screen). The two capabilities can also be combined.

[0169] Device 5 includes the following components connected to each other via an address and data bus 55 (which also transmits clock signals):

[0170] - Microprocessor 51 (or CPU);

[0171] - Graphics card 52, including several graphics processing units (or GPUs) 520 and graphics random access memory (GRAM) 521;

[0172] -ROM-type non-volatile memory 56;

[0173] -RAM 57;

[0174] - One or more I / O (input / output) devices 54, such as, for example, a keyboard, mouse, joystick, webcam; other modes for introducing commands, such as, for example, voice recognition, are also possible;

[0175] - Power supply 58; and

[0176] -RF Unit 59.

[0177] Device 5 also includes a display device 53, a display screen type, directly connected to graphics card 52, for displaying composite images calculated and constructed in the graphics card, such as live images. Connecting display device 53 to graphics card 52 using a dedicated bus provides the advantage of a much higher data transfer bit rate and thus reduces the latency for displaying images constructed by the graphics card. According to a variant, the display device is external to device 5 and connected to device 5 via cable or wirelessly for transmitting display signals. Device 5 includes, for example, an interface via graphics card 52 for transmission or connection, adapted to transmit display signals to external display components, such as, for example, LCD or plasma screens or video projectors. In this regard, RF unit 59 can be used for wireless transmission.

[0178] In an alternative implementation, the display device 53 corresponds to a glass through which the user sees the real environment, and the device 5 also includes an optical projection system (not shown) that is capable of projecting, for example, the generated virtual image or content onto the glass.

[0179] Note that the term "register" used in the descriptions of memories 521, 56, and 57 refers to both low-capacity storage areas (containing some binary data) and high-capacity storage areas (enabling the storage of an entire program or the calculation or display of all or part of the data representing the data). Furthermore, the registers represented for GRAM 521 can be arranged and configured in any manner, and each of them does not necessarily correspond to an adjacent memory location, but can be distributed in other ways (this particularly covers the case where a register comprises several smaller registers).

[0180] When switched on, the microprocessor 51 loads and executes the instructions of the program contained in RAM 57.

[0181] Random access memory 57 particularly includes:

[0182] - Register 570 contains the operating program of the microprocessor 51 responsible for activating device 5.

[0183] - In register 571, parameters representing the real 3D scene (such as the model and lighting parameters of objects in the scene);

[0184] - In register 572, parameters representing one or more MR scenarios are stored;

[0185] - In register 573, constraints on the real 3D scene associated with the MR scene;

[0186] - In register 574, parameters for the virtual adaptation relating to the real 3D scene and the constraints corresponding to the MR scene.

[0187] The steps specific to the method of this disclosure and the algorithms described above are stored in the memory GRAM 521 of the graphics card 52 associated with the device 5 that implements those steps. When switched on and once parameters 571, 572, and 574 are loaded into RAM 57, the graphics processor 520 of the graphics card 52 loads those parameters into GRAM 521 and executes the instructions of those algorithms, for example, in the form of a microprogram of the “shader” type using HLSL (High-Level Shader Language) or GLSL (OpenGL Shading Language).

[0188] The random access memory GRAM 521 specifically includes:

[0189] - In register 5211, parameters representing the real 3D scene are stored;

[0190] - In register 5212, parameters representing the MR scenario are stored;

[0191] - In register 5213, the parameters represent the virtual adaptation to be applied in relation to the scenario constraints.

[0192] According to a variant, at least some primitive-related data is stored in RAM 57 and processed by microprocessor 51. However, this variant results in a greater latency in constructing the image, which includes a representation of the environment constructed from microprograms contained in GPU 520, because data must be transferred from the graphics card to random access memory 57 via bus 55, which typically has less transfer capability than the graphics card's capacity available for transferring data from GPU 520 to GRAM 521 (and vice versa).

[0193] As technicians will understand, the existence of the graphics card 52 is not mandatory and can be replaced by a simpler visual implementation.

[0194] According to another variation, the power supply 58 is located outside the device 1.

[0195] Based on this disclosure and the detailed embodiments, other implementations are possible and within the scope of those skilled in the art without departing from the scope of the invention. The specified elements can be interchanged or associated, in particular, in any manner that remains within the framework of this disclosure. Furthermore, elements of different implementations can be combined, supplemented, modified, or removed to produce other implementations. All such possibilities are considered through this disclosure.

Claims

1. A method for executing a mixed reality scene in the presence of a real 3D scene, the mixed reality scene including at least one real-world constraint requiring the presence of objects, the method comprising: Based on real-world information captured in the real 3D scene, determine whether the object exists in the real 3D scene; When it is determined that the object exists in the real 3D scene, the mixed reality scene is executed using the object existing in the real 3D scene; as well as When it is determined that the object does not exist in the real 3D scene, the mixed reality scene is executed using an added virtual version of the object.

2. The method of claim 1, further comprising adding the virtual version of the object by overlaying the virtual version of the object onto the representation of the real 3D scene.

3. A device configured to perform a mixed reality scene in the presence of a real 3D scene, the mixed reality scene including at least one real-world constraint requiring the presence of objects, the device including at least one processor configured to: Based on real-world information captured in the real 3D scene, determine whether the object exists in the real 3D scene; When it is determined that the object exists in the real 3D scene, the mixed reality scene is executed using the object existing in the real 3D scene; as well as When it is determined that the object does not exist in the real 3D scene, the mixed reality scene is executed using an added virtual version of the object.

4. The device according to claim 3, wherein, The at least one processor is further configured to add the virtual version of the object by overlaying the virtual version of the object onto a representation of the real 3D scene.

5. A method for performing a mixed reality scene in the presence of a real 3D scene, the mixed reality scene including at least one real-world constraint associated with an object, the method comprising: Process the received real-world information to detect the presence of the object in the real 3D scene; When it is determined that the object detected in the real scene satisfies the at least one real-world constraint associated with the object, the mixed reality scene is executed using the object; as well as When it is determined that the object detected in the real scene does not satisfy the at least one real-world constraint associated with the object, the mixed reality scene is executed using a virtual adapted version of the object that satisfies the at least one real-world constraint associated with the object.

6. The method of claim 5, further comprising generating the virtual adapted version of the object that satisfies the at least one real-world constraint associated with the object.

7. The method according to claim 6, wherein, The real-world constraints involve at least one of the object's size, position, and orientation, and the virtual adapted version of the object has been moved, scaled, and rotated at least one of the following.

8. The method according to claim 7, wherein, The virtual adaptation version of the object is scaled up by increasing the scale when the object is too small to meet the real-world constraints, or by decreasing the scale when the object is too large to meet the real-world constraints.

9. The method according to claim 8, wherein, When scaled down, repair is used to fill in the missing portions of the object in the real-world environment after scaling down.

10. The method according to claim 7, wherein, In the event that the object moves or rotates, repair is used to fill the portions that are no longer filled by the object in the real environment.

11. A device configured to perform a mixed reality scene in the presence of a real 3D scene, the mixed reality scene including at least one real-world constraint associated with an object, the device including at least one processor configured to: Process the received real-world information to detect the presence of the object in the real 3D scene; When it is determined that the object detected in the real scene satisfies the at least one real-world constraint associated with the object, the mixed reality scene is executed using the object; as well as When it is determined that the object detected in the real scene does not satisfy the at least one real-world constraint associated with the object, the mixed reality scene is executed using a virtual adapted version of the object that satisfies the at least one real-world constraint associated with the object.

12. The device according to claim 11, wherein, The at least one processor is also configured to generate the virtual adapted version of the object that satisfies the at least one real-world constraint associated with the object.

13. The device according to claim 12, wherein, The real-world constraints involve at least one of the object's size, position, and orientation, and the virtual adapted version of the object has been moved, scaled, and rotated at least one of the following.

14. The device according to claim 13, wherein, The at least one processor is further configured to scale the virtual adaptation version of the object by increasing the scale if the object is too small to meet the real-world constraints, or by decreasing the scale if the object is too large to meet the real-world constraints.

15. The device according to claim 14, wherein, The at least one processor is also configured to use repair to fill in portions of the object that are missing in the real environment after the scaling down, in the case of scaling down.

16. The device according to claim 13, wherein, The at least one processor is also configured to use repair to fill portions that are no longer filled by the object in the real environment, in the event that the object is moved or rotated.

Images