Testing of collaborative mixed reality objects

Abstract

To provide a method for testing and debugging the interaction of collaborative mixed reality objects.SOLUTION: In one embodiment, such a method comprises receiving inputs including a first mixed reality object expressed by a first set of attributes, a second mixed reality object expressed by a second set of attributes, a first individual test case associated with the first mixed reality object, and a second individual test case associated with the second mixed reality object. The method comprises automatically generating, from the inputs, a collaborative mixed reality test case to evaluate the interaction of the first mixed reality object with the second mixed reality object within a collaborative mixed reality environment. In certain embodiments, a generative-AI-based encoder-decoder architecture is used to generate the collaborative mixed reality test case from the inputs. A corresponding system and a corresponding computer program product are also disclosed.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates generally to mixed reality environments, and more particularly to a system and method for testing and debugging interactions of collaborative mixed reality objects originating from different sources. [Background technology]

[0002] A collaborative mixed reality (MR) environment is a digital space in which multiple users, whether in the same physical location or remotely, can come together to interact and collaborate in a setting that combines elements of the physical world with virtual, computer-generated information and objects. In such immersive environments, users typically wear a mixed reality headset or use a mixed reality device that allows them to simultaneously perceive and interact with both the real world and digital elements. The technology enables real-time collaboration between users, which may allow users to engage with each other, manipulate virtual objects, and share information within a shared virtual space. Summary of the Invention [Problem to be solved by the invention]

[0003] Systems and methods are provided that may enable real-time collaboration between users, thereby enabling users to engage with each other, manipulate virtual objects, and share information within a shared virtual space. [Means for solving the problem]

[0004] The present invention has been developed in response to the state of the art, and in particular to problems and needs in the art that have not yet been fully addressed by currently available systems and methods. Accordingly, a system and method for testing and debugging collaborative mixed reality object interactions has been developed. Features and advantages of the present invention will become more fully apparent from the following description and appended claims, or may be learned by the practice of the invention as set forth hereinafter.

[0005] Consistent with the foregoing, a method is disclosed for testing and debugging collaborative mixed reality object interactions. In one embodiment, such a method comprises receiving an input including a first mixed reality object represented by a first set of attributes, a second mixed reality object represented by a second set of attributes, a first individual test case associated with the first mixed reality object, and a second individual test case associated with the second mixed reality object. The method automatically generates collaborative mixed reality test cases from the input that evaluate interactions between the first and second mixed reality objects in a collaborative mixed reality environment. In a particular embodiment, a generative AI-based encoder-decoder architecture is used to generate the collaborative mixed reality test cases from the input.

[0006] Corresponding systems and computer program products are also disclosed and claimed herein. [Brief description of the drawings]

[0007] So that the advantages of the present invention may be readily understood, the invention, briefly described above, will now be more particularly described with reference to specific embodiments which are illustrated in the accompanying drawings, with the understanding that these drawings depict only typical embodiments of the invention and are therefore not to be considered as limiting its scope, and the embodiments of the invention will be explained and described with additional specificity and detail using the accompanying drawings, in which:

[0008] [Figure 1]FIG. 1 is a schematic block diagram illustrating an example of a computing system for use in implementing embodiments of the present invention.

[0009] [Diagram 2] FIG. 1 is an overview block diagram illustrating a method for training a generative AI based encoder-decoder architecture to capture embeddings of mixed reality objects.

[0010] [Diagram 3] FIG. 1 is an overview block diagram illustrating a method for training a generative AI based encoder-decoder architecture to capture embeddings for individual test cases of mixed reality objects.

[0011] [Figure 4] FIG. 1 is an overview block diagram illustrating a method for training a generative AI based encoder-decoder architecture to generate collaborative mixed reality test cases.

[0012] [Diagram 5] FIG. 1 is a schematic block diagram illustrating a generative AI-based encoder-decoder architecture configured to output predicted collaborative mixed reality test cases upon receiving a pair of mixed reality objects represented as a set of attributes and individual test cases associated with the mixed reality objects.

[0013] [Figure 6] FIG. 2 illustrates an example of an individual test case for a first mixed reality object, in this example a virtual laptop. [Figure 7] FIG. 2 illustrates an example of an individual test case for a first mixed reality object, in this example a virtual laptop.

[0014] [Figure 8] FIG. 13 illustrates an example of an individual test case for a second mixed reality object, in this example a virtual mouse. [Figure 9] FIG. 13 illustrates an example of an individual test case for a second mixed reality object, in this example a virtual mouse.

[0015] [Figure 10] FIG. 10 illustrates an example of a collaborative mixed reality test case for evaluating the interaction between the virtual laptop and virtual mouse described in FIGS. 6-9. [Figure 11] FIG. 10 illustrates an example of a collaborative mixed reality test case for evaluating the interaction between the virtual laptop and virtual mouse described in FIGS. 6-9. [Figure 12] FIG. 10 illustrates an example of a collaborative mixed reality test case for evaluating the interaction between the virtual laptop and virtual mouse described in FIGS. 6-9. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] It will be readily understood that the components of the present invention, as generally described herein and illustrated in the figures, could be arranged and designed in a wide variety of different configurations. Thus, as shown in the figures, the following more detailed description of embodiments of the present invention is not intended to limit the scope of the invention as claimed, but is merely representative of some examples of presently contemplated embodiments according to the present invention. The presently described embodiments can be best understood by reference to the drawings, where like parts are designated with like numerals throughout.

[0017] Various aspects of the disclosure are described by text, flow charts, block diagrams of computer systems, and / or block diagrams of machine logic included in embodiments of computer program products (CPPs). For any flow chart, depending on the technology involved, operations may be performed in a different order than that shown in a given flow chart. For example, two operations shown in successive flow chart blocks may be performed in reverse order, as a single integrated step, simultaneously, or in an at least partially overlapping manner, also depending on the technology involved.

[0018] A computer program product embodiment ("CPP embodiment" or "CPP") is a term used in this disclosure to describe any set of one or more storage media (also referred to as "media") collectively contained in a set of one or more storage devices that collectively contain machine-readable code corresponding to instructions and / or data for performing computer operations specified in a given CPP claim. A "storage device" is any tangible device that can hold and store instructions for use by a computer processor. The computer-readable storage medium may be, but is not limited to, an electronic storage medium, a magnetic storage medium, an optical storage medium, an electromagnetic storage medium, a semiconductor storage medium, a mechanical storage medium, or any suitable combination of the foregoing. Some known types of storage devices that include these media include diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanically encoded devices (such as punch cards or pits / lands formed on the major surface of a disk), or any suitable combination of the foregoing. Computer-readable storage media, as the term is used in this disclosure, is not to be construed as storage in the form of a transitory signal per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide, light pulses passing through fiber optic cables, electrical signals transmitted through wires, and / or other transmission media. As will be appreciated by those skilled in the art, data is typically moved at some infrequent time during normal operation of the storage device, such as during access, defragmentation, or garbage collection, but the above does not render the storage device transitory since the data is not transitory while it is stored.

[0019] The computing environment 100 includes an example of an environment for executing at least a portion of computer code involved in implementing the methodology of the present invention, such as code 150 for testing and debugging collaborative mixed reality object interactions (i.e., “test and debug module 150”). In addition to block 150, the computing environment 100 includes, for example, a computer 101, a wide area network (WAN) 102, an end user device (EUD) 103, a remote server 104, a public cloud 105, and a private cloud 106. In this embodiment, the computer 101 includes a processor set 110 (including processing circuitry 120 and cache 121), a communication fabric 111, a volatile memory 112, a persistent storage 113 (including an operating system 122 and block 150 shown above), a peripheral device set 114 (including a user interface (UI) device set 123, storage 124, and an Internet of Things (IoT) sensor set 125), and a network module 115. The remote server 104 includes a remote database 130. The public cloud 105 includes a gateway 140, a cloud orchestration module 141, a set of host physical machines 142, a set of virtual machines 143, and a set of containers 144.

[0020] Computer 101 may take the form of a desktop computer, a laptop computer, a tablet computer, a smartphone, a smartwatch or other wearable computer, a mainframe computer, a quantum computer, or any other form of computer or mobile device now known or to be developed in the future that can execute a program, access a network, or query a database, such as remote database 130. As is well understood in the field of computer technology, and depending on the technology, the performance of a computer-implemented method may be distributed among multiple computers and / or among multiple locations. On the other hand, in this description of computing environment 100, in order to keep the description as simple as possible, the detailed discussion focuses on a single computer, specifically computer 101. Although computer 101 is not shown in the cloud in FIG. 1, it may be located in the cloud. On the other hand, computer 101 is not required to be present in the cloud, except to any extent that may be categorically indicated.

[0021] Processor set 110 includes one or more computer processors of any type now known or to be developed in the future. Processing circuitry 120 may be distributed across multiple packages, e.g., multiple linked integrated circuit chips. Processing circuitry 120 may implement multiple processor threads and / or multiple processor cores. Cache 121 is memory located within the processor chip package and is typically used for data or code that should be available for fast access by threads or cores executing on processor set 110. Cache memory is typically organized into multiple levels depending on relative proximity to the processing circuitry. Alternatively, some or all of the cache for a processor set may be located "off-chip." In some computing environments, processor set 110 may be designed to operate with qubits and perform quantum computing.

[0022] Computer-readable program instructions are typically loaded into the computer 101 and cause the processor set 110 of the computer 101 to perform a sequence of operational steps, thereby executing the computer-implemented method, and thus the instructions so executed will instantiate the method specified in the computer-implemented method flowcharts and / or descriptions contained in this document (collectively, the "methods of the present invention"). These computer-readable program instructions are stored in various types of computer-readable storage media, such as the cache 121 and other storage media discussed below. The program instructions and associated data are accessed by the processor set 110 to control and direct the implementation of the methods of the present invention. In the computing environment 100, at least some of the instructions for implementing the methods of the present invention may be stored in the block 150 of the persistent storage 113.

[0023] Communications fabric 111 is the signal-conducting pathway that allows the various components of computer 101 to communicate with one another. Typically, this fabric is made up of switches and conductive pathways, such as those that make up buses, bridges, physical input / output ports, and the like. Other types of signal communication pathways may be used, such as fiber optic and / or wireless communication pathways.

[0024] Volatile memory 112 is any type of volatile memory now known or to be developed in the future. Examples include dynamic random access memory (RAM) or static RAM. Typically, volatile memory 112 is characterized by random access, although this is not required unless expressly indicated. In computer 101, volatile memory 112 is located in a single package and is internal to computer 101, although alternatively or additionally, volatile memory may be distributed across multiple packages and / or located external to computer 101.

[0025] Persistent storage 113 is any form of non-volatile storage for a computer, now known or later developed. The non-volatility of this storage means that the stored data is maintained regardless of whether power is provided to computer 101 and / or directly to persistent storage 113. Persistent storage 113 may be read-only memory (ROM), but typically at least a portion of persistent storage allows data to be written, data to be deleted, and data to be rewritten. Some well-known forms of persistent storage include magnetic disks and solid-state storage devices. Operating system 122 may take several forms, such as various known proprietary operating systems employing a kernel or open source Portable Operating System Interface type operating systems. The code contained in block 150 typically includes at least some of the computer code involved in implementing the method of the present invention.

[0026] The peripheral device set 114 includes a set of peripheral devices of the computer 101. Data communication connections between the peripheral devices and other components of the computer 101 may be implemented in various ways, such as Bluetooth connections, Near Field Communication (NFC) connections, connections made by cables (such as Universal Serial Bus (USB) type cables), insertion type connections (e.g., Secure Digital (SD) cards), connections made through local area communication networks, and even connections made through wide area networks such as the Internet. In various embodiments, the UI device set 123 may include components such as display screens, speakers, microphones, wearable devices (such as goggles and smart watches), keyboards, mice, printers, touch pads, game controllers, and haptic devices. The storage 124 is external storage, such as an external hard drive, or insertable storage, such as an SD card. The storage 124 may be persistent and / or volatile. In some embodiments, the storage 124 may take the form of a quantum computing storage device for storing data in the form of qubits. In embodiments where computer 101 is required to have a large amount of storage (e.g., computer 101 stores and manages a large database locally), then this storage may be provided by a peripheral storage device designed to store very large amounts of data, such as a storage area network (SAN) shared by multiple geographically distributed computers. IoT sensor set 125 is made up of sensors that may be used in Internet of Things applications. For example, one sensor may be a thermometer and another sensor may be a motion detector.

[0027] The network module 115 is a collection of computer software, hardware, and firmware that enables the computer 101 to communicate with other computers over the WAN 102. The network module 115 may include hardware such as a modem or Wi-Fi® signal transceiver, software for packetizing and / or depacketizing data for communication network transmission, and / or web browser software for communicating data over the Internet. In some embodiments, the network control and network forwarding functions of the network module 115 are implemented on the same physical hardware device. In other embodiments (e.g., those that utilize software-defined networking (SDN)), the control and forwarding functions of the network module 115 are implemented on physically separate devices, such that the control function manages several different network hardware devices. Computer-readable program instructions for implementing the methods of the present invention may be downloaded to the computer 101, typically from an external computer or external storage device, through a network adapter card or network interface included in the network module 115.

[0028] WAN 102 is any wide area network (e.g., the Internet) capable of communicating computer data over non-local distances by any technology for communicating computer data now known or to be developed in the future. In some embodiments, WAN 102 may be replaced and / or supplemented by a local area network (LAN) designed to communicate data between devices located in a local area, such as a Wi-Fi network. WANs and / or LANs typically include computer hardware such as copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, and edge servers.

[0029] End-user device (EUD) 103 is any computer system used and controlled by an end user (e.g., a customer of the business operating computer 101) and may take any of the forms discussed above in connection with computer 101. EUD 103 typically receives useful and useful data from the operation of computer 101. For example, in a hypothetical case where computer 101 is designed to provide recommendations to the end user, the recommendations would typically be communicated from network module 115 of computer 101 over WAN 102 to EUD 103. In this manner, EUD 103 may display or otherwise present the recommendations to the end user. In some embodiments, EUD 103 may be a client device such as a thin client, a heavy client, a mainframe computer, a desktop computer, or the like.

[0030] Remote server 104 is any computer system that provides at least some data and / or functionality to computer 101. Remote server 104 may be controlled and used by the same entity that operates computer 101. Remote server 104 refers to a machine that collects and stores useful and useful data for use by other computers, such as computer 101. For example, in the hypothetical case where computer 101 is designed and programmed to provide recommendations based on historical data, then this historical data may be provided to computer 101 from remote database 130 of remote server 104.

[0031] A public cloud 105 is any computer system available for use by multiple entities that provides on-demand availability of computer system resources and / or other computer capabilities, particularly data storage (cloud storage) and computing power, without direct active management by users. Cloud computing typically leverages resource sharing to achieve consistency and economies of scale. The direct, active management of the computing resources of the public cloud 105 is performed by computer hardware and / or software of a cloud orchestration module 141. The computing resources provided by the public cloud 105 are typically implemented by virtual computing environments running on various computers that make up the computers of the host physical machine set 142, which is the universe of physical computers that are in and / or available to the public cloud 105. The virtual computing environments (VCEs) typically take the form of virtual machines from the virtual machine set 143 and / or containers from the container set 144. It is understood that these VCEs may be stored as images and can be transferred among and between various physical machine hosts, either as images or after instantiation of the VCEs. Cloud orchestration module 141 manages the transfer and storage of images, deploys new instantiations of VCE, and manages active instantiations of VCE deployments. Gateway 140 is a collection of computer software, hardware, and firmware that enables public cloud 105 to communicate over WAN 102.

[0032] Next, some further explanation is provided about Virtualized Computing Environments (VCEs). A VCE can be stored as an "image". A new active instance of a VCE can be instantiated from an image. Two well-known types of VCEs are virtual machines and containers. A container is a VCE that uses operating system level virtualization. This refers to a feature of an operating system where the kernel allows multiple isolated user space instances called containers to exist. These isolated user space instances typically behave as real computers from the perspective of the programs running within them. A computer program running on a typical operating system can utilize all the resources of that computer, such as connected devices, files and folders, network shares, CPU power, and quantifiable hardware capabilities. However, a program running inside a container can only use the contents of the container and the devices assigned to the container, a feature known as containerization.

[0033] A private cloud 106 is similar to a public cloud 105, except that the computing resources are only available for use by a single enterprise. Although the private cloud 106 is shown as being in communication with the WAN 102, in other embodiments, the private cloud may be completely disconnected from the Internet and only accessible via a local / private network. A hybrid cloud is a composite of multiple clouds of different types (e.g., private, community or public cloud types), often each implemented by a different vendor. While each of the multiple clouds remains a separate and discrete entity, the larger hybrid cloud architecture is bound together by standardized or proprietary technologies that enable orchestration, management, and / or data / application portability between the multiple constituent clouds. In this embodiment, both the public cloud 105 and the private cloud 106 are part of a larger hybrid cloud.

[0034] Referring to FIG. 2, as previously mentioned, in a collaborative mixed reality environment, objects from different sources may interact with each other to achieve a collaborative goal. These objects may be developed and tested separately. Because the interactions between these objects may be new and may not have been previously tested, they may expose bugs or other undesirable behaviors that may require resolution. Currently, there is no established or agreed upon testing framework for testing and debugging collaborative mixed reality objects from different sources.

[0035] In certain embodiments, a neural network, more specifically a generative artificial intelligence based encoder-decoder architecture, can be trained to generate collaborative mixed reality test cases from various types of inputs. These inputs may include, for example, individual test cases for pairs of mixed reality objects, and the mixed reality objects themselves represented as a set of attributes. The collaborative mixed reality test cases can be used to evaluate interactions between the mixed reality objects in a collaborative mixed reality environment.

[0036] In general, mixed reality objects have various attributes that define their behavior and appearance in a mixed reality environment. These attributes may include one or more of location, scale, orientation, animation, interactivity, material, audio, and the like. For example, mixed reality objects may be placed at specific locations in the real world, and they may be fixed to real-world surfaces or left free-floating. This allows users to interact with virtual objects at specific locations, providing a sense of presence and immersion. Mixed reality objects may also be sized and scaled to match the size and scale of real-world objects, thereby enhancing the realism and immersion of the mixed reality experience. Mixed reality objects may be rotated and oriented in 3D space, allowing users to view them from different angles and perspectives. In certain cases, mixed reality objects may be animated to simulate real-world behaviors, such as movement, lighting, and particle effects. They may be designed to respond to user interactions, such as hand gestures or controller inputs, thereby allowing users to manipulate and interact with virtual objects in real time. In certain cases, mixed reality objects may be designed with specific materials, such as glass or metal, that define their appearance and how they respond to lighting and shadows. In certain cases, they may be designed with audio effects, such as sound effects or voices, that enhance their behavior and appearance in the mixed reality environment.

[0037] Similarly, mixed reality objects may interact with each other in various ways depending on the particular mixed reality technology being used and the design of the mixed reality objects. For example, mixed reality objects may interact with each other by colliding or intersecting in 3D space. This may be used to simulate real-world interactions such as objects bouncing off each other or blocking each other's motion. Mixed reality objects may be constrained by each other such that their motion and / or behavior are linked in some way. For example, one mixed reality object may be attached to another such that it follows the motion of the other object. Mixed reality objects may interact with each other by simulating physical forces such as gravity, friction, and spring forces. This allows mixed reality objects to behave in realistic and intuitive ways, which can be used to create complex interactions between mixed reality objects. In certain cases, mixed reality objects may interact with each other using audio, such as making a sound when they collide or react to each other in some way. In other cases, mixed reality objects may interact with each other by exchanging data, such as triggering events or changing the behavior of one mixed reality object based on the state of another mixed reality object. Mixed reality objects may interact with each other by communicating over a network, such as exchanging data between multiple devices or triggering events. These are just a few of the ways mixed reality objects may interact and are not intended to be limiting.

[0038] To generate collaborative mixed reality test cases using neural networks, and more specifically, the generative AI-based encoder-decoder architecture, the generative AI-based encoder-decoder architecture can be trained with input data, also referred to herein as "training data." As mentioned above, mixed reality objects may have various attributes that can be used to classify the mixed reality objects. These attributes may provide one type of training data. Similarly, because mixed reality objects are usually developed and tested separately, mixed reality objects will usually have their own test cases. These test cases may provide another type of training data. Furthermore, in a collaborative mixed reality environment, when mixed reality objects from different sources interact with each other, logs describing the interactions may be generated and stored. These logs may provide yet another type of training data.

[0039] FIG. 2 is a schematic block diagram illustrating an initial stage of training a generative AI-based encoder-decoder architecture to capture an embedding of a mixed reality object 202. As shown, the generative AI-based encoder-decoder architecture 200 may receive as input a mixed reality object 202 represented as a set of attributes. In this initial stage, a variational autoencoder (VAE) may be trained to effectively capture the mixed reality object 202 in a Gaussian embedding space. To achieve this, the attributes of the mixed reality object 202 may be tokenized and fed to a VAE encoder 204 to generate a bottleneck embedding 206. This bottleneck embedding 206 may then be fed to a decoder 208 to reconstruct the original mixed reality object 202 represented as a set of attributes. In a particular embodiment, the VAE encoder 204 and the decoder 208 may utilize a multilayer perceptron (MLP) architecture. The VAE encoder 204 may be configured to minimize reconstruction loss and Kullback-Leibler (KL) information loss to ensure Gaussian embedding. Once the system is trained, the encoder 204 may be used to generate embeddings 206 for mixed reality objects, and the decoder 208 may be ignored.

[0040] FIG. 3 is a schematic block diagram showing the next stage of training a generative AI-based encoder-decoder architecture to capture embeddings for individual test cases of a mixed reality object 202. At this stage, the individual test cases 302 may be represented as a sequence of intent tokens 304a and mixed reality object tokens 304b, which may then be converted to embeddings 306 via a learnable embedding layer. In a particular embodiment, the system 300 may use the encoder 202 described in connection with FIG. 2 as an embedding layer for the mixed reality object to capture attribute correlations. The system 300 may use a masked autoencoder 308 that masks a percentage (e.g., 75 percent) of the tokens 304, which the encoder and decoder may then reconstruct in box 310. The masked autoencoder 308 may be built on top of a transformer and may be powerful enough to capture cross-channel correlations. By enabling masked token generalization, the system 300 may learn better and more compact generalized representations of the test cases. The embeddings of the test cases can be obtained using the encoder of the masked autoencoder 308. For later sampling requirements, a normal distribution may be applied in the encoder space.

[0041] FIG. 4 is a schematic block diagram showing the next stage of training a generative AI-based encoder-decoder architecture to generate collaborative mixed reality test cases. Once the embedding of the test cases is learned by the training described in connection with FIG. 3, a simple supervised fine-tuning approach can be followed. Given two individual test cases 402a, 402b, the system 400 can generate a collaborative test case 410. A ground truth collaborative test case 414 can be used for this training to minimize a loss 412 (e.g., cross entropy or softmax loss) and train the model. A fine-tuning encoder 406 receives embeddings 404a, 404b for the two test cases 402a, 402b, and a fine-tuning decoder 408 generates a generated raw collaborative test case 410 from the embedding space. The weights of the fine-tuning decoder can be initialized from a mean absolute error (MAE) decoder where the embedding space is constrained to be normally distributed. This allows the fine-tuning decoder 408 to efficiently sample and generate new collaborative test cases 410 from the embedding space. The end-to-end process is trained in a supervised setting and fine-tuned to the problem of interest.

[0042] 5 provides an overview of the generative AI-based encoder-decoder architecture 500 after training has been performed and the generative AI-based encoder-decoder architecture is in inference or prediction mode. As shown, the generative AI-based encoder-decoder architecture 500 is configured to output a predicted collaborative mixed reality test case 510 upon receiving a pair of mixed reality objects 502 (A and B) represented as a set of attributes and an individual test case 502 associated with the mixed reality objects 502. As shown, the encoder 504 outputs embeddings (E(A) and E(B)) for each of the mixed reality objects 502 and associated individual test cases. These embeddings may be concatenated and passed to a decoder 508 to generate and predict the collaborative mixed reality test case (A, B) 510.

[0043] With reference to FIGS. 6-12, an example of two mixed reality objects and associated individual test cases are provided that are used to generate a collaborative mixed reality test case for evaluating interactions between the mixed reality objects. In this example, the first mixed reality object is a virtual laptop and the second mixed reality object is a virtual mouse. These mixed reality objects may be developed and tested separately. The generated collaborative mixed reality test case can be used to evaluate the interaction of the virtual laptop and the virtual mouse to determine or predict whether they are operating correctly relative to each other. FIGS. 6 and 7 show an example individual test case for the virtual laptop; FIGS. 8 and 9 show an example individual test case for the virtual mouse; and FIGS. 10-12 show an example collaborative mixed reality test case for evaluating interactions between the virtual laptop and the virtual mouse.

[0044] 6 and 7, to verify functionality and interaction of the virtual laptop mixed reality object within the mixed reality environment, individual test cases for the virtual laptop may include: launching the mixed reality application 602; verifying that the mixed reality headset is properly connected and calibrated 604; verifying that the mixed reality environment loads successfully 606; identifying and selecting 608 the "virtual laptop" mixed reality object from available options; verifying that the virtual laptop object is accurately placed within the mixed reality environment 610; interacting with the virtual laptop using available gestures or controller inputs 612; verifying that the virtual laptop responds appropriately to interactions such as opening and closing the lid or pressing virtual keys 614; testing functionality of the virtual laptop features such as the touchpad, keyboard, and screen 616; verifying that the touchpad accurately tracks finger movements and performs actions such as scrolling and clicking 618; test typing on the virtual laptop keyboard and verify that keystrokes are registered correctly 620; interact with the virtual laptop screen by launching applications or navigating menus 622; verify that the virtual laptop screen accurately displays content and responds to user input 702; test the virtual laptop's interaction with other mixed reality objects, such as dragging and dropping files onto the virtual laptop's screen 704; verify that the virtual laptop correctly handles interactions with other objects and responds accordingly 706; test resizing and repositioning the virtual laptop within the mixed reality environment 708; verify that the virtual laptop can be resized without any issues and retains its image quality and interaction 710; verify that the placement and orientation of the virtual laptop can be adjusted accurately 712; test object collision detection by overlapping the virtual laptop with other objects in the environment 714;Verify that the virtual laptop behaves realistically when it collides or overlaps with other objects 716; test the persistence of the virtual laptop by saving the current mixed reality scene and exiting the application 718; restarting the application and reloading the saved scene 720; and verify that the previously placed virtual laptop is correctly restored and retains its properties and interactions 722.

[0045] Expected results for the virtual laptop individual test case may include the following: the virtual laptop mixed reality object is accurately positioned within the mixed reality environment; the virtual laptop responds appropriately to interactions such as opening and closing the lid or pressing virtual keys; the virtual laptop's touchpad accurately tracks finger movements and performs actions such as scrolling and clicking; typing on the virtual laptop's keyboard registers keystrokes correctly; the virtual laptop's screen accurately displays content and responds to user input; interactions with other mixed reality objects, such as dragging and dropping files, are correctly handled by the virtual laptop; resizing and repositioning the virtual laptop works without any issues and retains its image quality and interactions; the virtual laptop behaves realistically when colliding or overlapping with other objects; and saved mixed reality scenes correctly restore the previously placed virtual laptop, including its properties and interactions.

[0046] 8 and 9, to verify functionality and interaction of the virtual mouse mixed reality object within a mixed reality environment, individual test cases for the virtual mouse may include: launching a mixed reality application 802; verifying that the mixed reality headset is properly connected and calibrated 804; verifying that the mixed reality environment loads successfully 806; identifying and selecting a "virtual mouse" mixed reality object from available options 808; verifying that the virtual mouse object is accurately placed within the mixed reality environment 810; interacting with the virtual mouse using available gestures or controller inputs 812; verifying that the virtual mouse cursor accurately follows the movements of the user's hand or controller 814; testing functionality of virtual mouse features such as left click, right click, and scroll wheel 816; verifying that a left click of the virtual mouse triggers the expected action, such as selecting an object or interacting with a user interface element 818; verifying that a right click of the virtual mouse triggers an action in a content menu 819; 904; verify that the virtual mouse cursor accurately interacts with the environment and triggers the appropriate actions, such as displaying a menu or performing a secondary action 820; test scrolling with the virtual mouse scroll wheel and verify that the scrolling action is registered correctly 822; interact with the virtual mouse cursor on the mixed reality environment, such as hovering over an interactive element or dragging an object 902; verify that the virtual mouse cursor accurately interacts with the environment and triggers the appropriate actions 904; test resizing and repositioning the virtual mouse within the mixed reality environment 906; verify that the virtual mouse can be resized without any issues and retains its image quality and interaction 908; verify that the placement and orientation of the virtual mouse can be accurately adjusted 910; test object collision detection by overlapping the virtual mouse with other objects in the environment 912; verify that the virtual mouse behaves realistically when it collides with or overlaps other objects 914; test virtual mouse persistence by saving the current mixed reality scene and exiting the application 916;Restart the application and reload the saved scene 918; and verify that the previously placed virtual mouse is correctly restored and retains its properties and interactions 920.

[0047] Expected results for the virtual mouse individual test cases may include the following: the virtual mouse mixed reality object is accurately placed within the mixed reality environment; the virtual mouse cursor accurately follows the movements of the user's hand or controller; a left click of the virtual mouse triggers the expected action; a right click of the virtual mouse triggers the expected action; scrolling the virtual mouse scroll wheel correctly registers the scrolling action; the virtual mouse cursor accurately interacts with the mixed reality environment and triggers the appropriate action; resizing and repositioning the virtual mouse works without any issues and retains its image quality and interaction; the virtual mouse behaves realistically when colliding or overlapping with other objects; and saved mixed reality scenes correctly restore the previously placed virtual mouse, including its characteristics and interactions.

[0048] 10-12, to verify the functionality of and interactions between the previously discussed virtual laptop and virtual mouse within the collaborative mixed reality environment, collaborative mixed reality test cases may be generated as follows: launch the collaborative mixed reality application 1002; verify that all users have properly connected and calibrated their mixed reality headsets 1004; verify that the collaborative mixed reality environment loads successfully 1006; establish a connection between multiple users within the application 1008; identify and select a "virtual laptop" mixed reality object from available choices 1010; verify that the virtual laptop object is correctly placed within the mixed reality environment and is visible to all users 1012; interact with the virtual laptop using available gestures or controller inputs 1014; verify that all users can see the virtual laptop and their interactions in real time 1016; functionality of the virtual laptop features, such as opening and closing the lid, pressing virtual keys, and using the touchpad. 1018; verify that all users see the virtual laptop features being used correctly and in sync 1020; identify and select a "virtual mouse" mixed reality object from available choices 1022; verify that the virtual mouse object is correctly placed in the mixed reality environment and is visible to all users 1102; interact with the virtual mouse using available gestures or controller inputs 1104; verify that all users can see the virtual mouse cursor movement and interaction in real time 1106; test functionality of the virtual mouse features such as left click, right click, and scroll wheel 1108; verify that all users see the virtual mouse features being used correctly and in sync 1010; test cooperative interaction between the virtual laptop and virtual mouse 1112; have one user control the virtual laptop using the virtual mouse cursor to perform actions such as opening applications, typing, or clicking objects 1114;Verify that all users see the virtual laptop actions being performed by the users using their virtual mice to control the virtual laptop 1116; test simultaneous virtual laptop and virtual mouse interactions from different users 1118; have multiple users control the virtual laptop using their respective virtual mouse objects to perform actions such as typing or clicking 1120; verify that all users see the virtual laptop actions being performed by each user using their own virtual mouse 1202; test resizing and repositioning of the virtual laptop and virtual mouse within the mixed reality environment 1204; verify that the virtual laptop and virtual mouse are correctly resized and repositioned. 1206; test object collision detection by overlapping the virtual laptop and virtual mouse with other objects in the environment 1208; verify that the virtual laptop and virtual mouse behave realistically when colliding or overlapping with other objects 1210; test virtual laptop and virtual mouse persistence by saving the current collaborative mixed reality session and exiting the application 1212; restart the application and join the saved collaborative session 1214; and verify that the previously positioned virtual laptop and virtual mouse are correctly restored and retain their properties and interactions 1216.

[0049] Expected results for the virtual mouse individual test cases may include the following: multiple users can interact with the virtual laptop and virtual mouse simultaneously and see real-time updates; the virtual laptop responds properly to interactions such as opening and closing the lid or pressing virtual keys performed by a user controlling it with the virtual mouse; the virtual mouse cursor accurately follows the movements of the user's hand or controller and interacts with the mixed reality environment; collaborative interactions between the virtual laptop and virtual mouse performed by different users are synchronized and visible to all users; resizing and repositioning of the virtual laptop and virtual mouse works without any issues and retains their image quality and interactions; the virtual laptop and virtual mouse behave realistically when colliding or overlapping with other objects; saved collaborative sessions correctly restore previously positioned virtual laptop and virtual mouse, including their properties and interactions.

[0050] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of code, comprising one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, depending on the functionality involved, two blocks shown in succession may in fact be executed substantially simultaneously, or sometimes the blocks may be executed in the reverse order. Other implementations may not require all of the disclosed steps to achieve the desired functionality. It should also be noted that each block of the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, may be implemented by a dedicated hardware-based system that performs the specified functions or acts, or a combination of dedicated hardware and computer instructions.

Claims

1. receiving input including a first mixed reality object represented by a first set of attributes, a second mixed reality object represented by a second set of attributes, a first individual test case associated with the first mixed reality object, and a second individual test case associated with the second mixed reality object; and automatically generating collaborative mixed reality test cases from the input that evaluate interactions between the first mixed reality object and the second mixed reality object within a collaborative mixed reality environment.

16. A method for testing and debugging collaborative mixed reality object interactions, comprising:

2. The method of claim 1 , wherein the automatically generating step comprises automatically generating using a generative AI-based encoder-decoder architecture.

3. 3. The method of claim 2, further comprising training the generative AI-based encoder-decoder architecture using at least one of attributes of a mixed reality object, individual test cases of a mixed reality object, and collaboration logs of a mixed reality object in a collaborative mixed reality environment.

4. The method of claim 2 , wherein the inputs are concatenated before being input to the generative AI-based encoder-decoder architecture.

5. 3. The method of claim 2, wherein the first mixed reality object and the second mixed reality object are developed and tested separately before being input into the generative AI based encoder-decoder architecture.

6. The method of claim 1 , wherein the first mixed reality object and the second mixed reality object originate from different sources.

7. The method of claim 1 , wherein the first and second sets of attributes include at least one of physical attributes and meta attributes.

8. 1. A computer program for testing and debugging collaborative mixed reality object interactions, comprising computer usable program code that, when executed by at least one processor, causes the at least one processor to: receiving input including a first mixed reality object represented by a first set of attributes, a second mixed reality object represented by a second set of attributes, a first individual test case associated with the first mixed reality object, and a second individual test case associated with the second mixed reality object; and and automatically generating from the input a collaborative mixed reality test case that evaluates an interaction between the first mixed reality object and the second mixed reality object within a collaborative mixed reality environment. A computer program configured to cause a computer to perform the steps of:

9. 9. The computer program product of claim 8, wherein the step of automatically generating comprises the step of automatically generating using a generative AI based encoder-decoder architecture.

10. 10. The computer program product of claim 9, wherein the computer usable program code, when executed by the at least one processor, is configured to cause the at least one processor to further perform a procedure of training the generative AI based encoder-decoder architecture using at least one of attributes of a mixed reality object, individual test cases of a mixed reality object, and collaboration logs of a mixed reality object in a collaborative mixed reality environment.

11. 10. The computer program product of claim 9, wherein the inputs are concatenated before being input to the generative AI-based encoder-decoder architecture.

12. 10. The computer program product of claim 9, wherein the first mixed reality object and the second mixed reality object are developed and tested separately before being input into the generative AI based encoder-decoder architecture.

13. 13. The computer program product of claim 8, wherein the first mixed reality object and the second mixed reality object originate from different sources.

14. 13. The computer program product of claim 8, wherein the first and second sets of attributes comprise at least one of physical attributes and meta attributes.

15. at least one processor; at least one memory device operatively coupled to the at least one processor and storing instructions for execution by the at least one processor, the instructions including instructions to the at least one processor to: receiving input including a first mixed reality object represented by a first set of attributes, a second mixed reality object represented by a second set of attributes, a first individual test case associated with the first mixed reality object, and a second individual test case associated with the second mixed reality object; and and automatically generating, from the input, a collaborative mixed reality test case that evaluates an interaction between the first mixed reality object and the second mixed reality object within a collaborative mixed reality environment. A system for testing and debugging collaborative mixed reality object interactions comprising:

16. The system of claim 15, wherein the step of automatically generating comprises the step of automatically generating using a generative AI based encoder-decoder architecture.

17. 17. The system of claim 16, wherein the instructions further cause the at least one processor to train the generative AI based encoder-decoder architecture using at least one of attributes of a mixed reality object, individual test cases of a mixed reality object, and collaboration logs of a mixed reality object in a collaborative mixed reality environment.

18. 17. The system of claim 16, wherein the inputs are concatenated before being input to the generative AI based encoder-decoder architecture.

19. 17. The system of claim 16, wherein the first mixed reality object and the second mixed reality object are developed and tested separately before being input into the generative AI based encoder-decoder architecture.

20. The system of claim 15 , wherein the first mixed reality object and the second mixed reality object originate from different sources.

Images