Testing of collaborative mixed reality objects

Abstract

A method for testing and debugging interactions of collaborative mixed reality objects is disclosed. In one embodiment, the method includes receiving an input including a first mixed reality object expressed by a first set of attributes, a second mixed reality object expressed by a second set of attributes, a first individual test case associated with the first mixed reality object, and a second individual test case associated with the second mixed reality object. The method automatically generates a collaborative mixed reality test case from the input to evaluate an interaction of the first mixed reality object with the second mixed reality object within the collaborative mixed reality environment. In some embodiments, a generated AI-based encoder-decoder architecture is used to generate collaborative mixed reality test cases from inputs. A corresponding system and computer program product are also disclosed.

Description

Technical Field

[0001] The present invention relates generally to mixed reality environments and, more particularly, to systems and methods for testing and debugging the interaction of collaborative mixed reality objects originating from different sources. Background Art

[0002] A collaborative mixed reality (MR) environment is a digital space in which multiple users can gather together in the same physical location or remotely to interact and collaborate in a setting that combines elements of the physical world with virtual computer-generated information and objects. In such an immersive environment, users typically wear mixed reality headsets or use mixed reality devices, allowing them to perceive and interact with both the real world and digital elements simultaneously. The technology can enable real-time collaboration between users, allowing them to engage with each other, manipulate virtual objects, and share information within a shared virtual space. Summary of the invention

[0003] The present invention has been developed in response to the current state of the art, and in particular, in response to problems and needs in the art that have not yet been fully solved by currently available systems and methods. Accordingly, systems and methods for testing and debugging the interaction of collaborative mixed reality objects have been developed. The features and advantages of the present invention will become more fully apparent from the following description and appended claims, or may be learned by the practice of the invention as described below.

[0004] Consistent with the foregoing, a method for testing and debugging the interaction of collaborative mixed reality objects is disclosed. In one embodiment, the method includes receiving an input, the input including a first mixed reality object expressed by a first set of attributes, a second mixed reality object expressed by a second set of attributes, a first separate test case associated with the first mixed reality object, and a second separate test case associated with the second mixed reality object. The method automatically generates collaborative mixed reality test cases from the input to evaluate the interaction of the first mixed reality object with the second mixed reality object within a collaborative mixed reality environment. In some embodiments, an encoder-decoder architecture based on generative AI is used to generate collaborative mixed reality test cases from the input.

[0005] Corresponding systems and computer program products are also disclosed and claimed herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] In order to easily understand the advantages of the present invention, a more particular description of the invention briefly described above will be presented by reference to specific embodiments shown in the accompanying drawings. It is understood that these drawings depict only typical embodiments of the present invention and are not to be considered limiting of its scope. Embodiments of the present invention will be described and explained in more detail and in detail through the use of the accompanying drawings, in which:

[0007] Figure 1 is a high-level block diagram illustrating one example of a computing system for implementing embodiments of the present invention;

[0008] Figure 2 is a high-level block diagram illustrating a method for training a generative AI-based encoder-decoder architecture to acquire embeddings of mixed reality objects;

[0009] Figure 3 is a high-level block diagram illustrating a method for training a generative AI-based encoder-decoder architecture to capture embeddings of individual test cases of mixed reality objects;

[0010] Figure 4 is a high-level block diagram illustrating a method for training a generative AI-based encoder-decoder architecture to generate collaborative mixed reality test cases;

[0011] Figure 5 is a high-level block diagram illustrating a generative AI-based encoder-decoder architecture configured to output a predicted collaborative mixed reality test case upon receiving a pair of mixed reality objects expressed as a set of attributes and a separate test case associated with the mixed reality object;

[0012] Figure 6 and 7 An example of an individual test case for a first mixed reality object, in this example a virtual laptop computer, is shown;

[0013] Figure 8 and 9 An example of a separate test case for a second mixed reality object, in this example a virtual mouse, is shown; and

[0014] Figures 10 to 12 Shown for evaluation Figures 6 to 9 An example of a collaborative mixed reality test case of interaction between a virtual laptop and a virtual mouse as described in . DETAILED DESCRIPTION

[0015] It is readily understood that the components of the present invention, as generally described and illustrated in the drawings herein, may be arranged and designed in a variety of different configurations. Therefore, the following more detailed description of the embodiments of the present invention, as represented in the drawings, is not intended to limit the scope of the claimed invention, but is merely representative of certain examples of the presently contemplated embodiments of the present invention. The presently described embodiments will be better understood by reference to the drawings, wherein like parts are represented by like reference numerals throughout the drawings.

[0016] Various aspects of the present disclosure are described by narrative text, flow charts, block diagrams of computer systems, and / or block diagrams of machine logic included in computer program product (CPP) embodiments. With respect to any flow chart, depending on the technology involved, the operations may be performed in an order different from the order shown in a given flow chart. For example, again depending on the technology involved, two operations shown in consecutive flow chart blocks may be performed in reverse order, as a single integrated step, simultaneously, or in a manner that at least partially overlaps in time.

[0017] The term computer program product embodiment ("CPP embodiment" or "CPP") as used in this disclosure is used to describe any collection of one or more storage media (also referred to as "media") collectively included in a collection of one or more storage devices that collectively include machine-readable code corresponding to instructions and / or data for performing the computer operations specified in a given CPP claim. A "storage device" is any tangible device that can hold and store instructions for use by a computer processor. Without limitation, a computer-readable storage medium may be an electronic storage medium, a magnetic storage medium, an optical storage medium, an electromagnetic storage medium, a semiconductor storage medium, a mechanical storage medium, or any suitable combination of the foregoing. Some known types of storage devices that include these media include: a magnetic disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device (such as a punch card or pits / land formed in a major surface of a disk), or any suitable combination of the foregoing. The term computer-readable storage medium as used in this disclosure is not to be interpreted as storage in the form of a transitory signal per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides, light pulses through fiber optic cables, electrical signals transmitted through wires, and / or other transmission media. As will be appreciated by those skilled in the art, data is typically moved at certain occasional points in time during normal operation of the storage device, such as during access, de-fragmentation, or garbage collection, but this does not make the storage device transitory because the data is not transitory when it is stored.

[0018] The computing environment 100 includes an example of an environment for executing at least some of the computer codes involved in performing the methods of the present invention, such as code 150 for testing and debugging the interaction of collaborative mixed reality objects (i.e., "testing and debugging module 150"). In addition to block 150, the computing environment 100 includes, for example, a computer 101, a wide area network (WAN) 102, an end-user device (EUD) 103, a remote server 104, a public cloud 105, and a private cloud 106. In this embodiment, the computer 101 includes a processor group 110 (including a processing circuit 120 and a cache 121), a communication structure 111, a volatile memory 112, a persistent storage 113 (including an operating system 122 and block 150, as described above), a peripheral device group 114 (including a user interface (UI) device group 123, a storage 124, and an Internet of Things (IoT) sensor group 125), and a network module 115. The remote server 104 includes a remote database 130. The public cloud 105 includes a gateway 140 , a cloud coordination module 141 , a host physical machine group 142 , a virtual machine group 143 , and a container group 144 .

[0019] Computer 101 may take the form of a desktop computer, laptop computer, tablet computer, smart phone, smart watch or other wearable computer, mainframe computer, quantum computer, or any other form of computer or mobile device now known or developed in the future that is capable of running programs, accessing a network, or querying a database such as remote database 130. As is well known in the art of computer technology, and depending on the technology, the performance of computer-implemented methods may be distributed among multiple computers and / or among multiple locations. On the other hand, in this presentation of computing environment 100, the detailed discussion focuses on a single computer (particularly computer 101) to keep the presentation as simple as possible. Computer 101 may be located in the cloud, even though Figure 1 On the other hand, unless to any extent that can be positively indicated, there is no requirement that computer 101 is in the cloud.

[0020] Processor group 110 includes one or more computer processors of any type now known or to be developed in the future. Processing circuit 120 may be distributed over multiple packages, such as multiple cooperating integrated circuit chips. Processing circuit 120 may implement multiple processor threads and / or multiple processor cores. Cache 121 is a memory located in the processor chip package and is typically used for data or code that should be quickly accessed by threads or cores running on processor group 110. Cache memory is typically organized into multiple levels based on relative proximity to the processing circuit. Alternatively, some or all of the caches of the processor group may be located "off chip". In some computing environments, processor group 110 may be designed to work with qubits and perform quantum computing.

[0021] Computer readable program instructions are typically loaded onto computer 101 to cause processor group 110 of computer 101 to perform a series of operating steps to implement a computer-implemented method, such that the instructions so executed will instantiate the method specified in the flow chart and / or the narrative description of the computer-implemented method included in this document (collectively referred to as "the present method"). These computer readable program instructions are stored in various types of computer readable storage media, such as cache 121 and other storage media discussed below. The program instructions and related data are accessed by processor group 110 to control and direct the execution of the present method. In computing environment 100, at least some of the instructions for performing the present method in block 150 may be stored in persistent storage 113.

[0022] Communications fabric 111 is the signaling pathways that allow the various components of computer 101 to communicate with each other. Typically, the fabric is comprised of switches and conductive pathways, such as those that constitute a bus, a bridge, physical input / output ports, etc. Other types of signal communication pathways may be used, such as fiber optic communication pathways and / or wireless communication pathways.

[0023] The volatile memory 112 is any type of volatile memory now known or developed in the future. Examples include dynamic random access memory (RAM) or static RAM. Typically, the volatile memory 112 is characterized by random access, but this is not required unless expressly stated. In the computer 101, the volatile memory 112 is located in a single package and is internal to the computer 101, but, alternatively or additionally, the volatile memory may be distributed among multiple packages and / or located externally relative to the computer 101.

[0024] Persistent storage 113 is any form of non-volatile memory for computers known now or developed in the future. The non-volatility of the memory means that the stored data is maintained regardless of whether power is supplied to the computer 101 and / or directly to the persistent storage 113. Persistent storage 113 can be a read-only memory (ROM), but typically at least a portion of the persistent storage allows the writing of data, the deletion of data, and the rewriting of data. Some common forms of persistent storage include disks and solid-state storage devices. Operating system 122 can take several forms, such as various known proprietary operating systems or operating systems of the open source portable operating system interface type using a kernel. The code included in box 150 typically includes at least some of the computer codes involved in executing the method of the present invention.

[0025] The peripheral device group 114 includes a group of peripheral devices of the computer 101. The data communication connection between the peripheral devices and other components of the computer 101 can be implemented in various ways, such as a Bluetooth connection, a near field communication (NFC) connection, a connection made by a cable (such as a universal serial bus (USB) type cable), a plug-in type connection (e.g., a secure digital (SD) card), a connection made through a local area communication network, and even a connection made through a wide area network such as the Internet. In various embodiments, the UI device group 123 may include components such as display screens, speakers, microphones, wearable devices (such as goggles and smart watches), keyboards, mice, printers, touchpads, game controllers, and tactile devices. Storage 124 is an external storage device, such as an external hard drive, or a pluggable storage device, such as an SD card. Storage 124 can be persistent and / or volatile. In some embodiments, storage 124 can take the form of a quantum computing storage device for storing data in the form of quantum bits. In embodiments where the computer 101 needs to have a large amount of storage (e.g., where the computer 101 locally stores and manages a large database), the storage may be provided by a peripheral storage device designed to store very large amounts of data, such as a storage area network (SAN) shared by multiple geographically distributed computers. The IoT sensor group 125 is composed of sensors that can be used in IoT applications. For example, one sensor may be a thermometer, while another sensor may be a motion detector.

[0026] The network module 115 is a collection of computer software, hardware, and firmware that allows the computer 101 to communicate with other computers via the WAN 102. The network module 115 may include hardware such as a modem or a Wi-Fi signal transceiver, software for packetizing and / or depacketizing data transmitted over a communication network, and / or a network browser software for transmitting data over the Internet. In some embodiments, the network control function and the network forwarding function of the network module 115 are executed on the same physical hardware device. In other embodiments (e.g., embodiments utilizing software defined networks (SDN)), the control function and the forwarding function of the network module 115 are executed on physically separated devices so that the control function manages several different network hardware devices. Computer-readable program instructions for executing the method of the present invention can generally be downloaded to the computer 101 from an external computer or an external storage device via a network adapter card or a network interface included in the network module 115.

[0027] WAN 102 is any wide area network (e.g., the Internet) capable of transmitting computer data over non-local distances by any technology now known or developed in the future for transmitting computer data. In some embodiments, WAN 102 may be replaced and / or supplemented by a local area network (LAN) designed to transmit data between devices located in a local area, such as a Wi-Fi network. WANs and / or LANs typically include computer hardware, such as copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and edge servers.

[0028] End-user device (EUD) 103 is any computer system used and controlled by an end-user (e.g., a business customer operating computer 101), and may take any of the forms discussed above in connection with computer 101. EUD 103 typically receives helpful and useful data from the operation of computer 101. For example, in the hypothetical case where computer 101 is designed to provide recommendations to an end-user, the recommendations would typically be transmitted from network module 115 of computer 101 to EUD 103 via WAN 102. In this manner, EUD 103 may display or otherwise present the recommendations to the end-user. In some embodiments, EUD 103 may be a client device, such as a thin client, a heavy client, a mainframe computer, a desktop computer, etc.

[0029] Remote server 104 is any computer system that provides at least some data and / or functionality to computer 101. Remote server 104 may be controlled and used by the same entity that operates computer 101. Remote server 104 represents a machine that collects and stores helpful and useful data for use by other computers, such as computer 101. For example, in the hypothetical case where computer 101 is designed and programmed to provide recommendations based on historical data, then the historical data may be provided to computer 101 from remote database 130 of remote server 104.

[0030] The public cloud 105 is any computer system available for use by multiple entities that provides on-demand availability of computer system resources and / or other computer capabilities, particularly data storage (cloud storage) and computing capabilities, without direct active management by users. Cloud computing typically leverages the sharing of resources to achieve consistency and economy of scale. The direct and active management of the computing resources of the public cloud 105 is performed by computer hardware and / or software of the cloud coordination module 141. The computing resources provided by the public cloud 105 are typically implemented by virtual computing environments running on various computers that constitute the host physical machine group 142, which is the universe of physical computers in and / or available to the public cloud 105. Virtual computing environments (VCEs) typically take the form of virtual machines from the virtual machine group 143 and / or containers from the container group 144. It will be appreciated that these VCEs can be stored as images and can be transferred between various physical machine hosts as images or after instantiation of the VCEs. The cloud coordination module 141 manages the transmission and storage of images, deploys new instantiations of VCEs, and manages active instantiations of VCE deployments. Gateway 140 is a collection of computer software, hardware, and firmware that allows public cloud 105 to communicate over WAN 102 .

[0031] Some further explanation of virtualized computing environments (VCEs) will now be provided. A VCE can be stored as an "image". A new active instance of the VCE can be instantiated from the image. Two common types of VCEs are virtual machines and containers. Containers are VCEs that use operating system-level virtualization. This refers to an operating system feature in which the kernel allows the existence of multiple isolated user space instances, called containers. From the perspective of the programs running in them, these isolated user space instances typically behave like actual computers. Computer programs running on ordinary operating systems can utilize all of the resources of the computer, such as connected devices, files and folders, network shares, CPU capabilities, and quantifiable hardware capabilities. However, programs running within a container can only use the contents of the container and the devices assigned to the container, a feature known as containerization.

[0032] Private cloud 106 is similar to public cloud 105, except that the computing resources are only available to a single enterprise. Although private cloud 106 is depicted as communicating with WAN 102, in other embodiments, the private cloud can be completely disconnected from the Internet and only accessible through a local / private network. A hybrid cloud is a combination of multiple clouds of different types (e.g., private, community, or public cloud types), typically implemented by different vendors. Each of the multiple clouds remains a separate and discrete entity, but the larger hybrid cloud architecture is bound together by standardized or proprietary technologies that enable coordination, management, and / or data / application portability between the multiple constituent clouds. In this embodiment, both public cloud 105 and private cloud 106 are part of a larger hybrid cloud.

[0033] refer to Figure 2 As previously mentioned, in a collaborative mixed reality environment, objects from different sources may interact with each other to achieve collaborative goals. These objects may be developed and tested individually. Because the interactions between these objects may be new and may not have been previously tested, their interactions may expose vulnerabilities or other undesirable behaviors that may need to be addressed. Currently, there is no established or agreed-upon testing framework for testing and debugging collaborative mixed reality objects originating from different sources.

[0034] In certain embodiments, neural networks (and more specifically encoder-decoder architectures based on generative artificial intelligence) can be trained to generate collaborative mixed reality test cases from various types of inputs. These inputs can include, for example, individual test cases for mixed reality object pairs and the mixed reality objects themselves expressed as a set of properties. The collaborative mixed reality test cases can be used to evaluate the interaction of mixed reality objects in a collaborative mixed reality environment.

[0035] Generally speaking, mixed reality objects have various properties that define their behavior and appearance in a mixed reality environment. These properties may include one or more of position, scale, orientation, animation, interactivity, material, audio, etc. For example, mixed reality objects may be placed at specific locations in the real world, and they may be anchored to the real world surface or remain free-floating. This allows users to interact with virtual objects at specific locations and provide a sense of presence and immersion. Mixed reality objects may also be sized and scaled to match the size and scale of real-world objects, thereby enhancing the authenticity and immersion of the mixed reality experience. Mixed reality objects may be rotated and oriented in 3D space, allowing users to view them from different angles and perspectives. In some cases, mixed reality objects may be animated to simulate real-world behaviors such as movement, lighting, and particle effects. They may also be designed to respond to user interactions, such as gestures or controller inputs, allowing users to manipulate and interact with virtual objects in real time. In some cases, mixed reality objects may also be designed with specific materials such as glass or metal that define their appearance and how they respond to lighting and shadows. In some cases, they can also be designed with audio effects, such as sound effects or speech, that enhance their behavior and appearance in the mixed reality environment.

[0036] Similarly, depending on the specific mixed reality technology used and the design of the mixed reality object, the mixed reality objects can interact with each other in various ways. For example, mixed reality objects can interact with each other by colliding or intersecting in 3D space. This can be used to simulate real-world interactions, such as objects bouncing off each other or blocking each other's movement. Mixed reality objects can also constrain each other so that their movement and / or behavior are linked in some way. For example, one mixed reality object can be attached to another so that it follows the movement of another object. Mixed reality objects can also interact with each other by simulating physical forces (such as gravity, friction, and spring forces). This allows mixed reality objects to behave in a realistic and intuitive way and can be used to create complex interactions between mixed reality objects. In some cases, mixed reality objects can interact with each other using audio, such as making sounds when they collide or react with each other in some way. In other cases, mixed reality objects can interact with each other by exchanging data, such as changing the behavior of another mixed reality object or triggering an event based on the state of one mixed reality object. Mixed reality objects can interact with each other by communicating via a network, such as exchanging data or triggering an event between multiple devices. These only represent some of the ways in which mixed reality objects can interact, and are not intended to be limiting.

[0037] In order to generate collaborative mixed reality test cases using a neural network and more specifically using an encoder-decoder architecture based on generative AI, the encoder-decoder architecture based on generative AI can be trained using input data, which is also referred to as "training data" in this article. As described above, mixed reality objects may have different attributes that can be used to classify mixed reality objects. These attributes can provide a type of training data. Similarly, because mixed reality objects are typically developed and tested separately, mixed reality objects will typically have their own test cases. These test cases can provide another type of training data. In addition, in a collaborative mixed reality environment, when mixed reality objects from different sources interact with each other, logs describing the interaction can be generated and stored. These logs can provide yet another type of training data.

[0038] Figure 2 is a high-level block diagram illustrating the initial steps of training a generative AI-based encoder-decoder architecture to capture an embedding of a mixed reality object 202. As shown, the generative AI-based encoder-decoder architecture 200 may receive as input a mixed reality object 202 expressed as a set of attributes. In this initial step, a variational autoencoder (VAE) may be trained to efficiently capture the mixed reality object 202 in a Gaussian embedding space. To achieve this, the attributes of the mixed reality object 202 may be tokenized and fed to a VAE encoder 204 to generate a bottleneck embedding 206. The bottleneck embedding 206 may then be fed to a decoder 208 to reconstruct the original mixed reality object 202 expressed as a set of attributes. In certain embodiments, the VAE encoder 204 and decoder 208 may utilize a multi-layer perceptron (MLP) structure. The VAE encoder 204 may be configured to minimize a reconstruction loss and a Kullback-Leibler (KL) divergence loss to ensure that the embedding is in a Gaussian distribution. After the system is trained, the encoder 204 may be used to generate an embedding 206 of a mixed reality object, and the decoder 208 may be ignored.

[0039] Figure 3 2 is a high-level block diagram illustrating the next step of training an encoder-decoder architecture based on generative AI to collect embeddings of individual test cases of mixed reality objects 202. In this step, the individual test cases 302 may be represented as a sequence of intent tokens 304a and mixed reality object tokens 304b, which may then be converted into embeddings 306 via a learnable embedding layer. In some embodiments, the system 300 may use a combination of Figure 2The encoder 202 described is used as an embedding layer for mixed reality objects to capture attribute correlations. The system 300 can use a masked autoencoder 308 that masks a certain percentage (e.g., 75%) of the tokens 304, and the encoder and decoder can then be reconstructed at box 310. The masked autoencoder 308 can be built on top of the transformer and is powerful enough to capture cross-channel correlations. By enabling masked token summarization, the system 300 can learn better and more compact summary representations of test cases. The encoder of the masked autoencoder 308 can be used to obtain test case embeddings. For the latter sampling requirements, a normal distribution can be implemented in the encoder space.

[0040] Figure 4 is a high-level block diagram showing the next steps in training a generative AI-based encoder-decoder architecture to generate collaborative mixed reality test cases. Figure 3 The described training learns the test case embeddings and can follow a simple supervised fine-tuning method. When provided with two separate test cases 402a, 402b, the system 400 can generate a collaborative test case 410. The ground truth collaborative test case 414 can be used for this training to minimize the loss 412 (e.g., cross entropy or softmax loss) and train the model. The fine-tuning encoder 406 receives the embeddings 404a, 404b of the two test cases 402a, 402b, and the fine-tuning decoder 408 generates the original generated collaborative test case 410 from the embedding space. The fine-tuning decoder weights can be initialized from the mean absolute error (MAE) decoder, where the embedding space is constrained to a normal distribution. This enables the fine-tuning decoder 408 to effectively sample from the embedding space and generate new collaborative test cases 410. The end-to-end process is trained in a supervised setting to fine-tune to the problem of interest.

[0041] Figure 5 A general overview of a generative AI-based encoder-decoder architecture 500 is provided after training is performed and the generative AI-based encoder-decoder architecture is placed in inference or prediction mode. As shown, the generative AI-based encoder-decoder architecture 500 is configured to output a predicted collaborative mixed reality test case 510 upon receiving a pair of mixed reality objects 502 (A and B) expressed as a set of attributes and a separate test case 502 associated with the mixed reality object 502. As shown, the encoder 504 outputs an embedding (E(A) and E(B)) for each of the mixed reality objects 502 and the associated separate test case. These embeddings can be connected and passed to the decoder 508 to generate and predict the collaborative mixed reality test case (A, B) 510.

[0042] refer to Figure 6-12, provides an example of two mixed reality objects and associated individual test cases for generating a collaborative mixed reality test case for evaluating the interaction of the mixed reality objects. In this example, the first mixed reality object is a virtual laptop computer and the second mixed reality object is a virtual mouse. These mixed reality objects can be developed and tested individually. The generated collaborative mixed reality test case can be used to evaluate the interaction of the virtual laptop computer and the virtual mouse to determine or predict whether they operate correctly with each other. Figure 6 and 7 An exemplary individual test case for a virtual laptop is shown; Figure 8 and 9 An exemplary individual test case for a virtual mouse is shown; and Figures 10 to 12 An exemplary collaborative mixed reality test case for evaluating the interaction between a virtual laptop and a virtual mouse is shown.

[0043] refer to Figure 6 and 7, to verify the functionality and interaction of the virtual laptop mixed reality object within the mixed reality environment, a separate test case for the virtual laptop may include the following: launching 602 a mixed reality application; verifying 604 that the mixed reality headset is properly connected and calibrated; verifying 606 that the mixed reality environment is successfully loaded; locating 608 and selecting 608 a "virtual laptop" mixed reality object from the available options; verifying 610 that the virtual laptop object is accurately placed within the mixed reality environment; interacting 612 with the virtual laptop using available gestures or controller input; verifying 614 that the virtual laptop responds appropriately to interactions, such as opening and closing the lid or pressing virtual keys; testing 616 the functionality of the virtual laptop's features, such as the touchpad, keyboard, and screen; verifying 618 that the touchpad accurately tracks finger movements and performs actions such as scrolling and clicking; testing 620 typing on the virtual laptop's keyboard and verifying that keystrokes are properly registered; interacting 622 with the virtual laptop's screen by launching applications or navigating through menus; verifying 702 that the virtual laptop the virtual laptop's screen accurately displays content and responds to user input; tests 704 the virtual laptop's interaction with other mixed reality objects, such as dragging and dropping files onto the virtual laptop's screen; verifies 706 that the virtual laptop properly handles interactions with other objects and responds accordingly; tests 708 the virtual laptop is resized and repositioned in the mixed reality environment; verifies 710 that the virtual laptop can be resized without any problems and maintains its visual quality and interactions; verifies 712 that the position and orientation of the virtual laptop can be accurately adjusted; tests 714 object collision detection by overlapping the virtual laptop with other objects in the environment; verifies 716 that the virtual laptop behaves realistically when the virtual laptop collides or overlaps with other objects; tests 718 the persistence of the virtual laptop by saving the current mixed reality scene and exiting the application; restarts 720 the application and reloads the saved scene; and verifies 722 that a previously placed virtual laptop is correctly restored and maintains its properties and interactions.

[0044] Expected results for a separate test case for a virtual laptop may include the following: the virtual laptop mixed reality object is accurately placed within the mixed reality environment; the virtual laptop responds appropriately to interactions, such as opening and closing the lid or pressing virtual keys; the touchpad on the virtual laptop accurately tracks finger movements and performs actions like scrolling and clicking; typing on the virtual laptop's keyboard correctly registers keystrokes; the virtual laptop's screen accurately displays content and responds to user input; the virtual laptop correctly handles interactions with other mixed reality objects, such as dragging and dropping files; resizing and repositioning the virtual laptop works without any issues and maintains its visual quality and interactions; the virtual laptop behaves realistically when colliding or overlapping with other objects; and a saved mixed reality scene correctly restores a previously placed virtual laptop, including its properties and interactions.

[0045] refer to Figure 8 and 9, in order to verify the functionality and interaction of the virtual mouse mixed reality object within the mixed reality environment, separate test cases for the virtual mouse may include the following: launching 802 the mixed reality application; verifying 804 that the mixed reality headset is properly connected and calibrated; verifying 806 that the mixed reality environment is successfully loaded; locating 808 and selecting the "virtual mouse" mixed reality object from the available options; verifying 810 that the virtual mouse object is accurately placed within the mixed reality environment; interacting with the virtual mouse using available gestures or controller input 812; verifying 814 that the virtual mouse cursor accurately follows the movements of the user's hand or controller; testing 816 the functionality of features of the virtual mouse, such as left-click, right-click, and scroll wheel; verifying 818 that left-clicking the virtual mouse triggers the expected action, such as selecting an object or interacting with a user interface element; verifying 820 that right-clicking the virtual mouse triggers the expected action, such as displaying a context menu or performing a secondary action; testing 822 that scrolling the virtual mouse Scroll wheel and verify that scrolling actions are registered correctly; interact 902 with the virtual mouse cursor on the mixed reality environment, such as hovering over interactive elements or dragging objects; verify 904 that the virtual mouse cursor interacts accurately with the environment and triggers appropriate actions; test 906 resizing and repositioning of the virtual mouse within the mixed reality environment; verify 908 that the virtual mouse can be resized without any issues and maintains its visual quality and interactions; verify 910 that the position and orientation of the virtual mouse can be accurately adjusted; test 912 object collision detection by overlapping the virtual mouse with other objects in the environment; verify 914 that the virtual mouse behaves realistically when colliding or overlapping with other objects; test 916 the persistence of the virtual mouse by saving the current mixed reality scene and exiting the application; restart 918 the application and reload the saved scene; and verify 920 that the previously placed virtual mouse is correctly restored and retains its properties and interactions.

[0046] Expected results for individual test cases for the virtual mouse may include the following: the virtual mouse mixed reality object is accurately placed within the mixed reality environment; the virtual mouse cursor accurately follows the movements of the user's hand or controller; left-clicking the virtual mouse triggers the expected action; right-clicking the virtual mouse triggers the expected action; scrolling the virtual mouse's wheel correctly registers the scrolling action; the virtual mouse cursor accurately interacts with the mixed reality environment and triggers appropriate actions; resizing and repositioning the virtual mouse works without any issues and maintains its visual quality and interactions; the virtual mouse behaves realistically when colliding or overlapping with other objects; and a saved mixed reality scene correctly restores the previously placed virtual mouse, including its properties and interactions.

[0047] refer to Figures 10 to 12, in order to verify the functionality and interaction of the previously discussed virtual laptop and virtual mouse within the collaborative mixed reality environment, a collaborative mixed reality test case may be generated as follows: launch 1002 the collaborative mixed reality application; verify 1004 that all users are properly connected and calibrated with their mixed reality headsets; verify 1006 that the collaborative mixed reality environment is successfully loaded; establish 1008 connections between multiple users within the application; locate 1010 and select a "virtual laptop" mixed reality object from the available options; verify 1012 that the virtual laptop object is accurately placed within the mixed reality environment and visible to all users; interact 1014 with the virtual laptop using available gestures or controller input; verify 1016 All users can see the virtual laptop and interact with it in real time; test 1018 the functionality of features of the virtual laptop, such as opening and closing the lid, pressing virtual keys, and using the touchpad; verify 1020 that all users see features of the virtual laptop being used correctly and synchronously; locate 1022 and select a "virtual mouse" mixed reality object from available options; verify 1102 that the virtual mouse object is accurately placed within the mixed reality environment and visible to all users; interact with the virtual mouse 1104 using available gestures or controller input; verify 1106 that all users can see the movement and interaction of the virtual mouse cursor in real time; test 1108 the functionality of features of the virtual mouse, such as left-click, right-click, and wheel; verify 1010 that all users see features of the virtual mouse being used correctly and synchronously; test 1112 the cooperative interaction between the virtual laptop and the virtual mouse; have 1114 one user control the virtual laptop using the cursor of the virtual mouse, performing actions such as opening applications, typing, or clicking objects; verify 1116 that all users see actions of the virtual laptop performed by the user controlling the virtual laptop using the virtual mouse; test 1118 simultaneous interaction with the virtual laptop and the virtual mouse from different users; have 1120 multiple users control the virtual laptop using their respective virtual mouse objects, performing actions such as typing or clicking; verify 1202 that all The users see the actions of the virtual laptop performed by each user using their virtual mouse; testing 1204 resizing and repositioning of the virtual laptop and virtual mouse within the mixed reality environment; verifying 1206 that the virtual laptop and virtual mouse can be accurately resized and repositioned and that all users see the changes in real time; testing 1208 object collision detection by overlapping the virtual laptop and virtual mouse with other objects in the environment; verifying 1210 that the virtual laptop and virtual mouse behave realistically when colliding or overlapping with other objects; testing 1212 persistence of the virtual laptop and virtual mouse by saving the current collaborative mixed reality session and exiting the application;Restart 1214 the application and join the saved collaboration session; and verify 1216 that the previously placed virtual laptop and virtual mouse are correctly restored and maintain their properties and interactions.;

[0048] Expected results for a separate test case for the virtual mouse may include the following: multiple users are able to interact with the virtual laptop and virtual mouse simultaneously and see real-time updates; the virtual laptop responds appropriately to interactions performed by users controlling it with the virtual mouse, such as opening and closing the lid or pressing virtual keys; the virtual mouse cursor accurately follows the movements of the user's hand or controller and interacts with the mixed reality environment; collaborative interactions between the virtual laptop and virtual mouse performed by different users are synchronized and visible to all users; resizing and repositioning the virtual laptop and virtual mouse works without any issues and maintains their visual quality and interactions; the virtual laptop and virtual mouse behave realistically when colliding or overlapping with other objects; a saved collaborative session correctly restores a previously placed virtual laptop and virtual mouse, including their properties and interactions.

[0049] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention.In this regard, each frame in the flow chart or block diagram can represent a module, segment or part of a code, which includes one or more executable instructions for realizing the specified (multiple) logical functions.It should also be noted that in some alternative implementations, the function marked in the frame may not occur in the order marked in the figure.For example, the two frames shown in succession can actually be performed substantially at the same time, or these frames can sometimes be performed in reverse order, depending on the functions involved.Other implementations may not require all disclosed steps to realize the desired functions.It will also be noted that each frame of the block diagram and / or flow chart and the combination of frames in the block diagram and / or flow chart can be realized by a combination of a system based on special-purpose hardware or special-purpose hardware and computer instructions that performs a specified function or action.

Claims

1. A method for testing and debugging the interaction of collaborative mixed reality objects, the method comprising: receiving an input including a first mixed reality object expressed by a first set of attributes, a second mixed reality object expressed by a second set of attributes, a first individual test case associated with the first mixed reality object, and a second individual test case associated with the second mixed reality object; as well as A collaborative mixed reality test case is automatically generated from the input to evaluate interaction between the first mixed reality object and the second mixed reality object within a collaborative mixed reality environment.

2. The method according to claim 1, wherein: Automatic generation includes automatic generation using an encoder-decoder architecture based on generative AI.

3. The method according to claim 2, wherein: Further comprising training the generative AI-based encoder-decoder architecture using a collaborative log of mixed reality objects in a collaborative mixed reality environment, a single test case of the mixed reality objects, and at least one of properties of the mixed reality objects.

4. The method according to claim 2, wherein: The inputs are connected before being input into the generative AI based encoder-decoder architecture.

5. The method according to claim 2, wherein: The first mixed reality object and the second mixed reality object are developed and tested separately before being input into the generative AI based encoder-decoder architecture.

6. The method according to claim 1, wherein: The first mixed reality object and the second mixed reality object originate from different sources.

7. The method according to claim 1, wherein: The first set of attributes and the second set of attributes include at least one of physical attributes and metadata attributes.

8. A computer program product for testing and debugging the interaction of collaborative mixed reality objects, the computer program product comprising a computer readable storage medium having computer usable program code embodied therein, the computer usable program code being configured to, when executed by at least one processor, perform the following operations: receiving an input including a first mixed reality object expressed by a first set of attributes, a second mixed reality object expressed by a second set of attributes, a first individual test case associated with the first mixed reality object, and a second individual test case associated with the second mixed reality object; and A collaborative mixed reality test case is automatically generated from the input to evaluate interaction between the first mixed reality object and the second mixed reality object within a collaborative mixed reality environment.

9. The computer program product according to claim 8, wherein: Automatic generation includes automatic generation using an encoder-decoder architecture based on generative AI.

10. The computer program product according to claim 9, wherein: The computer usable program code is further configured to train the generative AI based encoder-decoder architecture using at least one of a collaboration log of mixed reality objects in a collaborative mixed reality environment, a single test case of the mixed reality objects, and properties of the mixed reality objects.

11. The computer program product of claim 9, wherein: The inputs are connected before being input into the generative AI based encoder-decoder architecture.

12. The computer program product of claim 9, wherein: The first mixed reality object and the second mixed reality object are developed and tested separately before being input into the generative AI based encoder-decoder architecture.

13. The computer program product of claim 8, wherein: The first mixed reality object and the second mixed reality object originate from different sources.

14. The computer program product of claim 8, the first set of attributes and the second set of attributes comprising at least one of physical and metadata attributes.

15. A system for testing and debugging the interaction of collaborative mixed reality objects, comprising: Hardware processor; as well as A memory storing a computer program product which, when executed by the hardware processor, causes the hardware processor to perform the steps of the method according to any one of claims 1 to 7.

16. A system comprising modules respectively configured to perform the steps of the method according to any one of claims 1 to 7.

17. A system for testing and debugging the interaction of collaborative mixed reality objects, the system comprising: at least one processor; at least one memory device operably coupled to the at least one processor and storing instructions for execution on the at least one processor, the instructions causing the at least one processor to: receiving an input including a first mixed reality object expressed by a first set of attributes, a second mixed reality object expressed by a second set of attributes, a first individual test case associated with the first mixed reality object, and a second individual test case associated with the second mixed reality object; as well as A collaborative mixed reality test case is automatically generated from the input to evaluate interaction between the first mixed reality object and the second mixed reality object within a collaborative mixed reality environment.

18. The system of claim 17, wherein: Automatic generation includes automatic generation using an encoder-decoder architecture based on generative AI.

19. The system of claim 18, wherein: The instructions further cause the at least one processor to train the generative AI-based encoder-decoder architecture using a collaborative log of mixed reality objects in a collaborative mixed reality environment, a single test case of the mixed reality objects, and at least one of properties of the mixed reality objects.

20. The system of claim 18, wherein: The inputs are connected before being input into the generative AI based encoder-decoder architecture.

21. The system of claim 18, wherein: The first mixed reality object and the second mixed reality object are developed and tested separately before being input into the generative AI based encoder-decoder architecture.

22. The system of claim 17, wherein: The first mixed reality object and the second mixed reality object originate from different sources.

Images