Generating interactive and immersive virtual and augmented reality environments corresponding to digital twins of real living elements

Abstract

A method and system for generating digital twins of live events for an interactive virtual environment. The system includes a server configured to receive a video data stream and an audio data stream. The server runs a computing module configured to: analyze a video data stream and corresponding metadata; determining a spatial depth between images of the video data stream; matching the object in each image with the coordinate matrix of the virtual space; matching the 3-D audio field with the coordinate matrix of the virtual space; a digital twinning of the live event is generated based on the image, the corresponding metadata, the spatial depth, the predetermined coordinate set and the coordinate matrix. The server is configured to generate a plurality of digital representations from a plurality of directions of a live event, the plurality of digital representations including at least one 360-degree digital representation and a plurality of unidirectional digital representations.

Description

Cross Reference to Related Applications

[0001] This application claims priority to U.S. Provisional Patent Application 63 / 504,614, filed May 26, 2023. The contents of this U.S. Provisional Patent Application are incorporated herein by reference. TECHNICAL FIELD

[0002] The present technology relates to electronic systems and methods for generating interactive virtual or augmented reality environments corresponding to digital twins of real life elements, particularly real life elements related to live events, including live events of performing arts, music, and sports, among others. BACKGROUND

[0003] Augmented reality and virtual reality applications are well known in the art. Some of these applications utilize augmented reality technology to enhance real life experiences through digital overlays. Other applications digitize real life elements to simulate them through virtual reality technology. For example, it has been recognized that digital elements can be integrated into a metaverse, a digital virtual world, designed to simulate, for example, physical laws, in which users can use objects similar to real life objects. This is well known in the field of electronic games.

[0004] However, in order to digitize large scale live events, there are still many technical problems to be solved, as these live events are difficult to compute and feed back to multiple users. In the future, the development of technology can provide suitable technical solutions. Generally, many patent documents teach various aspects of augmented reality and virtual reality technology.

[0005] For example, U.S. Patent No. 10,650,590, entitled “Method and system for fully immersive virtual reality,” issued May 12, 2020 to Pankaj N. Topiwala et al., teaches a method and system for creating a model-based reality view with a grid of video sensors over an area and extensive signal processing techniques. The method and system employs grid-based simultaneous capture, point cloud generation and refinement, morphological, polygon tiling and surface representation, texture mapping, data compression, and system-level components for user-directed signal processing to create a virtual world from user demand that can be viewed from any location within the area, with any gaze direction, at any time within the capture interval. The data stream is transmitted for near-term network-based delivery and 5G. Finally, since the virtual world is model-based in nature, it can be combined with augmentation (or deletion) functionality, creating a harmonious and realistic blend of real and synthetic worlds. This provides a fully immersive mixed reality world that supports full interaction using gestures.

[0006] For example, U.S. Patent No. US 11,196,964, entitled “Merged reality live event management system and method,” issued December 7, 2021 to Cevat Yerli, teaches a precise and flexible merged reality system and method configured to enable remote viewing and participation in real or virtual events. In the merged reality system, at least a portion of the real or virtual world can be replicated or streamed, respectively, into corresponding sub-universes included within a virtual world system, where some of the sub-universes include events that customers can view and interact from one or more associated customer physical locations. Other virtual elements, such as pure virtual objects or graphical representations of applications and games, can also be included in the virtual world system. Virtual objects include logic, virtual data, and models that provide self-computing capabilities and autonomous behavior. The system enables customers to virtually access, interact, and transact within events through the virtual world system.

[0007] For example, US Patent No. 11,202,037, granted to Cevat Yerli on December 14, 2021, entitled "Virtual Presence System and Method Through Merged Reality," teaches a virtual presence system that merges reality. This system includes a server comprising at least one processor and memory, the memory including a data storage device for storing a persistent virtual world system comprising one or more virtual copies of real-world elements. The virtual copies provide self-computing capabilities and autonomous behavior. The persistent virtual world system includes virtual copies of the physical locations hosting live events, wherein the persistent virtual world system is configured to communicate via a network with multiple connected devices, the multiple connected devices including sensing mechanisms configured to capture real-world data of the live events, enabling updates to the persistent virtual world system. This system allows clients to virtually access, interact with, and transact within a live event through the persistent virtual world system. A computer-implemented method is also provided.

[0008] For example, U.S. Patent Application No. US20200401576, published on December 24, 2021, by Cevat Yerli entitled "Interacting with real-world items and corresponding databases through a virtual twin reality," teaches a system comprising at least one cloud server of a cloud server computer system. The at least one cloud server includes at least one processor and memory for storing a persistent virtual world system. The persistent virtual world system includes one or more virtual objects, which include virtual data and models. The virtual objects include one or more virtual twins, purely virtual objects, or applications, wherein at least one of the virtual objects represents a store of real-world items connected to a periodically updated database associated with products in the at least one store. A user can access the store via a user device through the persistent virtual world system, the user device enabling interaction with elements within the store and between elements within the store.

[0009] For example, US Patent No. 11,245,872, granted to Cevat Yerli on February 8, 2022, entitled "Merged Reality Spatial Streaming of Virtual Spaces," teaches a merged reality system comprising at least one server storing a virtual world system including one or more virtual objects, each virtual object including a virtual copy at least at a first location, a virtual copy at a second location, and virtual copies of real-world elements located at at least the first and second locations. The at least one server is configured to: receive real-world data from real-world elements at the first and second locations from a plurality of connected devices communicating with the at least one server via a network; enrich the virtual copies using the real-world data from the first and second locations in the virtual world system and synchronize the virtual copies with the corresponding real-world elements; and overlay and stream at least a portion of the real-world data from the second location onto one or more surfaces, such as the virtual copy at the first location.

[0010] For example, U.S. Patent No. 11,032,588, granted to Haritaoglu et al. on June 8, 2021, entitled "Method and apparatus for spatially enhanced adaptive bitrate live streaming for 360 degree videoplayback," discloses an apparatus and method for realizing a spatially enhanced live streaming experience for virtual reality or 360-degree video live streaming. The live streaming video signal is encoded into multiple streams at different resolutions. A portion of the high-resolution video stream corresponding to the field of view across the entire 360-degree view is merged with the low-resolution video stream. The resulting video stream is called a spatially adaptive video stream. Multiple spatially adaptive video streams are generated to provide a high-resolution field of view across the entire 360 ​​degrees. When a viewer watches from different directions, the video player plays back one of the spatially adaptive video streams based on the direction the viewer is currently viewing.

[0011] In another example, U.S. Patent Publication No. 2020 / 0082389, entitled “Payment system for augmented, mixed, or virtual reality platforms integrated with cryptocurrency wallet,” filed by inventor Regev on September 9, 2019, and published on March 12, 2020, discloses an improved electronic reality system (e.g., augmented reality, mixed reality, and / or virtual reality system) that integrates a cryptocurrency wallet for transferring funds between a sender and a receiver based on interactions performed within the electronic reality system (e.g., drag-and-drop interactions, in which an image of an item to be purchased or an image representing funds to be transferred is dragged and dropped onto an image of a cryptocurrency wallet on an AR display).

[0012] In addition, companies like Google and Apple are developing devices designed to enable users to utilize virtual reality and augmented reality applications, such as Google Glass, Apple Glasses, Apple's AR / VR headset, and Magic Leap VR. Other companies are also developing various virtual reality and augmented reality devices for users. These devices all feature application programming interfaces (APIs), which allow developers to create compatible virtual reality and augmented reality applications.

[0013] The inventions to date are known to have many shortcomings, including a lack of sufficient computing power to run data-intensive virtual reality environments, insufficient flexibility in data flow control, and high cost of implementing these technologies in large-scale live events.

[0014] There is a need for a method and / or system for solving one or more problems described herein and / or for solving one or more problems that a person skilled in the art may notice upon familiarity with this specification.

[0015] The objective of this invention is to provide an efficient method and system for generating interactive virtual or augmented reality environments that correspond to digital twins of real-life elements and / or live events, and for effectively integrating these interactive virtual reality environments into the metaverse. This method and system must be more user-friendly for internet users and more reliable, easier to install, and more cost-effective than existing systems and methods.

[0016] Another objective of this invention is to provide an efficient method and system for applying techniques for generating interactive virtual or augmented reality environments to the computation of real-time digital live events using spatial networks. This method and system must be more user-friendly for internet users and more reliable, easier to install, and more cost-effective than existing systems and methods. Summary of the Invention

[0017] Therefore, the purpose of this invention is to improve some of the inconveniences existing in the prior art.

[0018] This invention relates to generating graphical representations of physical space and / or large-scale events from video feeds, with the aim of generating stable digital virtual reality representations of these representations so that users can access them and process sufficient bandwidth to interact with digital elements of the environment generated by the digital virtual reality representation.

[0019] The embodiments of this invention are developed based on researchers' understanding of at least one technical problem related to existing methods for generating and maintaining digital virtual or augmented reality. Engineers have considered the possibility of applying distributed ledger technology, blockchain technology, and automation technologies related to video game environment programming.

[0020] For example, existing technological systems do not appear to take into account the advancements in modern computing power and the enhanced security of blockchain technology.

[0021] Researchers also found that it would be beneficial to provide a system capable of generating digital representations of the spatial environment using a network configuration of multiple independent servers connected to the same network.

[0022] In the context of this specification, unless otherwise expressly stated, the words “first,” “second,” “third,” etc., when used grammatically as adjectives, are used only to distinguish the nouns they modify from one another, and not to describe any particular relationship between those nouns. Therefore, for example, it should be understood that the use of the terms “first server” or “second server” is not intended to imply any particular function, order, type, chronological order, hierarchy, or ranking of the servers / servers (for example), nor is its use (alone or in combination) intended to imply the necessary existence of any “first server” or “second server” in any given situation. Furthermore, as described elsewhere in this document, references to the “first” and “second” elements do not preclude the two elements from being the same actual element. Therefore, for example, in some cases, a “first” server or a “second” server may have the same software and / or hardware, while in others they may employ different software and / or hardware.

[0023] According to a first aspect of the present invention, a system is provided for generating digital twins in an interactive virtual or augmented reality environment. The digital twin corresponds to a physical live event, such as a live performance, a live art event, a sporting event, a concert, a public event, a private event, etc. The digital twin can also correspond to a portion of a live event, for example, any real element of the live event whose parameters have been recorded and digitized; this could be: the geometry of a concert hall, event venue, entertainment venue, amusement park, stadium, arena, or a performance by an artist, athlete, or their group, a portion of an exhibition displayed in a concert hall or stadium, etc. The system includes multiple video capture devices connected to a network. Each video capture device is adapted to operate within a predetermined coordinate system for the live event. This coordinate system defines the physical space where the live event occurs. Each video capture device is adapted to be remotely controlled. The video capture devices can be controlled independently of each other or as a group of devices. The control method can be pre-programmed into the system or partially or fully controlled by an operator. Each of the video capture devices is adapted to capture video of the live event. It should be understood that video capture devices can include any device suitable for capturing data and digitizing it to create a graphical representation of a virtual reality environment. For example, a video capture device can be a camera, an infrared camera, a dedicated (distance) measuring camera, a drone, etc. Those skilled in the art will understand that these specific methods may evolve and change with the development of future technologies, but will always remain within the scope of this invention.

[0024] For ease of understanding, "video" in this article refers to any data that a video capture device can capture and that can be used to create a graphical representation of a virtual reality environment.

[0025] Continuing with the first major aspect of the invention, each captured video has corresponding metadata. Metadata, as used herein, is data that provides information about other data, rather than the content of the data itself. For example, the metadata corresponding to a video includes other data related to that video, such as a description of the artist, athlete, or performance captured in the video, as well as the location where the video was captured, the type of device used to capture it, any encoders or libraries, or other data that may help generate a digital twin based on the captured video.

[0026] The system also features multiple audio recording devices, which can be adapted to record one or more audio tracks of a live event, or various parts of the live event, such as music, vocals, and ambient sounds. These audio recording devices can also be adapted to record a three-dimensional audio field of the live event, and to record different intensities of different audio tracks at different locations in the physical space where the live event occurs.

[0027] The system also includes one or more servers configured to receive data from each video capture device and each audio recording device. The servers operate a computing module configured to analyze the captured video and corresponding metadata, including but not limited to: determining the spatial depth between images captured by different video capture devices; matching each captured video with the coordinate matrix of the physical space where the live event occurred; matching a three-dimensional audio field with the coordinate matrix of the physical space; and generating a digital twin of at least a portion of the live event based on the captured video, corresponding metadata, spatial depth, a predetermined set of coordinates, and the coordinate matrix. The servers are configured to generate multiple digital representations from multiple directions of the live event, including at least one 360-degree digital representation and multiple unidirectional digital representations, each digital representation corresponding to the viewpoint of the digital twin within an interactive virtual reality environment. The server can employ any suitable computing technology to determine the dimensions of the physical space where the life event occurs, determine the shapes existing within that physical space and the distances between each of these shapes, and generate a digital twin containing a graphical representation of the physical space where the life event occurs, the shapes existing within that physical space, and the distances between each of these shapes. Furthermore, the system is configured to transmit the digital representation and audio to the receiving devices of users participating in the interactive virtual reality environment of the life event.

[0028] In another aspect of the technology, the system is also configured to receive data from user devices to feed that data back to the system, wherein the data may correspond to the user’s interaction with the virtual reality environment, including, for example, data related to the user’s position, user actions, and user gaze.

[0029] The device can be any wearable device with integrated motion capture sensors, AI glasses, virtual reality glasses, AI headphones for binaural stereo, virtual reality headsets, mobile phones, computers, laptops, smartwatches, game consoles, and cross-network devices.

[0030] It should be understood that the user equipment is not part of the protection claimed in this patent application, and the mention of the user, user equipment, or data generated and sent to the system by the user equipment is not part of the system, but is mentioned only for ease of understanding and does not limit the scope of the claims.

[0031] In another aspect of this technology, the module is also configured to determine real-life elements suitable for inclusion in the graphical representation of a digital twin within a virtual reality environment. Therefore, the module is configured to determine which real-life elements potentially present in the captured video need to be omitted from the digital representation and which real-life elements need to be retained as part of that representation. The module can use various parameters to determine which real-life elements need to be digitized and retained in the digital twin. For example, the module can determine the user's viewpoint and retain only real-life elements within the user's line of sight. For example, the module can estimate the computational complexity of digitizing and retaining a particular real-life element in the digital twin; if the complexity exceeds a certain threshold, the computation module will ignore the real-life element, i.e., it will neither digitize it nor add it to the digital twin.

[0032] In another aspect of this technology, a digital representation can be transmitted to the device in real time or with a delay after the live event. For example, a live event can be digitized to create a digital twin in a virtual reality environment, which the user can access in real time. Other events can also be digitized to create digital twins in a virtual reality environment, which the user can access after the live event ends. Therefore, a digital twin can be a permanent virtual reality environment that the user can access at any time, allowing the user to participate in any part of the live event they wish to participate in.

[0033] In another aspect of this technology, the digital representation, audio recording, corresponding metadata, and corresponding coordinate matrix data are parsed by the system and stored in a database for at least partial redistribution on demand. For example, a digital twin of a virtual reality environment can be added to different metaverses hosted on separate servers and running on different protocols. Therefore, any part of the digital twin of a live event can be used to generate a virtual reality environment accessible to users in one or more metaverses.

[0034] In another aspect of this technology, digital representations, audio recordings, corresponding metadata, and corresponding coordinate matrix data are stored in a distributed manner using blockchain technology. It should be understood that digital twins can be stored in a distributed or centralized manner, depending on the technical characteristics required by the virtual reality environment. In some cases, using blockchain technology to store digital twins in a distributed manner may have technical advantages, aiming to achieve decentralized use of resources, thereby improving the computing power of the virtual reality environment that generates and runs the digital twin of a live event. For example, it could be beneficial to allow users to access the virtual reality environment of a digital twin of a live event in real time.

[0035] In another aspect of this technology, non-fungible tokens (NFTs) are allocated to at least a portion of the digital twin, including digital representations, audio recordings, corresponding metadata, and corresponding coordinate matrix data. For example, a first NFT may be allocated to a first audio recording, a second NFT to a second audio recording, a third NFT to a third audio recording, a fourth NFT to a first video recording, a fourth NFT to a second video recording, a fifth NFT to the first metadata, a sixth NFT to the first set of coordinate matrix data, and so on. Non-fungible tokens are commonly abbreviated as NFTs.

[0036] It should be understood that in some implementations, the system can also use fungible tokens as a payment tool, an exchange tool, or a communication tool between users or between the system and users.

[0037] In another aspect of this technology, the system includes a payment processor adapted to identify non-fungible tokens (NFTs) purchased by a user and to allocate the corresponding NFT rights to the user. For example, the payment processor may be integrated into the system, or a third-party payment processor may communicate with the system via a network. It should be understood that the term "payment processor" is used herein in the broadest possible sense, including any technology that operates with monetary and non-monetary instruments such as cryptocurrencies, digital coupons, discount cards, points, etc. It should be understood that purchasing non-fungible tokens (NFTs) is understood herein as any person skilled in the art of using NFTs will understand, as NFTs may be associated with such unique rights, and these rights can be transferred to the purchaser via any suitable computational protocol, including any suitable blockchain protocol.

[0038] In another aspect of this technology, the system distributes non-fungible tokens to the digital twin of the live event. For example, rights can be allocated to the entire digital twin, allowing the NFT owner of the entire digital twin to control the allocation of further sub-rights to various parts of the digital twin. Alternatively, rights can be allocated to the entire digital twin, allowing the NFT owner of the entire digital twin to control the allocation of further sub-rights to various parts of the digital twin to multiple users, also known as partial ownership.

[0039] In another aspect of this technology, the system uses artificial intelligence to generate graphical representations. For example, AI algorithms can be used to determine how to approximate the shape and size of digital real-life elements in order to generate a digital twin of the entire live event before constructing its digital twin. For instance, AI can approximate the shape of the stage, the figures of the performers, the size of the stadium, the body types of the athletes, the seating arrangement of the audience, the dance floor, the mash pit, the crowd, the restaurant area with tables and chairs, walls, windows, balconies, musical instruments, sports equipment, and so on.

[0040] In another aspect of this technology, the system generates a data stream received by a user device. The data stream instructs the user device to generate augmented reality effects for the user, including: (a) projecting a portion of a digital twin of a live event or real-life element graphically onto a physical element or as a hologram or audio stream; (b) the user device presenting the user with a graphical representation and audio stream of a portion of the digital twin of the live event or real-life element; or (c) a combination of (a) and (b) described herein. Thus, the augmented reality effects generated for the user enable the user device to provide the user with a physical location experience digitized into a digital twin augmented reality environment.

[0041] According to a second major aspect of the invention, a method is provided for generating an interactive virtual reality environment in a metaverse. This environment includes multiple digital twins of real-life elements, which have been digitized into graphical representations. The real-life elements may include at least one of visual elements, audio elements, spatial elements, and tactile elements. For example, a visual element is a graphical representation of captured video, i.e., a graphical representation generated after the captured video has been processed and digitized into a suitable file format that can be integrated into the metaverse. For example, it could be a graphical representation similar to a video game, or a graphical representation similar to computer-generated virtual reality. Visual elements are graphical representations that a user can see through their user device from the perspective of an avatar, as the avatar observes the metaverse from different perspectives. Audio elements can be audio tracks heard by the user through their user device. For example, spatial elements can be any computer code that allows a user's avatar to determine the distance the avatar needs to travel from one location to another in the virtual reality environment of the metaverse, or the direction the avatar can look in, the direction in which objects can be thrown at it, the direction in which sounds can be heard, and so on. For example, a tactile element can be computer code designed to generate commands to a user device to simulate a latent sensation that a user might experience through the user device interface when interacting with the metaverse. This latent sensation could be, for example, vibration, pushing, electrical impulses, temperature differences, etc. The tactile element can be generated by a mechanism integrated into the user device and controlled by the system by sending commands to the device to engage the user. For example, if a user touches a vibrating object in the metaverse, the user's device might vibrate accordingly. The digital twin's virtual reality environment can connect to distributed ledger technology to facilitate complex computational operations and / or communication between the user device and the metaverse hosting server. The method includes the following steps: (i) Generate digital twins of real-life elements by performing the following operations: (a) Capturing video and generating visual elements (i.e., digital representations of the captured video), recording audio and generating audio elements (i.e., digital audio elements corresponding to the recorded audio), determining the tactile features of objects in the digital twin and generating tactile elements, and measuring the distances and spatial coordinates between shapes and / or spaces in the virtual reality environment of the digital twin and generating spatial elements; and (b) Record metadata corresponding to one of the visual, audio, spatial, and tactile elements; (ii) Store the digital twin on a server; (iii) Assign tokens to at least one of video elements, audio elements, spatial elements, and haptic elements; (iv) Connect the server to the Metaverse hosting server; (v) Integrating digital twins into the metaverse by generating virtual reality environments that correspond to the digital twins; (vi) To enable interactive experiences in a virtual reality environment by providing user devices with access to a virtual reality environment by allowing users to interact with at least one of visual, audio, spatial, and tactile elements.

[0042] In another aspect of this technology, the step of storing the digital twin on a server also includes storing the digital twin in a distributed manner using blockchain technology.

[0043] In another aspect of this technology, the method also includes communicating token data via a user device. For example, the token data can be used by the user to interact with and / or manage permissions for at least one of visual, audio, spatial, and tactile elements.

[0044] In another aspect of this technology, the token is a non-fungible token.

[0045] In another aspect of this technology, the step of providing user devices with access to a virtual reality environment includes receiving commands from the user via a metaverse hosting server or pre-programmed server from wearable devices with integrated motion capture sensors, AI glasses, virtual reality glasses, AI headphones for binaural sound, virtual reality headsets, mobile phones, computers, laptops, smartwatches, game consoles, and cross-network devices.

[0046] In another aspect of the technology, the step of measuring distance and spatial coordinates includes determining at least one of the following: the shape of the visual element, the distance between the first visual element and the second visual element, and the speed associated with the moving visual element.

[0047] In another aspect of the technology, the method also includes connecting a payment processor adapted to identify the purchased tokens and the corresponding rights associated with the purchased tokens.

[0048] In another aspect of this technology, the step of providing access to a virtual reality environment includes allowing the user to send their digital self to any location within the virtual reality environment.

[0049] In another aspect of this technology, the method also includes generating a portion of the metaverse based on a digital twin-based virtual reality environment.

[0050] In another aspect of the technology, the method also includes generating multiple virtual reality environments corresponding to multiple digital twins.

[0051] In another aspect of this technology, the method also includes integrating multiple digital twins into a virtual reality environment.

[0052] In another aspect of this technology, the method also includes selecting real-life elements for digitization in the digital twin. For example, this step includes determining which real-life elements will be digitized and added as digitized elements to the virtual reality environment, and which elements that are part of captured video, recorded audio, measured spatial data, or measured haptic data will be omitted from the digital twin and / or removed from the final virtual reality environment generated in the metaverse. In another embodiment, selecting real-life elements includes generating approximations of the real-life elements, which can be represented as visual elements, audio elements, haptic elements, spatial elements, or combinations thereof. Depending on the desired characteristics of the virtual reality environment, this approximation can be coarse or fine.

[0053] According to a third major aspect of the present invention, a method is provided for generating a three-dimensional graphical representation in a virtual reality environment, operated by a system. The three-dimensional graphical representation corresponds to real-life elements or live events. The system is connected to a network. The system includes a video capture device, a corresponding metadata recording device, an audio recording device, and a server configured to receive data from the video capture device, the audio recording device, and the metadata recording device. The server operates a computing module configured to analyze the captured video, the recorded audio, and the recorded corresponding metadata, and digitize objects displayed in the images of the captured video and at least partially identified by the corresponding metadata, thereby creating a three-dimensional graphical representation of real-life elements or live events. The method includes the following steps: a. Generate a 3D graphical representation model based on the objects displayed in the images of the captured video, the determination of the distances between the objects displayed in the images of the captured video, and the corresponding metadata analyzed; b. Link the audio tracks to elements in the 3D graphical representation; c. Store the 3D graphic representation on a server; d. Connect the server to the Metaverse hosting server; e. Integrate the 3D graphical representation into the metaverse by generating a virtual reality environment that corresponds to the 3D graphical representation; f. Provide user devices with access to the virtual reality environment.

[0054] This method can also generate a coordinate matrix of the physical space corresponding to real-life elements, and generate a three-dimensional graphical representation of at least a portion of the real-life elements.

[0055] The method can also determine the tactile features of objects displayed in images in the captured video and assign these features to 3D graphical representations for future interaction with the user's device or the user's virtual avatar.

[0056] This method can also determine the spatial coordinates of a three-dimensional graphical representation and assign these coordinates to objects displayed in images within the captured video.

[0057] This method can also enable interactive experiences in virtual reality environments by allowing users to interact with digital twins of objects displayed in images within captured videos.

[0058] According to a fourth aspect of the present invention, a system is provided for generating digital twins in an interactive virtual reality environment, the digital twin corresponding to at least one real-life element. The system is connected to a network. The system includes a video capture device, a corresponding metadata recording device, an audio recording device, and a server configured to receive data from the video capture device, the audio recording device, and the metadata recording device. The server operates a computing module configured to: (a) analyze the captured video, the recorded audio, and the recorded corresponding metadata; (b) determine the shapes of objects in the captured video and the distances between objects in the captured video; and (c) determine the coordinate matrix of a three-dimensional virtual space containing the real-life element.

[0059] According to a fifth aspect of the present invention, a system is provided for integrating an interactive virtual reality environment corresponding to a digital twin of real-life elements. The system is connected to a network. The system includes a video capture device, a corresponding metadata recording device, an audio recording device, and a server configured to receive data from the video capture device, the audio recording device, and the metadata recording device. The server operates a computing module configured to: (a) analyze the captured video, the recorded audio, and the recorded corresponding metadata to determine the spatial depth between real-life elements captured by the video capture device, recorded by the audio device, or recorded by the metadata device; (b) approximate the shape of the real-life elements; (c) determine real-life elements suitable for digitization; and (d) generate a digital twin of at least a portion of the real-life elements. The server is configured to send data to a metaverse hosting server to integrate the interactive virtual reality environment into the metaverse.

[0060] According to a sixth major aspect of the present invention, a system for generating computational space is provided, which integrates augmented reality, mixed reality, virtual reality, geofencing, cryptocurrency, and other technologies to provide a segmented database of digital space that simulates, is specifically designed for, and serves live music events, live performing arts, and all events categorized as live sporting events.

[0061] In another embodiment of the present invention, a system and method are provided for providing a shared virtual or augmented reality environment in which a digital overlay skin on a user's virtual body can be viewed by other participants in the shared virtual or augmented reality environment.

[0062] In another embodiment of the technology, a system and method are provided for informing participants in an augmented reality or virtual reality environment about upcoming events such as music, live entertainment, and sporting events, including dynamic pop-ups and / or advertisements.

[0063] In another embodiment of this technology, a system and method are provided for integrating an application programming interface into a digital twin, which allows communication via a protocol compatible with the “Play Money-Making Game Framework” integrated in the metaverse.

[0064] In another embodiment of the technology, a system and method are provided that allows participants to purchase event-related souvenirs, event tickets, or other items in an augmented reality or virtual reality environment, including dynamic pop-ups and / or advertisements.

[0065] In another aspect of this technology, metaverse events may differ from live events corresponding to a digital twin. Besides the virtual or augmented reality environment of the live event venue, which is recorded, digitized, and broadcast as the primary digital twin of the live event, virtual or augmented reality immersive events can also be held in multiple virtual or augmented reality environments. For example, a digital metaverse event space may contain auxiliary virtual or augmented reality environments generated by system 100, which can be represented in the metaverse as another room or another venue where virtual objects related to and / or unrelated to the primary live event can be displayed. For example, these virtual objects may include digitized NFT art galleries generated by the methods described herein, which are offset from the original venue metaverse of the digital twin corresponding to the virtual or augmented reality environment of the live event.

[0066] In another aspect of this technology, the invention includes an augmented reality environment configured to interact with a real-world live physical environment (e.g., buildings, urban spaces, parks, rooms, stadiums, etc.). This augmented reality environment runs on a server with a computer processor and a database. The server is connected to a network, enabling the augmented reality environment to communicate with multiple user devices (e.g., smartphones, computers, AR / VR headsets, etc.). The augmented reality environment can communicate with user devices via any suitable computer program and / or mobile application. The user device is configured to display content captured by its camera, whereby the augmented reality environment adds at least one digital overlay to the image. In one embodiment, the camera and / or lidar system associated with the user device is operable to generate a point cloud that maps a space. This point cloud provides the system not only with an image of the environment but also with the depth range and relative angular range of objects within the environment, thereby constructing the environment in three-dimensional space. U.S. Patent Publications 2020 / 0158869 and 2019 / 0244378 describe methods for generating LiDAR point clouds, the entire contents of each of which are incorporated herein by reference.

[0067] In another aspect of this technology, virtual or augmented reality environments are configured to record the profiles of users interacting with them. Each user profile contains the user's own associated data, such as preferences, age, gender, associated digital wallets (which are configured to store information related to one or more cryptocurrencies and / or one or more fiat currencies), and so on.

[0068] In another aspect of this technology, a virtual or augmented reality environment is associated with a pre-defined cryptocurrency configured to be used to purchase goods within the virtual or augmented reality environment, pay for activities within the environment, bet on activities and / or games within the environment, and / or access specific areas within the environment. For example, the cryptocurrency may be a cryptocurrency native to the virtual or augmented reality environment and / or the metaverse, and can only be exchanged within the virtual or augmented reality environment. In one implementation, the area where the cryptocurrency can be used is defined by at least one geofence.

[0069] In another aspect of this technology, different areas of the virtual or augmented reality environment are marked with unique identifiers, such as ID numbers, cryptocurrency tokens, QR codes, barcodes, etc. For example, the identifiers can be images, sounds, 3D objects, point cloud representations, etc.

[0070] It should be understood that data within or associated with a virtual or augmented reality environment is stored in a centralized or distributed database on a server hosting the virtual or augmented reality environment. Each virtual object in the virtual or augmented reality environment is indexed by a unique identifier. In some implementations, this unique identifier may be associated with relative lighting levels, altitude, location color, location purpose (e.g., game room, shop, corridor, etc.), and / or its relative location information relative to other unique identifiers. The information associated with the unique identifier is used to determine which digital overlays should be provided on these unique identifiers.

[0071] In another aspect of this technology, the virtual or augmented reality environment allows selection of a geographic location as a destination. After selecting a geographic location, the virtual reality or augmented reality environment automatically determines the starting point of selection (e.g., the user's location when selecting the target geographic location) by collecting sensor data from at least one geographic location sensor in the user device used to make the selection. In another embodiment, the starting point is determined using methods known in the art through received signal strength indications (RSSI) between the user device and multiple beacons (e.g., based on Wi-Fi, Bluetooth, WiMAX, etc.), such as the method in U.S. Patent Publication No. 2017 / 0295461, the entire contents of which are incorporated herein by reference. Using the selected target geographic location and the determined starting geographic location, the augmented reality platform automatically generates an optimal path (e.g., the shortest path considering traffic) between the two locations using any method in the art (e.g., the method described in U.S. Patent No. 9,886,036, the entire contents of which are incorporated herein by reference).

[0072] In another aspect of this technology, geofencing technology is utilized. The geofencing described herein is generated in accordance with the manner described in related technologies, such as U.S. Patent Nos. 10,375,514, 10,841,734, 10,834,212, and 10,979,849, the entire contents of each of which are incorporated herein by reference.

[0073] In another aspect of this technology, the virtual or augmented reality environment uses any facial recognition technology known in the art to identify other users based on the user's facial recognition, such as the technology described in U.S. Patent No. 9,275,269, the entire contents of which are incorporated herein by reference.

[0074] Virtual or augmented reality environments run on computer systems connected to a network, which have multiple computing devices, servers, and databases. The servers are configured to communicate with the multiple computing devices over the network. Servers typically include processing units with an operating system known in the art. The operating system executes computer programs to create and operationally maintain the virtual or augmented reality environment. Databases are typically used to store the data required to run the operating system, memory, and the programs needed to create and maintain the virtual or augmented reality environment.

[0075] In the context of this specification, unless otherwise expressly stated, "server" means a computer program running on suitable hardware and capable of receiving and executing requests (e.g., requests from devices) over a network. The hardware may be a physical computer or a physical computer system, but neither is necessary for the purposes of this invention. In the context of this specification, the use of the term "server" is not intended to imply that every task (e.g., receiving an instruction or request) or any particular task is received, executed, or caused to be executed by the same server (i.e., the same software and / or hardware); rather, it is intended to imply that any number of software elements or hardware devices may be involved in receiving / sending, executing any task or request or the consequences thereof, or causing any task or request or the consequences thereof to be executed; and all such software and hardware may be one server or multiple servers, both of which are included in the phrase "at least one server."

[0076] In the context of this specification, unless otherwise expressly stated, a "module" means a computer program that runs on suitable hardware and is capable of performing certain computational tasks or enabling such tasks to be performed. The term "module" has a broader technical meaning than "server" and is not associated with any specific hardware. The term "module" may mean one or more modules, and is not limited to any combination of modules or hardware used with them.

[0077] In the context of this specification, unless otherwise expressly stated, "device" means any computer hardware capable of running software suitable for the present and relevant task. Therefore, (non-limiting) examples of devices include personal computers (desktops, laptops, netbooks, etc.), smartphones and tablets, as well as network devices such as routers, switches, and gateways. It should be noted that the use of the term "device" in the context of this specification does not exclude its interaction with other devices as a server or module. The use of the term "device" does not exclude the use of multiple devices in receiving / sending, performing any task or request, the consequences of any task or request, or the steps of any method described herein, or in causing any task or request, the consequences of any task or request, or the steps of any method described herein to be performed.

[0078] In the context of this specification, unless otherwise expressly stated, "database" means any structured collection of data, regardless of its specific structure, database management software, or computer hardware that stores, implements, or otherwise provides the data. A database may reside on the same hardware as the processes that store or use the information in the database, or it may reside on separate hardware, such as a dedicated server or multiple servers.

[0079] In the context of this specification, unless otherwise expressly stated, the term "information" includes information of any nature or kind that can be stored in a database. Therefore, information includes, but is not limited to, audiovisual works (images, films, recordings, presentations, etc.), data (location data, numerical data, etc.), text (opinions, comments, questions, messages, etc.), documents, spreadsheets, etc.

[0080] In the context of this specification, unless otherwise expressly stated, the term "component" means including the software (suitable for the particular hardware environment) necessary and sufficient to perform the specific function referenced.

[0081] In the context of this specification, unless otherwise expressly stated, the terms "numerical representation" and "graphical representation" are used interchangeably.

[0082] In the context of this specification, unless otherwise expressly stated, the term "computer-usable information storage medium" is intended to include media of any nature and kind, including RAM, ROM, disks (CD-ROM, DVD, floppy disk, hard disk, etc.), USB flash drives, solid-state drives, tape drives, etc.

[0083] Various implementations of the present invention have at least one of the above-mentioned objectives and / or aspects, but not all of them are required. It should be understood that some aspects of the present invention, developed to achieve the above objectives, may not achieve those objectives, and / or may achieve other objectives not specifically listed herein.

[0084] Additional and / or alternative features, aspects and advantages of the implementation of the present invention will become apparent from the following description, drawings and appended claims. Attached Figure Description

[0085] To better understand the technical aspects and other features of the present invention, reference will be made to the following description, used in conjunction with the accompanying drawings, wherein: Figure 1 This is a schematic diagram of a system that communicates with a metaverse and a user equipment implemented according to an embodiment of the present invention; Figure 2 This is a schematic diagram of an alternative implementation of a system for communicating with a user equipment according to an embodiment of the present invention; Figure 3 This is a schematic diagram of another alternative implementation of a system for communicating with a user equipment according to an embodiment of the present invention; Figure 4 This is a schematic diagram of another alternative implementation of a system for communicating with a user equipment according to an embodiment of the present invention; Figure 5 A block diagram depicts an alternative method implemented according to an embodiment of the present invention; Figure 6 A block diagram depicts an alternative method implemented according to an embodiment of the present invention; Figure 7 A block diagram depicts an alternative method implemented according to an embodiment of the present invention; Figure 8 A block diagram depicts a method for generating a three-dimensional digital twin according to an embodiment of the present invention; Figure 9 The diagram depicts a block diagram of another method for generating a three-dimensional digital twin, implemented according to an embodiment of the present invention. Figure 10 The diagram depicts a block diagram of another method for generating a three-dimensional digital twin, implemented according to an embodiment of the present invention. Figure 11 The diagram depicts a method for user interaction with a virtual reality environment implemented according to an embodiment of the present invention. Figure 12 This is a schematic diagram of a user equipment implemented according to an embodiment of the present invention. Detailed Implementation

[0086] The following detailed reference will be made to some specific examples of embodiments of the present invention, including several embodiments that the inventors believe are suitable for understanding the technology. Examples of specific embodiments are shown in the accompanying drawings. Although the technology has been described in conjunction with these specific embodiments, it should be understood that this is not intended to limit the invention to the described embodiments. Rather, the invention is intended to cover alternatives, modifications, and equivalents that may be included within the scope of the invention as defined by the appended claims.

[0087] The techniques and mechanisms of this invention can be described within the context of this document. However, it should be noted that the techniques and mechanisms of this invention are suitable for various combinations of modes and are not limited to the examples and embodiments listed. Specific details are set forth in the following description to provide a comprehensive understanding of the invention. Specific exemplary embodiments of the invention may be implemented without some or all of these specific details. In other instances, some well-known process operations have not been described in detail to avoid unnecessarily obscuring the invention.

[0088] This disclosure describes flexible virtual reality and augmented reality systems and methods configured to remotely generate fully immersive virtual reality environments and partially immersive augmented reality environments that users can choose to view and / or participate in. The virtual reality and augmented reality systems are generated by fusing the digitization of real-life elements and spaces and generating artificial objects within the real-world environment itself. At least a portion of the real world or virtual world can be copied or streamed separately to corresponding spaces in a metaverse, thereby creating sub-universes and / or subspaces of a larger virtual world. These sub-universes and / or subspaces can be adapted to host digital twins of live events that users can view and / or interact with from one or more associated user physical locations. Virtual elements, i.e., purely virtual objects, as well as graphical representations of applications, games, and characters, can be generated by the system or by third-party resources accessing the virtual world via a network.

[0089] Figure 1 An exemplary implementation of System 100 is illustrated. System 100 is configured to generate digital twins of real-life elements and / or events, and digitize them in real time to create interactive 3D digital representations. These digital representations are combined into 3D models, which can be integrated as virtual or augmented reality environments into a metaverse and / or sub-universe or virtual space. System 100 can be hosted on a single server or a combination of multiple servers. In this embodiment, System 100 has a device control and data collection server 101 and a computing server 106. Those skilled in the art will understand that servers 101 and 106 can both be multiple servers and / or a cloud-based configuration hosting multiple modules, as described in detail below, and / or a blockchain technology configuration hosted on a distributed ledger configured to run multiple modules, as described in detail below.

[0090] System 100 is connected to a network (e.g., the Internet) for wireless or wired communication and is processed by at least one mobile communication computing device. Alternatively, the wireless and wired communication and connectivity between devices and components described herein include wireless network communication such as Wi-Fi, Worldwide Interoperability for Microwave Access (WIMAX), radio frequency (RF) communication (including RF identification (RFID), near field communication (NFC), Bluetooth (including Bluetooth Low Energy, BLE), ZigBee, infrared (IR) communication, cellular communication, satellite communication, Universal Serial Bus (USB), Ethernet communication, communication over fiber optic cables, coaxial cables, twisted-pair cables, and / or any other type of wireless or wired communication. System 100 may be a virtualized computing system capable of executing any or all of the software and / or application components presented herein on device control and data collection server 101 and computing server 106. In some aspects, computer system 100 may be operable to be implemented either in hardware or a combination of software and hardware, either deployed in a dedicated computing device, integrated into other entities, or distributed across multiple entities or computing devices.

[0091] Device control and data collection server 101 and computing server 106 can be any suitable electronic device including at least a processor and memory, such as a server, blade server, mainframe, mobile phone, personal digital assistant (PDA), smartphone, desktop computer, netbook, tablet computer, workstation, laptop computer, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the embodiments of the invention described and / or claimed herein. For example, device control and data collection server 101 and / or computing server 106 include components such as a processor, system memory having random access memory (RAM) and read-only memory (ROM), and a system bus (not shown) connecting the memory to the processor. Device control and data collection server 101 and computing server 106 can be configured to include components such as storage devices for storing an operating system and one or more applications, network interface units, and / or input / output controllers (not shown). Each of these components is operable to be interconnected via at least one bus. Input / output controllers are operable to receive and process input from or to provide output to multiple other devices, including but not limited to alphanumeric input devices, mice, electronic pens, display units, touch screens, signal generating devices (e.g., speakers), or printers. Processors can be general-purpose microprocessors (e.g., central processing units (CPUs)), graphics processing units (GPUs), microcontrollers, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), programmable logic devices (PLDs), controllers, state machines, gate or transistor logic circuits, discrete hardware components, or any other suitable entity or combination thereof capable of performing computations, processing instructions for execution, and / or other information operations.

[0092] It should be understood that there may be multiple processors, multiple buses, and multiple different types of memory (e.g., a combination of DSP and microprocessor, multiple microprocessors, or a combination of one or more microprocessor and DSP cores). Multiple computing devices may be operable to be interconnected, with each device providing some of the necessary operations (e.g., server groups, blade server groups, or multiprocessor systems). Alternatively, certain steps or methods may be operable by circuitry specific to a given function.

[0093] System 100 generates instructions operable to be implemented in hardware, software, firmware, or any combination thereof. A computer-readable medium is operable to provide volatile or non-volatile storage for one or more sets of instructions (e.g., operating systems, data structures, program modules, application programs, or other data embodying one or more methods or functions described herein). The computer-readable medium is operable to include memory, processor, and / or storage media, and is operable to be a single medium or multiple media (e.g., a centralized or distributed computer system) for storing one or more sets of instructions. Non-transitory computer-readable media includes all computer-readable media, with the sole exception of transient propagation signals themselves. Instructions can be transmitted or received over a network via a network interface unit operable to contain modulated data signals (e.g., carrier waves or other transmission mechanisms) and include any transport medium. The term "modulated data signal" means a signal whose one or more characteristics are altered or set in a manner that encodes information in the signal.

[0094] Storage devices and memories include, but are not limited to, volatile and non-volatile media such as caches, RAM, ROM, EPROM, EEPROM, flash memory or other solid-state storage technologies; optical discs (such as digital versatile optical discs (DVD), HD-DVD, BLU-RAY, optical discs (CD) or CD-ROM) or other optical storage devices; magnetic tape cassettes, magnetic tapes, disk storage devices, floppy disks or other magnetic storage devices; or any other medium that can be used to store computer-readable instructions and is accessible by system 100.

[0095] The device control and data collection server 101 has a live data collection module 105 and a control module 105a. It should be understood that the data collection module 105 may be located on one device control and data collection server 101, while the control module 105a may be located on another device control and data collection server 101.

[0096] Control module 105a controls multiple video capture devices 102, multiple audio recording devices 103, and multiple sensors 104. It should be understood that any number of video capture devices 102, audio capture devices 103, and sensors 104 can exist, from one to hundreds. The number of video capture devices 102, audio capture devices 103, and sensors 104 depends on the physical space and / or real-life elements that need to be captured and digitized. Sometimes, a single video capture device 102, a single audio capture device 103, and a single sensor 104 are sufficient. Sometimes, one or more video capture devices 102 are sufficient, without the need for other devices. Sometimes, one or more audio capture devices 103 are sufficient, without the need for other devices. The combination of video capture devices 102, audio capture devices 103, and sensors 104 depends on the amount of data stream required to digitize the live event or real-life elements. Video capture devices 102 include various types of camera devices, such as video cameras, infrared cameras, ultraviolet cameras, etc. Audio capture device 103 includes devices for recording various sounds that can be used to map space and / or generate during live events. Sensors 104 include motion sensors, position sensors, proximity sensors, temperature sensors, accelerometers, etc. Sensors 104 typically serve as auxiliary data stream sources, which can help identify the type, shape, distance, and coordinates of objects in the physical space where the live event is taking place, as well as the position and movement of performers, athletes, decorations, equipment, etc., and the brightness, texture, temperature, and other properties of objects digitized to be included in a 3D model of a virtual or augmented reality environment.

[0097] The video capture device 102, audio recording device 103, and sensor 104 are remotely controlled by the control module 105a. Typically, each of these devices has a predetermined position in the physical space where the live event occurs, known coordinates within that physical space, a known recording range, a known recording angle (e.g., viewing angle), known sensitivity, and a known range of motion. These known parameters are referred to herein as corresponding metadata. Therefore, each of these devices has corresponding metadata specific to that particular device. Each of these devices is calibrated for each live event, and these calibration parameters are also part of the metadata.

[0098] The live data collection module 105 of the device control and data collection server 101 is configured to receive data streams from the video capture device 102, audio recording device 103, and sensor 104 during live event recording. In some embodiments of the invention, different configurations of these devices can be arranged to record data for a live event. For example, the video capture device 102 can be integrated with the audio recording device 103 and sensor 104. Thus, it can be a single device (e.g., a smartphone, video camera, or drone) that records video, audio, position, motion, acceleration, etc. In other configurations, it can be a video camera that records both video and audio, but its position and motion within the physical space of the live event venue can be measured by a separate sensor 104, which can be located on a tripod or drone, or otherwise attached to a device that moves the camera. In other configurations, the motion of the video camera can be pre-recorded, and the video camera will move along a specific trajectory. In other configurations, audio can be recorded by the audio recording device 103, independent of the video capture device 102 and sensor 104. For example, such audio recording device 103 could be a microphone attached to a performer during a live event. It should be understood that the number of audio recording devices 103 (e.g., microphones) can be the same as the number of sound sources (e.g., musical instruments, performers, audience members, etc.).

[0099] Those skilled in the art will understand that any number of configurations of the video capture device 102, audio recording device 103, and sensor 104 are possible, and all such configurations are within the scope of the present invention. It should be understood that, regardless of the configuration, the live data collection module can collect a sufficient number of data streams, which may include captured video, recorded audio, and sensor data (illuminance, motion, position, acceleration, temperature, etc.), thereby generating a physical space of the live event venue and / or a three-dimensional model of the performers / athletes / spectators / staff participating in the live event.

[0100] The live data collection module 105 receives all data streams in digital formats suitable for processing by the system 100. These data streams are stored in the live data collection module 105 after being indexed by the system 100. Each individual data stream is identified by all necessary information that can be estimated by those skilled in the art, such as file type, file size, information about the device recording the data, and corresponding metadata. The information used to identify the data stream depends on a computer model used by the computing server 106 to analyze the data stream and create a 3D model of the physical space of the live event, incorporating real-life elements to create a digital twin for a virtual or augmented reality environment.

[0101] Computing server 106 receives data from device control and data collection server 101. Computing server 106 can process the received data in two ways: (a) The computing server 106 will use the data / information (such as data stored in the data storage module 107) to... Figure 1 , Figure 2 and Figure 3 (As shown) to supplement the data received from the live data collection module 105, and the computing server 106 applies one or more techniques to the data supplemented by the data stored in the storage data module 107 to generate a 3D model and digital twin of the live event; or (b) The computing server 106 will not access information located in the storage data module 107 (such as...). Figure 4 (As shown) Supplement the data received from the live data collection module 105, and the computing server 106 will apply one or more technologies to the data received from the live data collection module 105 only to generate 3D models and digital twins of the live events.

[0102] The technologies used to generate 3D models and digital twins for live events include at least one of the following: (a) Image-Based Modeling and Rendering Techniques (IBMR), which can be computed on the image-based rendering module 108, is described in Manuel M. Oliveira. Image-Based Modeling and Rendering Techniques: A Survey. The full text of the above two articles is incorporated herein by reference in UFRGS, Caixa Postal 15064, CEP 91501-970, Porto Alegre, RS, Brasil; Heung-Yeung Shum and Sing Bing Kang. A Review of Image-based Rendering Techniques. Microsoft Research.

[0103] (b) Photogrammetry and close-range photogrammetry techniques, which can be computed on photogrammetry module 109, described in T Luhmann, S Robson, S Kyle and I Harley. Close Range Photogrammetry. Principles, techniques and applications. Published by Whittles Publishing, Dunbeath Mains Cottages, Dunbeath, Caithness KW6 6EY, Scotland, UK, 2006, ISBN 1-870325-50-8; and Surendra Pal Singh, Kamal Jain and V. Ravibabu Mandla. A new approach towards image based virtual 3d city modeling by using close range photogrammetry. Geomatics Engineering Section, Department of Civil Engineering, Indian Institute of Technology, Roorkee. All the contents of the above references are incorporated into this paper by way of citation.

[0104] (c) Hybrid technologies, including the following: SFM-based multi-view Figure 3 3D reconstruction techniques, the combination of IBMR and close-range photogrammetry, laser scanning techniques, computer vision techniques, etc., these techniques can be computed on the hybrid module 110, and their methods are described in at least part in the following documents, the entire contents of each of which are incorporated herein by reference: a. Lei Gao, Yingbao Zhao, Jingchang Han and Huixian Liu. Research on Multi-View 3D Reconstruction Technology Based on SFM. School of Electrical Engineering, Hebei University of Science and Technology, Shijiazhuang 050018, China. b. Shen, XL; Dou, Y.; Mills, S.; Eyers, DM; Feng, H.; Huang, Z. “Distributed sparse bundle adjustment algorithm based on three-dimensional alpoint partition and asynchronous communication”. Front. Inf. Technol. Electron. Eng. 2018, 19, 889–904. c. Crosilla, F.; Beinat, A.; Fusiello, A.; Maset, E.; Visintini, D. Basics of computer vision. In Advanced Procrustes Analysis Models in Photogrammetric Computer Vision; Springer International Publishing: Cham, Switzerland, 2019. d. DeTone, D.; Malisiewicz, T.; Rabinovich, A. Superpoint: Self-supervised interest point detection and description. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Salt Lake City, UT, USA, 18–23 June 2018; pp. 224–236 e.Zhu, S.; Shen, T.; Zhou, L.; Zhang, R.; Wang, J.; Fang, T.; Quan, L. Parallel structure from motion from local increment to global averaging.arXiv 2017, arXiv:1702.08601 (parallel structure from motion from local increment to global averaging). f. Schonberger, JL; Hardmeicr, H.; Sattler, T.; Pollefeys, M. Comparative evaluation of hand-crafted and learned local features. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017; pp. 1482–1491. g.Bian, JW; Lin, WY; Matsushita, Y.; Yeung, SK; Nguyen, TD; Cheng, MM GMS: Grid-based motion statistics for fast, ultra-robust feature correspondence. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017; pp. 4181–4190 h.Sweeney, C.; Sattler, T.; Hollerer, T.; Turk, M.; Pollefeys, M. Optimizing the viewing graph for structure-frommotion. In Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile, 7–13 December 2015 i. Wilson, K.; Snavely, N. Robust global ranslations with 1dsfm. In European Conference on Computer Vision; Springer: Cham, Switzerland, 2014. J. Sweeney, C.; Fragoso, V.; Hollerer, T.; Turk, M. Large-scale SfM with the distributed camera model. In Proceedings of the 2016 4th International Conference on 3D Vision (3DV), Stanford, CA, USA, 25–28 October 2016; pp. 230–238. K. Schönberger, JL; Frahm, JM. Structure-from-motion revisited. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 27–30 June 2016. L. Yao, Y.; Luo, Z.; Li, S.; Fang, T.; Quan, L. Mvsnet: Depthinference for unstructured multi-view stereo. In Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany, 8–14 September 2018; pp. 767–783. m.Knapitsch, A.; Park, J.; Zhou, QY; Koltun, V. Tanks and temples: Benchmarking large-scale scene reconstruction. ACM Trans. Graph. 2017, 36,78–90; n. Berger, M.; Tagliasacchi, A.; Seversky, LM. A survey of surface reconstruction from point clouds. Comput. Graph. Forum 2017, 36, 301–329. o.Zhu, S.; Zhang, R.; Zhou, L.; Shen, T.; Fang, T.; Tan, P.; Quan, L. Very large-scale global SfM by distributed motion averaging. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Salt Lake City, UT, USA, 18–23 June 2018 p.Kasten, Y.; Geifman, A.; Galun, M.; Basri, R. GPSFM: Globalprojective SFM using algebraic constraints on multi-view fundamentalmatrices. In Proceedings of the IEEE Computer Society Conference on ComputerVision and Pattern Recognition, Long Beach, CA, USA, 15–20 June 2019 Liu, H.; Zhang, G.; Bao, H. Robust keyframe-based monocular SLAM for augmented reality. In Proceedings of the 2016 IEEE International Symposium on Mixed and Augmented Reality, ISMAR Adjunct 2016, Merida, Mexico, 19–23 September 2016. r. Ke, T.; Roumeliotis, SI An efficient algebraic solution to the perspective-three-point problem. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017; pp. 7225–7233 s. Cefalu, A.; Haala, N.; Fritsch, D. Hierarchical structure from motion combining global image orientation and structureless bundle adjustment. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. 2017, 42, 535–542 (Hierarchical Structure from Motion Combining Global Image Orientation and Structureless Bundle Adjustment); t. Wang, R.; Lin, J.; Li, L.; Xiao, Z.; Hui, Y.; Xin, Y. A revised orientation-based correction method for SfM-MVS point clouds of outcrops using ground control planes with marks. J. Struct. Geol. 2021, 143, 104266 (A Revised Orientation-Based Correction Method for SfM-MVS Point Clouds of Outcrops Using Ground Control Planes with Marks, Acta Sedimentologica Sinica, 2021, 143, 104266); u. Dugan, U.; Sangsoo, L. Microscopic structure from motion (SfM) for microscale 3D surface reconstruction. Sensors 2020, 20, 5599 (Microscopic Structure from Motion (SfM) for Microscale 3D Surface Reconstruction); v. Khalil, M.; Ismanto, I.; Fu’ad, M.N. 3D reconstruction using structure from motion (SFM) algorithm and multi view stereo (MVS) based on computer vision. IOP Conf. Ser. Mater. Sci. Eng. 2021, 1073, 012066 (3D Reconstruction Using Structure from Motion (SFM) Algorithm and Multi View Stereo (MVS) Based on Computer Vision); w.Mali, VK; Venu, P.; Nagaraj, MK; Kuiry, SN Demonstration of structure-from-motion (SfM) and multi-view stereo (MVS) close rangephotogrammetry technique for scour hole analysis. 2021, 46, 227 (Demonstration of moving structure (SfM) and multi-view stereo (MVS) close-range photogrammetry techniques for scour pit analysis); x.Yu, Q.; Yang, C.; Wei, H. Part-wise AtlasNet for 3D point cloud reconstruction from a single image. Knowl.-Based Syst. 2022, 242, 108395 y.Mur-Artal, R.; Tardos, JD ORB-SLAM2: An open-source SLAM system for monocular, stereo, and RGB-D cameras. IEEE Trans. Robot. 2017, 33, 1255–1262.

[0105] The computing server 106 also includes an artificial intelligence computing module 111 (AI computing module 111), which can be used to enhance data received from the live streaming data collection module 105 with information about the objects, shapes, and depths of real-life elements captured on images of live events received from the video capture device 102. The AI ​​computing module 111 can work in conjunction with the storage data module 107 to locate and match seemingly similar real-life elements captured in images from the data stream from the video capture device 102 with known samples of digitized objects in the library of the storage data module 107. For example, the shape and structure of a specific chair captured in an image captured by the video capture device 102 and processed by the live streaming data collection module 105 can be located and matched with similar chair shapes and forms in the storage data module 107, thereby facilitating the processing of chair digitization in a digital twin 3D model, since the size, texture, and digital representation of these chairs were already stored in the storage data module 107 before the video capture device 102 captured the live event.

[0106] Figure 1 The computing server 106 is shown to have a 3D model rendering module 112. The 3D model rendering module 112 generates a digital twin based on digital representations of real-life elements in the live event venue and digital representations of the live event venue itself.

[0107] After the computing server 106 generates the digital twin, the system 100 generates a virtual or augmented reality environment and then transmits it to or builds it within the metaverse on the metaverse server 113. The metaverse server 113 may be configured as a server running as a centralized database or a distributed ledger.

[0108] Metaverse server 113 communicates with user device 114 to provide user device 114 with a digital twin virtual or augmented reality environment for live events.

[0109] In some implementations, the virtual or augmented reality environment of the digital twin of the live event may be stored on the metaverse server 113. In other implementations, a portion of the virtual or augmented reality environment of the digital twin of the live event may be stored on the metaverse server 113, while other portions of the virtual or augmented reality environment of the digital twin of the live event are uploaded to the metaverse server 113 upon request between the system 100 and the metaverse server 113.

[0110] Figure 2An embodiment in which system 100 includes a video capture device 102, an audio recording device 103, and a sensor 104 is illustrated. In this embodiment, system 100 has an integrated group of video capture devices 102, audio recording devices 103, and sensors 104, which may be located at a given venue (e.g., a stadium) and are designed to generate a 3D model of events at the venue by generating a digital twin of a live event taking place at the venue.

[0111] Figure 3 An implementation of a virtual or augmented reality whereby system 100 generates a digital twin of a live event hosted within system 100 and allows user device 114 to communicate with system 100 via communication module 115 is shown.

[0112] It should be understood that system 100, metaverse server 113, communication module 115, and user equipment 114 communicate via a network, which may be the Internet, Ethernet, etc. Video capture device 102, audio recording device 103, and sensor 104 may also communicate with control and data collection server 101 via a network or in any other manner known in the art.

[0113] Figure 4 An exemplary implementation is shown in which the data storage module 107 is not present. In this implementation, the AI ​​computing module 111 can access any other data required to generate the 3D model via the Internet.

[0114] Figure 5 An exemplary embodiment of a method 400 for generating digital twins to create virtual or augmented reality environments according to the present invention is shown. The method 400 includes the following exemplary steps: a) Capture video and generate visual elements, record audio and generate audio elements, determine tactile features and generate tactile elements, and measure distance and spatial coordinates and generate spatial elements; b) Record metadata corresponding to one of the visual, audio, spatial, and tactile elements; c) Generate digital twins of real-life elements; d) Store the digital twin on a server; e) Assign tokens to at least one of the visual, audio, spatial, and tactile elements; f) Connect the server to the Metaverse hosting server; g) Integrating digital twins into the metaverse by generating virtual reality environments that correspond to the digital twins; h) Provide user devices with access to the virtual reality environment corresponding to the digital twin.

[0115] Figure 6 An exemplary implementation of method 500 for generating a virtual or augmented reality environment with multiple real-world live elements is shown, each of which has been assigned cryptocurrency tokens. The virtual or augmented reality environment of method 500 is integrated into a metaverse that provides access to the virtual or augmented reality environment to multiple user devices. The method includes the following steps: a) Receive digital video streams with corresponding metadata, receive digital audio streams with corresponding metadata, and receive supplementary data from sensors; (b) (1) Generate video elements based on images in the video stream, (2) Generate audio elements based on the audio stream, and (3) Generate tactile and spatial elements based on the combination of data; c) Generate digital twins of multiple real-life elements captured in the images of the video stream, and assign corresponding video elements, audio elements, tactile elements, and spatial elements to each digital twin; d) Store references to each generated digital twin in a distributed manner on the blockchain; e) Assign cryptocurrency tokens to at least one digital twin visual object; f) Generate a virtual or augmented reality environment for hosting digital twins; g) Integrating virtual or augmented reality environments into the metaverse; h) Provides access to the virtual reality environment corresponding to the user's device.

[0116] Method 600 assigns cryptocurrency tokens to different virtual elements in a virtual or augmented reality environment. In this document, a digital twin refers to a virtual object that may include any of the following: a digital portion of a live performance, a digital song or performance, a specific seating arrangement in a digital performance, a specific perspective of a digital performance, a specific object (which may be a digital twin of a real live performance or a purely virtual object, such as a dragon, avatar, fireball, etc.), a series of digital images of a performance, a specific digital space corresponding to a real physical space in the venue of a live event, and any other virtual object in a virtual or augmented reality environment. The cryptocurrency token may be a fungible token or a non-fungible token.

[0117] In some embodiments of System 100, the cryptocurrency payment processing system is also integrated into the generated virtual or augmented reality environment, allowing users to interact with the virtual or augmented reality environment via user devices to acquire any digital twin within the virtual or augmented reality environment, as well as the virtual or augmented reality environment itself, using cryptocurrency tokens. To achieve this, System 100 assigns a permanent or semi-permanent reference to each digital twin in the virtual or augmented reality environment on a blockchain. Those skilled in the art will understand that System 100 can assign references to any blockchain currently existing on the token or that may exist in the future. Assigning a reference includes providing a unique identifier for the digital twin in a given blockchain.

[0118] Virtual or augmented reality environments can also be hosted on a distributed ledger and operate entirely based on blockchain technology. The computing power of computing server 106 will be distributed across a large number of machines, each machine performing a separate task, which is written to the blockchain, and system 100 can access these tasks upon request.

[0119] Figure 7 A method 700 is illustrated that enables a user to transfer token data between a user device and a virtual or augmented reality environment via blockchain after system 100 generates a virtual or augmented reality environment in the metaverse. It should be understood that numerous metaverses and numerous virtual or augmented reality environments may allow a user to only view and listen to digital twins within the virtual or augmented reality environment, thus limiting other interactions between the user and the metaverse or virtual / augmented reality environment. In some cases, this may be done to optimize the computing power of the server 113 hosting the metaverse or the communication module 115 communicating with the user device 114.

[0120] Figure 8An exemplary implementation of method 800 for generating a digital twin of a virtual or augmented reality environment for a live event is shown. System 100 first generates a 3D model of the live event venue, which includes digital representations of appropriate real-life elements within the venue, as well as virtual objects integrated into the virtual or augmented reality environment by system 100 or a metaverse server 113. In some cases, it may be more useful to first generate the complete 3D virtual or augmented reality environment of the live event and then add audio elements to it. Audio elements can be added to different parts of the virtual or augmented reality environment of the live event in different ways; for example, the sound may be quieter in some locations and louder in others. Furthermore, the audio can be divided into different audio tracks, each corresponding to a separate sound source. As shown in method 800, after generating the digital twin, audio tracks corresponding to the audio elements in the virtual or augmented reality environment of the live event can be added. In some implementations, the actual files containing the audio tracks may not be hosted on system 100's server but may be accessible via a network link to system 100.

[0121] It should be understood that in some implementations, system 100 may store captured video data streams, audio data streams, and data streams from sensors; in other implementations, system 100 may use these data streams to generate 3D models of the virtual or augmented reality environment of the live event, while the actual video, audio, and sensor data may be stored on a separate server and connected to its computing module 106 to generate digital representations of the real live elements and live events.

[0122] Figure 9 An exemplary implementation of a method 900 for generating a 3D model digital twin of real-life elements via a computing server 106 is shown, the method comprising the following steps: a) Receive video data streams, audio data streams, and sensor data streams; b) Live streaming data processing; c) Depth / distance processing; d) Artificial intelligence analysis and data augmentation processing; e) Mesh generation; f) Point cloud generation; g) Approximation of the three-dimensional model; h) Viewpoint generation; i) Compression / transmission.

[0123] It should be understood that those skilled in the art can modify method 900, but it will still remain within the scope of the present invention.

[0124] Figure 10A method for using multi-view based Figure 3 An exemplary implementation of a method 1000 for generating digital twin 3D models of real live-stream elements or live-stream event venues from multi-view images using 3D reconstruction technology.

[0125] Figure 11 An exemplary implementation of method 1100 in which a user device interacts with a virtual or augmented reality environment is shown. It should be understood that any suitable model of user device interaction with a virtual or augmented reality environment falls within the scope of this invention.

[0126] Figure 12 An exemplary embodiment of user equipment 1200 is shown. User equipment 1200 is shown as a virtual reality headset; however, user equipment may also include smartphones, smart glasses, computers, tablets, game controllers, etc. (not shown here).

[0127] It should be understood that, in some implementations, system 100 may have a 3D model of the physical space of the venue where the live event takes place, stored in storage data module 107. Therefore, system 100 will only capture video, audio, and sensor data streams relevant to the actual performance taking place within the venue. Device control and data collection server 101 will operate live data collection module 105 to capture video, audio, and sensor data streams of performers, athletes, and / or decorations and animations present within the live event venue. Computation server 106 then matches the 3D model of the digital twin of the venue's physical space stored in storage data module 107 with a digital twin generated by any of the image-based rendering module 108, photogrammetry module 109, and / or mixing module 110. AI computing module 111 may be activated to facilitate and / or enhance compatibility between the pre-stored 3D digital twin of the physical space and the generated 3D digital twin of the live event. In some implementations, the use of AI computing module 111 can be omitted. 3D model rendering module 112 generates a final 3D model of a digital twin corresponding to the physical space of the site and the live event, thereby generating a virtual or augmented reality environment.

[0128] The computing server 106 enables the system 100 to generate sufficiently accurate virtual or augmented reality environments in real-time or near real-time by integrating information to determine which real-life elements of a live event will be digitized, which will not be digitized by the system 100, and which may be digitized by the system 100 during the live event to be integrated into the virtual or augmented reality environment at a given time during the live event. This technology can be adapted to selectively identify real-time elements that require significant computing resources to digitize. For example, fireworks, flames, fountains, etc., can be omitted from digitization and instead replaced by virtual objects stored in a virtual object library in the storage data module 107 or virtual objects accessible via the Internet.

[0129] In some embodiments of the present invention, system 100 can generate a digital twin 3D model of the physical venue (e.g., a stadium or concert hall) for a large live event. The generated virtual or augmented reality environment will be matched to the actual size of the venue. When a user places their avatar in the virtual or augmented reality environment, the user can experience the live event from the location of their avatar. It is possible that multiple users will place their avatars in the virtual or augmented reality environment, and each user may be granted different rights to interact with the virtual or augmented reality environment. To determine which rights users have and how they interact with the virtual or augmented reality environment, system 100 can generate a virtual geofence around each avatar of each user or user group. The virtual geofence can also be generated at different locations within the virtual or augmented reality environment, so that each location can be assigned the same or different rights. Furthermore, the geofence can be associated with cryptocurrency tokens. Therefore, the geofence can allow users to use the tokens to purchase or redeem virtual objects within the geofence. Additionally, to enter a particular geofence, a user may need to purchase or redeem cryptocurrency tokens, just as a user in real life might enter a VIP area of ​​a live event.

[0130] It should be understood that System 100 may use any suitable geofencing technology or equivalent, not limited to using a Global Positioning System (GPS) satellite network and / or local radio frequency identifiers (such as Wi-Fi nodes or Bluetooth beacons), to create virtual boundaries around a specific location. Geofencing in a virtual or augmented reality environment can be generated using a metaverse coordinate system and matrix and other types of boundary calculation techniques. The geofencing can then be paired with a software application or cryptocurrency token on a user's device, which responds to the boundary in a certain way based on parameters of the metaverse or virtual / augmented reality environment.

[0131] System 100 can be designed to operationally record live events, generate their digital twins, and transmit the data to the metaverse for AR / VR / mixed reality broadcasting. Therefore, examples of user equipment 1200 include devices that support AR / VR / mixed reality broadcasting. The system 100 and methods described herein provide a technical solution for associating live event broadcasts, or portions thereof, with digital twins and utilizing them to generate virtual or augmented reality environments.

[0132] In some implementations, the systems and methods described herein can combine digital twins with broadcasts or portions thereof by leveraging geofencing or equivalent technologies, as well as cryptocurrency technologies including NFTs, and blockchain. This enables System 100 to integrate any suitable combination of "play-to-earn" features into a virtual or augmented reality environment for use in metaverse live events. It should be understood that any combination of NFTs can be attached to virtual objects integrated into the virtual or augmented reality environment of the live event.

[0133] In some implementations, the live data collection module 105 is configured to run within the WEB3 framework and generate data suitable for efficient transmission to the metaverse.

[0134] In some implementations, system 100 uses blockchain technology to catalog and redistribute virtual objects in a virtual or augmented reality environment during or after a live event.

[0135] As described above, System 100 can use any suitable 3D modeling technique to generate digital twins of real-life elements.

[0136] For clarity, the various techniques and mechanisms of this invention are sometimes described in a single form. However, it should be noted that, unless otherwise stated, some implementations involve multiple iterations of the techniques or multiple instantiations of the mechanisms. For example, the system uses a processor in various situations. However, it should be understood that, unless otherwise stated, the system may use multiple processors, cloud computing, distributed ledger technology, multi-core processors, graphics cards or graphics accelerators, quantum computers, or combinations thereof, while still remaining within the scope of this invention. Furthermore, the techniques and mechanisms of this invention sometimes describe connections between two entities. It should be noted that a connection between two entities does not necessarily mean a direct, unobstructed connection, as various other entities may exist between the two entities. For example, a processor may be connected to memory, but it should be understood that various bridges and controllers may exist between the processor and memory. Therefore, unless otherwise stated, a connection does not necessarily mean a direct, unobstructed connection.

[0137] Many specific details have been listed in the above description, but embodiments of the invention may be implemented without these specific details. To avoid hindering understanding of this description, well-known circuits, structures, and techniques have not been described in detail. Terms such as "implementation," "various embodiments," etc., indicate that the embodiments thus described may include specific features, structures, or characteristics, but not every embodiment necessarily includes these specific features, structures, or characteristics. Some embodiments may have some, all, or none of the features described in other embodiments. "Connection" can mean that elements are in direct physical or electrical contact with each other, and "linkage" can mean that elements cooperate or interact with each other, but they may or may not be in direct physical or electrical contact. Furthermore, although similar or identical numbers may be used in different figures to represent the same or similar components, this does not mean that all figures containing similar or identical numbers constitute a single or the same embodiment.

[0138] Those skilled in the art will understand that when this specification refers to "receiving data from a user," the electronic device performing the action of receiving data from a user may receive electronic (or other) signals from the user. Those skilled in the art will also understand that displaying data to a user through a user graphical interface (e.g., the screen of an electronic device) may involve sending a signal to the user graphical interface containing data that can be processed, and at least a portion of said data that can be displayed to the user through the user graphical interface.

[0139] Some of the steps and signal transmission / reception methods described herein are well known in the art; therefore, for the sake of simplicity, certain parts of this specification omit these steps and signal transmission / reception methods. Signals can be transmitted / received using optical methods (e.g., fiber optic connections), electronic methods (e.g., wired or wireless connections), and mechanical methods (e.g., based on pressure, temperature, or any other suitable physical parameter).

[0140] Modifications and improvements to the implementation of the present invention described above may be apparent to those skilled in the art. The above description is intended to be exemplary and not to limit the invention. Therefore, the scope of the present invention is intended to be defined only by the scope of the appended claims.

Claims

1. A system for generating digital twins in an interactive virtual reality environment, the digital twins corresponding to live events, the system comprising: Multiple video capture devices are connected to a network, each video capture device is used to capture video of the live event, and each captured video has corresponding metadata; Multiple audio recording devices are used to record the audio of the live event; A server configured to receive data from each video capture device and each audio recording device; The server runs the computing module, which is configured to: Analyze the captured video and its corresponding metadata. The spatial depth between images captured by different video capture devices is determined. Each captured video is matched against a coordinate matrix in virtual space. Match the audio with the coordinate matrix of the virtual space. Based on the captured video, the corresponding metadata, the spatial depth, the predetermined coordinate set, and the coordinate matrix, a digital twin of at least a portion of the live event is generated; The server is configured to generate multiple digital representations from multiple directions of the live event, the multiple digital representations including at least one 360-degree digital representation and multiple unidirectional digital representations, each digital representation corresponding to the viewpoint of the digital twin within the interactive virtual reality environment; During operation, the digital representation and the audio can be transmitted to the devices of users participating in the interactive virtual reality environment of the live event.

2. The system according to claim 1, wherein, The system is also configured to receive data from the user's device, the data corresponding to the user's interaction with the virtual reality environment.

3. The system according to claim 2, wherein, The computing module is also configured to calculate thresholds to determine real-life elements suitable for digital representations of the digital twin existing in the virtual reality environment.

4. The system according to claim 1, 2 or 3, wherein, The server is also configured to transmit the digital representation to the device in real time or with a delay after the live event.

5. The system according to any one of claims 1 to 4, wherein, The digital representation, audio recording, corresponding metadata, and corresponding coordinate matrix data are parsed by the system and stored in the database for at least partial redistribution on demand.

6. The system according to any one of claims 1 to 5, wherein, The digital representation, audio recording, corresponding metadata, and corresponding coordinate matrix data are stored in a distributed manner using blockchain technology.

7. The system according to claim 6, wherein, Non-fungible tokens are allocated to at least a portion of the digital twin, the at least a portion of which includes the digital representation, the audio recording, the corresponding metadata, and the corresponding coordinate matrix data.

8. The system according to claim 7, wherein, The system includes a payment processor adapted to identify non-fungible tokens purchased by the user and to allocate the corresponding non-fungible token rights to the user.

9. The system according to any one of claims 1 to 8, wherein, The system allocates non-fungible tokens to the digital twin of the live event.

10. The system according to any one of claims 1 to 9, wherein, The server uses artificial intelligence to generate digital representations.

11. A method for generating an interactive virtual reality environment in a metaverse, the environment comprising digital twins of real-life elements, the real-life elements comprising at least one of visual elements, audio elements, spatial elements, and tactile elements, the environment being connected to a distributed ledger technology, the method comprising: Digital twins of the real-life elements are generated based on at least one of the following: (a) Capturing video and generating visual elements, recording audio and generating audio elements, determining tactile features and generating tactile elements, and measuring distance and spatial coordinates and generating spatial elements, and (b) Record metadata corresponding to one of the visual element, the audio element, the spatial element, and the tactile element; The digital twin is stored on a server; Tokens are assigned to at least one of the visual element, the audio element, the spatial element, and the tactile element; Connect the server to the Metaverse hosting server; By generating a virtual reality environment corresponding to the digital twin, the digital twin is integrated into the meta-universe; Provide user devices with access to the virtual reality environment corresponding to the digital twin.

12. The method according to claim 11, wherein, Storing the digital twin on a server also includes storing it in a distributed manner using blockchain technology.

13. The method according to claim 11 or 12, wherein, The method also includes communicating token data via user equipment.

14. The method according to claim 13, wherein, The token is a non-fungible token.

15. The method according to any one of claims 11 to 14, wherein, Measuring distance and spatial coordinates includes determining at least one of the following: the shape of the visual element, the distance between the first visual element and the second visual element, and the speed associated with the moving visual element.

16. The method according to any one of claims 11 to 15, the method further comprising connecting a payment processor adapted to identify the purchased token and the corresponding rights of the purchased token.

17. The method according to any one of claims 11 to 16, wherein, Providing access to the virtual reality environment includes allowing the user to send their digital self to any location within the virtual reality environment.

18. The method of any one of claims 11 to 17, further comprising generating a portion of the meta-universe based on the virtual reality environment of the digital twin.

19. The method according to any one of claims 11 to 18, the method further comprising generating a plurality of virtual reality environments corresponding to a plurality of digital twins.

20. The method according to any one of claims 11 to 19, the method further comprising integrating a plurality of digital twins into the virtual reality environment.

21. The method according to any one of claims 11 to 20, the method further comprising selecting real-life elements for digitization in the digital twin.

22. A method for generating a three-dimensional graphical representation in a virtual reality environment, the three-dimensional graphical representation corresponding to real-life elements, the system being connected to a network, the system including a video capture device, a corresponding metadata recording device, an audio recording device, and a server, the server being configured to receive data from the video capture device, the audio recording device, and the metadata recording device, the server running a computing module configured to analyze the captured video, the recorded audio, and the recorded corresponding metadata, and digitize objects displayed in images in the captured video, thereby creating a three-dimensional graphical representation of real-life elements, the method comprising: a. Generate a three-dimensional graphical representation model based on the objects displayed in the images of the captured video, the determination of the distances between the objects displayed in the images of the captured video, and the corresponding metadata analyzed; b. Link the audio track to the elements in the three-dimensional graphical representation; c. Store the three-dimensional graphic representation on the server; d. Connect the server to the Metaverse hosting server; e. Integrate the three-dimensional graphic representation into the metaverse by generating a virtual reality environment corresponding to the three-dimensional graphic representation; f. Provide the user's device with access to the virtual reality environment.

23. A system for generating digital twins in an interactive virtual reality environment, the digital twins corresponding to real-life elements, the system being connected to a network, the system comprising: The system includes a video capture device, a corresponding metadata recording device, an audio recording device, and a server. The server is configured to receive data from the video capture device, the audio recording device, and the metadata recording device. The server runs a computing module configured to analyze the captured video, the recorded audio, and the recorded corresponding metadata. Determine the shapes of objects in the captured video and the distances between objects in the captured video; and determine the coordinate matrix of a three-dimensional virtual space containing the real-life elements.

24. A system for integrating an interactive virtual reality environment corresponding to a digital twin of real-life elements into a metaverse, the system being connected to a network, the system comprising: The system includes a video capture device, a corresponding metadata recording device, an audio recording device, and a server. The server is configured to receive data from the video capture device, the audio recording device, and the metadata recording device. The server runs a computing module configured to: analyze the captured video, the recorded audio, and the recorded corresponding metadata; determine the spatial depth between real-life elements captured by the video capture device, recorded by the audio device, or recorded by the metadata device; and approximate the shapes of the real-life elements. Identify real-life elements suitable for digitalization; And to generate digital twins of at least a portion of the life elements, the server being configured to send data to a metaverse hosting server to integrate the interactive virtual reality environment into the metaverse.

25. A system for generating digital twins in an interactive virtual reality environment, the digital twins corresponding to live events, the system comprising: Multiple video data streams received via the network, each video stream corresponding to a predetermined set of coordinates in the physical space of the live event and having corresponding metadata; Multiple audio data streams corresponding to the 3D audio field of the live event; A server configured to receive the video data stream and the audio data stream. The server runs the computing module, which is configured to: The video data stream and corresponding metadata are analyzed. The spatial depth between images in the video data stream is determined. Match the objects in each image with the coordinate matrix in the virtual space. Match the 3D audio field with the coordinate matrix of the virtual space. Based on the image, the corresponding metadata, the spatial depth, the predetermined coordinate set, and the coordinate matrix, a digital twin of at least a portion of the live event is generated; The server is configured to: Multiple digital representations are generated from multiple directions of the live event, including at least one 360-degree digital representation and multiple unidirectional digital representations, each digital representation corresponding to the viewpoint of the digital twin within the interactive virtual reality environment; as well as The digital representation and audio are transmitted to the user device via a network.

26. The system according to claim 25, wherein, The system is also configured to receive data from the user device, the data corresponding to the user's interaction with the virtual reality environment.

27. The system according to claim 26, wherein, The computing module is also configured to compute thresholds to determine real-life elements and approximate digital representations of the real-life elements that are suitable for a graphical representation of the digital twin existing in the virtual reality environment.

28. The system according to claim 25, 26 or 27, wherein, The numbers represent data transmitted to the device in real time or within a time delay following the live event.

29. The system according to any one of claims 25 to 28, wherein, The digital representation, audio, corresponding metadata, and corresponding coordinate matrix data are parsed by the system and stored in the database for at least partial redistribution on demand.

30. The system according to any one of claims 25 to 29, wherein, The digital representation, audio, corresponding metadata, and corresponding coordinate matrix data are stored in a distributed manner using blockchain technology.

31. The system according to claim 30, wherein, Non-fungible tokens are allocated to at least a portion of the digital twin, the at least a portion of which includes the digital representation, audio recordings, corresponding metadata, and corresponding coordinate matrix data.

32. The system according to claim 31, wherein, The system includes a payment processor adapted to identify non-fungible tokens purchased by the user and allocate the corresponding non-fungible token rights to the user.

33. The system according to any one of claims 25 to 32, wherein, The system allocates non-fungible tokens to the digital twin of the live event.

34. The system according to any one of claims 25 to 33, wherein, The server uses artificial intelligence to generate digital representations.

Images