Techniques for preloading and displaying high-quality image data

Abstract

A system can include one or more processors 107 and a memory 105 storing instructions that, when executed by the processor 107, cause the processor 107 to perform operations including receiving data associated with a user; determining that the received data corresponds to a direction of movement of the user through a virtual reality (VR) environment, an augmented reality (AR) environment, or a mixed reality (MR) environment. The operations can also include transmitting a tile of high-quality image data to a display device 108 based on the direction of movement of the user; transmitting a command to the display device 108 to display one or more aspects of a region of the VR environment, AR environment, or MR environment based on the tile of high-quality image data; and preloading one or more additional tiles of high-quality image data into a preloader 103 based on the tile of high-quality image data transmitted to the display device.

Description

[0001] Cross-reference to related applications This application claims the benefit of U.S. Provisional Application No. 63 / 002,056, filed on March 30, 2020, entitled “Technique for Preloading and Displaying High-Quality Image Data,” which is hereby incorporated in its entirety by reference for all purposes. Background Technology

[0002] This disclosure generally relates to displaying image data in a virtual reality environment, an augmented reality environment, or a mixed reality environment. More specifically, this disclosure relates to an imaging system that can preload image data to facilitate the display of certain aspects of one or more areas of a virtual reality environment, an augmented reality environment, or a mixed reality environment on a display device.

[0003] As new imaging technologies are developed, higher-quality images or videos can be generated and subsequently displayed to users within virtual reality (“VR”), augmented reality (“AR”), or mixed reality (“MR”) environments via display devices. However, due to the amount of data in each image or video, these higher-quality images or videos can have very large file sizes. Therefore, displaying VR / AR / MR environments on a large scale using such image or video files generated from high-quality imaging technologies can involve significant computational resources (e.g., terabytes or more), which can be costly or resource-intensive.

[0004] This section aims to introduce the reader to various aspects of the technology that may be related to the aspects of the present technology described and / or claimed below. This discussion is intended to help provide the reader with background information to facilitate a better understanding of the various aspects of this disclosure. Therefore, it should be understood that these statements should be interpreted in this context and not as an admission of prior art. Summary of the Invention

[0005] The following section outlines certain embodiments disclosed herein. It should be understood that these aspects are presented merely to provide the reader with a brief overview of these embodiments, and that they are not intended to limit the scope of this disclosure. In fact, this disclosure may include a wide variety of aspects that may not be set forth below.

[0006] In one embodiment, a system may include one or more processors and a memory storing instructions that, when executed by the processor, cause the processor to perform operations including: receiving data associated with a user; and determining that the received data corresponds to a direction of movement of the user through a virtual reality (VR), augmented reality (AR), or mixed reality (MR) environment. The operations also include: transmitting tiles of high-quality image data to a display device based on the user's direction of movement; transmitting commands to the display device to display one or more aspects of a region of the VR, AR, or MR environment based on the tiles of high-quality image data; and preloading one or more additional tiles of high-quality image data into a preloader based on the tiles of high-quality image data.

[0007] In another embodiment, a method may include: receiving user-associated data from at least one or more input devices or one or more sensors via one or more processors; and determining, via the processor, that the received data corresponds to a direction of movement of the user through a virtual reality (VR), augmented reality (AR), or mixed reality (MR) environment. The method further includes: transmitting one or more high-quality image data files to a display device via the processor based on the user's direction of movement; transmitting commands via the processor to the display device to display one or more virtual objects in a region of the VR, AR, or MR environment based on the one or more high-quality image data files transmitted to the display device; and preloading one or more additional high-quality image data files into a preloader via the processor based on the user's direction of movement.

[0008] In another embodiment, a non-transitory computer-readable medium containing instructions, when executed by one or more processors, causes the processors to perform operations including: receiving user-associated data from one or more input devices, one or more sensors, or both; and determining that the received data corresponds to a specific orientation in the user's field of vision within a virtual reality (VR), augmented reality (AR), or mixed reality (MR) environment. The operations also include: transmitting tiles of high-quality image data to a display device based on the specific orientation in the user's field of vision; transmitting a command to the display device to display one or more aspects of a region of the VR, AR, or MR environment based on the tiles of high-quality image data transmitted to the display device; and preloading one or more additional tiles of high-quality image data into a preloader based on the tiles of high-quality image data transmitted to the display device.

[0009] Various modifications to the features described above in relation to various aspects of this disclosure may exist. Similarly, additional features may be incorporated into these various aspects. These modifications and additional features may exist individually or in any combination. For example, various features discussed below with respect to one or more embodiments described may be incorporated individually or in any combination into any aspect of the foregoing aspects of this disclosure. The brief overview presented above is intended only to familiarize the reader with certain aspects and context of embodiments of this disclosure and is not intended to limit the claimed subject matter. Attached Figure Description

[0010] These and other features, aspects, and advantages of the invention will become more readily understood when the following detailed description is read with reference to the accompanying drawings, in which the same characters denote the same parts throughout the drawings, wherein: Figure 1 A block diagram illustrating an imaging system for preloading high-quality image data corresponding to one or more regions of a virtual reality (VR), augmented reality (AR), or mixed reality (MR) environment, according to embodiments described herein; Figure 2 A schematic diagram illustrating an exemplary VR / AR / MR environment according to embodiments described herein; Figure 3 This describes the user navigation according to the embodiments described herein. Figure 2 The first set of areas in an exemplary VR / AR / MR environment Figure 1 A block diagram of the preloader and display device of the imaging system; Figure 4 This describes the user navigation according to the embodiments described herein. Figure 2 The second set of areas in an exemplary VR / AR / MR environment Figure 1 A block diagram of the preloader and display device of the imaging system; and Figure 5 A flowchart illustrating a method for displaying high-quality image data corresponding to a specific region of a VR / AR / MR environment, and for preloading high-quality image data corresponding to one or more regions that intersect with the specific region in a preloader, according to embodiments described herein. Detailed Implementation

[0011] One or more specific embodiments of this disclosure will now be described. To provide a concise description of these embodiments, not all features of the actual implementation may be described in this specification. It should be appreciated that, as in any engineering or design project, many implementation-specific decisions may be made in the development of any such implementation to achieve specific goals that may differ from the developer's, such as compliance with system-related constraints and business-related constraints. Furthermore, it should be appreciated that such development efforts may be complex and time-consuming, but for those skilled in the art who benefit from this disclosure, such development efforts will be nothing more than routine tasks of design, fabrication, and manufacturing.

[0012] When describing the elements of various embodiments of this disclosure, the articles “a,” “an,” and “the” are intended to mean the presence of one or more of the elements. The terms “comprising,” “including,” and “having” are intended to be inclusive and mean that additional elements may be present in addition to those listed. One or more specific embodiments described herein will be described below. To provide a concise description of these embodiments, not all features of the actual implementation may be described in this specification. It should be appreciated that, as in any engineering or design project, many implementation-specific decisions must be made to achieve the developer’s specific goals, which may vary depending on the implementation, such as compliance with system-related constraints and business-related constraints. Furthermore, it should be appreciated that such development efforts may be complex and time-consuming, but for those skilled in the art who benefit from this disclosure, such development efforts will be nothing more than routine tasks of design, fabrication, and manufacturing.

[0013] As described above, high-quality images or videos can be generated from various imaging techniques and subsequently displayed to a user in a virtual reality (“VR”), augmented reality (“AR”), or mixed reality (“MR”) environment via a display device. Such high-quality imaging techniques can include those such as light field techniques, point cloud modeling, voxelization, and so on. By using image or video files generated from these techniques and subsequently displayed to the user, the user can perceive a level of detail in digitally presented objects in a VR / AR / MR environment similar to the level of detail typically perceived by the user in the real world. For example, a light field volume can be generated by capturing one or more images of a real-world object, including the brightness and color values ​​of each ray received by an image sensor and the direction and / or angle of each ray received by the image sensor. In this way, the light field volume can contain sub-images that are slightly different from each other based on the direction and / or angle of each ray associated with the sub-image. Based on the desired depth of focus or desired viewpoint of the object, the light field volume can be processed to produce a two-dimensional (“2D”) or three-dimensional (“3D”) image of the object corresponding to the desired depth of focus or desired viewpoint of the object. In this way, images or videos generated from light field technology can be displayed in VR / AR / MR environments to provide users with a perspective similar to that of objects that users would have in the real world.

[0014] However, due to the amount of image data in each light field volume, the image or video files providing the light field volume are very large (e.g., terabytes or more). Similarly, the image and video files generated via point cloud modeling, voxelization, and other high-quality imaging techniques can also be very large. Consequently, using such files generated from high-quality imaging techniques to display VR / AR / MR environments on a large scale can involve significant computational resources, which can be costly or resource-intensive. For example, in an interactive ride setup where a user can move around within a VR / AR / MR environment and the user's field of view can change within the VR / AR / MR environment, multiple high-quality image files can be loaded to display the scene based on the user's position within the VR / AR / MR environment, the user's field of view within the VR / AR / MR environment, or both. Furthermore, the size and number of high-quality image files can grow exponentially with the size of the VR / AR / MR environment in which the user is allowed to move around.

[0015] Therefore, embodiments of this disclosure generally refer to imaging systems that can preload or cache (e.g., into a high-speed data storage device) one or more tiles of high-quality image data corresponding to a region within a VR / AR / MR environment, through which a user can potentially navigate or perceive the region from neighboring regions. Preloading tiles of high-quality image data enables faster access and transfer of tiles compared to storing tiles in other storage devices (e.g., non-volatile memory). After preloading the tiles of high-quality image data, the imaging system can cause a display device to display specific tiles of the preloaded tiles based on user input indicating that the user will be walking through the VR / AR / MR environment or that the user is facing a desired direction within the VR / AR / MR environment (e.g., by transmitting specific tiles to the display device). Then, when the user is facing and walking in the desired direction toward a region of the VR / AR / MR environment, or when the user is facing the region of the VR / AR / MR environment from a neighboring region in the desired direction, the imaging system can cause the display device to display one or more aspects of the corresponding region of the VR / AR / MR environment based on the received tiles. For example, an imaging system can cause a display device to show a user a cinematic scene or image that corresponds to an area of ​​the VR / AR / MR environment.

[0016] As mentioned herein, "high-quality" image or video data refers to 2D image data, 2D video data, 3D image data, 3D video data, etc., generated through light field technology, point cloud modeling, voxelization, etc. In some embodiments, the resolution of a VR / AR / MR environment with high-quality image data is at least 4K pixels (e.g., 3840 pixels x 2160 pixels or 4096 pixels x 2160 pixels), including 8K pixels (e.g., 7680 pixels x 4320 pixels). Additionally, as mentioned herein, a "tile" refers to one or more high-quality image data files that can be used to present aspects of a specific area of ​​a VR / AR / MR environment to a user on the display device after the imaging system transmits the tile to the display device. For example, a VR / AR / MR environment can be divided into a predetermined number of areas corresponding to the respective tiles. Each tile can contain high-quality image data that the display device can use to display aspects of the corresponding area of ​​the VR / AR / MR environment. In particular, tiles can be used to display virtual objects in the VR / AR / MR environment, virtual space above the user, virtual space below the user, user modifications, etc., on the display device. In some embodiments, a tile may have at least 1 terabyte (TB) of data, at least 850 megabytes (MB) of data, at least 750 MB of data, at least 600 MB of data, at least 500 MB of data, etc.

[0017] In one embodiment, the imaging system may cause a display device to show a city-based VR / AR / MR environment through which a user can navigate (e.g., move) as part of an interactive ride or experience within an amusement park. At the start of the interactive ride, the imaging system can render tiles from a database corresponding to a default or predetermined area within the city-based VR / AR / MR environment in which the user is located. The imaging system may also preload one or more tiles from a database corresponding to a specific area (e.g., a neighboring or adjacent area) within the city-based VR / AR / MR environment, which the user can potentially navigate through or perceive from the default area. The area corresponding to the preloaded tiles may border a default area within the city-based VR / AR / MR environment. The user may be allowed to move through such an area after leaving the boundary of the default area, and / or may be allowed to perceive such an area while at or near the boundary of the default area. For example, when a user approaches a virtual or displayed street intersection at the boundary of a default area, the imaging system can render the street intersection and portions of the intersection relative to the user along a first street in the forward direction, a second street relative to the user in the left direction, and / or a second street relative to the user in the right direction, based on tiles corresponding to the default area. The system can then preload tiles from a database corresponding to the rendered portions passing through the first and second streets.

[0018] In response to user input instructing the user to proceed in a desired direction through an intersection, the imaging system may cause a display device to display preloaded tiles corresponding to the area through the intersection (i.e., the selected area) in the desired direction (e.g., by transmitting the preloaded tiles to the display device). As the user moves toward and navigates within the selected area, the imaging system may cause the display device to present various aspects of the selected area based on the received preloaded tiles from high-quality image data. Additionally, the imaging system may preload one or more additional tiles from a database, which correspond to corresponding areas in a city-based VR / AR / MR environment that the user may potentially navigate through or perceive from the selected area (e.g., areas the user may subsequently move to). The area corresponding to the preloaded tiles may border the selected area within the city-based VR / AR / MR environment. The user may be allowed to proceed through such additional areas after leaving the boundary of the selected area, and / or may be allowed to perceive such additional areas at or near the boundary of the selected area. In some embodiments, the imaging system may also discard previously preloaded tiles corresponding to areas in a direction the user has not yet decided to take.

[0019] The imaging system can then repeat this process until the user has completed the interactive ride. That is, the imaging system can repeatedly preload one or more tiles of high-quality image data from a database, the tiles corresponding to corresponding areas in the VR / AR / MR environment through which the user can potentially navigate and / or perceive the corresponding areas from neighboring areas within the VR / AR / MR environment. After preloading the tiles of high-quality image data, the imaging system can cause the display device to display specific tiles of the preloaded tiles based on user input indicating that the user will travel through the VR / AR / MR environment or the desired direction the user is facing within the VR / AR / MR environment (e.g., by transmitting specific tiles to the display device). When the user faces and travels within an area of ​​the VR / AR / MR environment in the desired direction, and / or when the user faces an area of ​​the VR / AR / MR environment from a neighboring area in the desired direction, the imaging system can cause the display device to display the corresponding aspect of the area of ​​the VR / AR / MR environment based on the received tiles. In this way, the imaging system can process and transmit smaller amounts of high-quality image data multiple times during an interactive ride, instead of processing and loading the entire high-quality image dataset (e.g., a large number of tiles corresponding to high-quality image data) all at once. Accordingly, the techniques described herein continuously reduce and / or optimize the amount of computational resources the imaging system uses to provide a seamless VR / AR / MR experience to the user, based on user input indicating the desired orientation for user navigation through the VR / AR / MR environment and / or the desired orientation in the user's field of vision within the VR / AR / MR environment.

[0020] As an introduction, Figure 1This is a block diagram of an imaging system 100 according to embodiments described herein, which can preload one or more tiles of high-quality image data. These tiles can be used to display specific aspects of one or more corresponding areas of a VR / AR / MR environment on a user-accessible display device 108. For example, a control system 102 of the imaging system 100 can retrieve one or more tiles of high-quality image data from a database 104 communicatively coupled to the control system 102 via a network 106 and store the tiles in a preloader 103 of the control system 102. The preloader 103 can be any suitable high-speed data storage device capable of preloading or caching tiles of high-quality image data from the database 104. For example, the preloader 103 can include a caching device, such as a computer processing unit (CPU) cache device. In some embodiments, the CPU cache device can include an L1, L2, or L3 cache. Preloading tiles of high-quality image data into the preloader 103 can enable faster access and transfer of the tiles compared to storing them in other storage devices, such as non-volatile memory. For example, the preloader 103 may have a data transfer rate of at least 10 gigabytes per second (GB / s). In some embodiments, the preloader 103 may have a data transfer rate of at least 25 GB / s, at least 250 GB / s, or at least 1 terabyte per second (TB / s), etc. Additionally, the preloader 103 may have any suitable size to store an appropriate number of preloaded tiles. In some embodiments, the preloader 103 may have a data storage size of at least 5 TB, at least 2 TB, or at least 1 TB, etc.

[0021] The control system 102 may also include a memory 105 and a processor 107. The processor 107 of the control system 102 may include one or more of any suitable type of computer processor or microprocessor capable of executing computer-executable code, including but not limited to one or more field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), programmable logic devices (PLDs), programmable logic arrays (PLAs), etc. In some embodiments, the processor 105 may include multiple processors. The memory 105 may include any suitable article of manufacture used as a medium to store processor-executable code, data, etc. The memory 105 may store non-transitory processor-executable code used by the processor 107 to implement the techniques currently disclosed.

[0022] As described above, each tile of high-quality image data can be generated using any suitable high-quality imaging technique, such as light field technology, point cloud modeling, voxelization, etc. After the tiles of high-quality image data have been generated, they can be stored in database 104. In some embodiments, the tiles can be indexed in database 104 using tile identifiers, region identifiers associated with corresponding areas of the VR / AR / MR environment, one or more boundary region identifiers associated with corresponding boundary areas of the VR / AR / MR environment, etc. Control system 102 can transmit requests with indications of tile identifiers, region identifiers, and / or boundary region identifiers (e.g., pointers to tile identifiers, region identifiers, and / or boundary region identifiers) to database 104, and database 104 can transmit a response with the corresponding tile to control system 102.

[0023] Display device 108 can be any suitable device for displaying VR / AR / MR content to a user, such as smart glasses, a virtual retina display, one or more contact lenses, a computer, a mobile device, a head-mounted device, etc. Display device 108 may optionally have one or more sensors 110 that can acquire data associated with the user and transmit that data via network 106 to control system 102 for analysis. Control system 102 can use the received data associated with the user to determine the direction the user is looking in (e.g., a specific direction in the user's field of vision) or the direction the user intends to move in (e.g., the direction the user desires to move). For example, sensors 110 may include: one or more image sensors that can acquire image or video data associated with the user's eyes, head, limbs, etc.; one or more microphones that can acquire sound data associated with the user; one or more motion sensors (e.g., velocity sensors, position sensors, or accelerometers) that can acquire motion data associated with the user, etc. In some embodiments, sensors 110 may optionally or additionally be attached to the user's body. For example, one or more motion sensors may be located on the user's hand, wrist, arm, fingers, leg, foot, torso, or any other suitable part of the user's body to acquire the user's motion data. Sensor 110 may be optionally or additionally located in the user's physical environment. For example, in an interactive riding setup, sensor 110 may be located along a predetermined path that the user may physically walk along, or within the riding vehicle associated with the user.

[0024] After acquiring user-associated data, sensor 110 can transmit the user-associated data to control system 102 via network 106 for analysis to determine different types of user input that the user can provide to modify the VR / AR / MR environment or otherwise control the user's experience of the VR / AR / MR environment. In some embodiments, control system 102 can determine the user's desired direction of movement through the VR / AR / MR environment based on the analysis of user-associated data, or determine the desired direction in the user's field of vision within the VR / AR / MR environment based on the analysis of data received from display device 108. For example, control system 102 can determine one or more user characteristics based on the analysis of data received from display device 108, such as user position (e.g., the position of the user's eyes, arms, legs, head, or body), user movement (e.g., the user's eyes, arms, legs, head, or body), or user orientation (e.g., the user's eyes, arms, legs, head, or body (e.g., directional tilt, pitch, yaw, or roll)). Control system 102 can then determine the user's desired direction of movement or the desired direction in the user's field of vision based on the determined user characteristics. In some embodiments, the control system 102 may compare the determined user characteristics with one or more stored, learned, or otherwise interpretable directional movements of the user or desired orientations in the user's field of vision stored in memory accessible by the control system 102. The control system 102 may also use image and / or pattern recognition techniques to determine the user's position, user movement, or user orientation based on analysis of data received from the display device 108. For example, image and / or pattern recognition techniques or algorithms may include machine learning, artificial intelligence, deep learning, convolutional neural networks, etc. The memory accessible by the control system 102 may store image recognition models, speech recognition models, etc. Such models can be trained by inputting human sample data (e.g., images) and indications of user position, user movement, or user orientation in the data. After training such models, the control system 102 can then use one or more of the models to determine a specific user position, specific user movement, or specific user orientation in the data received from the display device 108. The control system 102 can then determine that the determined user characteristics are associated with the user's desired directional movement or desired orientations in the user's field of vision.

[0025] Alternatively or concurrently, the control system 102 may receive user input or commands from one or more user input devices 112, such as gesture commands or voice commands from the user. For example, the control system 102 may analyze the user input or commands and determine the gesture or voice commands via image and / or pattern recognition techniques or algorithms (including machine learning, artificial intelligence, deep learning, convolutional neural networks, etc.). For example, a memory accessible to the control system 102 may store image recognition models, speech recognition models, etc. Such models can be trained by inputting human sample data (e.g., images or speech) and indications of various gesture or voice commands in the data. After training such a model, the control system 102 can then use one or more models in the model to determine a specific gesture or voice command in the user input received from the user input device 112.

[0026] Based on determined user characteristics or user commands, the control system 102 can correspondingly modify the VR / AR / MR environment perceived by the user. For example, after receiving tiles from the control system 102, the control system 102 can cause the display device 108 to display tiles of high-quality image data from the preloader 103 based on the user's movement in the physical environment. The high-quality image data tiles may correspond to regions of the VR / AR / MR environment that intersect with the current region of the VR / AR / MR environment in which the user is virtually located. As the user approaches the boundary of the current region in the VR / AR / MR environment, the control system 102 can cause the display device 108 to present various aspects of the region intersecting with the current region based on the received tiles of the high-quality image data, to provide the user with a continuous experience of the VR / AR / MR environment as if the user were experiencing it in the real world. That is, the control system 102 can cause the display device 108 to gradually display certain aspects of the boundary region of the VR / AR / MR environment to the user as the user moves toward the boundary region of the VR / AR / MR environment.

[0027] Based on determined user characteristics or user commands, the control system 102 can also preload one or more tiles from the database 104, which the user can potentially navigate through (e.g., an area the user may move to next) or perceive from an area the user has already virtually moved into. For example, the control system 102 can determine, based on determined user characteristics or commands, that the user intends to virtually move in a desired direction toward and / or to a specific area in the VR / AR / MR environment. Based on the user's desired direction, the control system 102 can preload one or more tiles corresponding to areas that respectively border the specific areas in the VR / AR / MR environment toward which and / or to which the user intends to move.

[0028] Furthermore, the control system 102 can determine, based on determined user characteristics or user commands, the user's field of vision (e.g., viewing angle) that modifies the VR / AR / MR environment displayed via the display device 108. For example, based on the position or movement of the user's eyes, the control system 102 can modify the appearance of the high-quality image data displayed to the user via the display device 108, allowing the user to perceive the VR / AR / MR environment with a field of vision similar to that the user would have in the real world. That is, in response to the user changing their viewing angle of objects within the VR / AR / MR environment, the control system 102 can modify the appearance of objects in the VR / AR / MR environment (similar to how objects with similar changes in viewing angle would appear to the user in the real world), and cause the display device 108 to display the modified appearance.

[0029] As illustrated in the described embodiments, the control system 102 may be communicatively coupled to one or more user input devices 112 associated with a user. For example, user input devices 112 may include one or more input devices such as joysticks, steering wheels, touchscreen displays, mobile phones, or any other suitable means for providing user input. In some embodiments, user input devices 112 may be communicatively coupled to a display device 108. In any case, the control system 102 may receive one or more user commands from user input devices 112. After receiving a user command from user input devices 112, the control system may modify the VR / AR / MR environment, adjust the user's field of view of the VR / AR / MR environment, or otherwise control the user experience of the VR / AR / MR environment based on the received user command. For example, the control system 102 may compare the received user command with one or more stored, learned, or otherwise interpretable user commands in memory accessible to the control system 102, and modify the VR / AR / MR environment based on the comparison. In some embodiments, memory may include memory 105, read-only memory (ROM) of the control system 102, or database 104.

[0030] It should be noted that any suitable network 106 can be employed in the embodiments described herein. For example, network 106 may include any wired or wireless communication network implemented, such as a local area network (LAN), a wide area network (WAN), etc. Network 106 can be implemented via any suitable communication protocol, such as Wi-Fi, mobile telecommunications protocols (e.g., 2G, 3G, 4G, 5G, LTE, NR), Bluetooth®, near field communication protocols, etc.

[0031] Considering the foregoing, Figure 2A schematic diagram 200 illustrates an exemplary VR / AR / MR environment that a user can experience in an interactive ride setting according to the embodiments described herein. In the illustrated embodiment, the schematic diagram 200 of the VR / AR / MR environment may be designed as a city in which a user can navigate between multiple areas (i.e., 202, 204, 206, 208, 210, 212, 214, 216, 218, 220, 222, 224, 226, 228) of the VR / AR / MR environment at designated intersections (i.e., A, B, C, D, E, F, G, H, I, J, K) between adjacent areas. At the start of the interactive ride, the user can begin at a default location 201 in a default area 202 within the VR / AR / MR environment. Although the VR / AR / MR environment is described as a city-based VR / AR / MR environment, it should be understood that such embodiments are intended to be exemplary and non-limiting. In other embodiments, the VR / AR / MR environment can have any other suitable design that can be divided into multiple regions corresponding to tiles of high-quality image data, such as mazes, castles, forests, external spaces, etc. In any case, intersections in the VR / AR / MR environment can prompt the user to make a decision about the next region the user will move to after passing through the intersection. For example, each intersection can be presented to the user in the VR / AR / MR environment as the user approaches the boundary of the region the user is currently in. As the user approaches an intersection, the user can provide one or more types of user input to indicate the desired direction the user will take when passing through the intersection. As described above, user input can include physical movement of the user in the desired direction, gesture commands, voice commands, etc.

[0032] Before or during a suitable period after the start of an interactive ride, the control system 102 of the imaging system 100 may cause the display device 108 to display tiles of high-quality image data from the preloader 103, the tiles corresponding to a default area 202 in the city-based VR / AR / MR environment in which the user is located (e.g., by transmitting the tiles to the display device 108). For example, after the display device 108 receives the tiles of high-quality image data, the control system 102 may cause the display device 108 to display various aspects of the default area 202 of the city-based VR / AR / MR environment to the user based on the tiles of high-quality image data. These aspects of the default area 202 may include buildings, vehicles, streets, sidewalks, storefronts, sky, etc. During the same time period, the control system 102 can preload one or more additional tiles of high-quality image data from the database 104 into the preloader 103. These additional tiles correspond to specific areas 204, 206, and 208 in a city-based VR / AR / MR environment, which the user can potentially travel into or perceive from the default area 202. That is, the control system 102 can preload areas 204, 206, and 208 that intersect with, are adjacent to, and / or border with the default area 202 within the city-based VR / AR / MR environment.

[0033] In the illustrated embodiment, when a user approaches intersection A at the boundary of default area 202, control system 102 may cause display device 108 to display portions of intersection A and a first street relative to the user in the forward direction (i.e., leading to area 206), a second street relative to the user in the left direction (i.e., leading to area 204), and a second street relative to the user in the right direction (i.e., leading to area 208), based on tiles corresponding to high-quality image data corresponding to default area 202. In some embodiments, when the user approaches the boundary of default area 202 or travels into intersection A, control system 102 may also cause display device 108 to gradually display portions of areas 204, 206, and 208 to the user based on corresponding tiles of high-quality image data received from control system 102, to provide the user with a continuous VR / AR / MR environment experience. For example, when a user is moving toward the center of intersection A, the control system 102 may cause the display device 108 to display an increased portion of the area 204, 206, 208 along the first street and the second street based on the user's distance from the center of the intersection, the user's field of vision (e.g., the direction the user is facing), etc.

[0034] Additionally, users may be allowed to move forward (e.g., as indicated by a single-sided arrow), backward (e.g., as indicated by a single-sided arrow), or both (e.g., as indicated by a double-sided arrow) within each area relative to their default position 201 in the VR / AR / MR environment. In the illustrated embodiments, for example, a user may be allowed to move forward from their default position 201 through area 202 to intersection A, while simultaneously being allowed to move forward from intersection A through area 206 to intersection C and / or backward from intersection C through area 206 to the intersection. In some embodiments, based on the storyline of the VR / AR / MR environment, the user may be restricted to one or more directions of movement. Furthermore, although the schematic diagram 200 of the VR / AR / MR environment is illustrated as a grid, it should be understood that such embodiments are intended to be exemplary and non-limiting. For example, at a decision point (e.g., an intersection), the user may have more than three or fewer options regarding the direction of movement through the decision point. In some embodiments, in addition to moving left, right, forward, or backward, users may also be allowed to move up or down (e.g., virtual stairs or virtual ramps).

[0035] Considering the preceding text, Figure 3 This is a block diagram 300 of a preloader 103 for a display device 108 and a control system 102 according to embodiments described herein. Referring to the above... Figure 2 As described in the example, when user 304 approaches intersection A from the default area 202 of VR / AR / MR environment 302, control system 102 can cause display device 108 to display aspects of the default area 202 of VR / AR / MR environment 302 based on tiles 308 of high-quality image data received by control system 102 from preloader 103. Additionally, control system 102 can store tiles (e.g., 310, 312, 314) of high-quality image data from database 104 in preloader 103, which correspond to areas 204, 206, 208 of VR / AR / MR environment 302 that the user may potentially navigate through or perceive from the default area 202 (e.g., areas the user may subsequently move to).

[0036] In response to a user input instructing a forward direction 306 through intersection A, control system 102 can transfer high-quality image data tiles 308 from preloader 103 to display device 108. High-quality image data tiles 308 correspond to area 206 of the VR / AR / MR environment the user will be traveling towards in the forward direction from intersection A. Additionally, control system 102 can preload one or more additional high-quality image data tiles (e.g., 306, 308, 310), which correspond to areas 214, 216, 218 (e.g., adjacent areas) of the VR / AR / MR environment that user 404 can potentially navigate through or perceive from area 206. For example, control system 102 can receive additional tiles 306, 308, 310 from database 104 and store them in preloader 103. In some embodiments, the preloader 103 may discard unused tiles 310, 314 corresponding to regions 204, 208 of the high-quality image data remaining in the preloader 103 before, simultaneously with, or after receiving additional tiles 306, 308, 310. That is, the preloader 103 may discard tiles 310, 314 of high-quality image data that have not been transmitted to the display device 108 before, simultaneously with, or after receiving additional tiles 306, 308, 310. In other embodiments, the preloader 103 may retain certain tiles of high-quality image data corresponding to regions 204, 208, which are adjacent to region 206, toward which the user has chosen to move, and / or may retain certain tiles of high-quality image data corresponding to regions 204, 208 based on whether the user is allowed to move backward from region 206 to those regions 204, 208. In some embodiments, the preloader 103 also stores tiles corresponding to regions frequently accessed by the user. For example, additional or alternative locations corresponding to adjacent areas, such as tiles corresponding to virtual shops, save points, starting points, etc., can be stored in the preloader 103. In this way, the control system 102 of the imaging system 100 can minimize the retrieval of duplicate data from the database 104.

[0037] As the user moves toward the boundary of region 202, the control system 102 can cause the display device 108 to display one or more aspects of region 206 based on tiles 312 of high-quality image data received from the preloader 103. In this way, the control system 102 can cause the display device 108 to present certain aspects of region 206 that intersect with region 202 to provide the user with a continuous experience of a VR / AR / MR environment as the user would experience in the real world. That is, the control system 102 can cause the display device 108 to gradually display aspects of region 206 to the user as the user moves toward region 206 of the VR / AR / MR environment. In some embodiments, the control system 102 can cause the display device 108 to display certain aspects of region 202 based on tiles 308 of high-quality image data until the user has moved to a threshold distance within region 206. For example, the threshold distance could correspond to the distance the user could travel before such an aspect typically disappears from view in the real world. In this way, the control system 108 can simulate the sensory experience the user would typically receive in the real world. Additionally, in some embodiments, the display device 108 may retain tiles 308 of high-quality image data corresponding to region 202 until the user has entered a region that is not adjacent to region 202 (e.g., 218, 220, 222, 224, 226, or 228) (e.g., in order to render faster and more efficiently if the user returns to region 202).

[0038] Similarly, Figure 4 This is a block diagram 400 of the display device 108 and the preloader 103 of the control system 102 when a user 404 approaches an intersection F from region 218 of a VR / AR / MR environment 402, according to the embodiments described herein. When the user 404 approaches the intersection F, the control system 102 may cause the display device 108 to display aspects of region 218 of the VR / AR / MR environment 402 based on tiles 408 of high-quality image data received from the preloader 103 via the control system 102. Additionally, the preloader 103 of the control system 102 may store tiles 410 of high-quality image data corresponding to region 222 of the VR / AR / MR environment, through which the user may potentially navigate (e.g., the user may move to the next region) or perceive region 222 from region 218.

[0039] In response to a user input instructing the user to take a forward direction 406 through intersection F, control system 102 can transmit high-quality image data tiles 410 from preloader 103 to display device 108. High-quality image data tiles 410 correspond to area 222 of the VR / AR / MR environment the user will be traveling towards from intersection F in the forward direction. Furthermore, control system 102 can preload high-quality image data tiles 412, which correspond to area 228 of the VR / AR / MR environment that the user 404 can potentially navigate through or perceive from area 222. As the user moves towards the boundary of area 218, control system 102 can cause display device 108 to begin displaying one or more aspects of area 222 based on tiles 410 of the high-quality image data received from preloader 103. Because the user is not allowed to move backward towards intersection F in area 222, control system 102 can discard any copies of tiles 408 corresponding to area 218 in preloader 103.

[0040] Considering the preceding text, Figure 5 A flowchart illustrating a method 500 according to embodiments described herein is provided for: preloading one or more tiles of high-quality image data, each tile corresponding to a region within a VR / AR / MR environment through which a user can potentially navigate or perceive the region from a neighboring region within the VR / AR / MR environment; and instructing a display device to display the preloaded tiles of high-quality image data for displaying aspects of the corresponding regions of the VR / AR / MR environment based on user input instructing the user to traverse the VR / AR / MR environment or a desired direction in which the user is facing within the VR / AR / MR environment. Although the following description of method 500 is given in a specific order, it should be noted that method 500 is not limited to the depicted order, and instead, method 500 can be implemented in any suitable order. In fact, at least some steps of method 500 can be skipped entirely. Furthermore, although method 500 is described as being implemented by the control system 102 of imaging system 100, it should be noted that it can be implemented by any suitable computing device.

[0041] As mentioned above, during an interactive ride in the amusement park, users can navigate through areas within a VR / AR / MR environment. Now refer to Figure 5At block 502, when a user approaches the boundary of the area the user is navigating through or a nearby intersection, the control system 102 of the imaging system 100 can receive user-associated data from one or more input devices 112, one or more sensors 110, or both. For example, the user input device 112, sensor 110, or both can continuously acquire user-associated data and transmit the data to the control system 102. At block 504, after the control system 102 receives user-associated data from the input device 112, sensor 110, or both, the control system 102 can determine that the received data corresponds to user input indicating the desired direction in which the user intends to proceed through the intersection. For example, user-associated data may include image or video data associated with the user's eyes, head, limbs, etc., voice data associated with the user, motion data associated with the user, etc. The control system 102 can determine one or more user characteristics (such as the position of the user's eyes, arms, legs, head, or body, the movement of the user's eyes, arms, legs, head, or body, or the orientation of the user's eyes, arms, legs, head, or body (e.g., directional tilt, pitch, yaw, or roll) or user commands (such as gesture commands, voice commands, etc.) based on the analysis of received data associated with the user. The control system 102 can then determine the desired direction in which the user intends to move across the intersection based on the one or more user characteristics and / or one or more user commands. For example, the desired direction could be left, right, forward, up, down, etc. In some embodiments, the control system 102 can determine the desired direction via any suitable image recognition technology or algorithm (including machine learning, artificial intelligence, deep learning, convolutional neural networks, etc.).

[0042] After determining that the received data corresponds to user input indicating the user's intended direction of travel through the intersection, at block 506, the control system 102 may transfer tiles of high-quality image data from the preloader 103 to the display device 108 based on the user input indicating the intended direction. For example, the high-quality image data tiles may correspond to an area along the intended direction of travel through the intersection. As the user travels towards and navigates within that area in the intended direction, the control system 102 may send commands to cause the display device 108 to present various aspects of that area based on the received tiles of high-quality image data. For example, the control system 102 may cause the display device 108 to use the high-quality image data tiles to display buildings, vehicles, streets, sidewalks, storefronts, sky, etc. In some embodiments, the control system 102 may cause the display device 108 to display one or more videos or animations corresponding to a storyline associated with the interactive ride.

[0043] At block 508, control system 102 can preload one or more additional tiles of high-quality image data into preloader 103 based on tiles of high-quality image data transmitted to display device 108 at block 506. For example, the additional tiles of high-quality image data may correspond to areas of a VR / AR / MR environment that a user can potentially travel toward or navigate through from an area via an intersection in a desired direction. In some embodiments, control system 102 can transmit a request for additional tiles to database 104. This request may include one or more identifiers associated with the tiles transmitted to display device 108 at block 506, the area corresponding to the tiles transmitted to display device 108, and / or one or more areas bordering the area corresponding to the tiles transmitted to display device 108. In response to receiving a request for additional tiles, database 104 can transmit a response with additional tiles to control system 102. Control system 102 can then store the additional tiles received from database 104 in preloader 103. In some embodiments, the control system 102 may discard unused tiles of high-quality image data existing in the preloader 103 before storing additional tiles in the preloader 103.

[0044] After preloading the additional tiles into the preloader 103, the control system 102 can repeat the method described above with respect to boxes 502 to 508. That is, the control system can repeat boxes 502 to 508 until the user has completed the interactive ride. In this way, the control system 102 of the imaging system 100 can process and transmit a smaller amount of high-quality image data multiple times during the interactive ride, instead of processing and loading the entire high-quality image dataset once. Therefore, the technique described herein continuously reduces and optimizes the amount of computational resources used by the imaging system 100 to provide a seamless experience of the VR / AR / MR environment to the user based on user input indicating the desired direction of user navigation through the VR / AR / MR environment.

[0045] While only certain features of this disclosure have been described and illustrated herein, many modifications and changes will be apparent to those skilled in the art. Therefore, it is to be understood that the appended claims are intended to cover all such modifications and changes that fall within the true spirit and scope of this disclosure.

[0046] The techniques proposed and claimed herein are referenced and applied to physical objects and specific examples of practical nature, which can demonstrably improve the technical field and are therefore not abstract, intangible, or purely theoretical. Furthermore, if any claim appended to this specification contains one or more elements designated as “component for [implementing]…[function]” or “step for [implementing]…[function]”, such elements are intended to be interpreted according to 35 USC 112(f). However, for any claim containing elements designated in any other way, such elements are intended not to be interpreted according to 35 USC 112(f).

Claims

1. A system for preloading high-quality image data, comprising: One or more processors; as well as A memory accessible by the one or more processors, the memory storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including: Data associated with the user is received from one or more sensors located in the user's physical environment, wherein the one or more sensors include one or more sensors located along a predetermined path that the user can physically walk on or located within a vehicle associated with the user. Determine that the received data corresponds to the user's direction of movement through a virtual reality (VR), augmented reality (AR), or mixed reality (MR) environment; Based on the user's direction of movement, tiles of high-quality image data associated with a region of the VR environment, AR environment, or MR environment are transmitted to the display device. The command is transmitted to the display device to display one or more aspects of a region of the VR environment, the AR environment, or the MR environment based on the tiles of high-quality image data; and Based on the tiles of the high-quality image data transmitted to the display device, one or more additional tiles of the high-quality image data are preloaded into the preloader, wherein the one or more additional tiles of the high-quality image data are associated with one or more neighboring regions of the VR environment, the AR environment, or the MR environment, and the one or more neighboring regions are at the boundary of the region of the VR environment, the AR environment, or the MR environment; The received data determines that the user is moving toward the first neighboring area among the one or more neighboring areas; and Additional tiles of high-quality image data associated with the first neighboring region are transferred from the preloader to the display device.

2. The system according to claim 1, wherein, The data associated with the user is received before the user virtually approaches the boundary of the area of ​​the VR environment, AR environment, or MR environment.

3. The system according to claim 1, wherein, The operation includes: Receive additional data associated with the user from the one or more input devices, the one or more sensors, or both; and The received additional data is determined to correspond to a second direction of movement of the user through the VR environment, the AR environment, or the MR environment.

4. The system according to claim 3, wherein, The operation includes: transferring a specific tile from the preloader to the display device from one or more additional tiles of high-quality image data based on the user's second movement direction; and transmitting a second command to the display device to display one or more additional aspects of a second region of the VR environment, the AR environment, or the MR environment based on the specific tile from the one or more additional tiles.

5. The system according to claim 1, wherein, The operation includes: Determine that the received data associated with the user corresponds to a specific direction in the user's field of vision within the area of ​​the VR environment, AR environment, or MR environment; Based on the specific direction within the user's field of view, a specific tile from one or more additional tiles of high-quality image data is transmitted from the preloader to the display device; and The second command is transmitted to the display device to display one or more additional aspects of a second region that intersects with the region in the direction of the field of view in the VR environment, the AR environment, or the MR environment, based on a specific tile in one or more additional tiles of high-quality image data.

6. The system according to claim 1, wherein, Based on the fact that the user is not allowed to move backward through the area of ​​the VR, AR, or MR environment, one or more additional tiles of the high-quality image data are deleted.

7. A method for preloading high-quality image data, comprising: Receive user-related data from at least one or more input devices or one or more sensors via one or more processors; The one or more processors determine that the received data corresponds to the user's direction of movement through a virtual reality (VR), augmented reality (AR), or mixed reality (MR) environment; Based on the user's direction of movement, one or more high-quality image data files are transmitted to the display device via the one or more processors; Commands are transmitted to the display device via the one or more processors to display one or more virtual objects in a region of the VR environment, AR environment, or MR environment based on the one or more high-quality image data files transmitted to the display device, wherein the user is not permitted to move backward through the region of the VR environment, AR environment, or MR environment; Based on the user's movement direction, one or more additional high-quality image data files are preloaded into the preloader via the one or more processors; and Based on the fact that the user is not allowed to move backward through the area of ​​the VR, AR, or MR environment, the preloaded additional high-quality image data file is deleted.

8. The method according to claim 7, wherein, The one or more additional high-quality image data files are associated with one or more neighboring regions of the VR environment, the AR environment, or the MR environment, and the one or more neighboring regions are adjacent to the regions of the VR environment, the AR environment, or the MR environment.

9. The method according to claim 7, wherein, The one or more high-quality image data files and the one or more additional high-quality image data files are generated via light field technology, point cloud modeling, voxelization, or a combination thereof.

10. The method according to claim 7, wherein, The data associated with the user is received before the user virtually approaches the boundary of the area of ​​the VR environment, AR environment, or MR environment.

11. The method of claim 7, comprising: Additional data associated with the user is received from the one or more input devices, the one or more sensors, or both via the one or more processors; as well as The received additional data is determined by the one or more processors to correspond to a second direction of movement of the user through the VR environment, the AR environment, or the MR environment.

12. The method of claim 11, comprising: Based on the user's second movement direction, a specific high-quality image data file from the preloader is transferred to the display device from the one or more additional high-quality image data files; And transmit a second command to the display device to display one or more additional aspects of a second region of the VR environment, the AR environment, or the MR environment based on the specific high-quality image data file in the one or more additional high-quality image data files.

13. The method according to claim 7, wherein, The preloading of the one or more additional high-quality image data files into the preloader via the one or more processors is based on the frequency with which the user accesses the region.

14. A non-transitory computer-readable medium comprising instructions, which, when executed by one or more processors, cause the one or more processors to perform operations including: Receive user-related data from one or more input devices, one or more sensors, or both; Determine that the received data corresponds to a specific direction in the user's field of vision within a virtual reality (VR), augmented reality (AR), or mixed reality (MR) environment; High-quality image data tiles are transmitted to the display device based on the specific direction in the user's field of view; The command is transmitted to the display device to display one or more aspects of a first region of the VR environment, the AR environment, or the MR environment based on the tiles of high-quality image data; Based on the tiles of high-quality image data being transmitted to the display device, angle and direction values ​​of one or more rays received by the image sensor are preloaded into a preloader, the angle and direction values ​​of the one or more rays corresponding to one or more additional tiles of the high-quality image data.

15. The non-transitory computer-readable medium according to claim 14, wherein, The operation includes: Receive additional data associated with the user from the one or more input devices, the one or more sensors, or both; and The received additional data is determined to correspond to a second direction in the user's field of vision within the VR environment, AR environment, or MR environment.

16. The non-transitory computer-readable medium according to claim 15, wherein, The operation includes: transferring a specific tile from the preloader to the display device from one or more additional tiles of high-quality image data based on the second direction in the user's field of view; and transmitting a second command to the display device to display one or more additional aspects of a second region of the VR environment, the AR environment, or the MR environment based on the specific tile in the one or more additional tiles.

17. The non-transitory computer-readable medium according to claim 14, wherein, The one or more additional tiles of the high-quality image data are associated with one or more neighboring regions of the VR environment, the AR environment, or the MR environment, and the one or more neighboring regions intersect with the region of the VR environment, the AR environment, or the MR environment.

18. The non-transitory computer-readable medium according to claim 14, wherein, The data associated with the user is received before the user virtually approaches the boundary of the area of ​​the VR environment, AR environment, or MR environment.

19. The non-transitory computer-readable medium according to claim 14, wherein, The operation includes deleting one or more preloaded additional high-quality image data tiles based on the fact that the user is not allowed to move backward through the area of ​​the VR, AR, or MR environment.

Images