Techniques for preloading and displaying high quality image data

Abstract

To provide a technique for preloading and displaying high-quality image data.SOLUTION: A system may include one or more processors and a memory storing instructions that, when executed by the processors, cause the processors to perform operations including receiving data associated with a user and determining that the received data corresponds to a direction of movement of the user through a virtual reality (VR) environment, an augmented reality (AR) environment, or a mixed reality (MR) environment. The operations may also include sending a tile of high quality image data to the display device based on the direction of movement of the user, sending a command to the display device to display one or more aspects of an area of the VR, AR, or MR environment based on the tile of high quality image data, and preloading one or more additional tiles into the preloader based on the tile of high quality image data.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of U.S. Provisional Patent Application Serial No. 63 / 002,056, filed March 30, 2020, entitled "TECHNIQUES FOR PRELOADING AND DISPLAYING HIGH QUALITY IMAGE DATA," which is incorporated herein by reference in its entirety for all purposes.

[0002] The present disclosure relates generally to displaying image data in a virtual, augmented, or mixed reality environment. Specifically, the present disclosure relates to an imaging system that can preload image data so that aspects of one or more regions of the virtual, augmented, or mixed reality environment can be easily displayed on a display device. [Background technology]

[0003] As new imaging technologies are developed, higher quality images or videos can be generated and then displayed to a user in a virtual reality ("VR"), augmented reality ("AR"), or mixed reality ("MR") environment via a display device. However, high quality images or videos can result in very large file sizes due to the amount of data in each image or video. Therefore, large-scale displays of VR / AR / MR environments provided by image or video files generated from such high quality imaging technologies can require significant computational resources (e.g., terabytes or more), which can be cost or resource prohibitive.

[0004] This section is intended to introduce the reader to various aspects of art that may be related to various aspects of the present disclosure, which are described and / or claimed below. This discussion is believed to be helpful in providing the reader with background information to facilitate a better understanding of the various aspects of the present disclosure. As such, it should be understood that these statements are to be read in this light, and not as admissions of prior art. Summary of the Invention

[0005] The following provides an overview of some embodiments disclosed herein. It should be understood that these aspects are merely intended to provide the reader with a summary of some embodiments and are not intended to limit the scope of the present disclosure. In fact, the present disclosure may include various aspects that may not be set forth below.

[0006] In one embodiment, a system can include one or more processors and a memory storing instructions that, when executed by the processor, cause the processor to perform operations including receiving data associated with a user and determining that the received data corresponds to a direction of movement of the user through a virtual reality (VR), augmented reality (AR), or mixed reality (MR) environment. The operations can also include transmitting tiles of high-quality image data to a display device based on the direction of movement of the user, transmitting commands to the display device to display one or more aspects of a region of the VR, AR, or MR environment based on the tiles of high-quality image data, and preloading one or more additional tiles into a preloader based on the tiles of high-quality image data.

[0007] In another embodiment, a method may include receiving, via one or more processors, data associated with a user from one or more input devices, one or more sensors, or both, and determining, via the processor, that the received data corresponds to a direction of movement of the user through a virtual reality (VR), augmented reality (AR), or mixed reality (MR) environment. The method may also include transmitting, via the processor, one or more high-quality image data files to a display device based on the direction of movement of the user, transmitting, via the processor, a command to the display device to display one or more virtual objects in an area of ​​the VR, AR, or MR environment based on the one or more high-quality image data files transmitted to the display device, and preloading, via the processor, one or more additional high-quality image data files into a preloader based on the direction of movement of the user.

[0008] In another embodiment, a non-transitory computer-readable medium includes instructions that, when executed by one or more processors, cause the processors to perform operations including receiving data related to a user from one or more input devices, one or more sensors, or both, and determining that the received data corresponds to a particular direction in a field of view of a user in a virtual reality (VR), augmented reality (AR), or mixed reality (MR) environment. The operations also include transmitting tiles of high-quality image data to a display device based on the particular direction in the field of view of the user, transmitting commands to the display device to display one or more aspects of a region of the VR, AR, or MR environment based on the tiles of high-quality image data, and preloading one or more additional tiles of high-quality image data into a preloader based on the tiles of high-quality image data transmitted to the display device.

[0009] Various refinements of the features described above may exist in connection with various aspects of the present disclosure. Furthermore, additional features may be incorporated into these various aspects. These refinements and additional features may exist individually or in any combination. For example, any of the above-described aspects of the present disclosure may incorporate, alone or in any combination, various features described below in connection with one or more of the illustrated embodiments. The foregoing summary is intended merely to familiarize the reader with certain aspects and contexts of embodiments of the present disclosure without limitation to the claimed subject matter.

[0010] These and other features, aspects and advantages of the present invention will be better understood from the following detailed description when read in conjunction with the accompanying drawings, in which like parts are designated by like numerals throughout. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a block diagram of an imaging system that preloads high-quality image data corresponding to one or more regions of a virtual reality (VR), augmented reality (AR), or mixed reality (MR) environment, according to embodiments described herein. [Figure 2] FIG. 1 is a schematic diagram of an exemplary VR / AR / MR environment, according to embodiments described herein. [Figure 3] 3 is a block diagram of a preloader and display device of the imaging system of FIG. 1 as a user navigates through a first set of regions of the example VR / AR / MR environment of FIG. 2, according to embodiments described herein. [Figure 4] 3 is a block diagram of a preloader and display device of the imaging system of FIG. 1 as a user navigates through a second set of regions of the example VR / AR / MR environment of FIG. 2, according to embodiments described herein. [Figure 5]1 is a flowchart of a method for displaying high-quality image data corresponding to a particular area of ​​a VR / AR / MR environment and preloading high-quality image data corresponding to one or more areas adjacent to the particular area into a preloader, according to embodiments described herein. DETAILED DESCRIPTION OF THE INVENTION

[0012]

[0013] One or more specific embodiments of the present disclosure will now be described. In the interest of brevity in describing these embodiments, not all features of an implementation may be described herein. It will be understood that the development of any such implementation, as in any engineering or design project, involves numerous implementation-specific decisions to achieve the developer's particular objectives, including compliance with system- and business-related constraints that may vary from implementation to implementation. Moreover, it will be understood that such a development effort may be complex and time-consuming, but would be a routine undertaking of design, fabrication, and manufacture for those of ordinary skill in the art having the benefit of this disclosure.

[0013] When introducing elements of various embodiments of the present disclosure, the articles “a,” “an,” “the,” and “said” are intended to mean the presence of one or more of the element. The terms “comprising,” “including,” and “having” are intended to be inclusive and mean that there may be additional elements other than the listed elements. One or more specific embodiments of the present embodiments described herein will now be described. In the interest of brevity, not all features of an implementation are described herein. It should be understood that the development of any such implementation, as in any engineering or design project, requires numerous implementation-specific decisions to achieve the developer's particular objectives, including compliance with system- and business-related constraints that may vary from implementation to implementation. Moreover, it should be understood that such a development effort might be complex and time-consuming, but would be a routine undertaking of design, fabrication, and manufacture for those of ordinary skill in the art having the benefit of this disclosure.

[0014] As described above, high-quality images or videos can be generated from various imaging technologies and then displayed to a user in a virtual reality ("VR"), augmented reality ("AR"), or mixed reality ("MR") environment via a display device. Such high-quality imaging technologies can include techniques such as light field technology, point cloud modeling, and voxelization. Through the use of image or video files generated from these technologies and then displayed to the user, a user can perceive a level of detail in digitally represented objects in a VR / AR / MR environment similar to the level of detail a user typically perceives in the real world. For example, a light field volume can be generated by capturing one or more images of a real-world object that include brightness and color values ​​of each light ray received by an image sensor, as well as the direction and / or angle of each light ray received by the image sensor. In this manner, the light field volume can include sub-images that differ slightly from each other based on the direction and / or angle of each associated light ray. This light field volume can be processed based on a desired depth of focus or a desired perspective of the object to generate a two-dimensional ("2D") or three-dimensional ("3D") image of the object that corresponds to the desired depth of focus or the desired perspective. In this manner, images or videos generated from light field techniques can be displayed within a VR / AR / MR environment to provide a user with a perspective of the object within the VR / AR / MR environment similar to what the user has for the object in the real world.

[0015] However, image or video files providing light field volumes are very large (e.g., terabytes or more) due to the amount of image data in each light field volume. Similarly, the sizes of image and video files generated via point cloud modeling, voxelization, and other high-quality imaging techniques can also be very large. Therefore, large-scale display of VR / AR / MR environments using files generated from such high-quality imaging techniques requires significant computational resources, which can be cost-prohibitive or resource-intensive. For example, in an interactive ride environment where a user moves around within the VR / AR / MR environment and the user's field of view within the VR / AR / MR environment may change, multiple high-quality image files may be loaded to display a scene based on the user's position within the VR / AR / MR environment, the user's field of view within the VR / AR / MR environment, or both. Furthermore, the size of the high-quality image files and the number of high-quality image files may increase exponentially with the size of the VR / AR / MR environment in which the user can move around.

[0016] Thus, in general, embodiments of the present disclosure relate to an imaging system that can preload or cache (e.g., in high-speed data storage) one or more tiles of high-quality image data corresponding to areas within a VR / AR / MR environment that a user may pass through or perceive from adjacent areas. Preloading the tiles of high-quality image data can enable faster access and transfer of the tiles compared to storing the tiles in other storage devices (e.g., non-volatile memory). After preloading the tiles of high-quality image data, the imaging system can cause a display device to display particular ones of the preloaded tiles (e.g., by sending particular tiles to the display device) based on user input indicating a desired direction in which the user is moving through or facing within the VR / AR / MR environment. The imaging system can then cause the display device to display one or more aspects of the corresponding area of ​​the VR / AR / MR environment based on the received tiles as the user moves toward a desired direction within the area of ​​the VR / AR / MR environment or as the user turns from an adjacent area toward an area of ​​the VR / AR / MR environment that is in a desired direction. For example, the imaging system may cause a display device to display to the user cinematic scenes or images corresponding to aspects of an area in the VR / AR / MR environment.

[0017] As used herein, "high-quality" image or video data refers to 2D image data, 2D video data, 3D image data, or 3D video data, etc., generated via light field technology, point cloud modeling, voxelization, etc. In some embodiments, the resolution of a VR / AR / MR environment including high-quality image data is at least 4K pixels (e.g., 3840 pixels by 2160 pixels, or 4096 pixels by 2160 pixels), including 8K pixels (e.g., 7680 pixels by 4320 pixels). Also, as used herein, a "tile" refers to one or more high-quality image data files that can be used to present aspects of a specific region of the VR / AR / MR environment to a user on a display device after an imaging system transmits the tile to the display device. For example, a VR / AR / MR environment can be divided into a predetermined number of regions, each corresponding to a tile. Each tile can include high-quality image data that a display device can use to display aspects of the corresponding region of the VR / AR / MR environment. Specifically, tiles can be used to display virtual objects in a VR / AR / MR environment, virtual space above a user, virtual space below a user, or modifications of a user, etc. In some embodiments, a tile can have at least 1 terabyte (TB) of data, at least 850 megabytes (MB) of data, at least 750 MB of data, at least 600 MB of data, or at least 500 MB of data, etc.

[0018] In one embodiment, an imaging system can cause a display device to display an urban VR / AR / MR environment through which a user can pass (e.g., move) as part of an interactive ride or experience within an amusement park. The imaging system can render tiles from a database corresponding to a default or predetermined area within the urban VR / AR / MR environment in which the user is located at the start of the interactive ride. The imaging system can also preload one or more tiles from the database corresponding to respective areas (e.g., adjacent or nearby areas) within the urban VR / AR / MR environment through which the user may pass or which the user may perceive from the default area. Areas corresponding to the preloaded tiles can border the default area within the urban VR / AR / MR environment. The user can navigate within such areas after leaving the boundaries of the default area and / or perceive such areas while residing at or near the boundaries of the default area. For example, as a user approaches a virtual or displayed street intersection at the boundary of a default region, the imaging system can render the street intersection, a portion of a first street that lies ahead of the user, a portion of a second street that lies to the left of the user, and / or a portion of a second street that lies to the right of the user based on tiles that correspond to the default region. The system can then preload tiles from a database that correspond to the respective areas past the rendered portions of the first and second streets.

[0019] In response to a user input indicating a desired direction in which the user intends to proceed through the intersection, the imaging system can cause the display device to display preloaded tiles corresponding to an area in the desired direction through the intersection (i.e., a selected area) (e.g., by transmitting preloaded tiles to the display device). As the user proceeds toward and passes through the selected area, the imaging system can cause the display device to present various aspects of the selected area based on the received preloaded tiles of high-quality image data. The imaging system can also preload one or more additional tiles from a database corresponding to respective areas in the urban VR / AR / MR environment that the user may pass through or perceive from the selected area (e.g., areas to which the user can move next). The areas corresponding to the preloaded tiles can border the selected area in the urban VR / AR / MR environment. The user can proceed through such additional areas after leaving the boundary of the selected area and / or perceive such additional areas while residing at or near the boundary of the selected area. In some embodiments, the imaging system can discard previously preloaded tiles corresponding to areas in respective directions in which the user did not decide to proceed.

[0020] The imaging system may then repeat this process until the user completes the interactive ride. That is, the imaging system may repeat the process of preloading from the database one or more tiles of high-quality image data corresponding to each area in the VR / AR / MR environment that the user may pass through and / or perceive from an adjacent area in the VR / AR / MR environment. After preloading the tiles of high-quality image data, the imaging system may cause the display device to display particular ones of the preloaded tiles (e.g., by sending the particular tiles to the display device) based on user input indicating a desired direction in which the user is moving through or facing within the VR / AR / MR environment. The imaging system may cause the display device to display corresponding aspects of an area of ​​the VR / AR / MR environment based on the received tiles as the user moves toward a desired direction within the area of ​​the VR / AR / MR environment and / or as the user turns toward an area of ​​the VR / AR / MR environment in a desired direction from an adjacent area. In this way, the imaging system can process and transmit smaller amounts of high-quality image data multiple times during the interactive ride, rather than processing and loading the entire high-quality image data set (e.g., corresponding to many tiles of high-quality image data) at once. Thus, the techniques described herein continuously reduce and / or optimize the amount of computational resources used by the imaging system to provide the user with a seamless experience of the VR / AR / MR environment based on user input indicating a desired direction of user navigation within the VR / AR / MR environment and / or a desired user viewing direction within the VR / AR / MR environment.

[0021] 1 is a block diagram of an imaging system 100 capable of preloading one or more tiles of high-quality image data that can be used to display particular aspects of one or more respective regions of a VR / AR / MR environment on a user-accessible display device 108, according to embodiments described herein. For example, a control system 102 of the imaging system 100 can retrieve one or more tiles of high-quality image data from a database 104 communicatively coupled to the control system 102 via a network 106 and store these tiles in a preloader 103 of the control system 102. The preloader 103 can be any suitable high-speed data storage capable of preloading or caching tiles of high-quality image data from the database 104. For example, the preloader 103 can include a cache device, such as a computer processing unit (CPU) cache device. In some embodiments, the CPU cache device can include an L1 cache, an L2 cache, or an L3 cache. Preloading tiles of high-quality image data into the preloader 103 can enable faster access and transfer of the tiles compared to storing the tiles in other storage devices (e.g., non-volatile memory). For example, the preloader 103 can have a data transfer rate of at least 10 gigabytes per second (GB / s). In some embodiments, the preloader 103 can have a data transfer rate of at least 25 GB / s, at least 250 GB / s, or at least 1 terabyte / s (TB / s), etc. Additionally, the preloader 103 can have any suitable size that stores an appropriate number of preloaded tiles. In some embodiments, the preloader 103 can have a data storage size of at least 5 TB, at least 2 TB, or at least 1 TB, etc.

[0022] The control system 102 may also include a memory 105 and a processor 107. The processor 107 of the control system 102 may include one or more of any suitable type of computer processor or microprocessor capable of executing computer-executable code, including, but not limited to, one or more field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), programmable logic devices (PLDs), programmable logic arrays (PLAs), etc. In some embodiments, the processor 107 may include multiple processors. The memory 105 may include any suitable article of manufacture that acts as a medium for storing processor-executable code, data, etc. The memory 105 may store non-transitory processor-executable code used by the processor 107 to perform the techniques of this disclosure.

[0023] As described above, each tile of high-quality image data may be generated via any suitable high-quality imaging technique, such as light field techniques, point cloud modeling, or voxelization. After the tiles of high-quality image data are generated, they may be stored in database 104. In some embodiments, the tiles may be indexed in database 104 with a tile identifier, a region identifier associated with a corresponding region of the VR / AR / MR environment, or one or more boundary region identifiers associated with a corresponding boundary region of the VR / AR / MR environment. Control system 102 may send a request to database 104 that includes an indication of (e.g., a pointer to) the tile identifier, region identifier, and / or boundary region identifier, and database 104 may send a response to control system 102 that includes the corresponding tile.

[0024] The display device 108 may be any suitable device for displaying VR / AR / MR content to a user, such as smart glasses, a virtual retinal display, one or more contact lenses, a computer, a mobile device, a head-mounted device, or the like. Optionally, the display device 108 may include one or more sensors 110 capable of acquiring data related to the user and transmitting the data to the control system 102 via the network 106 for analysis. The control system 102 may use the received data related to the user to determine the direction the user is looking (e.g., a particular direction within the user's field of view) or the direction the user intends to move (e.g., the user's desired direction of movement). For example, the sensors 110 may include one or more image sensors capable of acquiring image data or video data related to the user's eyes, the user's head, or the user's limbs, one or more microphones capable of acquiring sound data related to the user, or one or more motion sensors (e.g., speed sensors, position sensors, or accelerometers) capable of acquiring motion data related to the user. In some embodiments, the sensors 110 may alternatively or additionally be attached to the user's body. For example, one or more motion sensors may be placed on the user's hands, wrists, arms, fingers, legs, feet, torso, or any other suitable body part of the user to capture user motion data. Alternatively or additionally, sensors 110 may be placed in the user's physical environment. For example, in an interactive ride environment, sensors 110 may be placed along a predetermined path that the user may physically walk or within a ride vehicle associated with the user.

[0025] After obtaining data related to the user, the sensors 110 can transmit the data related to the user to the control system 102 over the network 106 for analysis to determine different types of user input the user can provide to modify or otherwise control the user's experience of the VR / AR / MR environment. In some embodiments, the control system 102 can determine the user's desired orientation through the VR / AR / MR environment based on an analysis of the data related to the user, or can determine the user's desired direction of view within the VR / AR / MR environment based on an analysis of data received from the display device 108. For example, the control system 102 can determine one or more user characteristics, such as the position of the user (e.g., of the user's eyes, arms, legs, head, or body), the movement of the user (e.g., of the user's eyes, arms, legs, head, or body), or the orientation of the user (e.g., of the user's eyes, arms, legs, head, or body (such as directional tilt, pitch, yaw, or roll)) based on an analysis of the data received from the display device 108. The control system 102 can then determine the user's desired orientation or the user's desired direction in the user's field of view based on the determined user characteristics. In some embodiments, the control system 102 can compare the determined user characteristics to one or more learned or otherwise interpretable user orientations or desired directions in the user's field of view stored in memory accessible to the control system 102. The control system 102 can also determine the user's position, user movement, or user orientation using image and / or pattern recognition methods based on an analysis of data received from the display device 108. For example, the image and / or pattern recognition methods or algorithms can include machine learning, artificial intelligence, deep learning, convolutional neural networks, or the like. The memory accessible to the control system 102 can store an image recognition model, a voice recognition model, or the like. Such models can be trained by inputting sample data (e.g., images) of people, such as users, and indicative of the user's position, user movement, or user orientation within the data.After such models are trained, control system 102 can use one or more of the models to determine a particular user's position, a particular user's movement, or a particular user's orientation within data received from display device 108. Control system 102 can then determine that such determined user characteristics are associated with a user's desired orientation or a desired direction within the user's field of view.

[0026] Additionally or alternatively, the control system 102 may receive user input or commands, such as gesture commands from a user or voice commands from a user, from one or more user input devices 112. For example, the control system 102 may analyze the user input or commands and determine gesture commands or voice commands through image and / or pattern recognition methods or algorithms, including machine learning, artificial intelligence, deep learning, or convolutional neural networks, etc. For example, a memory accessible to the control system 102 may store an image recognition model, a voice recognition model, etc. Such models may be trained by inputting sample data (e.g., images or audio) of people, such as users, and indicating various gesture commands or voice commands within the data. After such models are trained, the control system 102 may use one or more of the models to determine specific gesture commands or specific voice commands in user input received from the user input devices 112.

[0027] The control system 102 can appropriately modify the VR / AR / MR environment perceived by the user based on the determined user characteristics or user commands. For example, the control system 102 can cause the display device 108 to display tiles of high-quality image data from the preloader 103 based on the user's movement within the physical environment after receiving the tiles from the control system 102. The tiles of high-quality image data can correspond to areas of the VR / AR / MR environment that border a current area of ​​the VR / AR / MR environment in which the user is virtually located. As the user approaches the border of the current area in the VR / AR / MR environment, the control system 102 can cause the display device 108 to present aspects of the area that borders the current area based on the received tiles of high-quality image data, providing the user with a continuous experience of the VR / AR / MR environment similar to that experienced in the real world. That is, the control system 102 can cause the display device 108 to gradually display aspects of the border area of ​​the VR / AR / MR environment to the user as the user moves toward the border area of ​​the VR / AR / MR environment.

[0028] Based on the determined user characteristics or user commands, the control system 102 can also preload from the database 104 one or more tiles that the user may pass through (e.g., areas the user can move to next) or that the user may perceive from an area that the user has virtually entered. For example, based on the determined user characteristics or commands, the control system 102 can determine that the user intends to virtually move in a desired direction toward and / or into a particular area within the VR / AR / MR environment. Based on the user's desired direction, the control system 102 can preload one or more tiles that correspond to areas that respectively border the particular area within the VR / AR / MR environment that the user intends to approach and / or enter.

[0029] Control system 102 may also determine, based on determined user characteristics or user commands, to modify the user's field of view (e.g., viewing angle) of the VR / AR / MR environment displayed via display device 108. For example, control system 102 may modify the appearance of high-quality image data displayed to the user via display device 108 based on the user's eye position or eye movement so that the user perceives the VR / AR / MR environment with a field of view similar to the field of view the user would have in the real world. That is, in response to the user changing the viewing angle of an object in the VR / AR / MR environment, control system 102 may modify the appearance of the object in the VR / AR / MR environment to resemble how the object would appear to the user if the viewing angle were similarly changed in the real world, and display the modified appearance on display device 108.

[0030] As shown in the illustrated embodiment, the control system 102 may be communicatively coupled to one or more user input devices 112 associated with a user. For example, the user input devices 112 may include one or more input devices on a joystick, a steering wheel, a touchscreen display, a mobile phone, or any other suitable device for providing user input. In some embodiments, the user input devices 112 may be communicatively coupled to the display device 108. In either case, the control system 102 may receive one or more user commands from the user input devices 112. After receiving the user commands from the user input devices 112, the control system may modify the VR / AR / MR environment, adjust the user's view of the VR / AR / MR environment, or otherwise control the user's experience of the VR / AR / MR environment based on the received user commands. For example, the control system 102 may compare the received user commands to one or more learned or otherwise interpretable user commands stored in memory accessible to the control system 102 and modify the VR / AR / MR environment based on the comparison. In some embodiments, the memory may include memory 105 , a read-only memory (ROM) of control system 102 , or database 104 .

[0031] It should be noted that any suitable network 106 may be employed in the embodiments described herein. For example, the network 106 may include any implemented wired or wireless communication network, such as a local area network (LAN) and a wide area network (WAN). The network 106 may enable wired or wireless communication via any suitable communication protocol, such as Wi-Fi, mobile communication protocols (e.g., 2G, 3G, 4G, 5G, Long Term Evolution (LTE), New Radio (NR)), Bluetooth, and near field communication protocols.

[0032] With the above in mind, Figure 2 is a schematic diagram 200 of an exemplary VR / AR / MR environment that a user may experience in an interactive ride environment, according to embodiments described herein. In the illustrated embodiment, the schematic diagram 200 of the VR / AR / MR environment may be designed as a city, allowing a user to travel between multiple regions of the VR / AR / MR environment (i.e., 202, 204, 206, 208, 210, 212, 214, 218, 220, 222, 226, 228) at designated intersections between adjacent regions (i.e., A, B, C, D, E, F, G, H, I, J, K). The user may start at a default location 201 within a default region 202 within the VR / AR / MR environment at the beginning of the interactive ride. While the VR / AR / MR environment is described as an urban VR / AR / MR environment, it should be understood that such embodiments are intended to be exemplary and non-limiting. In other embodiments, the VR / AR / MR environment may have any other suitable design, such as a maze, castle, forest, or outer space, that can be divided into multiple regions corresponding to tiles of high-quality image data. In any case, intersections within the VR / AR / MR environment can prompt the user to determine the next region to proceed to after the user passes through the intersection. For example, each intersection can be presented to the user within the VR / AR / MR environment as the user approaches the boundary of the region in which they are currently located. As the user approaches an intersection, the user can provide one or more types of user input indicating the desired direction the user will select through the intersection. As described above, user input can include the user's physical movement in the desired direction, a gesture command, a voice command, or the like.

[0033] Before the interactive ride begins or during a suitable period after the interactive ride begins, the control system 102 of the imaging system 100 can cause the display device 108 to display tiles of high-quality image data from the preloader 103 corresponding to a default region 202 within the urban VR / AR / MR environment in which the user is located (e.g., by transmitting the tiles to the display device 108). For example, after the display device 108 receives the tiles of high-quality image data, the control system 102 can cause the display device 108 to display various aspects of the default region 202 of the urban VR / AR / MR environment to the user based on the tiles of high-quality image data. Such aspects of the default region 202 can include buildings, vehicles, streets, sidewalks, storefronts, sky, etc. During this period, the control system 102 can also preload one or more additional tiles of high-quality image data from the database 104 into the preloader 103 corresponding to respective regions 204, 206, 208 within the urban VR / AR / MR environment that the user may enter or perceive from the default region 202. That is, the control system 102 may preload regions 204, 206, 208 that border, are adjacent to, and / or are adjacent to the default region 202 within the urban VR / AR / MR environment.

[0034] In the illustrated embodiment, as the user approaches intersection A at the boundary of default region 202, control system 102 may cause display device 108 to display intersection A, a portion of a first street in a forward direction relative to the user (i.e., leading to region 206), a portion of a second street to the left of the user (i.e., leading to region 204), and a portion of a second street to the right of the user (i.e., leading to region 208) based on tiles of high-quality image data corresponding to default region 202. In some embodiments, as the user approaches the boundary of default region 202 or enters intersection A, control system 102 may cause display device 108 to gradually display portions of regions 204, 206, and 208 to the user based on corresponding tiles of high-quality image data received from control system 102, so as to provide the user with an experience of a continuous VR / AR / MR environment. For example, as the user moves toward the center of intersection A, the control system 102 may cause the display device 108 to display a greater portion of the regions 204, 206, 208 along the first and second streets based on, for example, the user's distance from the center of the intersection or the user's field of view (e.g., the direction the user is facing).

[0035] Additionally, the user can move forward within each region (e.g., as indicated by a single-headed arrow), backward within each region (e.g., as indicated by a single-headed arrow), or both (e.g., as indicated by a double-headed arrow) relative to the user's default position 201 within the VR / AR / MR environment. For example, in the illustrated embodiment, the user can move forward from default position 201 through region 202 to intersection A, but can also move forward from intersection A through region 206 to intersection C and / or backward from intersection C through region 206 to intersection A. In some embodiments, the user may be restricted to one or more directions of movement based on the storyline of the VR / AR / MR environment. Furthermore, while schematic representation 200 of the VR / AR / MR environment is depicted as a grid, it should be understood that such embodiments are intended to be illustrative and non-limiting. For example, at a decision point (e.g., an intersection), the user may have more or fewer than three options for the user's direction of movement through the decision point. In some embodiments, in addition to moving left, right, forward or backward, the user may also move up or down (e.g., a virtual staircase or a virtual ramp).

[0036] 3 is a block diagram 300 of the preloader 103 and display device 108 of the control system 102, according to an embodiment described herein. Referring to the example described in FIG. 2 above, as the user 304 approaches intersection A from the default region 202 of the VR / AR / MR environment 302, the control system 102 can cause the display device 108 to display an aspect of the default region 202 of the VR / AR / MR environment 302 based on tiles 308 of high-quality image data received from the preloader 103 via the control system 102. The control system 102 can also store tiles of high-quality image data (e.g., 310, 312, 314) from the database 104 in the preloader 103 that respectively correspond to regions 204, 206, 208 of the VR / AR / MR environment 302 that the user may pass through or perceive from the default region 202 (e.g., regions to which the user can next move).

[0037] In response to user input indicating a forward direction 306 through intersection A, the control system 102 can transmit a tile 308 of high-quality image data from the preloader 103 to the display device 108. The tile 308 of high-quality image data corresponds to a region 206 of the VR / AR / MR environment through which the user is traveling in a forward direction from intersection A. The control system 102 can also preload one or more additional tiles of high-quality image data (e.g., 306, 308, 310) corresponding to regions 214, 216, 218 (e.g., adjacent regions) of the VR / AR / MR environment through which the user 404 may pass or perceive from region 206. For example, the control system 102 can receive the additional tiles 306, 308, 310 from the database 104 and store the additional tiles 306, 308, 310 in the preloader 103. In some embodiments, the preloader 103 may discard unused tiles 310, 314 corresponding to regions 204, 208 of high-quality image data that remain in the preloader 103 before, during, or after receiving the additional tiles 306, 308, 310. That is, the preloader 103 may discard tiles 310, 314 of high-quality image data that were not sent to the display device 108 before, during, or after receiving the additional tiles 306, 308, 310. In other embodiments, the preloader 103 may retain some tiles of high-quality image data corresponding to regions 204, 208 adjacent to the region 206 to which the user has selected to proceed, and / or may retain some tiles of high-quality image data corresponding to regions 204, 208 based on whether the user can return to these regions 204, 208 from the region 206. In some embodiments, the preloader 103 also stores tiles corresponding to regions that the user frequently visits. For example, tiles corresponding to virtual stores, save points, starting points, etc. may be stored in the preloader 103 in addition to or instead of tiles corresponding to adjacent regions. In this manner, the control system 102 of the imaging system 100 may minimize retrieval of repetitive data from the database 104.

[0038] As the user moves toward the boundary of region 202, control system 102 can cause display device 108 to display one or more sides of region 206 based on tiles 312 of high-quality image data received from preloader 103. In this manner, control system 102 can cause display device 108 to present some sides of region 206 that border region 202 to provide the user with a continuous experience of the VR / AR / MR environment as the user would experience it in the real world. That is, control system 102 can cause display device 108 to gradually present the sides of region 206 to the user as the user moves toward region 206 of the VR / AR / MR environment. In some embodiments, control system 102 can cause display device 108 to display some sides of region 206 based on tiles 308 of high-quality image data until the user moves a threshold distance into region 206. For example, this threshold distance can correspond to a real-world or theoretical distance a user could travel before such sides would normally disappear from view in the real world. In this way, control system 102 can simulate the sensory experience that a user would typically have in the real world. Also, in some embodiments, display device 108 can retain tile 308 of high-quality image data corresponding to region 202 until the user enters a region (e.g., 218, 220, 222, 224, 226, or 228) that is not adjacent to region 202 (e.g., to increase the speed and efficiency of rendering if the user returns to region 202).

[0039] 4 is a block diagram 400 of the preloader 103 of the control system 102 and the display device 108 as a user 404 approaches intersection F from region 218 of the VR / AR / MR environment 402, according to an embodiment described herein. As the user 404 approaches intersection F, the control system 102 can cause the display device 108 to display an aspect of region 218 of the VR / AR / MR environment 402 based on tiles 408 of high-quality image data received from the preloader 103 via the control system 102. The preloader 103 of the control system 102 can also store tiles 410 of high-quality image data corresponding to regions 222 of the VR / AR / MR environment that the user may pass through or perceive from region 218 (e.g., areas to which the user can next move).

[0040] In response to user input indicating a forward direction 406 in which the user is traveling through intersection F, the control system 102 can transmit a tile 410 of high-quality image data from the preloader 103 to the display device 108. The tile 410 of high-quality image data corresponds to a region 222 of the VR / AR / MR environment through which the user is traveling in a forward direction from intersection F. The control system 102 can also preload a tile 412 of high-quality image data corresponding to a region 228 of the VR / AR / MR environment through which the user 404 may pass or perceive from region 222. As the user travels toward the boundary of region 218, the control system 102 can cause the display device 108 to begin displaying one or more sides of region 222 based on the tile 410 of high-quality image data received from the preloader 103. Because the user cannot backtrack within region 222 toward intersection F, the control system 102 can discard any copies of the tiles 408 in the preloader 103 that correspond to region 218.

[0041] In light of the above, FIG. 5 illustrates a flowchart of a method 500, according to embodiments described herein, for preloading one or more tiles of high-quality image data corresponding to areas in a VR / AR / MR environment that a user may pass through or perceive through adjacent areas in the VR / AR / MR environment, and instructing a display device to display the preloaded tiles of high-quality image data showing aspects of the corresponding areas of the VR / AR / MR environment based on user input indicating a desired direction in which the user is expected to move through the VR / AR / MR environment or in which the user is facing within the VR / AR / MR environment. Note that while the following description of method 500 is provided in a particular order, method 500 is not limited to the illustrated order and may be performed in any suitable order. In fact, at least some steps of method 500 may be skipped entirely. Furthermore, although method 500 is described as being performed by control system 102 of imaging system 100, method 500 may be performed by any suitable computing device.

[0042] As described above, a user can be considered to be passing through an area of ​​a VR / AR / MR environment during an interactive ride in an amusement park. Referring now to FIG. 5 , in block 502, the control system 102 of the imaging system 100 can receive data related to the user from one or more input devices 112, one or more sensors 110, or both, as the user approaches an intersection at or near the boundary of the area being passed through. For example, the user input devices 112, the sensors 110, or both, can continuously acquire data related to the user and transmit this data to the control system 102. After receiving the data related to the user from the input devices 112, the sensors 110, or both, the control system 102 can determine in block 504 that the received data corresponds to a user input indicating a desired direction of the user to proceed through the intersection. For example, the data related to the user can include image or video data related to the user's eyes, the user's head, or the user's limbs, sound data related to the user, motion data related to the user, etc. Based on an analysis of the received user-related data, the control system 102 can determine one or more user characteristics, such as the position of the user's eyes, arms, legs, head, or body; the movement of the user's eyes, arms, legs, head, or body; or the orientation of the user's eyes, arms, legs, head, or body (e.g., directional tilt, pitch, yaw, or roll); or a user command, such as a gesture command from the user or a voice command from the user. The control system 102 can then determine a desired direction in which the user intends to proceed through the intersection based on the one or more user characteristics and / or the one or more user commands. For example, the desired direction may be left, right, forward, up, down, etc. In some embodiments, the control system 102 can determine the desired direction via any suitable image recognition method or algorithm, including machine learning, artificial intelligence, deep learning, convolutional neural networks, etc.

[0043] After the control system 102 determines that the received data corresponds to a user input indicating a desired direction of the user navigating through the intersection, in block 506, the control system 102 can transmit tiles of high-quality image data from the preloader 103 to the display device 108 based on the user input indicating the desired direction. For example, the tiles of high-quality image data can correspond to an area in the desired direction through the intersection. As the user navigates toward and through the area in the desired direction, the control system 102 can transmit commands that cause the display device 108 to present various aspects of the area based on the received tiles of high-quality image data. For example, the control system 102 can cause the display device 108 to display buildings, vehicles, streets, sidewalks, storefronts, or the sky using the tiles of high-quality image data. In some embodiments, the control system 102 can cause the display device 108 to display one or more videos or animations corresponding to a storyline associated with the interactive ride.

[0044] At block 508, the control system 102 may preload one or more additional tiles of high-quality image data into the preloader 103 based on the tiles of high-quality image data sent to the display device 108 at block 506. For example, the additional tiles of high-quality image data may each correspond to an area of ​​the VR / AR / MR environment from which the user may travel in a desired direction through an intersection or through the area. In some embodiments, the control system 102 may send a request for the additional tiles to the database 104. The request may include one or more identifiers associated with the tile sent to the display device 108 at block 506, the area corresponding to the tile sent to the display device 108, and / or one or more areas bordering the area corresponding to the tile sent to the display device 108. In response to receiving the request for the additional tiles, the database 104 may send a response to the control system 102 including the additional tiles. The control system 102 may then store the additional tiles received from the database 104 in the preloader 103. In some embodiments, the control system 102 can discard any unused tiles of high quality image data present in the preloader 103 before storing additional tiles in the preloader 103 .

[0045] After preloading the additional tiles into the preloader 103, the control system 102 can repeat the method described above with respect to blocks 502-508. That is, the control system can repeat blocks 502-508 until the user completes the interactive ride. In this manner, the control system 102 of the imaging system 100 can process and transmit smaller amounts of high-quality image data multiple times during the interactive ride, rather than processing and loading the entire high-quality image data set at once. Thus, the techniques described herein continuously reduce and optimize the amount of computational resources used by the imaging system 100 to provide the user with a seamless VR / AR / MR environment experience based on user input indicating the user's desired direction through the VR / AR / MR environment.

[0046] While only certain features of the present disclosure have been illustrated and described herein, many modifications and changes will occur to those skilled in the art and it is, therefore, to be understood that the appended claims are intended to cover all such modifications and changes which comply with the true spirit of the present disclosure.

[0047] The technology shown and claimed herein refers to and applies to tangible objects and specific examples of a practical nature that positively improve the art and are therefore not abstract, intangible, or purely theoretical. Furthermore, where any claim appended at the end of this specification contains one or more elements designated as "means for [performing] ... [function]" or "step for [performing] ... [function]," such elements are to be construed pursuant to 35 U.S.C. 112(f). Conversely, for any claim containing elements designated in any other manner, such elements are not to be construed pursuant to 35 U.S.C. 112(f). [Explanation of symbols]

[0048] 100 Imaging System 102 Control System 103 Preloader 104 (singular and plural) databases 106 Network 105 memory 107 processors 108 Display device 110 (singular or plural) sensors 112 User input device(s)

Claims

1. A system for preloading high quality image data, comprising: one or more processors; a memory accessible by said one or more processors for storing instructions; the instructions, when executed by the one or more processors, cause the one or more processors to: receiving data related to a user from at least one or more input devices or one or more sensors; determining that the received data corresponds to a direction of movement of the user through a virtual reality (VR), augmented reality (AR), or mixed reality (MR) environment; transmitting tiles of high quality image data to a display device based on the direction of movement of the user; sending commands to the display device to display one or more aspects of an area of ​​the VR environment, the AR environment, or the MR environment based on the tiles of the high quality image data; preloading one or more additional tiles of high quality image data into a preloader based on the tiles of high quality image data transmitted to the display device; performing an action including system.

2. the one or more additional tiles of high-quality image data are associated with one or more adjacent regions of the VR environment, the AR environment, or the MR environment that border the region of the VR environment, the AR environment, or the MR environment; The system of claim 1 .

3. the tile of high quality image data and the one or more additional tiles of high quality image data are generated via light field techniques, point cloud modeling, voxelization, or any combination thereof. The system of claim 1 .

4. the data associated with the user is received before the user virtually approaches a boundary of the region of the VR environment, the AR environment, or the MR environment; The system of claim 1 .

5. The operation is receiving additional data related to the user from the one or more input devices, the one or more sensors, or both; determining that the received additional data corresponds to a second direction of movement of the user through the VR environment, the AR environment, or the MR environment; The system of claim 1 , comprising:

6. the operations include transmitting a particular tile of the one or more additional tiles of the high quality image data from the preloader to the display device based on the second direction of movement of the user, and transmitting a second command to the display device based on the particular tile of the one or more additional tiles to display one or more additional aspects of a second region of the VR environment, the AR environment, or the MR environment. The system of claim 5.

7. the operations include discarding one or more tiles of high quality image data that were not transmitted to the display device before preloading the one or more additional tiles of high quality image data. The system of claim 1 .

8. The operation is determining that the received data corresponds to a particular direction within the user's field of view in the region of the VR environment, the AR environment, or the MR environment; transmitting a particular tile of the one or more additional tiles of the high-quality image data from the preloader to the display device based on the particular direction in the field of view of the user, and transmitting a second command to the display device based on the particular tile of the one or more additional tiles of the high-quality image data to display one or more additional sides of a second region that borders the region of the VR environment, the AR environment, or the MR environment in the direction of the field of view; The system of claim 1 , comprising:

9. 1. A method for preloading high quality image data, comprising: receiving, via one or more processors, data relating to a user from at least one or more input devices or one or more sensors; determining, via the one or more processors, that the received data corresponds to a direction of movement of the user through a virtual reality (VR), augmented reality (AR), or mixed reality (MR) environment; transmitting, via the one or more processors, one or more high quality image data files to a display device based on the direction of movement of the user; transmitting, via the one or more processors, a command to the display device to display one or more virtual objects in the VR environment, the AR environment, or the MR environment based on the one or more high-quality image data files transmitted to the display device; via the one or more processors, preloading one or more additional high quality image data files into a preloader based on the direction of movement of the user; A method comprising:

10. the one or more additional high-quality image data files are associated with one or more adjacent regions of the VR environment, the AR environment, or the MR environment that border the region of the VR environment, the AR environment, or the MR environment; 10. The method of claim 9.

11. the one or more high-quality image data files and the one or more additional high-quality image data files are generated via light field techniques, point cloud modeling, voxelization, or a combination thereof; 10. The method of claim 9.

12. the data associated with the user is received before the user virtually approaches a boundary of the region of the VR environment, the AR environment, or the MR environment; 10. The method of claim 9.

13. receiving, via the one or more processors, additional data related to the user from the one or more input devices, the one or more sensors, or both; determining, via the one or more processors, that the received additional data corresponds to a second direction of movement of the user through the VR environment, the AR environment, or the MR environment; 10. The method of claim 9, comprising:

14. transmitting a particular high-quality image data file from the preloader to the display device based on the second direction of movement of the user, and transmitting a second command to the display device based on the particular high-quality image data file from the one or more additional high-quality image data files to display one or more additional aspects of a second region of the VR environment, the AR environment, or the MR environment. The method of claim 13.

15. 1. A non-transitory computer-readable medium comprising instructions for preloading high quality image data, the instructions, when executed by one or more processors, causing the one or more processors to: receiving user-related data from one or more input devices, one or more sensors, or both; determining that the received data corresponds to a particular direction within a field of view of the user in a virtual reality (VR), augmented reality (AR), or mixed reality (MR) environment; transmitting tiles of high quality image data to a display device based on the particular direction within the field of view of the user; sending commands to the display device to display one or more aspects of an area of ​​the VR environment, the AR environment, or the MR environment based on the tiles of the high quality image data; preloading one or more additional tiles of high quality image data into a preloader based on the tiles of high quality image data transmitted to the display device; A non-transitory computer-readable medium for causing a computer to perform operations including:

16. The operation is receiving additional data related to the user from the one or more input devices, the one or more sensors, or both; determining that the received additional data corresponds to a second direction within the field of view of the user in the VR environment, the AR environment, or the MR environment; 16. The non-transitory computer-readable medium of claim 15, comprising:

17. the operations include transmitting a particular tile of the one or more additional tiles of the high quality image data from the preloader to the display device based on the second direction within the field of view of the user, and transmitting a second command to the display device based on the particular tile of the one or more additional tiles to display one or more additional aspects of a second region of the VR environment, the AR environment, or the MR environment.

17. The non-transitory computer-readable medium of claim 16.

18. the one or more additional tiles of high-quality image data are associated with one or more adjacent regions of the VR environment, the AR environment, or the MR environment that border the region of the VR environment, the AR environment, or the MR environment; 16. The non-transitory computer-readable medium of claim 15.

19. the data associated with the user is received before the user virtually approaches a boundary of the region of the VR environment, the AR environment, or the MR environment; 16. The non-transitory computer-readable medium of claim 15.

20. the tile of high quality image data and the one or more additional tiles of high quality image data are generated via light field techniques, point cloud modeling, voxelization, or a combination thereof.

16. The non-transitory computer-readable medium of claim 15.

Images