Artificial intelligence-based image generation

Abstract

Generally described, one or more aspects of the present application relate to capturing and generating viewpoints of any given space. Pixel averaging and camera configurations, including microlens cameras, may be implemented to generate and capture viewpoints of any given space.

Description

ARTIFICIAL INTELLIGENCE-BASED IMAGE GENERATIONRELATED APPLICATIONS

[0001] This application claims priority to U.S. Provisional Application No. 63 / 540,614, entitled “ARTIFICIAL INTELLIGENCE-BASED IMAGE GENERATION” and filed on September 26, 2023, to U.S. Provisional Application No. 63 / 540,610, entitled “SELF- CONTAINED CAMERA STRIP” and filed on September 26, 2023, to U.S. Provisional Application No. 63 / 540,615, entitled “MICROLENS CAMERA” and filed on September 26, 2023, to U.S. Provisional Application No. 63 / 540,618, entitled “HEADSET FOR TRAVELING IN TIME AND SPACE” and filed on September 26, 2023, and to U.S. Provisional Application No. 63 / 540,612, entitled “VIRTUAL NAVIGATION OF A GEOGRAPHIC LOCATION” and filed on September 26, 2023, which are hereby incorporated by reference herein in their entireties. This application is also related to U.S. Provisional Application No. 63 / 327,216, entitled “TRAVELING IN TIME AND SPACE CONTINUUM” and filed on April 4, 2022, U.S. Provisional Application No. 63 / 343,957, entitled “TRAVELING IN TIME AND SPACE CONTINUUM II” and filed on May 19, 2022, U.S. Provisional Application No. 63 / 391,191, entitled “TRAVELING IN TIME AND SPACE CONTINUUM - TASC” and filed on July 21 , 2022, U.S. Provisional Application No. 63 / 421,495, entitled “TRAVELING IN TIME AND SPACE CONTINUUM - TASC” and filed on November 1, 2022, U.S. Provisional Application No. 63 / 493,135, entitled “TRAVELING IN TIME AND SPACE CONTINUUM - TASC” and filed on March 30, 2023, U.S. Patent Application No. 18 / 129,793, entitled “TRAVELING IN TIME AND SPACE CONTINUUM” and filed on March 31, 2023, U.S. Patent Application No. 18 / 129,766, entitled “TRAVELING IN TIME AND SPACE CONTINUUM” and filed on March 31, 2023, and U.S. Patent Application No. 18 / 129,786, entitled “TRAVELING IN TIME AND SPACE CONTINUUM” and filed on March 31, 2023, which are hereby incorporated by reference herein in their entireties.TECHNICAL FIELD

[0002] This disclosure relates to the field of camera systems, devices, software applications, and particularly to systems, methods, and techniques for realistic, immersive audiovisual experiences involving traveling through a virtual representation of a real space.BACKGROUND

[0003] Virtual environments and platforms have been developed to allow for a user to traverse various virtual spaces. For example, mapping tools exist in which various street-level images of a location are captured, the images are associated with a position on a map that corresponds with the location, and users are allowed to view the street-level images when selecting the position on the map.

[0004] However, these existing virtual environments and platforms suffer from technical limitations in image capture, image processing, and the recreation of three-dimensional spaces. For example, it can be difficult for users to navigate between different locations or to view real-time changes at various locations in existing virtual environments and platforms.SUMMARY

[0005] The systems, methods, and devices described herein each have several aspects, no single one of which is solely responsible for its desirable attributes. Without limiting the scope of this disclosure, several non-limiting features will now be discussed briefly.

[0006] In some aspects, the techniques described herein relate to a computer- implemented method for remote viewing, the computer-implemented method including; obtaining a sequence of images of a location, wherein the sequence of images is captured by at least one of a plurality of cameras positioned at one or more positions in the location; generating a virtual space, wherein the virtual space is a virtual representation of the location; determining at least one of a position, a direction of travel, or a speed of travel of a remote user within the virtual representation of the location based on one or more measurements obtained from a sensory input from the remote user; selecting a subset of cameras from the plurality of cameras positioned at one or more positions in the location based on at least one of the position, the direction of travel, or the speed of travel of the user within the virtual representation of the location; and causing a user device located at the remote location of the remote user to display one or more images in the sequence of images captured by the subset of cameras in an order based on the position, the direction of travel, and the speed of travel of the user within the virtual representation of the location.

[0007] The computer implemented method of the preceding paragraph can include any sub-combination of the following features: the computer-implemented method, further including: receiving an indication of an object to track and a first image in the sequence of images capturedby a first camera in the subset of cameras and displayed by the user device; applying image processing to the first image to identify a characteristic of the object; applying image processing to images in the sequence of images other than the first image to identify a second image in the sequence of images that depicts the object with the characteristic; and causing the user device to display the second image following the first image; the computer-implemented method, wherein applying image processing to the first image further includes applying the first image as an input to a trained object detection artificial intelligence model, wherein applying the first image as the input to the trained object detection artificial intelligence model causes the trained object detection artificial intelligence model to output an indication that the object with the characteristic is depicted in the first image; the computer-implemented method, wherein causing a user device to display one or more images in the sequence of images captured by the subset of cameras in an order based on the position, the direction of travel, and the speed of travel of the user within the virtual representation of the location further includes: determining a location of a first camera in the subset of cameras that captured a first image in the sequence of images that is displayed by the user device; determining a distance from the first camera based on at least one of the direction of travel of the user or the speed of travel of the user; determining that a second camera in the subset of cameras is located at a distance from the location of the first camera that matches the determined distance; and causing the user device to display a second image in the sequence of images captured by the second camera subsequent to the user device displaying the first image; the computer-implemented method, wherein the sensory input includes one of a touch input, a haptic input, a gesture input, a wearable input, or a voice input provided to the user device; the computer-implemented method, further including determining at least one of an updated position, an updated direction of travel, or an updated speed of a travel of the user within the virtual representation of the location based on one or more second measurements obtained from the sensory input and generated subsequent to the one or more measurements; the computer-implemented method, wherein each camera in the plurality of cameras is spaced evenly throughout the location; the computer-implemented method, wherein at least some cameras in the plurality of cameras are spaced unevenly throughout the location; the computer-implemented method, wherein the one or more images captured by the subset of cameras are displayed by the user device within a threshold time of the one or more images being captured by the subset of cameras; the computer-implemented method, wherein the one or more images captured by the subset of cameras are displayed by the user device at a ratethat is slower than a rate at which the one or more images are captured by the subset of cameras; the computer-implemented method, wherein the one or more images captured by the subset of cameras are displayed by the user device at a rate that is faster than a rate at which the one or more images are captured by the subset of cameras; the computer implemented method, wherein causing a user device to display one or more images in the sequence of images captured by the subset of cameras in an order based on the position, the direction of travel, and the speed of travel of the user within the virtual representation of the location further includes: determining a location of a first camera in the subset of cameras that captured a first image in the sequence of images that is displayed by the user device; determining a first position from the first camera based on at least one of the direction of travel of the user or the speed of travel of the user; determining that a second camera in the subset of cameras is located at a second position from the location of the first camera, wherein the second position's distance is greater from the position of the first camera than the first position, wherein the second camera is closest in location to the first position; generating a pixel- averaged frame; and causing the user device to display the pixel- averaged frame in the sequence of images subsequent to the user device displaying the first image; the computer implemented method, wherein the step of generating a pixel- averaged frame includes: retrieving one or more picture frames each from the first camera and the second camera; and averaging RGB values of one or more pixels including the one or more picture frames at corresponding relative coordinate positions from the one or more picture frames to generate the pixel- averaged frame; the computer implemented method, wherein averaging RGB values of one or more pixels includes: determining the individual red, green, and blue values of the one or more pixels including the one or more picture frames, determining an average red, green, and blue values for each of the one or more pixels, thereby generating an averaged pixel for each of the one or more pixels and generating the pixel averaged frame by positioning the averaged pixel for each of the one or more pixels; the computer implemented method, wherein there are no intervening cameras located between the positions of the first and the second cameras; the computer implemented method, wherein the pixel averaged frame approximates a view generated by a virtual third camera located in between the positions of the first and second cameras, wherein the virtual third camera is not present between the positions of the first and second cameras, wherein the virtual third camera is located at the first position; the computer implemented method, wherein the pixel averaged frame is image corrected before causing the user device to display the pixel-averaged frame, wherein image correctionincludes one or more of: distortion correction, perspective correction, or angle-distortion correction.

[0008] In some aspects, the techniques described herein relate to a system for remote viewing, the system including: memory configured to store computer-executable instructions; and a hardware processor in communication with the memory, wherein the computer-executable instructions, when executed by the hardware processor, cause the hardware processor to: obtain a sequence of images of a location, wherein the sequence of images is captured by at least one of a plurality of cameras positioned at one or more positions in the location; generate a virtual space, wherein the virtual space is a virtual representation of the location; determine at least one of a position, a direction of travel, or a speed of travel of a remote user within the virtual representation of the location based on one or more measurements obtained from a sensory input; select a subset of cameras from the plurality of cameras positioned at one or more positions in the location based on at least one of the position, the direction of travel, or the speed of travel of the user within the virtual representation of the location; and cause a user device to display one or more images in the sequence of images captured by the subset of cameras in an order based on the position, the direction of travel, and the speed of travel of the user within the virtual representation of the location.

[0009] The system of the preceding paragraph can include any sub-combination of the following features: the system, wherein the computer-executable instructions, when executed, further cause the hardware processor to: obtain an indication of an object to track and a first image in the sequence of images captured by a first camera in the subset of cameras and displayed by the user device; apply image processing to the first image to identify a characteristic of the object; apply image processing to images in the sequence of images other than the first image to identify a second image in the sequence of images that depicts the object with the characteristic; and cause the user device to display the second image following the first image; the system, wherein the computer-executable instructions, when executed, further cause the hardware processor to apply the first image as an input to a trained object detection artificial intelligence model, wherein application of the first image as the input to the trained object detection artificial intelligence model causes the trained object detection artificial intelligence model to output an indication that the object with the characteristic is depicted in the first image; the system, wherein the computerexecutable instructions, when executed, further cause the hardware processor to: determine alocation of a first camera in the subset of cameras that captured a first image in the sequence of images that is displayed by the user device; determine a distance from the first camera based on at least one of the direction of travel of the user or the speed of travel of the user; determine that a second camera in the subset of cameras is located at a distance from the location of the first camera that matches the determined distance; and cause the user device to display a second image in the sequence of images captured by the second camera subsequent to the user device displaying the first image; the system, wherein the sensory input includes one of a touch input, a haptic input, a gesture input, a wearable input, or a voice input provided to the user device; the system, wherein the computer-executable instructions, when executed, further cause the hardware processor to determine at least one of an updated position, an updated direction of travel, or an updated speed of a travel of the user within the virtual representation of the location based on one or more second measurements obtained from the sensory input and generated subsequent to the one or more measurements.

[0010] In some aspects, the techniques described herein relate to a non-transitory, computer-readable medium storing computer-executable instructions for remote viewing, wherein the computer-executable instructions, when executed, cause a computing system to: obtain a sequence of images of a location, wherein the sequence of images is captured by at least one of a plurality of cameras positioned at one or more positions in the location; generate a virtual space, wherein the virtual space is a virtual representation of the location; determine at least one of a position, a direction of travel, or a speed of travel of a remote user within the virtual representation of the location based on one or more measurements obtained from a sensory input; select a subset of cameras from the plurality of cameras positioned at one or more positions in the location based on at least one of the position, the direction of travel, or the speed of travel of the user within the virtual representation of the location; and cause a user device to display one or more images in the sequence of images captured by the subset of cameras in an order based on the position, the direction of travel, and the speed of travel of the user within the virtual representation of the location.

[0011] The non-transitory, computer-readable medium storing computer-executable instructions of the preceding paragraph can include any sub -combination of the following features: the non-transitory, computer-readable medium, wherein the computer-executable instructions, when executed, further cause the computing system to: obtain an indication of an object to trackand a first image in the sequence of images captured by a first camera in the subset of cameras and displayed by the user device; apply image processing to the first image to identify a characteristic of the object; apply image processing to images in the sequence of images other than the first image to identify a second image in the sequence of images that depicts the object with the characteristic; and cause the user device to display the second image following the first image; the computer-executable instructions, when executed, further cause the computing system to: determine a location of a first camera in the subset of cameras that captured a first image in the sequence of images that is displayed by the user device; determine a distance from the first camera based on at least one of the direction of travel of the user or the speed of travel of the user; determine that a second camera in the subset of cameras is located at a distance from the location of the first camera that matches the determined distance; and cause the user device to display a second image in the sequence of images captured by the second camera subsequent to the user device displaying the first image.

[0012] In some aspects, the techniques described herein relate to a system for capturing a scene, the system including: a plurality of cameras, wherein each camera in the plurality of cameras is distributed throughout a location, wherein each camera in the plurality of cameras is configured to capture an image at a predetermined frame per second value; and a image processing system including a hardware processor and in networked communication with the plurality of cameras, the image processing system configured with computer-executable instructions that, when executed by the hardware processor, cause the image processing system to integrate output from the plurality of cameras with a perspective view of a virtual representation of the location.

[0013] The system of the preceding paragraph can include any sub-combination of the following features: the system, wherein the plurality of cameras is evenly spaced throughout the location; the system, wherein the plurality of cameras is irregularly spaced throughout the location; the system, wherein the computer-executable instructions, when executed, further cause the image processing system to: select a subset of cameras from the plurality of cameras; and integrate output from the subset of cameras with the perspective view of the virtual representation of the location; the system, wherein the location includes one of an event hall, an aquarium, a mall, a grocery store, a grocery aisle, a concert hall, an outdoor space, a park, or a conference room; the system, wherein the plurality of cameras includes a microlens camera; the system, wherein the microlens cameraincludes a plurality of lenses coupled together to form one of a sphere, a cylinder, a three dimensional elliptical, or a cone.

[0014] In some aspects, the techniques described herein relate to a computer- implemented method of interacting with a virtual representation of a space, the computer- implemented method including: receiving a first set of user inputs, the first set of user inputs indicating a first point of view (POV) request; generating a first POV based on the first POV request, wherein the first POV is a virtual representation of a scene from a first angle, wherein the virtual representation of the scene includes one or more images captured by a plurality of imaging devices at a time at which the first POV request is received; receiving a second set of user inputs, the second set of user inputs indicating a second POV request; and generating a second POV based on the second POV request, wherein the second POV is an updated version of the virtual representation of the scene from a second angle, wherein the updated version of the virtual representation of the scene includes one or more second images captured by the plurality of imaging devices at a second time at which the second POV request is received.

[0015] The computer implemented method of the preceding paragraph can include any sub-combination of the following features: the computer- implemented method, wherein the first set of user inputs includes an indication of a time scale; the computer-implemented method, wherein the first set of user inputs includes an indication of a direction and a speed at which to move within the virtual representation of the scene; the computer-implemented method, wherein the plurality of imaging devices includes a microlens camera; the computer-implemented method, wherein the microlens camera includes a plurality of lenses coupled together to form one of a sphere, a cylinder, a three dimensional ellipse, or a cone; the computer-implemented method, wherein the first set of user inputs includes an indication of a first direction at which to move within the virtual representation of the scene, wherein the one or more images depict a portion of the scene in a direction of the first direction, wherein the second set of user inputs includes an indication of a second direction at which to move within the virtual representation of the scene, and wherein the one or more second images depict a portion of the scene in a direction of the second direction.

[0016] In some aspects, the techniques described herein relate to a system including: memory storing computer-executable instructions; and a hardware processor in communication with the memory, wherein the computer-executable instructions, when executed by the hardwareprocessor, cause the hardware processor to: generate a virtual representation of a location, wherein the location includes a plurality of image capture devices, wherein the plurality of image capture devices are each configured to record audiovisual content; obtain data indicative of a point of view (POV), wherein the POV is selected via user input; determine, based on the data indicative of the POV, a corresponding viewpoint in the virtual representation of the location; and update the virtual representation of the location with an image captured by an image capture device in the plurality of image capture devices at a current time.

[0017] The system of the preceding paragraph can include any sub-combination of the following features. In some aspects, the techniques described herein relate to a system, wherein the plurality of image capture devices includes a microlens camera; the system, wherein the microlens camera includes a plurality of lenses coupled together to form one of a sphere, a cylinder, a three dimensional ellipse, or a cone; the system, wherein the user input indicates a first direction at which to move within the virtual representation of the location, and wherein the image depicts a portion of the location in a direction of the first direction

[0018] In some aspects, the techniques described herein relate to a system for capturing a scene, the system including: a plurality of cameras, wherein each camera in the plurality of cameras is distributed throughout a location, wherein one or more subsets of cameras are located on a vector, wherein each camera in the plurality of cameras is configured to capture an image at a predetermined frame per second value, wherein the interval distance between adjacent cameras is determined by the speed of travel of the user and the effective frame rate of the system, wherein the one or more subsets of the plurality of cameras is arranged in an array along a line; and an image processing system including a hardware processor and in networked communication with the plurality of cameras, the image processing system configured with computer-executable instructions that, when executed by the hardware processor, cause the image processing system to integrate output from the plurality of cameras with a perspective view of a virtual representation of the location.

[0019] The system of the preceding paragraph can include any sub-combination of the following features: the system, wherein each camera in the plurality of cameras is fixed in position, wherein each camera is attached to a pole, a string, or a wall wherein each camera in the plurality of cameras captures a fixed field of view; the system, wherein each camera in the plurality of cameras is immobile, wherein each camera is attached to a pole, a string, or a wall, wherein thepole, the string, or the wall are immohile, wherein each camera in the plurality of cameras captures a fixed field of view; the system, wherein each camera in the plurality of cameras is attached to one or more poles; the system, wherein the plurality of cameras is mounted on a string; the system, wherein each camera in the plurality of cameras is attached to one or more drones, wherein the one or more drones are configured to fly in formation, wherein the one or more drones are equipped with GPS, wherein the system is configured to maintain the relative position of the one or more drones, wherein the one or more drones are configured to fly within a predefined space; the system, wherein the line wherein the one or more subsets of the plurality of cameras are located is a horizontal or vertical line; the system, wherein the system is configured for use within a retail environment, wherein the retail environment is a grocery, wherein the system is configured to store inventory status in an inventory system, wherein the system is configured to allow a user to observe the retail environment; the system, wherein the system is configured for use within a warehouse environment wherein the system is configured to store inventory status in an inventory system, wherein the system is configured to allow a user to observe the warehouse environment; the system, wherein the system is configured to identify and track items on shelves of the retail environment. ; the system, further including an object tracking system, the object tracking system configured with computer-executable instructions that, when executed by the hardware processor, causes the object tracking system to select an object for tracking; the system, further including a natural language processing system, in networked communication with the object tracking system, the natural language processing system configured with computer-executable instructions that, when executed by the hardware processor, causes the object tracking system to select the object for tracking; the system, wherein the system is configured for use on a vehicle; the system, wherein the system is configured for use on a bus, car, plane, train, or boat; the system, wherein the system is configured to display a user icon, wherein the user icon corresponds to the position of a user viewing the scene; the system, wherein the user icon further represents user input from the user viewing the scene, wherein the user input includes speaking, gesture, body position, or effects selected from a user menu ; the system, further including an object removal system, in networked communication with the object tracking system and the image processing system, the object removal system configured with computer-executable instructions that, when executed by the hardware processor, causes the object tracking system to select one or more objects for tracking and removal, wherein the object removal system is configured to remove one or more tracked objects from the scene, wherein theimage processing system is configured to generate an object corrected output, wherein the one or more tracked objects arc selected from a list including: a person, shopping carts, vehicles, or cars.

[0020] Another aspect of the disclosure provides a computer-implemented method for generating a new image. The computer-implemented method comprises: receiving a first image captured by a first sensor source from a first point of view and a second image captured by a second sensor source from a second point of view, wherein the first sensor source and the second sensor source are positioned within a first distance of each other at a geographic location, and wherein the first image and the second image each depict a first object; generating training data that includes the first image and the second image, wherein the second image is labeled with an indication that the second image is from the second point of view and is the first distance from the first image; training an artificial intelligence model using the training data; and providing a third image captured by a third sensor source from a third point of view and a second distance as an input to the trained artificial intelligence model, wherein providing the third image and the second distance as the input to the trained artificial intelligence model causes the trained artificial intelligence model to output a fourth image that is generated as if the fourth image was captured at a position that is the second distance from a position of the third sensor source.

[0021] The computer-implemented method of the preceding paragraph can include any sub-combination of the following features: where the second image is labeled with the indication that the second image from the second point of view, the indication that the second image is captured from the first distance from the first image, and an indication of one of a number of pixels in the second image, coordinates of the pixels in the second image, an intensity of each pixel in the second image, an angle from which the second image was captured relative to a plane bounded by positions of the first sensor source and the second sensor source, a depth represented by a distance from a position of the second sensor source relative to the plane bounded by the positions of the first sensor source and the second sensor source to a position of the first object, or an ambient light level of the geographic location; where the computer-implemented method further comprises: determining a coordinate of a third position from which the second image was captured, determining a third distance between the third position and a center position on a plane in which the third position is located, determining a percentage that the third distance is of a total distance between the center position on the plane and an edge position on the plane, wherein the edge position on the plane corresponds with a position of a second object captured by a third sensorsource that is at an end of a row of sensor sources that includes the first sensor source and the second sensor source, and determining the angle based on the determined percentage; where providing a third image captured by a third sensor source from a third point of view and a second distance as an input to the trained artificial intelligence model further comprises providing the third image, the second distance, and at least one of an angle, a depth, or an ambient light level; where the first sensor source, the second sensor source, and the third sensor source are aligned in a first plane, and wherein the position that is the second distance from the position of the third sensor source is aligned in the first plane; where the training data comprises a fifth image captured by a fourth sensor source within a second distance of the second sensor source; where the first sensor source comprises a camera; and where the first sensor source comprises a microlens camera.

[0022] Another aspect of the disclosure provides a system for generating a new image. The system comprises memory that stores computer-executable instructions. The system further comprises a processor in communication with the memory, wherein the computer-executable instructions, when executed by the processor, cause the processor to: receive a first image captured by a first sensor source from a first point of view and a second image captured by a second sensor source from a second point of view, wherein the first sensor source and the second sensor source are positioned within a first distance of each other at a geographic location, and wherein the first image and the second image each depict a first object; generate training data that includes the first image and the second image, wherein the second image is labeled with an indication that the second image is from the second point of view and is the first distance from the first image; train an artificial intelligence model using the training data; and provide a third image captured by a third sensor source from a third point of view and a second distance as an input to the trained artificial intelligence model, wherein providing the third image and the second distance as the input to the trained artificial intelligence model causes the trained artificial intelligence model to output a fourth image that is generated as if the fourth image was captured at a position that is the second distance from a position of the third sensor source.

[0023] The system of the preceding paragraph can include any sub-combination of the following features: where the second image is labeled with the indication that the second image from the second point of view, the indication that the second image is captured from the first distance from the first image, and an indication of one of a number of pixels in the second image, coordinates of the pixels in the second image, an intensity of each pixel in the second image, anangle from which the second image was captured relative to a plane bounded by positions of the first sensor source and the second sensor source, a depth represented by a distance from a position of the second sensor source relative to the plane bounded by the positions of the first sensor source and the second sensor source to a position of the first object, or an ambient light level of the geographic location; where the computer-executable instructions, when executed, further cause the processor to: determine a coordinate of a third position from which the second image was captured, determine a third distance between the third position and a center position on a plane in which the third position is located, determine a percentage that the third distance is of a total distance between the center position on the plane and an edge position on the plane, wherein the edge position on the plane corresponds with a position of a second object captured by a third sensor source that is at an end of a row of sensor sources that includes the first sensor source and the second sensor source, and determine the angle based on the determined percentage; where the computer-executable instructions, when executed, further cause the processor to provide the third image, the second distance, and at least one of an angle, a depth, or an ambient light level; where the first sensor source, the second sensor source, and the third sensor source are aligned in a first plane, and wherein the position that is the second distance from the position of the third sensor source is aligned in the first plane; where the training data comprises a fifth image captured by a fourth sensor source within a second distance of the second sensor source; where the first sensor source comprises a camera; and where the first sensor source comprises a microlens camera.

[0024] Another aspect of the disclosure provides a non-transitory, computer-readable medium comprising computer-executable instructions for generating a new image, wherein the computer-executable instructions, when executed by a processor, cause the processor to: receive a first image captured by a first sensor source from a first point of view and a second image captured by a second sensor source from a second point of view, wherein the first sensor source and the second sensor source are positioned within a first distance of each other at a geographic location, and wherein the first image and the second image each depict a first object; generate training data that includes the first image and the second image, wherein the second image is labeled with an indication that the second image is from the second point of view and is the first distance from the first image; train an artificial intelligence model using the training data; and provide a third image captured by a third sensor source from a third point of view and a second distance as an input to the trained artificial intelligence model, wherein providing the third image and the second distanceas the input to the trained artificial intelligence model causes the trained artificial intelligence model to output a fourth image that is generated as if the fourth image was captured at a position that is the second distance from a position of the third sensor source.

[0025] The non-transitory, computer-readable medium of the preceding paragraph can include any sub-combination of the following features: where the second image is labeled with the indication that the second image from the second point of view, the indication that the second image is captured from the first distance from the first image, and an indication of one of a number of pixels in the second image, coordinates of the pixels in the second image, an intensity of each pixel in the second image, an angle from which the second image was captured relative to a plane bounded by positions of the first sensor source and the second sensor source, a depth represented by a distance from a position of the second sensor source relative to the plane bounded by the positions of the first sensor source and the second sensor source to a position of the first object, or an ambient light level of the geographic location; where the computer-executable instructions, when executed, further cause the processor to: determine a coordinate of a third position from which the second image was captured, determine a third distance between the third position and a center position on a plane in which the third position is located, determine a percentage that the third distance is of a total distance between the center position on the plane and an edge position on the plane, wherein the edge position on the plane corresponds with a position of a second object captured by a third sensor source that is at an end of a row of sensor sources that includes the first sensor source and the second sensor source, and determine the angle based on the determined percentage; and where the computer-executable instructions, when executed, further cause the processor to provide the third image, the second distance, and at least one of an angle, a depth, or an ambient light level.

[0026] Another aspect of the disclosure provides a cassette unit comprising a housing, where the housing comprises: a first sensor source; a second sensor source; an electrical connection that couples the first sensor source to the second sensor source; a network interface coupled to the first sensor source and the second sensor source, wherein the network interface is configured to communicate with a device external to the cassette unit; and memory configured to store data captured by the first sensor source and the second sensor source. The cassette unit further comprises an input / output port coupled to the housing.

[0027] The cassette unit of the previous paragraph can include any sub-combination of the following features: where the housing comprises a flexible tube; where the flexible tube comprises a Teflon tube; where the housing comprises a glass casing; where the housing is transparent; where the first sensor source comprises a microlens camera; where the input / output port is configured to couple the cassette unit to a second cassette unit; where the second cassette unit comprises a third sensor source, and wherein the first sensor source is electrically coupled to the third sensor source when the input / output port couples the cassette unit to the second cassette unit; where the cassette unit comprises a circular’ shape, and wherein the electrical connection traverses a diameter of the circular shape; where a location of the first sensor source is 180 degrees opposite of a location of the second sensor source; and where the first sensor source and the second sensor source, when the cassette unit is placed in a geographic location, are configured to capture images of the geographic location.

[0028] Another aspect of the disclosure provides a camera strip comprising a housing, where the housing comprises: a first camera; a second camera; an optical connection that couples the first camera to the second camera; a network interface coupled to the first camera and the second camera, wherein the network interface is configured to communicate with a device external to the camera strip; and memory configured to store data captured by the first camera and the second camera. The camera strip further comprises an input / output port coupled to the housing.

[0029] The camera strip of the preceding paragraph can include any sub-combination of the following features: where the housing comprises a flexible tube; where the flexible tube comprises a Teflon tube; where the housing comprises a glass casing; where the housing is transparent; where the first camera comprises a microlens camera; where the input / output port is configured to couple the camera strip to a second camera strip; where the second camera strip comprises a third camera, and wherein the first camera is coupled to the third camera when the input / output port couples the camera strip to the second camera strip; and where the first camera and the second camera, when the camera strip is placed in a geographic location, are configured to capture images of the geographic location.

[0030] Another aspect of the disclosure provides a camera comprising an optical fiber. The camera further comprises a compound lens coupled to an end of the optical fiber, where the compound lens comprises a first lens and a second lens, where the first lens is positioned at a first angle and is configured to capture a first image of an object in a direction of the first angle, wherethe second lens is positioned at a second angle different than the first angle and is configured to capture a second image of the object in a direction of the second angle, and where the first lens is coupled to the second lens.

[0031] The camera of the preceding paragraph can include any sub-combination of the following features: where the first lens is coupled to the second lens via an optical connection; where the first lens is coupled to the second lens via an electrical connection; where the camera further comprises a network interface coupled to the optical fiber; where the first lens is configured to transmit the first image to the network interface via the optical fiber; where the network interface is coupled to a second end of the optical fiber that is different than the end of the optical fiber; where the compound lens is 100 micrometers wide; where the compound lens is configured to focus on an object at least a distance of 3 millimeters away from a position of the compound lens; where the first lens has a first size, and wherein the second lens has a second size that is different than the first size; and where the first lens has a first focus length, and wherein the second lens has a second focus length that is different than the first focus length.

[0032] Another aspect of the disclosure provides a camera comprising an optical fiber. The camera further comprises a first lens coupled to a first end of the optical fiber. The camera further comprises a second lens coupled to the first end of the optical fiber, where the first lens is positioned at a first angle and is configured to capture a first image of an object in a direction of the first angle, where the second lens is positioned at a second angle different than the first angle and is configured to capture a second image of the object in a direction of the second angle, and where the first lens is coupled to the second lens.

[0033] The camera of the preceding paragraph can include any sub-combination of the following features: where the first lens is coupled to the second lens via an optical connection; where the first lens is coupled to the second lens via an electrical connection; where the camera further comprises a network interface coupled to the optical fiber; where the first lens is configured to transmit the first image to the network interface via the optical fiber; where the network interface is coupled to a second end of the optical fiber that is different than the first end of the optical fiber; where the compound lens is 100 micrometers wide; where the compound lens is configured to focus on an object at least a distance of 3 millimeters away from a position of the compound lens; where the first lens has a first size, and wherein the second lens has a second size that is differentthan the first size; and where the first lens has a first focus length, and wherein the second lens has a second focus length that is different than the first focus length.

[0034] Another aspect of the disclosure provides a headset comprising a screen that produces a first image. The headset further comprises a magnifying device positioned between the screen and a wearer of the headset, wherein the magnifying device is configured to magnify the first image produced by the screen. The headset further comprises a gyroscope that detects a direction of movement of the wearer of the headset. The headset further comprises a network interface. The headset further comprises a processor configured with computer-executable instructions, wherein the computer-executable instructions, when executed by the processor, cause the processor to: determine that the wearer of the headset moved in a first direction based on a measurement captured by the gyroscope; cause the network interface to transmit an indication of the first direction to a location viewing system; and cause the screen to display a second image provided by the location viewing system in place of the first image.

[0035] The headset of the preceding paragraph can include any sub-combination of the following features: where the headset further comprises a second screen and a second magnifying device; where the screen and the magnifying device are positioned in front of a left eye of the wearer, and wherein the second screen and the second magnifying device are positioned in front of a right eye of the wearer; where the location viewing system obtains the second image captured by a sensor source in a direction that corresponds with the first direction; where the gyroscope is configured to capture the measurement in response to a movement of the wearer of the headset; where the headset further comprises a speaker that is configured to output audio captured at a location captured by the first image; where the first image is captured at a first ratio, and wherein the screen depicts the first image at a second ratio that is wider than the first ratio; where the first image is captured at a first zoom level, and wherein the screen depicts the first image at a second zoom level that is more zoomed in than the first zoom level; where the screen is curved at a first angle; and where the headset comprises a virtual reality headset.

[0036] Another aspect of the disclosure provides a headset comprising a projector that projects a first image. The headset further comprises a magnifying device positioned between the projector and a wearer of the headset, wherein the projector is configured to project the first image onto a first end of the magnifying device, and wherein the magnifying device is configured to magnify the first image projected by the projector. The headset further comprises a gyroscope thatdetects a direction of movement of the wearer of the headset. The headset further comprises a network interface. The headset further comprises a processor configured with computer- executable instructions, wherein the computer-executable instructions, when executed by the processor, cause the processor to: determine that the wearer of the headset moved in a first direction based on a measurement captured by the gyroscope; cause the network interface to transmit an indication of the first direction to a location viewing system; and cause the projector to project a second image provided by the location viewing system in place of the first image onto the first end of the magnifying device.

[0037] The headset of the preceding paragraph can include any sub-combination of the following features: where the headset further comprises a second projector and a second magnifying device; where the projector and the magnifying device are positioned in front of a left eye of the wearer, and wherein the second projector and the second magnifying device are positioned in front of a right eye of the wearer; where the location viewing system obtains the second image captured by a sensor source in a direction that corresponds with the first direction; where the gyroscope is configured to capture the measurement in response to a movement of the wearer of the headset; where the headset further comprises a speaker that is configured to output audio captured at a location captured by the first image; where the first image is captured at a first ratio, and wherein the projector projects the first image at a second ratio that is wider than the first ratio; where the first image is captured at a first zoom level, and wherein the projector projects the first image at a second zoom level that is more zoomed in than the first zoom level; where an eye of the wearer of the headset is positioned in front of a second end of the magnifying device that is different than the first end of the magnifying device; and where the headset comprises a virtual reality headset.

[0038] Another aspect of the disclosure provides a computer-implemented method for enabling a user to virtually navigate a geographic location. The computer-implemented method comprising: receiving an indication of a selection of the geographic location; causing a user device to display a first image of a first position at the geographic location, wherein the first image comprises an icon that represents a second user that is virtually navigating the geographic location, and wherein the icon is positioned in the first image at a location that represents a virtual location of the second user; receiving an indication of a selection of a sub-geographic location within the geographic location; receiving an indication of a selection to move in the geographic location withthe second user; and causing the user device and a second user device operated by the second user to display one or more second images captured between the first position and a second position of the sub-geographic location.

[0039] The computer-implemented method of the preceding paragraph can include any sub-combination of the following features: where the computer-implemented method further comprises: causing the user device to display a third image captured at the second position of the sub-geographic location, wherein the user device further displays a list of items present at the subgeographic location, receiving a selection of a first item in the list of items, and causing the user device to display a fourth image of the first item; where the fourth image comprises a stored image of the first item; where the fourth image comprises an image of an aisle in the sub-geographic location where the first item is located; where the computer-implemented method further comprises: receiving an indication of a selection of a request to converse with an individual located at the sub -geographic location, and causing an utterance spoken by the user to be output at the subgeographic location; where the computer-implemented method further comprises causing the user device to output a second utterance spoken by the individual located at the sub-geographic location; where the one or more second images depict the icon that represents the second user at one or more second locations that represent the virtual location of the second user as the second user navigates from the first position to the second position; where the icon comprises an avatar of the second user; where the user is a lead role user, and wherein the lead role user navigates for the second user within the geographic location; where the user device further displays a map of the geographic location; where the geographic location comprises one of a store or a mall; and where the geographic location comprises a mall, and wherein the sub-geographic location comprises a store in the mall.

[0040] Another aspect of the disclosure provides a system for enabling a user to virtually navigate a geographic location. The system comprises memory that stores computerexecutable instructions. The system further comprises a processor in communication with the memory, wherein the computer-executable instructions, when executed by the processor, cause the processor to: receive an indication of a selection of the geographic location; cause a user device to display a first image of a first position at the geographic location, wherein the first image comprises an icon that represents a second user that is virtually navigating the geographic location, and wherein the icon is positioned in the first image at a location that represents a virtual locationof the second user; receive an indication of a selection of a sub-geographic location within the geographic location; receive an indication of a selection to move in the geographic location with the second user; and cause the user device and a second user device operated by the second user to display one or more second images captured between the first position and a second position of the sub-geographic location.[ CX)41 1 The system of the preceding paragraph can include any sub-combination of the following features: where the computer-executable instructions, when executed, further cause the processor to: cause the user device to display a third image captured at the second position of the sub-geographic location, wherein the user device further displays a list of items present at the subgeographic location, receive a selection of a first item in the list of items, and cause the user device to display a fourth image of the first item; where the computer-executable instructions, when executed, further cause the processor to: receive an indication of a selection of a request to converse with an individual located at the sub-geographic location, and cause an utterance spoken by the user to be output at the sub-geographic location; where the computer-executable instructions, when executed, further cause the processor to cause the user device to output a second utterance spoken by the individual located at the sub-geographic location; where the one or more second images depict the icon that represents the second user at one or more second locations that represent the virtual location of the second user as the second user navigates from the first position to the second position; where the user device further displays a map of the geographic location; and where the geographic location comprises a mall, and wherein the sub-geographic location comprises a store in the mall.

[0042] Another aspect of the disclosure provides a non-transitory, computer-readable medium comprising computer-executable instructions for enabling a user to virtually navigate a geographic location, wherein the computer-executable instructions, when executed by a processor, cause the processor to: receive an indication of a selection of the geographic location; cause a user device to display a first image of a first position at the geographic location, wherein the first image comprises an icon that represents a second user that is virtually navigating the geographic location, and wherein the icon is positioned in the first image at a location that represents a virtual location of the second user; receive an indication of a selection of a sub-geographic location within the geographic location; receive an indication of a selection to move in the geographic location with the second user; and cause the user device and a second user device operated by the second userto display one or more second images captured between the first position and a second position of the sub-geographic location.BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Throughout the drawings, reference numbers may be re-used to indicate correspondence between referenced elements. The drawings are provided to illustrate example embodiments described herein and are not intended to limit the scope of the disclosure.

[0044] FIG. 1 is a block diagram of an illustrative operating environment in which a location viewing system uses data from multiple sources to visualize a remote location.

[0045] FIG. 2 illustrates an example diagram that visually depicts operations that may be performed by a location viewing system to visualize a remote location.

[0046] FIG. 3 illustrates an example diagram that visually depicts operations that may be performed by a location viewing system to visualize a remote location.

[0047] FIG. 4 is a flow diagram illustrating the operations performed by the components of the operating environment of FIG. 1 to visualize a remote location.

[0048] FIG. 5 is a flow diagram illustrating the operations performed by the components of the operating environment of FIG. 1 to visualize a remote location.

[0049] FIG. 6 is a flow diagram illustrating the operations performed by the components of the operating environment of FIG. 1 to visualize a remote location.

[0050] FIG. 7 is schematic illustration of an embodiment of the system presented herein.

[0051] FIG. 8 is a block diagram depicting an embodiment(s) of a computer hardware system configured to run software for implementing one or more embodiments of systems, devices, and methods for visualizing remote locations.

[0052] FIGS. 9-13 are example illustrations of an embodiment of the system presented herein.

[0053] FIG. 14A depicts a microlens camera with lenses arranged spherically.

[0054] FIG. 14B depicts a microlens camera with lenses arranged elliptically.

[0055] FIG. 15A depicts a microlens camera with lenses arranged conically.

[0056] FIG. 15B depicts a microlens camera with lenses arranged cylindrically.

[0057] FIG. 16A depicts a frame with pixels.

[0058] FIG. 16B depicts a frame with pixels.

[0059] FIG. 16C depicts a pixel- averaged frame with pixels.

[0060] FIG. 16D is a flow diagram illustrating the operations performed when pixelaveraging.

[0061] FIG. 16E is a flow diagram illustrating the operations performed when pixelaveraging using at least one prior averaged pixel frame.

[0062] FIG. 16F depicts an example arrangement of cameras, and potential viewpoints which can be generated according to an embodiment of the pixel-averaging system.

[0063] FIG. 17 is a flow diagram illustrating the operations performed in inputting object tracking training data.

[0064] FIG. 18 depicts example illustrations of an embodiment of the system presented herein.

[0065] FIG. 19 illustrates an example strip or cassette unit.

[0066] FIG. 20 illustrates a detailed view of the processor of the strip.DETAILED DESCRIPTIONIntroduction

[0067] As discussed above, it can be difficult for users to navigate between different locations or to view real-time changes at various locations in existing virtual environments and platforms. For example, existing virtual environments and platforms often capture images for display in the virtual environment using a moving vehicle. If a user wishes to navigate from point A to point B, the images displayed to the user correspond to images captured by the moving vehicle as the moving vehicle traveled from point A to point B (or from point B to point A). As a result, any obstructions (e.g., shadows, blocking of view, etc.) caused by the moving vehicle or another object that may be in motion at that location (e.g., another vehicle, a pedestrian, an animal, etc.) in viewing a particular location may be present each step of the way as the user navigates from point A to point B. In addition, the position from which the moving vehicle can capture an image is limited (e.g., one lane of a street), which therefore limits the view presented to the user as the user navigates from point A to point B. As the user navigates from point A to point B, the user may be interested in viewing street signs, the view from a particular lane, the view at a particular height,and / or the like, but the limitations in where a moving vehicle can capture an image may prevent the user from being able to view these points of interest.

[0068] As another example, existing virtual environments and platforms often display images to users that were captured at some time in the past. While some locations may not change often, other locations may change on a monthly, weekly, daily, etc. basis (e.g., construction sites, event sites, schools, parking lots, etc.). Because existing virtual environments and platforms capture and display historical images to users, users are unable to understand how a location may look or appear in real-time. In fact, because existing virtual environments and platforms are often limited to displaying historical images, it may not be possible for existing virtual environments and platforms to present to users other real-time information, including audiovisual data, smells, or other sensory inputs (e.g., real-time audio at a location (e.g., where real-time audio may include audio captured within a few seconds of a time at which the user hears the audio), real-time smells at a location (e.g., where real-time smells may include smells presented to a user that match detected smells at a location within a few seconds of when the smells are presented to the user), etc.).

[0069] To that end, systems, methods, and techniques relating to accurately capturing a space (which may include a specific space, locale, or place), including relevant three-dimensional metadata, may allow for more faithful and accurate reproductions of the space or locale, and may allow users to have a more realistic, immersive audiovisual experience while traveling through a virtual environment that represents the space or locale. In some embodiments, the system, methods, and techniques described herein allow for real-time or near real-time rendering of a real space. In some embodiments, the system, methods, and techniques described herein allow for a faithful reproduction (e.g., a realistic reproduction, a reproduction based on actual images, etc.) of a previously recorded real space (e.g., an historical version of the real space) as well. Thus, the present system as described comprises systems of cameras, lenses, user-input devices, picture / audio / video / haptic presentation devices, and a software system to process input data and provide appropriate output to simulate a space to users using the picture / audio / video / haptic presentation devices. Thus, the system, methods, and techniques disclosed herein resolve the technical deficiencies of existing virtual environments and platforms discussed above, and provides a solution for enabling a user to be present and move around in a real space in anydirection and speed, to visualize the real space and be present virtually, and interact with objects and people with haptic feedback.

[0070] As described herein, systems comprising a series of cameras and lenses allow for frame capturing and reproduction of viewpoints comprising a space, the cameras and lenses configured to capture frames at particular FPS speeds, and the cameras and lenses located at varying distances. To a user accessing the system herein, will be able to travel along a virtual representation of the real space, interact with, and track moving objects. Moreover, a user accessing the system will be able to interact with other remote users accessing the same space, and express motion, emotion, and other contextual gestures to the other remote users using an embodiment of the system described herein. The movement of a user traversing the virtual representation of the real space may be limited by the camera series set up within the real space, though methods described herein allow for pixel averaged approximations of viewpoints where cameras are not present, but alternate cameras are present, with frames from alternate cameras used to generate intermediate viewpoints within a space.

[0071] The present disclosure generally relates to a system for realistic, immersive audiovisual experiences, where a user is able to travel and observe virtually a rendered space, the rendered space created by capturing in real-time (e.g., capturing within a few nanoseconds, microseconds, milliseconds, etc. of the captured subject matter being presented to a user) or near real-time (e.g., capturing within a minute, 10 minutes, 1 hour, etc. of the captured subject matter being presented to a user) an existing space using a system of cameras and / or sensors. Particularly, the user may travel virtually through a real space in a flow of motion in that space (e.g. a mall, store, room, theater, performance hall, conference center, or street) in real-time, all from a remote location. The technology described herein uses a system of cameras, sensors, and / or video capture devices positioned at set time and / or space intervals to cover a given space. Image capture of a space may occur at an appropriate frequency of frames per second (e.g., 24 frames per second (fps), 30 fps, 60 fps, 120 fps, 240 fps, etc.) to enable a reproduction of an uninterrupted audiovisual experience for the user.

[0072] As used herein, a virtual representation or reproduction of a “real space” or a “rendered space,” a “reproduced space,” or a “simulated space” is a representation of a physical location that has been constructed by a system using one or more images or other audiovisual data captured at the physical location and that is viewable by a user and / or can be sensed by a user atany location (e.g., at the physical location, at a location remote from the physical location, etc.). The physical location may be a street, a street corner, a building, a structure, a town, a city, a zip code, a county, a river, a lake, an ocean, and / or any other place that can be defined by a boundary (e.g., a parcel boundary, a geographic boundary, a set of coordinates, one or more walls, etc.).

[0073] For example, while wearing a headset that displays images (e.g., a virtual reality headset, an augmented reality headset, etc.) or viewing a screen that displays images, the user may have the ability to navigate virtually to view a physical location at the same speed as, at a faster speed than, or at a slower speed than a moving object present at the physical location; to view a physical location while remaining at the same position; to view a physical location while moving in a direction opposite from a direction in which an object at the physical location is moving; to view a physical location while moving in any direction; to view a physical location at ground level or at one or many altitudes above ground level; to view a physical location as the physical location existed at a historical time, and / or the like. The instructions associated with the remote user for informing the system as to how the user wants to move or navigate within the virtual space can be obtained from the user in the form of data input from a keyboard, a touchscreen, a mouse, a microphone that detects voice commands, one or more sensors (e.g., accelerometers, gyroscopes, piezoelectric devices, motion sensors, smart watches, etc.) connected to an appendage or clothing of the user (e.g., the hand of the user, the foot of the user, a glove worn by the user, a face of the user, a head of the user, eye glasses worn by the user, a jacket worn by the user, etc.), one or more cameras that detect user gestures, and / or the like.

[0074] Use cases for the system described herein can include a user watching or tracking a moving object on a street, a user virtually visiting a shopping mall or store, a user virtually experiencing an underwater environment, a user virtually visiting an amusement park, a user virtually experiencing airspace, and / or the like. The system described herein can be implemented for use in a variety of contexts, including gaming, teleconferencing, virtual reality, augmented reality, and / or the like. Optionally, the system described herein can capture 360 degree images using 360 degree spatial cameras and / or audio systems, thereby creating a realistic immersive experience for a remote user that allows the user to experience a location as if the user is physically present at and navigating the location in real-time. In some embodiments, the system can present a contemporaneous (e.g., real-time) or near contemporaneous (e.g., near- real-time)representation of a real space to a user. In other embodiments, the system can present a recorded, archived representation of a real space to the user.

[0075] The system described herein can also include object tracking features, which track objects or subjects through time and space. For example, object tracking software can be implemented by a location viewing system described herein to identify specific stationary or moving objects (e.g., a vehicle, a person, an animal, a retail item, or any other type of object) in one or more images captured by one or more cameras. Object tracking can further approximate movement of tracked objects when objects leave a particular field of view of a camera, or are otherwise occluded in a scene. Therefore, an object traveling with a consistent direction and / or velocity can be accurately predicted to re-emerge at a particular location. In some embodiments of the present system, the movement of an object being followed by a user can be captured using a series of cameras instead of a single camera, where the series of cameras have differing fields of view. A tracked object can therefore be continuously tracked by a series of cameras in a real space where suitable sensors are present. In some embodiments, tracked objects may be identified by the system, and a user accessing the system may travel from their initial starting point to a second point where the user may view the tracked object. Thus, in some embodiments, appropriate cameras and lenses are selected to follow, or navigate a user to, a tracked object.

[0076] The aforementioned advantages, among others, are addressed in at least some embodiments by the disclosed methods, systems, and techniques for realistic, immersive audiovisual experiences. More specifically, the features disclosed herein address deficiencies that exist in existing virtual environments and platforms by allowing for the rendering of a three- dimensional environment or space designed such that users can navigate virtually one or more locations in a realistic, intuitive manner while viewing and / or sensing (e.g., hearing, smelling, etc.) the location in real-time as if the user is physically present at the location at a current time.

[0077] The present method and systems allow for obtaining 360 degree pictures and / or videos and utilizing techniques that stitch multiple images together to provide a realistic visualization of a space. In some embodiments, such a system can be used to recreate, for example, a conference room in which people from different remote locations can participate by using 3D emojis or icons of respective participants in real-time, enabling an experience of being in the same room, participating in a meeting, and / or collaborating in work together, and wherein the emojis or icons were previously assigned to identify individual users.

[0078] In some embodiments, the systems, methods, and techniques described herein also enable a user the freedom to choose one’s point of view from the surroundings, to aid in remote inspections of various types of spaces. For example, manufacturing or construction sites, day care centers and schools, and / or other places where a user may navigate through a space by choosing their own perspective can be represented in the virtual environment, enabling an interactive and immersive remote site experience as if the user were personally there. The system described herein may enable a user to conduct various types of video meetings, such as video meetings that include virtual reality (VR), augmented reality (AR), spatial audio, and / or the like.

[0079] The ability to recreate, approximate, or otherwise incorporate three- dimensional metadata, including information from one or more angles captured using one or more imaging devices, therefore allows a user to effectively tour a space, such as a street, mall, or shop, in real-time, move around the space in any direction as the user desires, and all from a location remote from the space being toured. Such a system may enable a remote user to experience traveling in a space (e.g., a road or a mall), as if the user is actually moving with the flow of the cars, walking on the sidewalk, walking adjacent to other individuals at the mall, and / or the like at the space being experienced.

[0080] These and other aspects of the disclosure will now be described with regard to certain examples and embodiments, which are intended to illustrate but not limit the disclosure. Although the examples and embodiments described herein will focus, for the purpose of illustration, on specific algorithms and data parsing, one of skill in the art will appreciate the examples are illustrative only, and are not intended to be limiting.

[0081] The foregoing aspects and many of the attendant advantages of this disclosure will become more readily appreciated as the same become better understood by reference to the following detailed description, when taken in conjunction with the accompanying drawings.Time and Space System

[0082] FIG. 1 is a block diagram of an illustrative operating environment 100 in which a location viewing system 120 uses sensor data and / or artificial intelligence to render and present a video stream, sequence of images, or other sensory output to user devices 102. The location viewing system 120 further includes a virtual render generator 121, sensor determination controller 123, object tracker 126, position parser 124, a sensor data store 125, and an image generator 127.In some embodiments, the location viewing system 1 0 further includes a virtual render and object tracking trainer 122. The location viewing system 120 is configured to communicate with network 110 to provide sensor data by a plurality of sensor sources 130. Moreover, user devices 102 are configured to request and receive output from the location viewing system 120. As can be appreciated herein, output sent to user devices 102 can include, but are not limited to, video, images, audio, tactile, atmospheric, ambient, and other outputs, which can be pulled from sensor sources 130, sensor data store 125, or preset by the location viewing system 120.

[0083] The location viewing system 120 can be a computing system configured to collect and generate a video stream, a sequence of one or more images, and / or another type of sensory output (e.g., audio, smell, touch, vibration, atmospheric readings, etc.) to be displayed or outputted on any number of user devices 102. As described herein, generation of the video stream, sequence of image(s), or other type of sensory output can include collecting sensor data from a plurality of sensor sources 130, object tracking, and / or using machine learning techniques mediated in part by the virtual render and object tracking trainer 122. For example, when using machine learning techniques, the virtual render and object tracking trainer 122 can obtain sensor data, generate or train a model using the sensor data, apply a version of the sensor data as an input to a trained artificial intelligence model (e.g., a virtual render machine learning model), and correct or supplement a resulting video stream, sequence of image(s), or other type of sensory output to be displayed or outputted on any number of user devices 102 based on the output of the trained artificial intelligence model. In some embodiments, the location viewing system 120 can receive from a user device initial position data, desired movement data, and / or object tracking data, and using the sensor determination controller 123, position parser 124, and / or object tracker 126, generate a video stream, sequence of image(s), or other type of sensory output for output with virtual render generator 121.

[0084] The location viewing system 120 may be a single computing device, or it may include multiple distinct computing devices, such as computer servers, logically or physically grouped together to collectively operate as a server system. The components of the location viewing system 120 can each be implemented in application- specific hardware (e.g., a server computing device with one or more ASICs) such that no software is necessary, or as a combination of hardware and software. In addition, the modules and components of the location viewing system can be combined on one server computing device or separated individually or into groupson several server computing devices. Tn some embodiments, the location viewing system 120 may include additional or fewer components than illustrated in FIG. 1.

[0085] The location viewing system 120 may include various modules, components, data stores, and / or the like to provide the image, video, and / or other sensory output generation functionality described herein. For example, the location viewing system 120 may include a virtual render generator 121, sensor determination controller 123, position parser 124, object tracker 126, sensor data store 125, and optionally a virtual render and object tracking trainer 122. The following description of the location viewing system 120 is described herein with respect to outputting visual content, but this is not meant to be limiting. For example, the location viewing system 120 can output other types of sensory data (e.g., audio, smell, touch, vibration, atmospheric readings, etc.) measured by the sensor source(s) 130 at locations corresponding to one or more positions that the user has selected or is viewing.

[0086] The virtual render generator 121 can generate a video stream and / or sequence of image(s) for output to one or more user devices 102 using sensor data from sensor sources 130 or sensor data store 125. Sensor data can include various types of output or measurements, such as video, images, audio, atmospheric (e.g., temperature, humidity, pressure, gravity, etc.), and / or other sensor data that can be captured. The video stream and / or sequence of image(s) generated by virtual render generator 121 corresponds to the spatial orientation, direction, speed of travel, and / or object tracking requested by user devices 102, and is generated from a plurality of sensor sources 130 placed throughout a real space. To generate the video stream and / or sequence of image(s) sent to user devices 102, the virtual render generator 121 first receives a request for a video stream, an image, or a sequence of images from user devices 102, the request comprising an initial position within a space (e.g., latitudinal and longitudinal coordinates, a street address, a name of a location, an identifier corresponding to a physical location, an identification of a sensor at a particular location, etc.), and optionally, a desired orientation, direction of travel, rate of travel, an object to track, and / or other inputs corresponding to movement or other interaction within a defined space.

[0087] After the location viewing system 120 receives a request for the video stream, the image, or a sequence of images, the position parser 124 identifies the initial position within the space identified in the request, and optionally, a direction and rate of travel identified in the request. From the initial position as identified by position parser 124, the sensor determination controller 123 selects one or more sensor sources 130 that have a location corresponding to the initial positionor that capture one or more measurements (e.g., sensor measurements, such as images, temperature readings, humidity readings, pressure readings, etc.) at a location corresponding to the initial position, and optionally, one or more other inputs corresponding to movement or other interaction within a defined space. The sensor determination controller 123 can then store the sensor data collected from the selected sensor sources 130 in sensor data store 125. Alternatively, the sensor determination controller 123 may store sensor data measured or captured by the sensor source(s) 130 in the sensor data store 125 as the sensor data is measured or captured, regardless of whether a request has been received to view a video stream, an image, or a sequence of images. The virtual render generator 121 can then retrieve the sensor data from the sensor data store 125, and analyze and / or splice the sensor data into a resulting video or image output. Optionally, multiple sensor data outputs are analyzed by virtual render generator 121, and the resulting video or image output is comprised of a composite of multiple sensor data outputs as determined by the location viewing system 120. In one embodiment, the virtual render generator 121 and / or the virtual render and object tracking trainer 122 may use a trained artificial intelligence model (e.g., a trained machine learning model) to correct, alter, or otherwise modify the resulting video or image output. For example, the virtual render and object tracking trainer 122 can train the artificial intelligence model using training data, such as the training data described herein. The virtual render generator 121 and / or the virtual render and object tracking trainer 122 can apply the sensor data as an input to the trained artificial intelligence model, compare the resulting video output generated by the virtual render generator 121 to an output of the trained artificial intelligence model, and render changes to the resulting video or image output based on the comparison (e.g., modify the resulting video or image output to replace one or more frames or images included in the resulting video or image output with one or more frames or images output by the trained artificial intelligence model) to create a corrected video or image output. The virtual render generator 121 and / or the virtual render and object tracking trainer 122 can then transmit either the resulting video or image output or the corrected video or image output (e.g., when using artificial intelligence) to a user device 102. As can be appreciated, the sensor data from selected sensor sources 130 can include values captured or measured in real-time, in near real-time, or from any time (e.g., where such sensor data may be stored in the sensor data store 125 if the values captured or measured are historical (e.g., not realtime or near real-time values)).

[0088] The object tracker 126 can be configured to identify one or more objects in one or more images captured by sensor sources 130 (c.g., cameras). For example, the request from a user device 102 to view a video stream or sequence of image(s) corresponding to an initial position may include an identification of an object to track. The identification can be in the form of a textual description of the object, a selection of an object from a menu of objects displayed by the user device 102, an identification of an object via a user gesture, an identification of an object via a voice command, an identification of an object via a button selection, and / or the like. The identification in the request can be forwarded by the location viewing system 120 to the object tracker 126. As another example, after the user device 102 requests a video stream or sequence of image(s) corresponding to an initial position and the location viewing system 120 provides the requested video stream or sequence of image(s), the user may identify an object in the displayed video or image. The user can provide an input to the user device 102 (e.g., a gesture input, a button selection, a menu item selection, an annotation drawn on a screen, a touch input, etc.) to select the identified object. The image or frame of the video in which the selection is made can be transmitted by the user device 102 to the object tracker 126. Alternatively, the virtual render generator 121 can provide the frame or image to the object tracker 126 based on an indication of the selection received by the virtual render generator 121 from the user device 102.

[0089] The object tracker 126 can then apply image processing to the image or frame provided by the user device 102 or an image captured by a sensor source 130 at a location corresponding to the initial position in the request to identify characteristics of the object (e.g., an outline of the object such as via edge detection applied to the image or frame, a color or shape of the object such as via edge detection applied to the image or frame and pixel analysis of pixels that fall within the identified edges or boundaries, an identification of the object via application of the image or frame as an input to a trained object detection artificial intelligence model, etc.). If the object is moving, the object may be captured by one or more other sensor sources 130 in addition to a sensor source 130 that captured a video or image displayed by the user device 102 at some time after the video or image displayed by the user device 102 was captured. Thus, once characteristics of the object are identified, the object tracker 126 can retrieve frames of video or images from the sensor data store 125 that were captured by sensor sources 130 other than the sensor source 130 that captured a video or image displayed by the user device 102, and perform the same or similar image processing on the retrieved frames or images to attempt to identify whichother frame(s) or image(s), if any, also depict the same object with the identified characteristics. In some embodiments, the object tracker 126 may limit the framc(s) or imagc(s) from the other sensor sources 130 that are analyzed to those frame(s) or image(s) captured by sensor source(s) 130 that are within a threshold distance (e.g., 5 feet, 50 feet, 100 feet, 200 feet, 1 mile, etc.) of the location of the sensor source 130 that captured the video or image displayed by the user device 102 such that the object tracker 126 reduces the number of false positive detections (e.g., reduces the number of times that an object in a frame or image is identified by the object tracker 126 as being the same as the selected object, but where the identified object is actually different than the selected object and / or cannot be the selected object given the distance between the two sensor sources 130 that captured the respective frame(s) or image(s)). The object tracker 126 can provide a copy of any frame(s) or image(s) that depict the same object to the virtual render generator 121.

[0090] The virtual render generator 121 can obtain the frame(s) or image(s) that depict the same object from the object tracker 126 and analyze the position from which and / or time at which the frame(s) or image(s) were captured to determine which frame(s) or image(s) to send to the user device 102 for display. For example, the location viewing system 120 may cause the user device 102 to display frame(s) or image(s) at a certain frame per second (e.g., 1, 12, 24, 30, 60, 120, etc.). In other words, the location viewing system 120 may cause the user device 102 to display N number of frames or images every second, where N represents the frame rate and can be any integer (e.g., 1, 12, 24, 30, 60, 120, etc.). A first frame or image displayed by the user device 102 (e.g., the frame or image at which the user made the object tracking selection) may correspond to a time 0. A second frame or image displayed by the user device 102 may then correspond to time 1 / N, a third frame or image displayed by the user device 102 may then correspond to time 2 / N, a fourth frame or image displayed by the user device 102 may then correspond to time 3 / N, and so on. The virtual render generator 121 may attempt to determine a frame or image captured at time 1 / N that the object tracker 126 identified as depicting the object. If the virtual render generator 121 determines a frame or image that was captured at time 1 / N that the object tracker 126 identified as depicting the object, then the virtual render generator 121 may cause the user device 102 to display the frame or image captured at time 1 / N subsequent to the display of the initial frame or image displayed by the user device 102. Optionally, the virtual render generator 121 may identify a position or location from which the frame or image captured at time 1 / N was captured, compare the identified position or location with the position or location from which theframe or image currently displayed by the user device 102 was captured, and cause the user device 102 to display the frame or image captured at time 1 / N if the distance between the two positions or locations is less than a threshold distance (e.g., 1 feet, 5 feet, 10 feet, or any other distance for which it would be reasonable for the object to move within a 1 / N time interval). If the virtual render generator 121 identifies two or more frames or images captured at time 1 / N that depict the object, the virtual render generator 121 may cause a frame or image closest to the object to be displayed by the user device 102, may cause a frame or image that captures the object from the same angle or distance as the previous frame or image to be displayed by the user device 102, may cause a frame or image that depicts the object in a center-most location of the frames or images to be displayed by the user device 102, may cause a frame or image captured by the same sensor source 130 as the previous frame or image to be displayed by the user device 102, may cause a frame or image captured by a sensor source 130 at an angle selected by a user (e.g., 45 degrees, 90 degrees, 180 degrees, etc.) to be displayed by the user device 102, and / or the like.

[0091] If no frame or image captured at time 1 / N (e.g., a current time) is determined to depict the object (or no frame or image captured at time 1 / N (e.g., a current time) that was captured within the threshold distance of the currently displayed frame or image is determined to depict the object), then the virtual render generator 121 can perform an analysis to determine whether the object is hidden from view of a sensor source 130 but is otherwise within a space defined by the sensor sources 130 (e.g., a space that is visible to and can be captured by the sensor sources 130) or whether the object has exited the space defined by the sensor sources 130. For example, the virtual render generator 121 can determine a speed or rate of movement of the object based on previous frame(s) or image(s) that depicted the object and the distance(s) between each sensor source 130 that captured the previous frame(s) or image(s). In particular, the virtual render generator 121 can determine the speed or rate of movement per second to be a distance between two sensor sources 130 that captured consecutively displayed frames or images depicting the object multiplied by N (e.g., the number for frames or images displayed per second). Because objects can move at variable speeds, the virtual render generator 121 can repeat this operation one or more times to determine multiple speeds or rates of movement, and can average the determined multiple speeds or rates of movements to identify an average speed or rate of movement. The virtual render generator 121 can then estimate where the object should be present at a current time given the speed or rate of movement or the average speed or rate of movement. If the virtual rendergenerator 121 estimates that the object should be depicted in a frame or image captured by a particular sensor source 130, then the virtual render generator 121 may cause the user device 102 to display the corresponding frame or image captured by the particular sensor source 130 even though the object may not be visible. If the virtual render generator 121 estimates that the object has moved to a location that is not captured by any sensor source 130, then the virtual render generator 121 may cause the user device 102 to display a notification identifying that the object is now out of view.

[0092] The virtual render generator 121 can repeat this process for each time interval (e.g., each 1 / N time interval) after the user selects an object to track until the user indicates via the user device 102 that the user no longer wishes to track the object (e.g., via a user gesture, a button input, a menu selection, a touch selection, a voice command, etc.), the user indicates via the user device 102 that the user wishes to track another object (e.g., via a user gesture, a button input, a menu selection, a touch selection, a voice command, etc.), or the object is no longer visible in any frame or image captured by the sensor sources 130 (e.g., the object has moved to a location outside the space defined by the sensor sources 130). The virtual render generator 121 can either cause a user device 102 to display each individual frame or image identified as depicting the object by transmitting each individual frame or image to the user device 102, or can stitch together multiple frames or images identified as depicting the object into a video or a sequence of images and cause a user device 102 to display each individual frame or image by transmitting the video or sequence of images to the user device 102. In other words, the virtual render generator 121 can stream frames or images to the user device 102 or generate and transmit a single video or sequence of images or sets of videos or sequences of images to the user device 102. Optionally, the virtual render generator 121 may annotate some or all of the frames or images to identify a location of the object within the respective frame or image (e.g., insert a bounding box that surrounds the object, highlight or change the color of the object, apply a text label adjacent to the object, etc.).

[0093] The virtual render and object tracking trainer 122 can train the object detection artificial intelligence model (e.g., neural network, machine learning model, etc.) using object detection training data. The object detection training data can include one or more images that are labeled to identify one or more objects depicted therein and / or that are labeled to identify one or more objects that are not depicted therein. Once trained, the object detection artificial intelligence model can receive, as an input, a frame or image and output an indication of one or more objectsdetected within the frame or image as a result. In order to determine the characteristics of an object selected by a user, the object tracker 126 can apply a frame or image as an input to the trained object detection artificial intelligence model, where the frame or image is the frame or image displayed by the user device 102 when the user selects an object to track. The trained object detection artificial intelligence model may output an indication of one or more objects detected within the frame or image, and the object tracker 126 can compare the location of the detected object(s) with the location within the frame or image where a user made a selection, can compare the characteristics of the detected object(s) with the characteristic(s) selected by a user, and / or the like. If the object tracker 126 detects a match (e.g., a detected object matches a location where a user made an object selection, a characteristic of a detected object matches a characteristic selected by a user, etc.), then the object tracker 126 may determine that the object selected by the user is the matching detected object, where the matching detected object may be defined by one or more characteristics. The object tracker 126 can then apply subsequent frames or images as an input to the trained object detection artificial intelligence model, where each application results in the trained object detection artificial intelligence model outputting an indication of one or more objects detected in the respective frame or image. For each output, the object tracker 126 can then compare the characteristic^ ) of the object selected by the user with the characteristic^ ) of each detected object, and determine which detected object has the same or similar characteristic(s) as the selected object. The object tracker 126 may determine that the detected object that has the same or similar characteristic(s) as the selected object is the selected object, and cause a user device 102 to display a frame or image that depicts the detected object in a manner as described herein.

[0094] By fitting a location, such as a street or a mall, with appropriate cameras (e.g., sensor sources 130), enabled with audio or spatial audio and video at appropriate distances and positions, the cameras can capture images and / or videos with one or more viewing angles (e.g., 120 degrees, 180 degrees, 360 degrees, etc.) periodically (e.g., every nanosecond, every millisecond, every second, every minute, 30 times a second, 60 times a second, 120 times a second, etc.). The images and / or videos obtained by virtual render generator 121 from multiple cameras located along a street or in the space within a mall can be presented to a user via a user device 102 in sequence as the images and / or videos are obtained, or stitched together in a sequence by virtual render generator 121 in such a way that enables the user to visualize and be immersed in the real-time experience of the happenings on the street or the mall (or other location) remotely through the user device 102.

[0095] A user can also enable one or more features via a user device 102 that allows the user to move virtually through a real space in tandem with a moving object (e.g., a car within the flow of traffic, a pedestrian, an animal, etc.), thereby watching the scene on the street as the scene unfolds in real-time. In addition, the user can select an option via a user device 102 to travel inside, above, below, and / or to the side of a vehicle selected by the user that is depicted in the virtual rendering of the real space and / or can select an option of a speed at which the user would like to travel. To travel in this way, the location viewing system 120 can select and / or alter the series of cameras or other sensor sources 130 from which the images displayed by the user device 102 are obtained such that future images match the location(s) to which the user would like to travel and / or speed at which the user would like to travel.

[0096] In some embodiments, the features and services provided by the location viewing system 120 may be implemented as web services consumable via the communication network 110. In further embodiments, the location viewing system 120 is provided by one or more virtual machines implemented in a hosted computing environment. The hosted computing environment may include one or more rapidly provisioned and released computing resources, which computing resources may include computing, networking and / or storage devices. A hosted computing environment may also be referred to as a cloud computing environment.

[0097] Thus, the present disclosure provides for the compilation of audiovisual data originating from a real space (e.g., a scenery) and captured by one or more pre-positioned sensor sources 130 (e.g., 1, 2, 3, etc. sensor sources 130, such as one or more cameras) located at a series of spatial distances. The audiovisual data may correspond to the motion of a moving object that is captured at a certain frames per second (FPS), such that the location viewing system 120 can generate a composite video output for display on a user device 102 that can produce a continuous audiovisual output that approximates the realistic motion of objects within the scenery in the view of the sensor sources 130. The user of the user device 102 can view the reproduced space as if the user is present and is participating in the scenery in real-time. In some embodiments, the scenery displayed to the user changes such that it appears, to the user, as if the user is propelled in tandem with a predefined or user-selected moving object. In some embodiments, the user is able to freely traverse and move relative to the reproduced space.

[0098] The operating environment 100 is also capable of enabling the user to enter a real space (c.g., a shopping mall or a shop) virtually via a user device 102, virtually walk around within the real space, and provide one or more inputs other than directional or position-based inputs. For example, a user may provide an input indicating a desire to shop for an item within a shop. The shop may have one or more sensor sources 130 pre-installed at one or more positions within the shop. The user, via a user device 102, may provide a position-based input to navigate to a particular location within the recreation of the shop. The user may also provide one or more further inputs to the location viewing system 120 via the user device 102 to select a specific good or item with which to shop or interact. Within the context of a shopping mall, for example, a plurality of sensor sources 130 (e.g., cameras, microphones, etc.) can be positioned within the mall or the shop space to such an extent that the remote user can walk around the shop virtually and visualize the items in real-time with complete three dimensional (3D) details of the items. Inputs obtained by the user device 102 can be generated using any available computing-based input device, such as a mouse, a cursor, a gesture-based input, haptic feedback, voice recognition, and / or the like. In some embodiments, voice recognition may include any number of computer voice input or natural language processing. Input devices can further include any user-based sensor device, such as a virtual reality controller, a haptic feedback vest or outerwear, or other peripheral devices used to provide interactivity within a virtual or computer environment.

[0099] In some embodiments, images and / or videos can be taken horizontally or vertically at different angles and / or at different heights by one or more sensor sources 130 to capture a 360 degree view of a space at which the sensor source(s) 130 are located. In some embodiments, the location viewing system 120 is able to apportion, slice, or cut the space within the purview of the camera system at different planes, such as the horizontal plane, vertical plane, or in any plane in 360 degrees. The location viewing system 120 enables the user to start at any place in the space, visualize the space at a defined starting position, and move along with the flow of the moving objects going forward or backward in time and space.

[0100] In some embodiments, the location viewing system 120 utilizes a plurality of static cameras positioned at various locations within a defined space. Each camera may capture sensor data, such as images and / or videos. The virtual render generator 121 can compile the sensor data obtained from the cameras, and cause the compiled sensor data to be displayed remotely on acomputer screen, an audiovisual display, a VR viewing system, and / or any other type of user device 102.Camera Systems

[0101] In some embodiments, a series of sensor sources 130, including cameras located within a given space are provided. In particular, sensor sources 130 comprising cameras can include still or video cameras configured to capture images at specified framerates, including from 1 FPS, to 60 FPS, to 200 FPS, and any other value in between or that exceeds 200 FPS.

[0102] For any space bounded by a series of sensor sources 130, and for any desired direction of travel (e.g., a vector), the number of sensor sources 130 may be determined using the desired FPS playback value, and the desired travel speed along the series of cameras. For example, where desiring to place cameras in a space and presenting a view to users at 24 FPS, following the average speed of objects within the space at 60 miles per hour (26.82 meters per second), a series of cameras may be placed equidistant from one another every 1.12 meters (112 cm) (26.82 mps / 24 cameras) in order for the system to track and display to a user a 60 mile per hour object. A general 5 formula for calculating the distance between successive cameras can be presented as D = -, where D is the distance between successive cameras, S is the speed of the user, and F is the desired FPS playback.

[0103] Therefore, sensor sources 130 can be placed at specific locations, including at regular or irregular intervals, at varying elevations and altitudes, and at varying angles and orientations, and can comprise cameras with varying fields of view (e.g., with lenses of varying focal lengths, with a field of view ranging from 1 to 360 degrees, or any value in between). When composited by virtual render generator 121, the sensor sources 130 can produce a seamless, continuous real-time immersive experience. In some embodiments, the series of cameras are placed at close space intervals, such as every centimeter, every meter, every 10 meters, every 15 meters, every 100 meters, every 1000 meters, or any value in between. The location viewing system 120 may be aware of the location, elevation, angle, and / or orientation of each sensor source 130, and can store this information in the sensor data store 125. As described herein, the location viewing system 120 can use the known location, elevation, angle, and / or orientation of each sensor source 130 to determine which frames or images to cause the user device 102 to display as a user navigates from an initial position to a final position.

[0104] In some embodiments, sensor sources 130 are placed statically within a space. In some embodiments, sensor sources 130 arc placed on moving or moveable objects within a space.

[0105] Thus, the system of cameras can record video or images of a space, and the location viewing system 120 can generate and provide multiple different video or image outputs for different remote users. Each video or image output may depend on a respective user’s desired movement or travel within a space, as indicated by input data received by the location viewing system 120 from the user device 102.

[0106] In some embodiments, sensor sources 130 can include cameras, including associated sensors and lenses, which can capture a specific viewpoint. Camera systems can include cameras with multiple sensors and multiple lenses, each with equivalent focal length, or each with differing focal lengths. In one embodiment, the sensor sources 130 include one or more microlens cameras. For example, a microlens camera may have a lens with a small diameter (e.g., less than 1 mm, less than 1 micrometer, less than 1 nm, etc.). The microlens camera can include a single lens or a series of lenses (e.g., 10 lens, 100 lens, 1000 lens, 10,000 lens, 100,000 lens, 1 million lens, etc.). In an embodiment in which a microlens camera includes multiple lenses, the lenses may be curved and / or coupled together to form a particular shape. FIGS. 14A-14B and FIGS. 15A-B display potential microlens camera arrangements, including where the lenses of the microlens camera are arranged spherically (FIG. 14A), elliptically (FIG. 14B), conically (FIG. 15A), or cylindrically (FIG. 15B). For example, microlens camera 1400 of FIG. 14A depicts multiple lenses 1410 that are coupled together to form a sphere. Microlens camera 1450 of FIG. 14B depicts multiple lenses 1460 that are coupled together to form a three-dimensional ellipse. With respect to FIG. 14B, arrangement 1470 represents a subset of multiple lenses that may be utilized by the microlens camera 1450 to capture a particular viewpoint. Arrangement 1480 represents an alternative subset of multiple lenses that may be utilized by the microlens camera 1450 to capture a different viewpoint compared to the viewpoint captured by arrangement 1480. Microlens camera 1500 of FIG. 15A depicts multiple lenses 1510 that are coupled together to form a cone. Microlens camera 1550 of FIG. 15B depicts multiple lenses 1560 that are coupled together to form a cylinder. With respect to FIGS. 14A-14B and FIGS. 15A-15B, any of the foregoing microlens designs may be oriented in any direction in space. For example, Microlens Camera 1450 may be oriented longitudinally so longest possible distance between the furthest two points on the ellipse issubstantially horizontal, or latitudinally so that the longest possible distance between the furthest two points is vertical. The field of view of such lens arrangements may range from 1 degree to 360 degrees, or any value in between. For each microlens camera, each view from an individual lens can be combined by the virtual render generator 121 to generate a larger image (and larger field of view). Image processing techniques performed by the virtual render generator 121 can further process imagery captured by microlens cameras in order to display any captured subset of the entire field of view as a traditional orthogonal projection.

[0107] As described with respect to FIG. 14B, individual lens arrangements of a microlens camera may be selected to generate any arbitrary view. Arrangement 1470 represents a subset of multiple lenses that may be utilized by the microlens camera 1450 to capture a particular viewpoint. Arrangement 1480 represents an alternative subset of multiple lenses that may be utilized by the microlens camera 1450 to capture a different viewpoint compared to the viewpoint captured by arrangement 1480. For example, a microlens camera comprising 100 microlenses may select a subset of one or more microlenses (e.g. 1 microlens, 10 microlenses, 20 microlenses, 50 microlenses, 100 microlenses) to visualize and capture a viewpoint. In one embodiment, an arrangement (first subset) of microlenses on one or more cameras may be selected to capture a particular viewpoint, and a subsequent arrangement (subsequent subset) of microlenses on one or more cameras may be selected in order to capture a second viewpoint. Different arrangements of microlenses may therefore be selected by an embodiment of the present system to capture a series of viewpoints to approximate a changing viewpoint of a scene. As described herein with respect to selecting cameras, so too can specific microlens or microlens arrangements be selected by an embodiment of the present system to present a continuous video stream of a changing viewpoint when a remote user is observing a scene. Thus, steps described herein for selecting a subset of cameras may also apply to selecting a subset of microlenses, or for selecting a subset of microlenses and cameras, in order to present a continuous series of captured frames approximating a desired viewpoint as a user observes a space. In some embodiments, described herein is a set of one or more microlens cameras, wherein an initial subset of microlenses on a microlens camera are selected to capture a first viewpoint. Upon receiving a user input from a remote user, which may be the user turning their head, moving a controller joystick, or moving their eyes, or any other movement corresponding to a user input to change the current view, an embodiment of the system described herein may select a subsequent subset of microlenses on the one or more microlenscameras, and present to the user the viewpoint generated by the subsequent subset of microlenses, in order to accommodate the change in view as parsed by the system from the user input.

[0108] The desired position data that is parsed by position parser 124 can be obtained from a mouse, keyboard, joystick, head or eye movement, hand or finger gesture, or a garment comprising sensors worn by the user. Any input can be recognized by the location viewing system 120, and video or image output is generated by virtual render generator 121 that corresponds with a desired position and movement. The user may follow an object moving within a reproduction of a real space. On the other hand, the user may want to move in a direction and speed that they desire within a reproduction of a real space, independent of other moving objects in the real space.

[0109] The location viewing system 120 can also employ spatial audio and / or spatial video to simulate a true immersive experience. For example, the location viewing system 120 can process the sensor data captured by the sensor sources 130 using an audio signal processing technique to simulate a binaural, spatially determined audio signal that is then output to a user device 102. Audio output by the location viewing system 120 can therefore allow for positional sounds to be generated and emitted by any number of user devices 102, such as user devices 102 that are or include speakers (e.g., a stereo system, a home theater, a headphone, an in-ear monitor, or any other conventional audio output device).

[0110] In one example of an embodiment of the operating environment 100, the user can visualize as a video or image output movement on a road with the flow of traffic, then subsequently input a U-turn to go in the opposite direction, thus updating the position requested by the user and the subsequent video output generated by location viewing system 120. The presentation or stitching software executed by the location viewing system 120 can generate images and / or videos for display by a user device 102 as determined by the sensor determination controller 123. In practice, with reference to the U-tum example described above, the location viewing system 120 may communicate with sensor sources 130 located along a road or street that cover a vehicle traveling in both directions (e.g., sensor sources 130 that are positioned such that motion of the vehicle can be captured regardless of the direction in which the vehicle is traveling), in such a way to enable the user a realistic and real-time experience of making a U-tum, and then presenting a video output of a traveler moving in the opposite direction.

[0111] The system also enables a user the ability to virtually travel to any captured environment. In one embodiment, a user may enter a mall like a shopping mall, and walk in anydirection they want, go into a store or shop, and move around in any direction, look at the items in the shop, visualize the people moving around, interact with any person in the shop, converse with staff or any person (if the staff or persons are connected with the system), purchase and order any item and pay for it, order said item for delivery, and / or the like.

[0112] The virtual render generator 121 can obtain information indicating a direction in which a user wishes to travel (e.g., North, South, East, West, or any combination thereof; angle at a location; vertical orientation at a location; horizontal orientation at a location; etc.) and / or a speed at which the user wishes to travel (e.g., from the request, from user inputs as the user navigates the space, etc.). Because the positions or locations of each sensor source 130 is known to the location viewing system 120 and stored in the sensor data store 125, the virtual render generator 121 can determine the location of a sensor source 130 that is capturing a scene to which the user should be moving from a previous position. For example, the location of a sensor source 130 that is capturing a scene to which the user should be moving may be determined by the virtual render generator 121 by dividing the speed of the user’s travel (e.g., in distance per second) by the frame rate (e.g., in frames per second) at which the frames or images are being displayed by the user device 102 to determine a distance, and identifying a location that is the determined distance from the sensor source 130 that captured the previous frame or image in a direction of the user’s travel. The virtual render generator 121 can then retrieve a frame or image captured by the sensor source 130 at the identified location at a time that is an inverse of the frame rate after the time instant corresponding to the time at which the previous frame or image displayed by the user device 102 was captured by a sensor source 130. The virtual render generator 121 can repeat these operations one or more times until the final frame or image that the virtual render generator 121 causes the user device 102 to display corresponds to a final location to which the user wishes to travel.

[0113] In some embodiments, the sensor sources 130 can be placed both vertically and horizontally at different positions in a space. The sensor sources 130 can be positioned to cover the view of the complete volume of a place (e.g., a room) or within a defined perimeter in a space. For example, the walls of a place can be covered by horizontal and / or vertical series of cameras to form a grid-like pattern of cameras. The grid-like pattern of cameras can enable the capture of images that cover a 360 degree view. The location viewing system 120 can stitch together one or more images captured by the cameras for display to a user to enable the user to virtually movearound freely in a seamless manner. Thus, when such a system is placed in a place like an aquarium, the complete visualization of every possible movement of organisms in the aquarium can be captured and presented to a user at a remote location, as if the user is present within the aquarium waters, interacting with the marine animals in real-time.

[0114] Additionally, the operating environment 100 can be configured with a variety of sensor sources 130, including cameras, microphones, global positioning system (GPS) devices, range finding devices, 3D mapping devices, geographic information system (GIS) devices, thermometers, barometers, hygrometers, and other ambient sensors in order to further provide or refine three-dimensional data, directionality data, or other inputs, like atmospheric conditions. In some embodiments, microphones can be placed at various positions throughout a scene. In some embodiments, the location viewing system 120 can perform further signal processing on various input signal data in order to approximate conditions at any point along a scene. In some embodiments, the location viewing system 120 performs 3D audio signal processing on the input signal data, where the resulting output approximates the range of sounds surrounding a user at any particular point or time within a scene.

[0115] Position data can also be automated independent of user input as well. Automatic navigation by road rules can be incorporated by implementing arbitrary road rules into the location viewing system 120.

[0116] In some embodiments, the location viewing system 120 can enable a surround audiovisual theater that can provide the experience of being in a space like Times Square, an aquarium, a concert hall, or an entertainment park, based on presentations from multiple projectors connected and corresponding to the series of camera systems in a space. This type of surround audio, visual, and haptic system can provide a realistic 3D immersive experience to a user.User Input

[0117] In some embodiments, the location viewing system 120 is configured in such a way to receive user input data in real-time from user devices 102. In some embodiments, user input data is provided via at least one of a mouse, a touchscreen, a gesture-detecting device (e.g., a camera, a motion sensing device, etc.), and / or the like. In some embodiments, gestural input comprises any gesture by which the user moves his or her hands, eyes, face, head, fingers, legs,feet, or any other part of the body. The location viewing system 120 (e.g., the position parser 124) can be configured to process the gestural input to generate a video output.

[0118] Furthermore, the operating environment 100 allows for a user to visualize and approximate a view as if the user were inside a car moving on a road. The user can then simulate driving the car by providing additional input, such as a request to move in one direction along the road.

[0119] In another use case, a pedestrian walkway can be configured to include one or more sensor sources 130. A user can request via a user device 102 to view the pedestrian walkway virtually and / or to move to different locations on the pedestrian walkway at varying speeds and / or pace and / or in varying directions.

[0120] The user devices 102 may use gyroscopic technologies to detect angular motions and / or movements to capture the motion of the user or an object in 360 degrees. Thus, the user may be able to provide the same type of input to the user device 102 regardless of whether the user is attempting to simulate driving a car on a road, walking on a pedestrian path, biking on a biking path, swimming in an underwater space, and / or the like.Example Block Diagrams for Generating a Video Output

[0121] FIG. 2 is a flow diagram 200 illustrating the operations performed by the components of the operating environment 100 of FIG. 1. As illustrated in FIG. 2, a user device 102 may request a video output from the location viewing system 120 (e.g., virtual render generator 121).

[0122] For example, the user device 102 may request a view (e.g., video output) at (1) from the virtual render generator 121. Before, during, and / or after the request by the user device 102, the user device 102 may send positioning data comprising an initial position and optionally directional, velocity, momentum, or other data corresponding to user movement to the position parser 124 at (2). The position parser 124 can use the received data to calculate an initial position at (3). At (4), the virtual render generator 121 requests an indication of the initial position calculated by the position parser 124. For example, the user device 102 may request a view (e.g., video output) of a real place (e.g., a street) starting at a position selected by the user, which is reflected by the request at operation (1). The position selected by the user may be transmitted at(2), processed by position parser 124 at (3) to calculate an initial position, and the resulting calculated position can be provided to the virtual render generator 121 at (4).

[0123] At (5), the sensor data store 125 receives sensor data (e.g., a stream of video, individual frames of a video, one or more images, etc.) from one or more sensor sources 130. The sensor data store 125 can store the sensor data (and any associated metadata) from the sensor source(s) 130. By way of example, the sensor sources 130 may be a set of cameras along a street, spaced at particular intervals and at particular elevations, with predetermined fields of view. As described herein, the sensor data may be received and stored at any time, asynchronous of user requests to view a location. The sensor data may be received and stored as the sensor data is captured by the sensor sources 130.

[0124] At (6), the virtual render generator 121 transmits a request for a sensor determination to sensor determination controller 123, where the request may include an indication of the initial position calculated by the position parser 124. The sensor determination controller 123 determines which sensor sources 130 from which to retrieve sensor data based on the calculated initial position at (7), and transmits an indication of which sensor sources 130 from which to retrieve sensor data to the virtual render generator 121 at (8).

[0125] The virtual render generator 121 can retrieve from the sensor data store 125 the sensor data captured by the sensor sources 130 indicated by the sensor determination controller 123 at (9). The virtual render generator 121 can then generate a frame of a requested view based on the retrieved sensor data at (10). Optionally, at (10.5), the virtual render generator 121 (or the user device 102) can transmit one or more frames or images to the object tracker 126 for use in object tracking. For example, the object tracker 126 can use artificial intelligence or other image processing techniques to identify a selected object in the received frame(s) or image(s) and to attempt to identify the same object in other frame(s) or image(s) captured by other sensor sources 130 other than the sensor source 130 that captured the received frame(s) or image(s), and as described herein. The frame generated at (10), optionally selected in accordance with tracking data generated and provided by the object tracker 126, may be one frame of a video that is output to the user device 102. For example, the virtual render generator 121 can transmit the generated frame to the user device 102 at (11), and the user device 102 can display the generated frame at (12). Operations (9), (10), (10.5), (11), and / or (12) can be repeated one or more times such that the virtual render generator 121 generates and transmits to the user device 102 one or more frames insuccession. As a result, the user may see a view of the initial position and / or other positions to which the user would like to move as a video without a noticeable interruption in playback.

[0126] FIG. 3 is a flow diagram 300 illustrating the operations performed by the components of the operating environment 100 of FIG. 1 with an optional machine learning component. As illustrated in FIG. 3, a user device 102 may request a video output from the location viewing system 120 (e.g., virtual render generator 121).

[0127] With respect to FIG. 3, operations (l)-(9) are as described above with respect to the flow diagram 200. At (10), the virtual render generator 121 transmits a machine learning request to the virtual render and object tracking trainer 122. The virtual render and object tracking trainer 122 may have previously trained a machine learning model using training data obtained from the sensor data store 125, where the training data includes one or more training data items. For example, each training data item may include one or more images captured by one or more sensor sources 130 at a particular location and may be labeled with an indication of a particular location to which a user would like to travel and an actual image viewable at a position at the particular location to which the user would like to travel (e.g., where the actual image may represent a composite image that could be formed from the image(s) included in the respective training data item). Thus, the virtual render and object tracking trainer 122 may have trained the machine learning model to output or predict a composite image based on one or more captured images provided as an input to the machine learning model. The machine learning request may therefore include the sensor data retrieved from the sensor data store 125 (e.g., one or more images captured by one or more sensor source(s) 130 indicated by the sensor determination controller 123) so that the sensor data can be applied as an input to the trained machine learning model.

[0128] The virtual render and object tracking trainer 122 can apply the sensor data and / or an indication of a particular location to which the user would like to travel as an input to the trained machine learning model at (11), which causes the trained machine learning model to output a generated, or composite frame (e.g., generated image). The virtual render generator 121 can generate a frame of a requested view based on the sensor data at (13). The virtual render generator 121 can then compare the generated frame to the composite frame output by the trained machine learning model at (14). If there is a discrepancy between the generated frame and the composite frame (e.g., one or more pixels of the generated frame have a different value than the corresponding pixels of the composite frame), the virtual render generator 121 can update thegenerated frame based on the composite frame at (15). For example, the virtual render generator 121 can replace the generated frame with the composite frame in response to a discrepancy. As another example, the virtual render generator 121 can modify some of, but not all of, the pixel values of the generated frame that are different than the composite frame such that the modified or updated generated frame is a blend of the initially generated frame and the composite frame. There may be a discrepancy in the frames due to a temporary obstruction in depicted in the sensor data, such as an unexpected shadow, a temporary obstructing object (e.g., a vehicle, a pedestrian, an animal, etc.), and / or the like.

[0129] The virtual render generator 121 can then transmit the updated generated frame to the user device 102 at (16), and the user device 102 can display the updated generated frame at (17). Some or all of the operations of the flow diagram 300 can be repeated one or more times such that the virtual render generator 121 generates and transmits to the user device 102 one or more frames in succession. As a result, the user may see a view of the initial position and / or other positions to which the user would like to move as a video without a noticeable interruption in playback.

[0130] Optionally, the virtual render and object tracking trainer 122 can re-train or update the trained machine learning model based on received feedback data, which can include one or more images captured by a sensor source 130 at a position corresponding to an outputted composite frame. For example, the virtual render and object tracking trainer 122 can determine any differences between the actual image captured at the position and the composite frame output by the trained machine learning model, and can re-train or update the trained machine learning model based on these difference(s).Example Video Output Generation

[0131] FIG. 4 is a flow diagram depicting an example video output generation routine 400 illustratively implemented by a location viewing system, according to one embodiment. For example, the location viewing system that implements the video output generation routine 400 may be the location viewing system 120 of FIG. 1. The generation routine 400 begins at block 410.

[0132] At block 420, the location viewing system receives a view request. For example, the view request can be obtained from a user device 102. The view request can include a request to view a particular location.

[0133] At block 430, a position is calculated. For example, the position may be an initial position calculated based on the location identified in the view request.

[0134] At block 440, a determination is made as to the sensor sources from which sensor data should be retrieved. For example, the determination may be made based on the initial position that is calculated.

[0135] At block 450, sensor data is retrieved that is captured by the determined sensor sources.

[0136] At block 460, a frame is generated based on the retrieved sensor data. For example, the frame may be formed from stitching together one or more images included in the retrieved sensor data, may be one of the images included in the retrieved sensor data, may be formed from blending pixel values from one image in the sensor data with pixel values from another image in the sensor data, and / or the like.

[0137] At block 470, the generated frame is transmitted to a user device. In response, the user device may cause the frame to be displayed for viewing by a user. After the frame is transmitted, the routine restarts at block 430 for the next position until the user decides to end the travel, wherein the routine 400 ends, as shown at block 490.

[0138] FIG. 5 is a flow diagram depicting another example video output generation routine 500 illustratively implemented by a location viewing system, according to one embodiment. For example, the location viewing system that implements the video output generation routine 500 may be the location viewing system 120 of FIG. 1. The generation routine 500 begins at block 502.

[0139] At block 504, the location viewing system receives a view request. For example, the view request can be obtained from a user device 102. The view request can include a request to view a particular location.

[0140] At block 506, a position is calculated. For example, the position may be an initial position calculated based on the location identified in the view request.

[0141] At block 508, a determination is made as to the sensor sources from which sensor data should be retrieved. For example, the determination may be made based on the initial position that is calculated.

[0142] At block 510, sensor data is retrieved that is captured by the determined sensor sources.

[0143] At block 512, the sensor data is applied as an input to a trained machine learning model to obtain a composite frame. Optionally, a position to which a user would like to move is also provided as an input to the trained machine learning model. For example, the machine learning model may have been previously trained to output a frame depicting a view from a position given one or more images provided as an input to the model.

[0144] At block 514, a frame is generated based on the retrieved sensor data. For example, the frame may be formed from stitching together one or more images included in the retrieved sensor data, may be one of the images included in the retrieved sensor data, may be formed from blending pixel values from one image in the sensor data with pixel values from another image in the sensor data, and / or the like.

[0145] At block 516, the composite frame is compared to the generated frame. For example, the comparison may include a comparison of a pixel value of the composite frame with a corresponding pixel value in the generated frame.

[0146] At block 518, the generated frame is updated based on the comparison. For example, the comparison may indicate a difference between one or more pixels in the composite and generated frames. The generated frame can be updated to include one or more pixel values found in the composite frame, the generated frame can be updated by being replaced with the composite frame, and / or the like.

[0147] At block 520, the updated generated frame is transmitted to a user device. In response, the user device may cause the frame to be displayed for viewing by a user. After the updated generated frame is transmitted, the routine 500 ends, as shown at block 522.

[0148] FIG. 6 is a flow diagram depicting another example video output generation routine 600 illustratively implemented by a location viewing system, according to one embodiment. For example, the location viewing system that implements the video output generation routine 600 may be the location viewing system 120 of FIG. 1. The generation routine 600 begins at block 610.

[0149] At block 610, a sequence of images of a location are obtained. The sequence of images of a location can be generated by one or more video cameras placed at specific distance intervals and / or at predefined positions. Optionally, one or more sensors in addition to a video camera can be placed at the predefined positions. The one or more sensors can include audio and / or atmospheric sensors. Moreover, the predefined positions of the video cameras and thecorresponding field of views (which can range up to 360 degrees) may be within a defined space (c.g., a physical location).

[0150] At block 620, a virtual space is generated. For example, the virtual space (e.g., the virtual reproduction or reconstruction of a real space) can be generated using the sequence of images of the location. In particular, video cameras may capture a variety of images, and by stitching or otherwise joining the images together, the location viewing system 120 can generate a virtual representation of a location (e.g., a virtual space).

[0151] At block 630, a subset of the cameras is selected. For example, the selected cameras may correspond to a specified position, such as a position corresponding to a location that a user would like to view.

[0152] At block 640, a display (e.g., of a user device 102) is caused to display one or more images captured by the subset of cameras in an order that is determined based on a position at which the user is viewing the real location.Additional Embodiments

[0153] Various example user devices 102 are shown in FIG. 1, including a desktop computer, laptop, and a mobile phone, each provided by way of illustration. In general, the user devices 102 can be any computing device such as a desktop, laptop or tablet computer, personal computer, wearable computer, server, personal digital assistant (PDA), hybrid PDA / mobile phone, mobile phone, electronic book reader, set-top box, voice command device, camera, digital media player, virtual reality headsets, virtual reality wearables, and the like. A user device 102 may execute an application (e.g., a browser, a stand-alone application, etc.) that allows a user to view a virtual reproduction or reconstruction of a real space.

[0154] The network 110 may include any wired network, wireless network, or combination thereof. For example, the network 110 may be a personal area network, local area network, wide area network, over-the-air broadcast network (e.g., for radio or television), cable network, satellite network, cellular telephone network, or combination thereof. As a further example, the network 110 may be a publicly accessible network of linked networks, possibly operated by various distinct parties, such as the Internet. In some embodiments, the network 110 may be a private or semi-private network, such as a corporate or university intranet. The network 110 may include one or more wireless networks, such as a Global System for Mobile Communications (GSM) network, a Code Division Multiple Access (CDMA) network, a LongTerm Evolution (LTE) network, or any other type of wireless network. The network 110 can use protocols and components for communicating via the Internet or any of the other aforementioned types of networks. For example, the protocols used by the network 110 may include Hypertext Transfer Protocol (HTTP), HTTP Secure (HTTPS), Message Queue Telemetry Transport (MQTT), Constrained Application Protocol (CoAP), and the like. Protocols and components for communicating via the Internet or any of the other aforementioned types of communication networks are well known to those skilled in the art and, thus, are not described in more detail herein.Examples

[0155] FIG 7. illustrates an example embodiment of the location viewing system 120 described herein. For example, the distance between the cameras depicted in FIG. 7 may correspond to the speed of an object traveling within the real space. If the speed of the object in the real space is 60 miles per hour, the object may travel 26.8 meters in one second. If the frame rate at which the cameras captured images is 24 FPS, then 24 cameras may be placed along a 26.8 meter stretch in the real space. Thus, at 60 miles per hour speed, the distance between cameras can be 1.12 meters (112 centimeters), and the time taken between images captured by consecutive cameras can be 41.7 milliseconds. When an image is captured by each of the successive cameras in the series of cameras at a time interval of about 40 milliseconds (e.g., 41.7 milliseconds) and presented at 24 FPS, the sequence of consecutive frames may produce a seamless video (e.g., a video that can playback without any user-noticeable interruption or skipping of frames) of length one second.Computer System

[0156] In some embodiments, the systems, processes, and methods described herein are implemented using a computing system 800, such as the one illustrated in FIG. 8. The example computer system 802 is in communication with one or more computing systems 820 and / or one or more data sources 822 via one or more networks 818. While FIG. 8 illustrates an embodiment of a computing system 802, it is recognized that the functionality provided for in the components and modules of computer system 802 can be combined into fewer components and modules, or further separated into additional components and modules.

[0157] The computer system 802 can comprise a TASM module 814 that carries out the functions, methods, acts, and / or processes described herein, such as the functionality described herein as being performed by the location viewing system 120. The TASM module 814 is executed on the computer system 802 by a central processing unit 806 discussed further below.

[0158] In general the word “module,” as used herein, refers to logic embodied in hardware or firmware or to a collection of software instructions, having entry and exit points. Modules are written in a program language, such as JAVA, C, or C++, or the like. Software modules can be compiled or linked into an executable program, installed in a dynamic link library, or can be written in an interpreted language such as BASIC, PERL, LAU, PHP or Python and any such languages. Software modules can be called from other modules or from themselves, and / or can be invoked in response to detected events or interruptions. Modules implemented in hardware include connected logic units such as gates and flip-flops, and / or can include programmable units, such as programmable gate arrays or processors.

[0159] Generally, the modules described herein refer to logical modules that can be combined with other modules or divided into sub-modules despite their physical organization or storage. The modules are executed by one or more computing systems, and can be stored on or within any suitable computer readable medium, or implemented in- whole or in-part within special designed hardware or firmware. Not all calculations, analysis, and / or optimization require the use of computer systems, though any of the above-described methods, calculations, processes, or analyses can be facilitated through the use of computers. Further, in some embodiments, process blocks described herein can be altered, rearranged, combined, and / or omitted.Computing System Components

[0160] The computer system 802 includes one or more processing units (CPU) 806, which can comprise a microprocessor. The computer system 802 further includes a physical memory 810, such as random access memory (RAM) for temporary storage of information, a read only memory (ROM) for permanent storage of information, and a mass storage device 804, such as a backing store, hard drive, rotating magnetic disks, solid state disks (SSD), flash memory, phase-change memory (PCM), 8D XPoint memory, diskette, or optical media storage device. Alternatively, the mass storage device can be implemented in an array of servers. Typically, the components of the computer system 802 are connected to the computer using a standards basedbus system. The bus system can be implemented using various protocols, such as Peripheral Component Interconnect (PCI), Micro Channel, SCSI, Industrial Standard Architecture (ISA) and Extended ISA (EISA) architectures.

[0161] The computer system 802 includes one or more input / output (I / O) devices and interfaces 812, such as a keyboard, mouse, touch pad, and printer. The I / O devices and interfaces 812 can include one or more display devices, such as a monitor, that allows the visual presentation of data to a user. More particularly, a display device provides for the presentation of GUIs as application software data, and multi-media presentations, for example. The I / O devices and interfaces 812 can also provide a communications interface to various external devices. The computer system 802 can comprise one or more multi-media devices 808, such as speakers, video cards, graphics accelerators, and microphones, for example.Computing System Device / Operating System

[0162] The computer system 802 can run on a variety of computing devices, such as a server, a Windows server, a Structure Query Language server, a Unix Server, a personal computer, a laptop computer, and so forth. In other embodiments, the computer system 802 can run on a cluster computer system, a mainframe computer system and / or other computing system suitable for controlling and / or communicating with large databases, performing high volume transaction processing, and generating reports from large databases. The computing system 802 is generally controlled and coordinated by an operating system software, such as z / OS, Windows, Linux, UNIX, BSD, PHP, SunOS, Solaris, MacOS, ICloud services or other compatible operating systems, including proprietary operating systems. Operating systems control and schedule computer processes for execution, perform memory management, provide file system, networking, and I / O services, and provide a user interface, such as a graphical user interface (GUI), among other things.Network

[0163] The computer system 802 illustrated in FIG. 8 is coupled to a network 818, such as a LAN, WAN, or the Internet via a communication link 816 (wired, wireless, or a combination thereof). Network 818 communicates with various computing devices and / or other electronic devices. Network 818 is communicating with one or more computing systems 820 and one or more data sources 222. The TASM module 814 can access or can be accessed by computingsystems 820 and / or data sources 822 through a web-enabled user access point. Connections can be a direct physical connection, a virtual connection, and other connection type. The web-enabled user access point can comprise a browser module that uses text, graphics, audio, video, and other media to present data and to allow interaction with data via the network 818.

[0164] The output module can be implemented as a combination of an all-points addressable display such as a cathode ray tube (CRT), a liquid crystal display (LCD), a plasma display, or other types and / or combinations of displays. The output module can be implemented to communicate with input devices 812 and they also include software with the appropriate interfaces which allow a user to access data through the use of stylized screen elements, such as menus, windows, dialogue boxes, tool bars, and controls (for example, radio buttons, check boxes, sliding scales, and so forth). Furthermore, the output module can communicate with a set of input and output devices to receive signals from the user.Other Systems

[0165] The computing system 802 can include one or more internal and / or external data sources (for example, data sources 822). In some embodiments, one or more of the data repositories and the data sources described above can be implemented using a relational database, such as DB2, Sybase, Oracle, CodeBase, and Microsoft® SQL Server as well as other types of databases such as a flat-file database, an entity relationship database, and object-oriented database, and / or a record-based database.

[0166] The computer system 802 can also access one or more databases 822. The databases 822 can be stored in a database or data repository. The computer system 802 can access the one or more databases 822 through a network 818 or can directly access the database or data repository through TO devices and interfaces 812. The data repository storing the one or more databases 822 can reside within the computer system 802.Example Embodiments

[0167] FIGs. 9-13 and 18 illustrate some embodiments of the present disclosure. For example, FIG. 9 illustrates a system of the present disclosure implemented on a city street. In particular, a pole or other vertically-positioned structure 904 comprises a plurality of cameras 902. Each camera 902 may include one or more lenses placed at different angles and / or directions in a camera series to enable the capturing of up to 360 degree angles of a space. In some embodiments,the cameras 902 can be positioned in a straight line (e.g., a vector), or a subset of cameras 902 can be positioned in a straight line (c.g., a vcctor).Thc cameras 902 can be positioned in different horizontal and / or vertical planes. In some embodiments, the cameras 902 can be positioned on a pole, along a wire, on a wall, or any suitable planar surface. In some embodiments, the cameras 902 can be positioned on any fixed location. In some embodiments, the cameras 902 can be immobile. In some embodiments, the cameras 902 can be positioned on one or more moving objects, including on a train, car, person, trolley, shopping cart, or drone. A series of similar cameras, microphones, and / or other sensors can also be placed on the pole 904 and / or on a string or wire that traverses a street and that is supported by at least one pole 904.

[0168] FIG. 10 illustrates a system of the present disclosure implemented in a conference setting. For example, a series of cameras 1002 are installed in a conference hall to cover the walls, the stage, and / or other portions of the hall. The cameras 1002 may be installed on one or more poles or other vertically-positioned structures 1004. Alternatively or in addition, the series of cameras 1002 can also be installed on or around one or more seats in the conference hall in such a way that the cameras 1002 can capture 360 degree images. The cameras 1002 can be positioned in different horizontal and / or vertical planes. A series of similar cameras, microphones, and / or other sensors can also be placed on the pole 1004 and / or on a string or wire that traverses a stage and that is supported by at least one pole 1004.

[0169] In comparison, FIG. 11 illustrates a system of the present disclosure implemented in a concert or performance venue. For example, a series of cameras 1102 are installed in a performance venue to cover the walls, the stage, and / or other portions of the venue. The cameras 1102 may be installed on one or more poles or other vertically-positioned structures 1104. Alternatively or in addition, the series of cameras 1102 can also be installed on or around one or more seats or the performance stage, in the conference hall in such a way that the cameras 1102 can capture 360 degree images. The cameras 1102 can be positioned in different horizontal and / or vertical planes. A series of similar cameras, microphones, and / or other sensors can also be placed on the pole 1104 and / or on a string or wire that traverses a stage and that is supported by at least one pole 1104.

[0170] In some embodiments, a system of the present disclosure can be implemented in an underwater setting, as illustrated in FIG. 12. For example, a marine underwater space in which marine organisms are present can be fitted with the audiovisual camera system of the presentdisclosure. A series of cameras 1202 can be installed in the underwater space on one or more poles or other similar structures 1204. The cameras 1202 can be installed in such a way that the cameras 1202 can capture 360 degree images. The cameras 1202 can be positioned in different horizontal and / or vertical planes. A series of similar cameras, microphones, and / or other sensors can also be placed on the pole 1204 and / or on a string or wire that is supported by at least one pole 1204.

[0171] In some embodiments, the system of the present disclosure can comprise one or more poles 1304 with multiple embedded camera elements 1302, as depicted in FIG. 13. In some embodiments, a pole 1304 with a series of multiple 360 degree cameras placed at various angles and / or heights can enable the visualization of a real space at different angles and / or heights. A user viewing an image captured by a camera 1302 on the top of the pole 1304, or any of the cameras 1302 located around the pole 1304, may feel as if the user is standing right at the pole 1304.

[0172] As described herein, some embodiments of the present system may further correct for obstructions within a virtual representation of a scene. For example, one embodiment of the system herein may digitally remove people or other obstructions when presenting a scene to a user, thus "filling in" any identified gaps, obstructions, or otherwise occluding objects within a scene. Thus, in a system set up in a grocery store for example, the system may present empty aisles to a remote user, despite the fact that one or more people may be positioned within a space captured by an embodiment of the system herein. In some embodiments, obstructing objects may be identified by an embodiment of the object tracking system herein, wherein the object removal system is configured to remove one or more tracked objects from the scene. In some embodiments, an image processing system is configured to generate an object corrected output, where the object corrected output removes certain tracked objects from generated frames, and wherein the object corrected output is configured to resemble a scene wherein the object is absent, In some embodiments, the system may sample surrounding objects to fill in any identified obstructions. In some embodiments, the system may sample previously captured or archival frames in order to fill in any identified obstructions. In some embodiments, the system may be trained to fill in obstructions based on a training data set of representative frames. For example, when observing a grocery space using an embodiment of the system herein, an empty grocery aisle may be used either as a direct comparator, or trained with a machine learning model, such that the system maygenerate an appropriate viewpoint to remove particular objects, and present the viewpoints to a remote user.

[0173] In some embodiments, the system of the present disclosure can be used to aid a user in shopping or selecting objects to purchase. In some embodiments, the system of the present disclosure can be implemented in a retail environment. In some embodiments, the system of the present disclosure can be implemented in a warehouse environment. As illustrated in FIG. 18, a retail environment can implement an embodiment of the system disclosed herein. In some embodiments, cameras 1802 can be arranged along an aisle on one side, and cameras 1804 can be arranged along the opposite side of the same aisle. In some embodiments, cameras 1806 may be placed substantially within the aisle. As described, cameras 1802 and 1804 may serve to approximate a viewpoint at the position of cameras 1806 using pixel-averaging as described with respect to FIGs. 16A-16F. Thus, the series of cameras 1802 and 1804 can generate a series of viewpoints at the midpoint, or any point in between cameras 1802 and 1804 to present to a user. In some embodiments, a user may, using an input device, a desire to see, view, or otherwise track an object within the boundaries of the system as described herein. For example, a user remotely viewing a grocery store, may ask to view a specific item or items, like laundry detergent, when they are currently viewing a scene in the cereal aisle. One or more cameras 1802 and / or 1804 may be labeled or otherwise associated with one or more items, such that the system may use the labeling or association to cause the user device 102 to display an image captured by a camera 1802 and / or 1804 associated with an item specified by the user. The system in response to the user’s input, may then select a direction and predetermined speed of travel, and move the user’s view through the system to navigate to their desired item, thereby traveling to the detergent aisle from the cereal aisle. Thus, an embodiment of the present system may be preprogrammed or trained to recognize specific objects, and then move users within the virtual space to a view the selected object. Some advantages of the present system allow for a user to assess stock values in real time, with the system configured to query a virtual inventory system to assess whether a desired good is available in storage or on the shelf, and to assess the relative quality of an object in real time. In some embodiments, the virtual inventory system is queried by the system when a user submits an input to view or move to an object. In some embodiments, the virtual inventory system is configured to communicate with the object tracking system to select suitable cameras to move a user through a space to arrive at the selected object. Moreover, objects that are recognized by thesystem described herein can visualize additional metadata, including estimated stock, nutrition information, price, discounts, and / or expiration dates, and present such information to the user when queried.

[0174] In some embodiments, the system of the present disclosure can comprise one or more mobile objects with multiple embedded camera elements. Mobile objects can include, cars, submarines, trains, buses, planes, drones, boats, people, animals, or any other non-sessile object. In some embodiments, the mobile objects may be equipped with GPS, and optionally, additional motion and position tracking equipment. In some embodiments, the camera series may be placed in various places within and external to a vehicle. Thus, a user may virtually tour and move within a vehicle, as well as observe the scene surrounding a traveling vehicle. In some embodiments, cameras comprising an embodiment of the system described herein may be affixed to mobile platforms like drones, which may move in response to an instruction from a user or an administrator of the drones such that users can view scenes from different defined spaces. For example, a space with multiple cameras may include any number of fixed, immobile cameras, and mobile camera platforms. Thus, drone-based cameras may be utilized to supplement or add additional views, or cameras from which a pixel- averaging system may be implemented according to embodiments described herein.

[0175] In some embodiments, a system of the present disclosure may be installed in a space dedicated for gaming. In some embodiments, a system of the present disclosure may be configured to display emoji characters in representations of the real space. In some embodiments, a system of the present disclosure may be configured to display video game or otherwise computer rendered images in place of, or augmenting certain objects within a scene. For example, a scene may be configured with cardboard representations and QR, or other proprietary codes, which instructs the system to insert a particular emoji, character, or computer-generated graphic. Such graphics may supplement, or supplant the tracked object, according to the pre-programmed functionality of the system. In some embodiments, the system may be further configured to provide haptic feedback. For example, a user wearing a vest with moveable haptic elements may perceive “pushes” or other physical forces, when interacting with select elements within a virtual space. As can be appreciated, the system described in the present disclosure may be implemented in a variety of settings, including amusement parks, amusement rides, walking, jogging, biking, or to tour hazardous or industrial locations.Pixel Averaging System

[0176] As described herein, a real space may be bounded by a system of cameras and / or sensors as described herein. In some embodiments, a first camera within a series of cameras may therefore capture an image or video stream from a fixed position, of a specific scenery or scene, thereby capturing a first frame. A second camera may be positioned within a set interval distance from the first camera, to capture a substantially similar scene as captured by the first camera, though the second camera’s viewpoint differs due to the fact that the second camera’s position is offset compared to the first camera, thereby capturing a second frame. Thus, different views of substantially the same scenery may be captured by multiple cameras and / or sensor sources, each camera capturing frames of substantially the same scene. Moreover, each frame captured by each camera may further be represented as a series of pixels according to known color models in the art (e.g., RGB, CMYK). In some embodiments, frames captured by two or more different cameras can have constituent pixel values averaged, to generate a pixel-averaged frame. Therefore, corresponding pixels of the two different frames averaged together will thus produce a new frame which will be the average of the two frames, and the new frame will represent the mid-point of view (mid POV) of the scenery between the two cameras. In effect, two cameras within an embodiment of the present system observing a single object or scene may generate a set of pixel- averaged frames, in order to approximate the view of a third camera located in the middle of the two cameras. In some embodiments, the approximated camera may be located at any arbitrary position between the first and second camera positions, based on the particular weight assigned when averaging pixels.

[0177] FIGs. 16A-16C represent various captured and pixel-averaged frames. Therefore, for conventional 2-dimensional image capture, every pixel within a captured frame maybe represented by a particular (x, y) coordinate on the frame. In FIG. 16A, 1610 represents a first frame captured by a first sensor source, with reference pixel 1600 shaded in. FIG. 16B depicts a second frame captured by a second sensor source, which possesses at analogous pixel coordinates as frame 1610, a shaded in pixel 1650. FIG. 16C depicts a pixel -a veraged frame 1675, wherein the pixel 1670 is averaged compared to pixel 1600 and pixel 1650. Therefore, for frames captured by two or more cameras, an average frame can be generated according to an embodiment of the present system. Therefore, representing an arbitrary pixel as an RGB value, for example, of(100,100,100), and a corresponding pixel on an additional captured frame of (200, 200, 200), each independent RGB channel may be averaged, to create an averaged pixel of (150, 150, 150). For example, the color models of pixels at the same x, y coordinates of a frame, representing 2 or more frames of a scenery on a plane, can be averaged in value to generate a pixel- averaged frame.

[0178] FIG. 16D is a flow diagram depicting an example pixel-average frame generation routine 1690 illustratively implemented by a location viewing system according to one embodiment. For example, the location viewing system that implements the pixel-average frame generation routine may be the location viewing system 120 of FIG. 1. The generation routine 1690 begins at block 1691, where the system loads a first and second frame from memory. In some embodiments, the first and second frame are sourced from two or more cameras, such as two or more cameras that may be oriented to capture the same or similar scene. For example, the boundaries of a scene captured by a camera may form a three-dimensional shape in space (e.g., where the boundaries include one or more edges from a lens of the camera to various points at a geographic location that are depicted within an image captured by the camera), and two or more cameras may be oriented to capture the same or similar scene if at least a portion of the three- dimensional shape representing a scene captured by one camera intersects the three-dimensional shape representing the scene captured by another camera. In other words, two or more cameras may be oriented to capture the same or similar scene if each image captured by the respective cameras includes at least one common object (which may be depicted from different angles in each image). In some embodiments, the first and second frame are sourced from a first and second camera. In some embodiments, the first frame is sourced from a pixel- averaged frame. In some embodiments, the second frame is sourced from a pixel-averaged frame.

[0179] At block 1692, a pixel average is calculated. For example, a pixel at a certain X,Y coordinate in the first frame may possess the RGB value of (200,200,200), while a corresponding pixel at the same relative X,Y coordinate in the second frame may possess an RGB value of (0,0,0), meaning a pixel- averaged pixel would possess RGB values of (100,100,100). As used herein, an X,Y coordinate of a pixel in a first frame may have the same relative X,Y coordinate of a pixel in the second frame if both pixels correspond to the same point or portion of the same object, the same point or portion of the same scene, etc. In some embodiments, the pixel average may assign different weights to the first and second frame pixels, generating a weighted average pixel. Differing weights values assigned to each frame are determined based on relativedistance from each camera (or viewpoint) to the desired generated viewpoint. For example, a pixel at a certain X,Y coordinate in the first frame may possess the RGB value of (200,200,200), while a corresponding pixel at the same relative X,Y coordinate in the second frame may possess an RGB value of (50,50,50). Where the system assigns a weight bias of 75% to pixels from frame 1, and a weight bias of 25% to pixels from frame 2, thereby a weighted average pixel would possess the RGB values

[0180] At block 1693, a pixel-averaged frame is generated based on the calculated pixel averaging.

[0181] FIG. 16E is a flow diagram depicting an example pixel-average frame generation routine 1695 for generating additional pixel-averaged frames illustratively implemented by a location viewing system according to one embodiment. For example, the location viewing system that implements the pixel-average frame generation routine may be the location viewing system 120 of FIG. 1. The generation routine 1695 begins at block 1696. With respect to FIG. 16D, steps 1695-1697 are described above as analogous to steps 1691-1693, respectively, to generate a first pixel-averaged frame.

[0182] At block 1698, the first pixel averaged frame and a third frame are loaded from memory. In some embodiments, the third frame is a pixel averaged frame. In some embodiments, the third frame is sourced from memory. In some embodiments, the third frame is sourced from a camera.

[0183] At block 1699A, a pixel average is calculated. For example, a pixel at a certain X,Y coordinate in the first frame may possess the RGB value of (200,200,200), while a corresponding pixel at the same relative X,Y coordinate in the second frame may possess an RGB value of (0,0,0), meaning a pixel-average pixel would possess RGB values of (100,100,100). In some embodiments, the pixel average may assign different weights to the first and second frame pixels, based on generating a viewpoint not at the midpoint between the location of the first frame and the second frame, thus generating a series of weighted average pixels. For example, a pixel at a certain X,Y coordinate in the first frame may possess the RGB value of (200,200,200), while a corresponding pixel at the same relative X,Y coordinate in the second frame may possess an RGB value of (50,50,50), and wherein the system is configured to generate a viewpoint at a point 374ththe distance between the first and second frame. Where the system assigns a weight bias of 75% to pixels from frame 1, and a weight bias of 25% to pixels from frame 2, thereby a weightedaverage pixel would possess the RGB values of (81 ,81 ,81) (e.g.,(200*0.75) , (50*0 25) (200*0.75) , (50*0.25) (200*0.75) (50*0.25)2 2 2 2 ’ 2 2

[0184] At block 1699B, a second pixel averaged frame is generated based on the calculated pixel averaging.

[0185] FIG. 16F illustrates an example embodiment of the location viewing system 120 described herein, comprising cameras arranged at pre-determined points. Cameras 1681, 1682, 1683, and 1684 are located at points A, B, C, and D, respectively. For camera 1681 at point A, a scries of cameras (e.g., two or more) extends in a linear direction towards point E. For camera 1682 at point B, a series of cameras (e.g., two or more) extends in a linear direction towards point F. The same camera arrangement exists for Camera 1684 at Point C, extending to point G, and for Camera 1683 at Point D, extending to point H. Thus, a plurality of cameras is arranged at vectors defined by points A-E, B-F, C-G, and D-H. With respect to the method disclosed in FIG. 16D, the method can generate a pixel-averaged frame to approximate a viewpoint as is from a virtual third camera located at viewpoint 1685 (which is the midpoint between the locations of camera 1681 and 1682). Viewpoint 1686 can be generated by at least two methods of the present disclosure. With respect to FIG. 16E, viewpoint 1686 may be generated using the pixel- averaged frame generated at 1685, averaged against a frame captured from camera 1682 at point B. Alternatively, viewpoint 1686 may be generated using weighted averages. As viewpoint 1686 is located 374ththe distance between camera 1681 and camera 1682, a percent weight bias of 75% is assigned to camera 1682, and a percent weight bias of 25% is assigned to camera 1681. For a viewpoint located at a point l / 3rd(“173rdpoint”) the distance between any two arbitrary cameras, the percent weight would be 66 and 33 percent, respectively, with the 66 percent weight applied to the camera closest in position to the l / 3ldpoint. Viewpoint 1687, which is the midpoint between viewpoints 1685 and 1688, represents a viewpoint that may be generated using the method described in FIG. 16D or 16E, using frames generated from viewpoints 1685 and 1688. Lastly, viewpoint 1689 may be generated by at least two methods, including using frames derived from cameras located along the C-G and D-H vectors, or alternatively, from cameras 1684 and 1690B, with the resulting pixel- averaged frame having further image processing, image correction (including perspective and distortion correction), and angle-distortion correction applied.

[0186] In some embodiments, a first frame and second frame can be averaged, based on the RGB average of the same pixel location in the first frame and the second frame. In someembodiments, a first frame and second frame can be averaged, based on the RGB average of the same relative pixel location in the first frame and the second frame. For example, pixels representing the image of an object within two or more frames can be shifted based on the angle and distance of the viewer in a space. Pixel shift due to camera location can therefore correct for difference in perspective based on camera effective focal length and position.

[0187] In some embodiments, the pixel averaging system permits camera arrays to be placed in non-linear arrangements. For example, one or more cameras may follow any regular or irregular pattern of placement, including into sinusoidal patterns, or otherwise offset at regular or irregular intervals in any horizontal plane, any vertical plane, or any combination thereof. For example, a string of 10 cameras may be arranged in a line, or alternatively, arranged in a sine pattern, with 5 cameras placed at each sign crest and trough respectively. As another example, a string of 10 cameras may be arranged in a line, with one or more of the cameras offset from the line in a horizontal, vertical, and / or horizontal and vertical direction. Using an embodiment of the presently described pixel-averaging system, approximate viewpoints between crest-placed cameras and trough-placed cameras may be generated and presented to a user. In one embodiment, where successive cameras are intended to be placed in close proximity, and to avoid obstruction by successive cameras, described herein is a system for alternating or offset camera positions, wherein cameras may be offset by 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 15, 20, 30, 40, 50, or 100 cameras in a series, or any value in between. Thus, in some embodiments, a camera series may comprise a system wherein every 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 15, 20, 30, 40, 50, 100 cameras, or any value in between, are offset compared to an initial camera series. Thus, in one embodiment, every 2ndcamera within a series (e.g. 2, 4, 6...) is offset compared a first camera series (e.g. 1, 3, 5..). Cameras may be positioned to avoid obstruction, or as local mounting options may prohibit mounting at certain positions. Therefore, cameras may be mounted to create multiple series of cameras positioned at offset horizontal and vertical positions. The frames from the different series of offset cameras can be shifted or averaged using pixel and frame averaging techniques to produce the view point from the middle point of any two, or greater sets of cameras. The averaged frames representing a mid-point scene can then be presented to the remote viewer.Producing New Points of View

[0188] As described herein, the location viewing system 120 allows for the remote visualization and experience of traveling in a location in real time, where a remote user can move around the location in a selected direction, speed, angle of motion, and / or the like via one or more sensory inputs provided by the user.

[0189] The location viewing system 120 can use artificial intelligence to create or synthesize one or more images representing a point of view (POV) from a position at which no sensor source 130 is present. For example, the location viewing system 120 can create or synthesize the image(s) using one or more images captured by sensor source(s) 130 in the vicinity of the position at which no sensor source 130 is present (e.g., from sensor source(s) 130 that are positioned along the same road, in the same building, etc. as the position at which no sensor source 130 is present). The location viewing system 120 can generate any number of images representing POVs from any number of different positions at which no sensor sources 130 are otherwise present. For example, the location viewing system 120 can generate a number of images representing a particular POV that corresponds with the number of images captured by a sensor source 130 located in the vicinity of the position of the POV in a given time period (e.g., 24 images per second, 30 images per second, 60 images per second, 120 images per second, etc.). Thus, the location viewing system 120 can generate a number of images representing a particular POV that, when viewed by a user, appears to the user as a moving image and how the user would perceive the position if the user was actually present at the position. The functionality of the location viewing system 120 has many advantages and applications in enabling a remote user to move in any direction and any path the user desires in a location and experience the audio-visualization of the scenery at that location as the user desires even if only a limited number of cameras are present in the location.

[0190] The location viewing system 120 can produce the image(s) from the POV of a position at which a sensor source 130 is not located by training an artificial intelligence model (e.g., a machine learning model) using training data that includes pre-existing images representing one or more other different POVs formed from images captured by sensor sources 130 at locations different from the different POVs, with or without using the pixel information of the images as described herein. The location viewing system 120 can train the artificial intelligence model using the features of the images in the training data (e.g., including the pixel details in the images) suchthat the artificial intelligence model, when receiving an input (e.g., one or more images captured by one or more sensor sources 130), recognizes the pixel features of one or more input images and produces the features of a new image representing a new POV where there is no sensor source 130 present. Optionally, the location viewing system 120 can train the artificial intelligence model to average pixels of input image(s), where the average values of the color model features of the corresponding pixels from one or more input images are obtained using a pre-determined algorithm that uses the position of the new POV relative to the pre-existing POVs represented by the input image(s).

[0191] The location viewing system 120 can train and use an artificial intelligence model to produce a new image representing a new POV by performing one or more operations. For example, the image generator 127 of the location viewing system can obtain an image captured by a first sensor source 130, an image captured by a second sensor source 130, and / or one or more images captured by one or more sensor sources 130 between the first sensor source 130 and the second sensor source 130. In particular, a side AB in a square ABCD (or rectangle or another shape) may be the face of a cube ABCDEFGH. The first sensor source 130 may be located at position A and the second sensor source 130 may be located at position B, which are at the ends of the side AB of the square ABCD (e.g., the front face of the cube ABCDEFGH). Any number of sensor sources 130 (e.g., 1, 2, 10, 100, etc.) may be located between position A and position B along an axis that passes through positions A and B. The sensor sources 130 may be at equal and / or non-equal distances from each other and from positions A and / or B. As an illustrative example, a distance between two sensor sources 130 can be 0.1 cm to cover a very slow movement of a remote user, such as a user that wishes to move at 0.1 miles per hour. The sensor sources 130 between positions A and B may be located in a straight line or in curvatures of various functions (e.g., sine, cosine, etc.). The sensor sources 130 between positions A and B may also be located in any direction from any plane to any other plane within the cube ABCDEFGH. For example, the sensor sources 130 can be located from the mid-point of the top of the front face ABCD to the mid-point of the top of the back face EFGH. The sensor sources 130 located between positions A and B may capture images of the scenery that is in the direction of the square EFGH (e.g., the back face of the cube ABCDEFGH).

[0192] The training data that the image generator 127 uses to train the artificial intelligence model can include details of one or more images, such as the number of pixels on thesquare or rectangle of the respective image, the coordinates (e.g., x, y, z) for each pixel in the respective image and / or the RGB color model of the respective pixel, an intensity of each pixel in the respective image, an angle from which the respective image was captured relative to a plane bounded by positions A and B (where the angle can be calculated based on a distance of the position of the respective image from a baseline image (e.g., an image captured by a middle sensor source 130 in an array of sensor sources 130) and a depth of the captured image), a depth represented by a distance from a position at which the respective image was captured relative to a plane bounded by positions A and B to a position of an object or point within the field of view of the respective image (e.g., where the distance may be a set distance that represents a depth captured by the respective image or a variable distance determined by a position of an object or point within the field of view of the respective image, such as a variable distance determined based on a range finder device, a light detection and ranging (LIDAR) device, a sound navigation and ranging (SONAR) device, a radar device, etc. that detects a distance to the object or point), an ambient light level of a location captured by the respective image, and / or the like. The images included in the training data can originate from the sensor source 130 located at position A, the sensor source 130 located at position B, and / or any of the sensor sources 130 located between positions A and B. For example, assuming that a sensor source 130 is located at position A, at position 1 that is adjacent to position A, at position 2 that is adjacent to position 1, at position 3 that is adjacent to position 2, and at position B that is adjacent to position 3, the image generator 127 can perform an initial training of the artificial intelligence model using an image captured at position A, an image captured at position 2, and an image captured at position 1 that is labeled to indicate that this image is a new POV derived from the images captured at positions A and 2. Similarly, the image generator 127 can perform a subsequent training of the artificial intelligence model using an image captured at position 1, an image captured at position 3, and an image captured at position 2 that is labeled to indicate that this image is a new POV derived from the images captured at positions 1 and 3. The image generator 127 can also perform a subsequent training of the artificial intelligence model using an image captured at position 2, an image captured at position B, and an image captured at position 3 that is labeled to indicate that this image is a new POV derived from the images captured at positions 2 and B, and so on for any number of images captured at any number of positions. The image generator 127 can perform the training in sequence as described above or concurrently.

[0193] While the above example describes the image generator 127 as using images captured from alternating positions to train the artificial intelligence model, this is not meant to be limiting. For example, the image generator 127 can use images captured at alternating positions (e.g., position A and position 2) and an image captured at an intervening position (e.g., position 1) that is labeled with an indication that the image is a new POV derived from the alternating positions to train the model, and / or can use images captured at positions that are 2 or more positions apart (e.g., position A and position 3, position A and position B, etc.) and one or more images captured at intervening position(s) (e.g., positions 1 and 2, positions 1-3, etc.) that are labeled with an indication that the image(s) are new POV(s) derived from the positions that are 2 or more positions apart to train the model. In addition, the image generator 127 can use images captured by sensor source(s) 130 at consecutive or adjacent positions (e.g., positions A and 1, positions 1 and 2, etc.) and an image captured by a sensor source 130 at a next adjacent position (e.g., position 2, position 3, etc.) that is labeled with an indication that the image is a new POV derived from the consecutive or adjacent positions to train the model. If the model is trained using images captured from consecutive positions, then the trained artificial intelligence model may be able to receive images captured from consecutive positions as an input and output a new image that appears to have been captured at the next adjacent position.

[0194] Optionally, some or all of the images in the training data set can be annotated or labeled to indicate additional information or parameters associated with the images, such as the number of pixels in the respective image, the color models of each of the pixels in the respective image, an intensity of each pixel in the respective image, an angle from which the respective image was captured relative to a plane bounded by positions A and B, a depth represented by a distance from a position at which the respective image was captured relative to a plane bounded by positions A and B to a position of an object or point within the field of view of the respective image, an ambient light level of a location captured by the respective image, and / or the like. The training data can also include image parameters and / or color models of other images other than the images captured along an axis that passes through a new POV (e.g., red, green, and / or blue values of pixels in x, y, and / or z positions in images other than images captured along the axis that passes through the new POV that correspond with (e.g., are aligned with) x, y, and / or z positions of images captured along the axis that passes through the new POV), such as images captured at different positions on the side AB or anywhere in the cube ABCDEFGH (e.g., images captured by sensorsource(s) 130 at different positions and / or angles within the cube ABCDEFGH, on different faces representing different cross sections at different angles within the cube (e.g., sensor sourcc(s) 130 located at a position parallel to ABCD, CDEF, etc.; sensor source(s) 130 located at a position not parallel to ABCD, CDEF, etc.; etc.), etc.).

[0195] To determine an angle of an image, the image generator 127 can determine an x, y, and z position of the position from which the image is captured, determine a distance between the x, y, and z position of the position from which the image is captured and a center position on a plane in which the position from which the image is captured is located, determine a percentage that the determined distance is of a total distance between the center position on the plane and an edge position on the plane (e.g., an edge of a location or scene), and estimate the angle to be the determined percentage of 90 degrees. For example, if the position of the image is 10% from a center point of a plane bounded by positions A and B (e.g., 10% of the total distance from the center point of the plane bounded by positions A and B to either position A or position B), the angle at the center point of the plane to a back face EFGH may be 90 degrees and the angle of the image may then be an angle that is 10% less than 90 degrees (or 90% of 90 degrees), which may be 81 degrees in this case.

[0196] Optionally, the images in the training data used to train the artificial intelligence model during an iteration may have been captured at about the same time (e.g., within 1ms, within 5ms, within 10ms, etc. of each other).

[0197] Once trained, the image generator 127 can apply one or more images captured by sensor source(s) 130 located at one or more positions and / or an indication of a distance from the position(s) of the one or more images at which a new image with a new POV is desired as an input to the trained artificial intelligence model, and the trained artificial intelligence model may output a new image from the perspective of the new POV as a result. For example, the image generator 127 can apply an image captured by a sensor source 130 located at position A and an indication of a distance from position A at which a new image with a new POV is desired (e.g., where the position that is the distance from position A may be position 1) as an input to the trained artificial intelligence model, and the trained artificial intelligence model may output a new image that appears to have been captured by a sensor source 130 at position 1 (but has not actually been captured by any sensor source 130). As another example, the image generator 127 can apply an image captured by a sensor source 130 located at position A and an image captured by a sensorsource 130 located at position 2 as inputs to the trained artificial intelligence model, and the trained artificial intelligence model may output a new image that appears to have been captured by a sensor source 130 at position 1 (but has not actually been captured by any sensor source 130).

[0198] Optionally, the image generator 127 can re-train the trained artificial intelligence model one or more times. The image generator 127 can use an updated set of training data to re-train the model, which can include new images captured by sensor source(s) 130, one or more images previously output by the trained artificial intelligence model that correspond to new POV(s), and / or one or more images produced by performing a pixel averaging operation.

[0199] In some cases, when trying to match images from 2, 3, or 4 corners of the face of a cube representing a space, the images may not exactly match at every pixel because the sensor source(s) 130 may not be exactly focused on the same location with precisely the same angle to capture the identical pixels representing the location (e.g., from the base face representing the cube ABCDEFGH). This slight discrepancy can be corrected by the image generator 127. For example, the image generator 127 can identify one or more objects in the one or more images, and match the different images based on the identified object(s) (e.g., align the images such that the identified object(s) are located in the same portion of each image, such as the center of the images, the bottom of the images, the top of the images, etc.). The image generator 127 can then crop the edges of the image(s) such that the image(s) represent exactly the same scenery. In addition, the image generator 127 can also match several objects, which enables the frame- shifting and matching to be more accurate. Additionally, the image generator 127 can run object tracking operations and / or artificial intelligence-based operations in this frameshifting and matching technique to accurately match multiple objects within the one or more images.

[0200] The image generator 127 can match one or more objects depicted in the image(s) by, for example, using a Fourier-domain image matching and / or stitching process to align different overlapping image segments (e.g., where each image segment may be considered overlapping if each segment depicts at least a portion of a same object). Alternatively or in addition, the image generator 127 can perform the object matching using publicly available tools, such as PtGui.

[0201] Generally, the greater the number of objects that the image generator 127 can match and the farther apart the objects are in the image(s), the more accurate the image-matching may be. After matching the one or more images based on multiple objects or features in the image(s), the image generator 127 can crop out unmatched pixels on the edges of images.Optionally, the image generator 127 can use standard image correction techniques to match images and / or to correct unmatched pixels within the images.

[0202] The image generator 127 can train an artificial intelligence model (e.g., a machine learning model, a neural network, etc.) to perform the object matching, image feature matching, and / or frameshifting to match different images. With a multi-sensor source 130 array architecture (e.g., multiple sensor sources 130 aligned in an array) in which adjacent sensor sources 130 are able to capture overlapping FOVs, the image generator 127 can produce a larger image for a particular point of view that can be applied in various use cases described herein.

[0203] The image generator 127 can also train the artificial intelligence model to perform an action that is similar to one or more standard image correction methods (e.g., flat-field correction, image geometry correction, lens correction, radiometric enhancement (e.g., image restoration, atmospheric correction, contrast enhancement, solar angle adjustment, etc.), spectral enhancement (e.g., spectral indices), spatial enhancement (e.g., focal analysis, edge detection, high pass filter, low pass filter, resolution merges, statistical filtering, adaptive filtering, texture filters, etc.), geometric enhancement (e.g., polynomial transformation, ground control points, reprojections, etc.), etc.). For example, the image generator 127 can take an image, run the image through one of the standard image correction methods to produce a corrected image, label the image with the corrected image, and include the image labeled with the corrected image in the training data. The image generator 127 can repeat this process for one or more images run through the selected standard image correction method to produce a full training data set. Training the artificial intelligence model using the full training data set may then result in an artificial intelligence model trained to produce an output that is similar to an output produced by the selected standard image correction method when receiving an image as an input. Together, some or all of the techniques described above can aid in the synthesis of a new POV from a point where no sensor source 130 is present using images capturing existing POVs and captured by existing sensor sources 130 and / or using images capturing synthesized POVs. Thus, the image generator 127 can produce an accurate and realistic image or series of images (e.g., video) from any point on any plane in the cross sections of the cube where no sensor sources 130 are present, and enable a remote viewer to visualize the audio-visual happenings of the scenery from any point and angle within the location.

[0204] As an illustrative example, the image generator 127 can match objects depicted in different images captured at different angles and by sensor sources 130 located at different positions within a location by performing one of more operations. For example, a series of sensor sources 130 may be in a straight line (or at an angle of varying sine or other functions or angular positions), with two additional sensor sources 130 on each side of a scene. The sensor sources 130 may capture objects in the scene, such as people, animals, cars, chairs, trees, lakes, fruits, vegetables, and buildings, at close-up distances and far away distances in the scenery. The image generator 127 can then use the images captured by the sensor sources 130 as training data to train the artificial intelligence model. In particular, the images can be labeled to indicate the object or group of objects depicted therein.

[0205] Once the artificial intelligence model is trained, the image generator 127 can apply one or more images captured by existing sensor source(s) 130 at one or more edges of a location or scene (e.g., where an edge may be a set distance from a middle sensor source 130 in an array of sensor sources 130, an edge of a field of view of a first or a last sensor source 130 in an array of sensor sources 130, a set distance from a first or last sensor source 130 in an array of sensor sources 130, a wall or other indoor boundary (e.g., if the location or scene is indoors), etc.) as an input to the trained artificial intelligence model, which may cause the trained artificial intelligence model to output a new image representing a new POV from a different position between the existing sensor sources 130. Alternatively or in addition, the images that the image generator 127 provides as an input to the trained artificial intelligence model can be captured from any number of corners of the location or scene (where the scene may be bounded by a shape, such as a straight line, square, pentagon, hexagon, circle ellipse, etc.). The new POV can be from different points within the “front face” or the face from any cross-section of the cube or other shape that bounds the location or scene (e.g., faces that are not parallel to the image of the location or scene). The edges of the shape that bounds the location or scene can be defined by the left (or right) edge of an image captured by a first sensor source 130 in an array of sensor sources 130 at the location or scene, the right (or left) edge of an image captured by a last sensor source 130 in an array of sensor sources 130 at the location or scene, a plane that intersects and is parallel to a depth of the images captured by the sensor sources 130 in the array, and a plane that intersects and is parallel to a position of the sensor sources 130 in the array.

[0206] In further embodiments, the image generator 127 can generate images depicting new POVs in situations in which a limited number of sensor sources 130 arc present in a location or scene. For example, the sensor sources 130 may not be a series of sensor sources 130 that are separated by specific distances, such as distances that would result in images being captured at a specific frame per second (e.g., 24 frames per second, 30 frames per second, 60 frames per second, etc.) when a remote user is moving at a certain speed (e.g., 30 mph, 60 mph, etc.) through the location or scene. Instead, one or more sensor sources 130 may be positioned at substantial distances, such as one every 1, 2, 3, etc. feet or meters.

[0207] The image generator 127 can generate new images depicting new POVs from images captured by existing sensor sources 130 using different methods, including pixel averaging, artificial intelligence, object tracking, and / or global positioning system (GPS). In each of these methods, one or more sensor sources 130 can be placed at short to long distances from each other. In pixel averaging, the image generator 127 can average the pixels from two or more images captured by two or more sensor sources 130 that are positioned to capture two or more POVs to produce a new image with a new POV, or can average the pixels from an image captured by a sensor source 130 and an image synthesized by the image generator 127 to produce a new image with a new POV.

[0208] In artificial intelligence, the image generator 127 trains one or more artificial intelligence models to produce a new image from a new POV when provided one or more images captured by existing sensor source(s) 130 and / or one or more images previously synthesized by the image generator 127 (or another system) as an input. For example, the image generator 127 can train the artificial intelligence model(s) using data from images captured by actual sensor sources 130 and / or images synthesized from images captured by actual sensor sources 130. Optionally, the training data may include images captured by actual sensor sources 130 that are positioned adjacent to (e.g., within 1, 2, 3, feet or meters) of a new POV for which a new image is requested. The sensor sources 130 that captured images used to train the artificial intelligence model(s) may be 0.1 inch, 1 inch, 1 foot, 100 feet, 1000 feet, etc. apart from each other.

[0209] The image generator 127 can also use an object tracking system (OTS) to produce a new image from a new POV using an image from a preexisting POV captured by an actual sensor source 130. The image generator 127 can use OTS to predict the change in different parameters of a moving object, such as the positions, angles, and lighting, from pre-existing dataof the movement of the object in a location. Here, the image generator 127 can use multiple preexisting images (such as in a video) to predict an image from a new POV.

[0210] As described herein, a remote user can provide an input to select an object to follow at a location or scene. Thus, the new images from new POVs produced by the image generator 127 can be presented to the remote user (e.g., automatically or in response to a user request) to show the remote user the location or scene from different POVs even if sensor sources 130 that are able to capture a particular POV do not exist in the location or scene.

[0211] The image generator 127 can also use GPS to produce new images from new POVs. For example, GPS may be capable of tracking an object within a location (e.g., based on a transponder coupled to the object, based on a detected signature of the object, etc.). Using the preexisting GPS data for various objects within the location, the image generator 127 can predict new GPS locations of the objects using one of several methods, including predictive algorithms and / or trained artificial intelligence models (e.g., models trained to predict a path that an object may take based on previous GPS locations of the object, where the previous GPS locations of the object can be provided as an input to the trained models to cause the trained models to output a new GPS location of the object).

[0212] In any of the above-described methods, the image generator 127 can produce a new image from a new POV using one or more pre-existing images and an algorithm that takes into account changes to various parameters, such as changes in the features of the pixels of a preexisting image by a percent value (e.g., 0.1%, 1%, 5%, 20%, 90%, etc.) based on the distance of the POV of the new image from the POV of the pre-existing image (e.g., where the percent value change in the features may be determined by the image generator 127 based on training an artificial intelligence model to predict the percent value change based on being trained with images taken from different POVs). For example, the image generator 127 can train the artificial intelligence model using sets of training images, where one training image in a set may be an image taken from a first POV and a second training image in the set may be an image taken from a second POV (and may be labeled as such). Both images in the set may capture at least a portion of the same content, with the red, green, and / or blue pixel values of the pixels that depict the same content being different in the two images (e.g., given changes in lighting captured from capturing the same content from different POVs). Thus, during the training, the artificial intelligence model may learn how the red, green, and / or blue pixel values change from one POV to another POV, such as thepercent value by which the pixel values change. The algorithm can change based on several different parameters, such as the distance or angle of the objects or lighting of the environment in the location or scene. The image generator 127 can also use an algorithm that compares the change in the parameters of the pre-existing image, such as the color models of the pixels, to those in the new image based on some specialized functions, such as a spline function, depending on the angle, lighting, distance and / or other parameters of the new image from the new POV.

[0213] Any of the above methods of synthesizing new images from pre-existing images enable the use of even a limited number of sensor sources 130 to be placed around a location, instead of a series of closely-spaced sensor sources 130, and allow for the presentation of a series of images (e.g., either individually or in the form of a video) from the same POV or from different POVs to be presented to a remote user. The remote user can move around in the location or scene in any direction speed, and / or angle of travel, thus enabling the complete seamless travel of a remote user within a location even if a limited number of sensor sources 130 are present in the location.

[0214] In response to a remote user requesting to view a location or scene, the image generator 127 can cause the remote user to see images captured by existing sensor sources 130 and / or images synthesized by the image generator 127 using any of the techniques described herein, where the image generator 127 can select the appropriate image based on the speed, angle, and / or direction of the movement of the remote user. The image generator 127 can cause the remote user to see the images in real-time (e.g., within a few seconds of the remote user requesting to view a location or scene and / or where the images are captured within a few seconds of the remote user request or synthesized based on images captured within a few seconds of the remote user request) so that a realistic experience is presented to a remote user in real-time.

[0215] In another embodiment, for viewing the inside of a space like mall or store from the sides (e.g., walls) or viewing the floor from the ceiling, the following method can be implemented by the image generator 127 to produce a seamless, continuous visualization from sensor sources 130 positioned linearly at substantial distance (e.g., 10 feet apart, 20 feet apart, etc.). First, the image generator 127 may obtain an image for a larger area (with a larger field of view) by combining two or more consecutive images (e.g., where consecutive images may be images captured by adjacent sensor sources 130 and / or images that capture overlapping content) captured by distantly positioned sensor sources 130 (e.g., using Fourier domain method). The distantlypositioned sensor sources 130 may each capture an image of the scene positioned along a same plane and in generally the same direction (e.g., where the angles of the lens of each sensor source 130 may be within a threshold angle of each other, such as within 1 degree of each other, 3 degrees of each other, 5 degrees of each other, etc.). A portion of the two or more consecutive images may capture the same content, but the color values of a pixel in the consecutive images that captures the same content may vary given the different POVs from which the sensor sources 130 may have captured the images. To create the combined image, in moving from one sensor source 130 to the next sensor source 130 in short successions, the image generator 127 can delete a small region (e.g., 2%) from an edge of a portion of the image captured by the respective sensor source 130 that depicts the same content as an image captured by the next sensor source 130 (e.g., crop a portion of the image captured by the respective sensor source 130), and add a small region (e.g., 2%) from an edge of a portion of the image captured by the next sensor source 130 that depicts the same content as the image captured by the respective sensor source 130. In addition, the image generator 127 can use a function to tilt a portion (e.g., the top 2%) of the image captured by the respective sensor source 130 by a predetermined or user-selected angle (by using a standard function or formula). Optionally, the image generator 127 can use a larger image to cover a portion (e.g., the next about 10%) of the image captured by the respective sensor source 130 so that the pixels from the image captured from the same sensor source 130 can be used in these pixel manipulations. This process enables the movement of the remote user to the next image as viewed from the position of a sensor source 130 placed at close proximity (e.g., 1 cm, 0.1 cm, etc.), which is presented from the synthesized image. In this manner, the image generator 127 can synthesize several consecutive images between the distantly placed sensor sources 130. When these consecutive images are presented to a remote user, the remote user can experience a seamless movement within the mall space as if the remote user were receiving the images from actual sensor sources 130 located consecutively between the two distantly spaced sensor sources 130.

[0216] When the sensor sources 130 are arranged in a circular manner, for example around a room or hall in a shop, then the image generator 127 can use an angular movement of the image (e.g., an angle at which the image was captured compared to a baseline angle assigned to one of the sensor sources 130) such that the subsequent image is tilted at an angle (e.g., matching a circular or elliptical or other shapes). The image generator 127 can calculate precisely these percentages and angles using corresponding mathematical formulas.Artificial Intelligence Trainer

[0217] Provided herein arc embodiments for training of an artificial intelligence (Al) and machine learning system. Specifically, training of the ML system to perform object tracking of an object residing within a space, the object traveling at a fixed, or variable speed, with fixed, or variable acceleration, and with fixed, or variable changes in direction. In some embodiments, the Al system is capable of automated selection of one or more tracked objects, traveling at variable speeds, directions, accelerations, or orientations.

[0218] FIG. 17 is a flow diagram depicting an example training dataset input workflow for an exemplary Al system of the present embodiment. For example, a system implementing the machine learning workflow 1700 may be the virtual render and object tracking trainer 122 of FIG. 1. The machine learning workflow 1700 begins at block 1710, wherein a training data set is input into an embodiment of the present disclosure. In some embodiments, the training data set are objects commonly found within a predefined space. Where the predefined space is a street within a city, for example, training data in the form of automobiles, clothing, pedestrians, or traffic signals may be provided to be trained. A training data set may take the form of, for example, multiple images of T-Shirts, where the space is expected to encompass a public pedestrian gathering place. In some embodiments, the training data set can comprise expected travel characteristics, including expected velocity, acceleration, position, or direction of travel. A training data set showing traffic flow within a scene for example, can provide initial travel directions and velocities of individual cars to an exemplary machine learning trainer as described herein.

[0219] At block 1720, the system loads one or more frames from a plurality of cameras according to an embodiment described herein.

[0220] At block 1730, the system then determines and recognizes an object’s presence within the retrieved one or more frames. For example, a training dataset of T-shirts as described herein and trained into an exemplary model may allow for an object detection system to recognize all instances of T-Shirts within any given frame. Therefore, the absence or presence of an object may be determined in part by image recognition from a trained machine learning module. In some embodiments, additional optional modules, include an optical character recognition (OCR) may be implemented to recognize and parse particular features within any given frame. For example, a16T-Shirt may be tracked based on not only parameters as defined in a machine-learning model, but also tracked based on positive recognition of a text phrase that appears on the T-Shirt.

[0221] At block 1740, the system proceeds to track the object’s velocity, acceleration, position, or direction of travel. In some embodiments, object tracking can further approximate movement of tracked objects when objects leave a particular field of view of a camera, or are otherwise occluded in a scene. Therefore, an object traveling with a consistent direction and / or velocity can be accurately predicted to re-emerge at a particular location. In some embodiments of the present system, the movement of an object being followed by a user can be captured using a series of cameras instead of a single camera, where the series of cameras have differing fields of view. A tracked object can therefore be continuously tracked by a series of cameras in a real space where suitable sensors are present.

[0222] At block 1750, the object is selected for tracking. In some embodiments, the selection is accomplished by a user indicating with an input device. In some embodiments, the selection is accomplished automatically by the system. For example, a scene captured by an exemplary system including a single car, wherein all other tracked objects are fixed or otherwise relatively static, can be automatically selected for tracking according to an embodiment of the invention herein. In some embodiments, automatic selection can involve pre-set criteria, including velocity, color, direction, acceleration, or type of object recognized within the scene.

[0223] In some embodiments, one or more elements of the location viewing system 120 may utilize machine learning methods to enhance functionality. For example, selection of cameras corresponding to a user directed pace of travel. In some embodiments, a machine learning system is provided to train a model for selecting cameras at certain positions, when a user is moving at a constant velocity in a constant direction, viewing a scene at a constant or variable angle. In some embodiments, a machine learning system is provided to train a model for selecting cameras at certain positions when the user is increasing or decreasing velocity. In some embodiments, a machine learning system is provided to train a model for selecting cameras at certain positions when the user is increasing or decreasing acceleration. In some embodiments, a machine learning system is provided to train a model for selecting cameras at certain positions when the user selects a pace of travel that has a variable velocity. In some embodiments, a machine learning system is provided to train a model for selecting cameras at certain positions when the user selects a pace of travel that has a variable acceleration. In some embodiments, a machinelearning system is provided to train a model for selecting cameras at certain positions when the user selects multiple directions of travel consecutively within a location. In some embodiments, the system is configured to anticipate likely user input from any user input device. Once trained, the system described herein can apply as an input a direction, angle, speed, and / or the like in which the user would like to move, and the trained model may output an indication of an image captured by a particular camera to cause the user device 102 to display in response. The system described herein can then cause the user device 102 to display an image captured by the particular camera.Emoji Placement

[0224] In some embodiments, user generated icons, avatars, or emoji may be placed within a representation of the real space, such that a user may view and optionally, interact with said icon, avatar, or emoji. In some embodiments, icons, avatars, or emoji may be implemented to track specific objects identified by the system, including cars, retail items, and users. In some embodiments, a system of the present disclosure may be configured to display emoji characters in representations of the real space. Where multiple users are accessing a space, and intend to visualize other users within the space, the system may assign an icon or user avatar to represent each individual user. In some embodiments, the icons may allow for users to easily access and view user information, or metadata pertaining to specific scenes. In some embodiments, user movements and additional inputs may be represented by the user icons, avatars, or emoji. For example, a user that inputs a desired change in position or orientation (e.g. moving from left to right) to view a scene from an alternate perspective, may have an icon representing them within the virtual space move in a corresponding manner (e.g. moving from right to left). A user icon, avatar, or emoji, may also be configured to express user output, including gesture, voice, head tilt, body orientation, body position, body shape, and the like. Thus, a user being tracked using any input device, may be represented to other users within a virtual space A remote user using an embodiment of the present disclosure may be represented by an icon, avatar, or emoji within any space wherein the present system is installed. Embodiment of the system allow for meetings of multiple users, and in some embodiments, converse and travel a space together.

[0225] In some embodiments, the system may be further configured to provide haptic feedback. For example, a user wearing a vest with moveable haptic elements may perceive “pushes” or other physical forces, when interacting with select elements within a virtual space. In someembodiments, the system may be further configured to animate contextual movements. For example, a user moving from point A to point B may input a request to the system to travel to point B, wherein the system moves an icon, avatar, or emoji representing the user to point B - depending on the context and pre-programmed system rules, the icon can be observed by other users as walking, jogging, running, swimming, kayaking, or any other ambulatory motion, to move from Point A and arrive at Point B. As can be appreciated, the system described in the present disclosure may be implemented in a variety of settings, including amusement parks, amusement rides, walking, jogging, biking, or to tour hazardous or industrial locations.Self-Contained Camera Strip with Electrical, OTS, Network, and Other Connections

[0226] A self-contained system described herein may include different devices of cameras, lenses, OTS, sensors, and / or other devices that are connected together by appropriate connections, such as in a cassette model. For example, a cassette unit as used herein may refer to a standalone set of a series of sensor sources 130 and / or the other devices described herein (e.g., electrical connections, network interfaces, controller units, etc.) that can operate in a manner as described herein without any further devices or components. The cassette unit length may be designed such that one cassette unit may be coupled to another cassette unit to make a longer sensor source 130 series system (e.g., longer camera series system). For example, one end of a housing of the cassette unit may be coupled to an outlet, plug, or other input / output port that can be mated to a plug, outlet, or other input / output port coupled to the other end of the housing of the cassette unit. Thus, one end of a first cassette unit can be coupled to the other end of a second cassette unit to form a longer series of sensor sources 130 (e.g., where the longer series of sensor sources 130 may include sensor sources 130 comprised within each cassette unit).

[0227] The series of sensor sources 130 can be connected electrically and / or electronically as well as to a network 110 (e.g., a publicly-accessible network like the Internet, a private network, a combination of a private and public network, etc.) via a network interface housed within the cassette unit to enable the operations of capturing audio, images, video, and / or other sensor information and transmitting the captured sensor information to the location viewing system 120 and / or the user device 102 via the network 110. The self-contained system can include a strip (e.g., cassette unit) of electrical and / or electronic connections linking one or more sensor sources 130, and connecting the sensor sources 130 with one or more network interfaces (e.g., awired network interface, a wireless network interface, etc., where each network interface can handle transmissions using an appropriate network protocol, such as an Ethernet protocol, an IEEE wireless protocol, etc.). The strip can encase the sensor sources 130 in, for example, Teflon tubing. Thus, 1, 10, 100, or 1000s of sensor sources 130 can be connected in this self-contained cassette unit. The individual sensor sources 130 in a cassette unit can be connected in several ways electrically, electronically, optically, and / or with the network interface(s) to fulfill various objectives. The sensor source 130 strip can be segmented into independent units that each include a controller unit for controlling the electronic and network operations. Each segment can be a certain length (e.g., 8 feet, 9 feet, 10 feet, 11 feet, etc.), and each segment can be coupled to the next segment through appropriate electrical, electronic, and / or network connections.

[0228] The various devices within a cassette unit and their corresponding connections can be combined within a transparent self-contained casing such that the casing could be placed in various places, heights, and / or angles in a location. One or more cassette units can be used in conjunction to be pail of various series of sensor sources 130. Because the sensor sources 130 can be very small, the encasing of the self-contained system can be small so that the strips can be easily placed in different places, such as a shopping mall, store, underwater, theme park, conference hall, and also in scientific, medical, and / or manufacturing systems. The strips can be transparent and constructed in such a manner to allow for the optical, electronic, and / or network connections to be seamless and integrable with other types of systems. When the strips are placed in close proximity to accommodate multiple series of sensor sources 130, the image generator 127 can reduce or eliminate obstruction by neighboring strips by using image processing techniques and / or artificial intelligence.

[0229] The strip can be arranged in a circular manner with one or more connections traversing the diameter of the circular shape (where the connections can include electrical components to couple the different sensor sources 130 and / or independent units) to form a design that resembles a spoke. For example, a first sensor source 130 may be located at about 0 degrees of the circular shape (e.g., within 1 degree of 0 degrees, within 2 degrees of 0 degrees, etc.) and a second sensor source 130 may be located at about 180 degrees of the circular shape (e.g., within 1 degree of 180 degrees, within 2 degrees of 180 degrees, etc.). A connection can traverse the diameter of the circular shape to couple the first sensor source 130 with the second sensor source 130.

[0230] Some or all of the sensor sources 130 and / or other systems can be coupled in various ways, such as around a middle connecting string (that provides electrical, electronic, and / or network connectivity). The sensor sources 130 may be housed within a transparent, flexible tube, such that the strip casing can be flexible and pliable, or rigid like a glass casing. The sensor sources 130 and / or lens therein can be arranged in 360 degrees within the tube. Thus, the flexible sensor source 130 tubes can house hundreds or thousands of sensor sources 130 that can be placed around a location or scene, such as a mall or street in any desired angle or height, or any other engineering, medical, and / or manufacturing systems. The data captured by the different sensor sources 130 can be stored centrally in the cassette device itself (e.g., in memory), such that the cassette device is a self-contained independent unit that includes a processor, memory, and / or similar computing components. The data captured by and / or computing components of independent units can be coupled to larger devices in a local hub or a cloud-based or centralized system (e.g., the location viewing system 120) to allow remote users access to such data.

[0231] Microlens or ultrasmall cameras also can be incorporated into the cassette unit. Just as the neuronal wiring behind the thousands of ommatidia in a dragonfly compound eye that are connected to its brain, a microlens camera can be wired from one or more microlenses to produce one or more images and be coupled to the location viewing system 120 via the network 110. The images can be processed by the location viewing system 120 in a manner as described herein to produce one or more points of views that a remote user is intending to view at a point in time and space. Ultrasmall cameras and lenses (described below) can also be incorporated into a cassette unit.

[0232] FIG. 19 illustrates an example strip or cassette unit 1900. For example, the strip 1900 may be a camera strip. As illustrated in FIG. 19, the strip 1900 may include one or more sensor source units 1902 that are each positioned within a housing of the strip 1900. For example, the sensor source units 1902 may be spaced a certain distance apart from each other. In some embodiments, each sensor source unit 1902 is an equal or nearly equal distance from the two sensor source units 1902 that neighbor the respective sensor source unit 1902.

[0233] Each sensor source unit 1902 may include one or more sensor sources 130. Each sensor source 130 may be positioned to face an exterior of the housing of the strip 1900. For example, in an embodiment in which the sensor sources 130 are cameras, the lens of each cameramay be positioned to face an exterior of the housing of the strip 1900. Thus, each sensor source 130 may be positioned to capture sensor data that originates external to the strip 1900.

[0234] Each sensor source unit 1902 may also include a processing unit 1904. Each sensor source 130 may be coupled to the processing unit 1904 via an electrical, optical, or wireless connection. The processing unit 1904 is further described below with respect to FIG. 20.

[0235] In an embodiment, the sensor source unit 1902 can be arranged in a two- dimensional or three-dimensional circular or oval shape, and the sensor sources 130 may be positioned along an interior side of an outer edge of the shape and face a direction away from an interior of the shape. Optionally, the sensor source unit 1902 may include a housing along the interior side of the outer edge of the shape within which the sensor sources 130 are located. The processing unit 1904 may be positioned in an interior of the shape and be coupled to each of the sensor sources 130.

[0236] FIG. 20 illustrates a detailed view of the processor 1904 of the strip 1900. As illustrated in FIG. 20, the processor 1904 may be a digital signal processor. The processor 1904 may include or be coupled with memory 2006, an internal interface 2008, and a transceiver 2010. The memory 2006 can store sensor data captured by the sensor sources 130. The internal interface 2008 may couple the processor 1904 to the sensor sources 130, such as via an electrical, optical, or wireless connection. The transceiver 2010 may be a wired or wireless transceiver that is configured to communicate with a component external to the sensor source unit 1902 (e.g., via Ethernet, Wi-Fi, Bluetooth, Zigbee, cellular, etc.). For example, the transceiver 2010 may communicate with another sensor source unit 1902, a router, a gateway, the location viewing system 120, and / or the like.Ultra Small Cameras and Lenses

[0237] Lens and camera technologies have advanced highly to the extent that a variety of camera and lens systems can be used herein. For example, a three-lens camera (e.g., a compound lens with at least three lenses) has been built that couples to (e.g., fits onto) the end of an optical fiber the width of two hairs by using 3-D printing. 3-D printing, which is also known as additive manufacturing, makes three-dimensional objects by depositing layer after layer of materials, such as plastic, metal, or ceramic. The compound lens can be just 100 micrometers (e.g., 0.1 millimeters or 0.004 inches) wide, and 120 micrometers with the casing of the compound lens.The compound lens can focus on objects at least a distance of 3.0 mm away from the compound lens, and relay the captured images over the length of a 1.7 meter (c.g., 5.6 foot) optical fiber to which the compound lens is coupled.

[0238] The optical fiber may be coupled to a network interface (e.g., a wired or wireless transceiver) so that images captured by individual lenses of the compound lens can be transmitted to the location viewing system 120 or other system for storage and / or processing. For example, the network interface may be coupled to one end of the optical fiber and the compound lens may be coupled to another end of the optical fiber (e.g., the network interface and the compound lens may be configured to different ends of the optical fiber). Each lens of the compound lens may be configured to transmit a captured image from the respective lens to the network interface via the optical fiber.

[0239] This type of tiny eyes yields high optical performance and tremendous compactness. Multiple compound lenses can also be printed onto image sensors other than optical fibers, such as those used in digital cameras. Using 3-D printing, micro and mini lenses of varying sizes can be built that are coupled to optical fibers or other types of image sensors and transmitters. Thus, a continuous strip of photosensing devices (e.g., a continuous strip of individual compound lenses) can be used by the location viewing system 120 to obtain images or frames instead of a series of sensor sources 130. A continuous strip of photosensing devices can be incorporated into the cassette units described above such that the cassette units can be placed in any location, including underwater and in the air, and be used in a variety of applications.

[0240] This type of device can have multiple lenses for each sensor source 130 positioned at multiple angles. For example, one compound lens may represent a single sensor source 130, where the individual lenses of the compound lens may face different angles to capture a scene from different POVs. In addition, lenses and sensor sources 130 of varying sizes with different focus lengths can be built. Optical, electrical, and electronic wiring of individual lenses and sensor sources 130 for network connectivity can be likewise built into the device. For example, an optical, electrical, or other type of connection can couple a first lens of the compound lens with a second lens of the compound lens. These types of devices can be used as a camera, lens, and / or other sensor systems. Thus, lenses and cameras with a wide range of resolution and field of view (FOV) can be used by the location viewing system 120 for various applications.

[0241] A multi-camera array architecture can also be created, where overlapping fields of view from adjacent cameras can produce a larger image for a particular point of view. The matching or stitching of the overlapping images can be achieved by the image generator 127 using several methods, such as the Fourier-domain image matching / stitching method, to align different overlapping image segments. The image generator 127 can also achieve the matching or stitching using commercially available software, such as PtGui.Television or Computer-Like Headset for Viewing Locations or Sceneries

[0242] Typical virtual reality (VR) headsets present a view that does not seem to be truly realistic. For example, there is a certain amount of warping of the images that are created by the hardware and software of the headset. In contrast, described herein are techniques that result in more realistic images and / or videos being displayed to a viewer.

[0243] For example, described herein is a method to provide a television-like screen within a headset (e.g., a VR headset), where the pixels are projected as if on a television screen. The headset can use a magnifying device to magnify the image on the screen such that the viewer can visualize and experience a larger projected image (e.g., to view an image as if the image is several feet wide). Just as the binocular vision of the human eyes is able to produce a seamless image, the projected images from the magnifying lenses can be adjusted by the headset to produce a single larger image seamlessly. The headset may present an image to a user on a screen having a square or rectangular shape and having a height of about 1 inch or 1 centimeter. The magnifying device may be positioned in front of the screen (e.g., from the perspective of the user, where the magnifying device may be between the screen and an eye of the user) such that the magnifying device can magnify the images depicted on the screen. Optionally, the headset may include two screens and two magnifying devices for each screen, where each screen and magnifying device pair is positioned such that each screen and magnifying device pair is in front of a user’s eye. Alternatively, the headset may not have any screens. Rather, the headset may include one or more projectors that each project images to be displayed to the user directly to a magnifying lens (e.g., where the projection may occur in a direction that the user is facing or in the opposite direction that the user is facing). Depending on the focal length of the lenses, the size of the projected image can be small or large. The headset can also use micropixels and / or microimage capture and projection devices.

[0244] The user can wear a headset that can both obtain the sensory inputs from the user, and also receive and present the audiovisuals to the user from the remote space in which the user is traveling. It is also possible for the headset to project the images on the eye glasses that can be worn by a user. In addition, the viewing angle can be adjusted to display the normal human eye system of -180 degrees without distortion or warping. The headset is designed to integrate the nuances of the different types of audio visualization from the location or scene in such a manner to present a realistic immersive visualization of the remote location, with all of the happenings of that location. For example, the headset may include one or more speakers that can output audio captured at the location or scene.

[0245] The headset can use gyroscopic and / or angular response (e.g., the headset can include a gyroscope or other sensor that can detect movement of a user), such that when a user turns his or her head in different angles (e.g., up, down, left, or right), the view will correspondingly change to present the corresponding audiovisual scenery in the remote location. For example, a processor housed within the headset can execute computer-executable instructions that, when executed, cause the processor to obtain a measurement from the gyroscope or other sensor and determine a direction in which the user moved his or head using the measurement. Once determined, the computer-executable instructions, when executed by the processor, may cause the processor to cause transmission of the determined direction to the location viewing system 120 via a network interface (e.g., a wired or wireless transceiver) housed within the headset. The location viewing system 120 can then obtain an image captured by a sensor source 130 that is positioned or angled at a location that corresponds with a direction in which the user moved in a manner as described herein. The location viewing system 120 can then transmit the obtained image to the processor of the headset via the network interface, and the computer-executable instructions, when executed by the processor, may cause the processor to instruct the screen(s) of the headset to display the obtained image or to instruct the projector(s) of the headset to project the obtained image. A wide-angle (e.g., 16:9 ratio, 21:9 ratio, etc. image of a scene or location, where the image may have been captured at a narrower ratio, such as 4:3 ratio, 16:9 ratio, etc., respectively) or zoomed view (e.g., a 2x, 4x, lOx, 20x, etc. optical or digital zoom level of a captured image, where the image may be captured at a lx, 2x, 4x, 8x, etc., optical or digital zoom level, respectively) can also be presented by the headset as the user desires (e.g., by the headset adjusting a focal length of the magnifying lens(es)). Using the coverage for a wider angle on the viewing screen will providea larger view of the location for a remote user. The headset can also provide a curved projection screen, such as at a 5 degree, 10 degree, 30 degree, etc. angle (c.g., where a normal, uncurvcd projection screen may be at an angle of 0 degrees). Thus, the headset can enable the wider angle such as 180 degrees or wider, instead of a smaller angular coverage of the scenery in the location.Use Case - Shopping with Family and Friends

[0246] Typical software applications for online shopping have several deficiencies. For example, typical software applications do not allow multiple people, such as family members and friends, to shop together online, seeing each other, conversing with each other, moving through the mall or shops together, and / or purchasing things together. Described herein is a use case in which family, friends, and / or other users are able to shop together remotely.

[0247] For example, the location viewing system 120 enables one or more remote users to move around a mall or store and visualize the items in a store or a mall, and for multiple users to connect together and move around the store or the mall and shop together. The different users can move around the store independently or in a group in a synchronized manner while shopping.

[0248] In this embodiment, multiple people can connect with each other remotely and see a map and / or list of locations when the user device 102 is booted and an application used to view remote locations is launched. Within the application, the users can select a city from among a plurality of cities (e.g., via a map or list), and / or select a mall from a plurality of malls available in the selected city (e.g., via a map or list). Once a mall is selected, each user may see a view of the mall and / or avatars or other icons representing the other users. The users can opt to move around the mall together while visualizing and conversing among themselves. For example, the users can go to a store they desire, look around at the items in the store, and / or purchase items from the store or the mall.

[0249] The shopping location can be visible to the user at the center of a user device 102 screen or a VR headset, while the one or more remote users will be visible to each other on one or more sides, below, or above the visual screen at the center. These users can view each other and converse among themselves, similar to a video chatting application. The users can view the remote location, such as a city, mall, or a store in the center of the user device 102 screen. As a starting point within the mall, the application can show a map of the type of stores in the mall, or a list of the types of items in the store or the mall.

[0250] The shopping travel can start with a location in the mall or store that is initially displayed on the user device 102 screen. Once the users choose a store, the application can automatically lead the user to the store (e.g., by showing images along a path from the starting location to the store). One of the remote users in a group may be assigned a lead role. The lead role user may navigate the shopping for the group. Once the users enter the store, the application may display the types of items at the center screen (e.g., in a list format). The lead role user or other users can then choose a type of item, and the user device 102 display may then switch to showing an image of the actual location in the store displaying the type of item captured by one or more sensor sources 130 that are pointing to the type of item. The lead role user can move around the type of item in real-time, or view the type of item from a stored set of images and / or videos. The stored images and / or videos can be viewed by the users by clicking on buttons with labels indicating stored images and / or videos. Once the lead role user or the other users choose an item, the sizes and colors of the item can be viewed in the user device 102 display. The lead role user or other users can then choose the item of their choice and place an order for the selected item.

[0251] Alternatively or in addition, when a type of item is chosen by the user, the sensor sources 130 pointed at the type of item can capture images of the aisle in which the type of item is present, and the location viewing system 120 can cause those images to be displayed in the user device 102 display. The lead role user or other users can move around by clicking on arrow buttons on their respective user device 102. The user can also wear a VR headset or gloves, and can point to the direction in which the user would like to move. In response, the application may cause the visual field or FOV to move accordingly (e.g., the user device 102 display can then display the moved visual field or FOV). The application may also allow users to converse with the store staff. For example, a user can click a button or select another user input provided on the user device 102 screen that corresponds with conversing with store staff. In response, an utterance spoken by the user that is captured by the user device 102 may be relayed by the user device 102 to a speaker present in the store (e.g., via the location viewing system 120) so that the utterance can be heard by the store staff. Similarly, a microphone present in the store may capture an utterance spoken by the store staff and the microphone may relay the spoken utterance to the user device 102 (e.g., via the location viewing system 120) so that the utterance can be heard by the user. As an illustrative example, the user can ask the store staff to display an item for the user to see, such as by picking up an item and moving the item closer to a sensor source 130 present inthe store. As another illustrative example, the user can negotiate a price for an item with the store staff.

[0252] The application running on user devices 102 may allow multiple people from remote locations to interact with each other while shopping together. Multiple mouse pointers visible on each user device 102 screen (color coded or initialed), one for each user, can show other users who is doing what in the store. The operations of multiple users can either be controlled by one leader of the group, or each of the users can shop independently, and the whereabouts of each user can be shown to the other users in the user device 102 screen.

[0253] Shopping can be performed by the users as a group, in sub-groups, or independently. In group shopping, some or all of the users who are in the video conference can be depicted on the same page. Some or all of the users may have the privilege to move to other categories of items and other users in the group may be able to watch the user(s) as those user(s) move to another location in the store that has the other category of items. Similarly, different combinations of sub-groups can be formed during the group shopping (e.g., sub-groups can be formed and re-formed during a single shopping session via the application), and each sub-group may have the ability to see different items in the same store or in different stores at the same time.

[0254] As an illustrative example, if there are 10 users in the video conference who are shopping online, all 10 users may be able to view one item in a store. If one user wants to look at a different item, that user can cause the user device 102 to show images that show the user navigating to another part of the store or another store that has the different item. At this stage, some or all of the users may see a split screen in which one screen shows the originally viewed item and the other screen shows the different item. If a second user wants to look at a third item, the same process may follow and may result in a three-part split screen being shown to some or all of the users, and so on.

[0255] In this mode, in one user device 102 screen there may be multiple mouse pointers displayed (e.g., initials of individual users, color codes for each user, or emojis corresponding to each user, pictures corresponding to each user, etc.), one for each user. Each user can opt to view all items being viewed by all users in the group, a subset of all items being viewed by all users in the group, or only the item that the respective user is viewing. Whether a user would like to shop in a group, in a sub-group, or independently, and / or whether the user wouldlike to see some or all of the items being viewed by other users can be customized by each user via the application.

[0256] The operations and functionality of the shopping use case described herein may be performed in part or in whole by the user device 102. For example, the application running on the user device 102 may perform the operations described independently without direction from another system, such as the location viewing system 120. The location viewing system 120 may still provide images, audio, and / or other sensor data to the user device 102, but in response to requests from the user device 102. Alternatively, the location viewing system 120 may instruct the user device 102 to perform some or all of the shopping use case operations described herein. For example, any inputs received by the user device 102 can be relayed to the location viewing system 120. The location viewing system 120 can then process the inputs and instruct the user device 102 to perform specific operations corresponding to the inputs in response. In addition, the location viewing system 120 may provide any images, audio, or other sensor data that the user device 102 may display, play, or otherwise present to the user. In either example, the location viewing system 120 can function as a relay, transmitting data between the user device 102 and the speakers, microphones, or other sensor sources 130 present at the mall.URLs and Cookies

[0257] In some embodiments, one or more features of the systems, methods, and devices described herein can utilize a URL and / or cookies, for example for storing and / or transmitting data or user information. A Uniform Resource Locator (URL) can include a web address and / or a reference to a web resource that is stored on a database and / or a server. The URL can specify the location of the resource on a computer and / or a computer network. The URL can include a mechanism to retrieve the network resource. The source of the network resource can receive a URL, identify the location of the web resource, and transmit the web resource back to the requestor. A URL can be converted to an IP address, and a Domain Name System (DNS) can look up the URL and its corresponding IP address. URLs can be references to web pages, file transfers, emails, database accesses, and other applications. The URLs can include a sequence of characters that identify a path, domain name, a file extension, a host name, a query, a fragment, scheme, a protocol identifier, a port number, a username, a password, a flag, an object, a resourcename and / or the like. The systems disclosed herein can generate, receive, transmit, apply, parse, serialize, render, and / or perform an action on a URL.

[0258] A cookie, also referred to as an HTTP cookie, a web cookie, an internet cookie, and a browser cookie, can include data sent from a website and / or stored on a user’s computer. This data can be stored by a user’s web browser while the user is browsing. The cookies can include useful information for websites to remember prior browsing information, such as a shopping cart on an online store, clicking of buttons, login information, and / or records of web pages or network resources visited in the past. Cookies can also include information that the user enters, such as names, addresses, passwords, credit card information, etc. Cookies can also perform computer functions. For example, authentication cookies can be used by applications (for example, a web browser) to identify whether the user is already logged in (for example, to a web site). The cookie data can be encrypted to provide security for the consumer. Tracking cookies can be used to compile historical browsing histories of individuals. Systems disclosed herein can generate and use cookies to access data of an individual. Systems can also generate and use JSON web tokens to store authenticity information, HTTP authentication as authentication protocols, IP addresses to track session or identity information, URLs, and the like.Terminology

[0259] All of the methods and tasks described herein may be performed and fully automated by a computer system. The computer system may, in some cases, include multiple distinct computers or computing devices (e.g., physical servers, workstations, storage arrays, cloud computing resources, etc.) that communicate and interoperate over a network to perform the described functions. Each such computing device typically includes a processor (or multiple processors) that executes program instructions or modules stored in a memory or other non- transitory computer- readable storage medium or device (e.g., solid state storage devices, disk drives, etc.). The various functions disclosed herein may be embodied in such program instructions, or may be implemented in application-specific circuitry (e.g., ASICs or FPGAs) of the computer system. Where the computer system includes multiple computing devices, these devices may, but need not, be co-located. The results of the disclosed methods and tasks may be persistently stored by transforming physical storage devices, such as solid state memory chips or magnetic disks, into a different state. In some embodiments, the computer system may be a cloud-based computing system whose processing resources are shared by multiple distinct business entities or other users.

[0260] The processes described herein or illustrated in the figures of the present disclosure may begin in response to an event, such as on a predetermined or dynamically determined schedule, on demand when initiated by a user or system administrator, or in response to some other event. When such processes are initiated, a set of executable program instructions stored on one or more non-transitory computer-readable media (e.g., hard drive, flash memory, removable media, etc.) may be loaded into memory (e.g., RAM) of a server or other computing device. The executable instructions may then be executed by a hardware-based computer processor of the computing device. In some embodiments, such processes or portions thereof may be implemented on multiple computing devices and / or multiple processors, serially or in parallel.

[0261] Depending on the embodiment, certain acts, events, or functions of any of the processes or algorithms described herein can be performed in a different sequence, can be added, merged, or left out altogether (e.g., not all described operations or events are necessary for the practice of the algorithm). Moreover, in certain embodiments, operations or events can be performed concurrently, e.g., through multi-threaded processing, interrupt processing, or multiple processors or processor cores or on other parallel architectures, rather than sequentially.

[0262] The various illustrative logical blocks, modules, routines, and algorithm steps described in connection with the embodiments disclosed herein can be implemented as electronic hardware (e.g., ASICs or FPGA devices), computer software that runs on computer hardware, or combinations of both. Moreover, the various illustrative logical blocks and modules described in connection with the embodiments disclosed herein can be implemented or performed by a machine, such as a processor device, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A processor device can be a microprocessor, but in the alternative, the processor device can be a controller, microcontroller, or logic circuitry that implements a state machine, combinations of the same, or the like. A processor device can include electrical circuitry configured to process computer-executable instructions. In another embodiment, a processor device includes an FPGA or other programmable device that performs logic operations without processing computer-executable instructions. A processor device canalso be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Although described herein primarily with respect to digital technology, a processor device may also include primarily analog components. For example, some or all of the rendering techniques described herein may be implemented in analog circuitry or mixed analog and digital circuitry. A computing environment can include any type of computer system, including, but not limited to, a computer system based on a microprocessor, a mainframe computer, a digital signal processor, a portable computing device, a device controller, or a computational engine within an appliance, to name a few.

[0263] The elements of a method, process, routine, or algorithm described in connection with the embodiments disclosed herein can be embodied directly in hardware, in a software module executed by a processor device, or in a combination of the two. A software module can reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of a non-transitory computer-readable storage medium. An exemplary storage medium can be coupled to the processor device such that the processor device can read information from, and write information to, the storage medium. In the alternative, the storage medium can be integral to the processor device. The processor device and the storage medium can reside in an ASIC. The ASIC can reside in a user terminal. In the alternative, the processor device and the storage medium can reside as discrete components in a user terminal.

[0264] Conditional language used herein, such as, among others, “can,” “could,” “might,” “may,” “e.g.,” and the like, unless specifically stated otherwise, or otherwise understood within the context as used, is generally intended to convey that certain embodiments include, while other embodiments do not include, certain features, elements or steps. Thus, such conditional language is not generally intended to imply that features, elements or steps are in any way required for one or more embodiments or that one or more embodiments necessarily include logic for deciding, with or without other input or prompting, whether these features, elements or steps are included or are to be performed in any particular embodiment. The terms “comprising,” “including,” “having,” and the like are synonymous and are used inclusively, in an open-ended fashion, and do not exclude additional elements, features, acts, operations, and so forth. Also, the term “or” is used in its inclusive sense (and not in its exclusive sense) so that when used, forexample, to connect a list of elements, the term “or” means one, some, or all of the elements in the list. The term “set” is used to include “one or more.” For example, a set of objects may include a single object or multiple objects.

[0265] Disjunctive language such as the phrase “at least one of X, Y, or Z,” unless specifically stated otherwise, is otherwise understood with the context as used in general to present that an item, term, etc., may be either X, Y, or Z, or any combination thereof (e.g., X, Y, or Z). Thus, such disjunctive language is not generally intended to, and should not, imply that certain embodiments require at least one of X, at least one of Y, and at least one of Z to each be present.

[0266] Any process descriptions, elements or blocks in the flow diagrams described herein and / or depicted in the attached figures should be understood as potentially representing modules, segments, or portions of code which include one or more executable instructions for implementing specific logical functions or elements in the process. Alternate implementations are included within the scope of the embodiments described herein in which elements or functions may be deleted, executed out of order from that shown, or discussed, including substantially concurrently or in reverse order, depending on the functionality involved as would be understood by those skilled in the art.

[0267] Unless otherwise explicitly stated, articles such as “a” or “an” should generally be interpreted to include one or more described items. Accordingly, phrases such as “a device configured to” are intended to include one or more recited devices. Such one or more recited devices can also be collectively configured to carry out the stated recitations. For example, “a processor configured to carry out recitations A, B, and C” can include a first processor configured to carry out recitation A working in conjunction with a second processor configured to carry out recitations B and C.

[0268] While the above detailed description has shown, described, and pointed out novel features as applied to various embodiments, it can be understood that various omissions, substitutions, and changes in the form and details of the devices or algorithms illustrated can be made without departing from the scope of the disclosure. As can be recognized, certain embodiments described herein can be embodied within a form that does not provide all of the features and benefits set forth herein, as some features can be used or practiced separately from others. All changes which come within the meaning and range of equivalency of the claims are to be embraced within their scope.

Claims

WHAT IS CLAIMED IS:

1. A computer-implemented method for generating a new image, the computer- implemented method comprising: receiving a first image captured by a first sensor source from a first point of view and a second image captured by a second sensor source from a second point of view, wherein the first sensor source and the second sensor source are positioned within a first distance of each other at a geographic location, and wherein the first image and the second image each depict a first object; generating training data that includes the first image and the second image, wherein the second image is labeled with an indication that the second image is from the second point of view and is the first distance from the first image; training an artificial intelligence model using the training data; and providing a third image captured by a third sensor source from a third point of view and a second distance as an input to the trained artificial intelligence model, wherein providing the third image and the second distance as the input to the trained artificial intelligence model causes the trained artificial intelligence model to output a fourth image that is generated as if the fourth image was captured at a position that is the second distance from a position of the third sensor source.

2. The computer-implemented method of Claim 1, wherein the second image is labeled with the indication that the second image from the second point of view, the indication that the second image is captured from the first distance from the first image, and an indication of one of a number of pixels in the second image, coordinates of the pixels in the second image, an intensity of each pixel in the second image, an angle from which the second image was captured relative to a plane bounded by positions of the first sensor source and the second sensor source, a depth represented by a distance from a position of the second sensor source relative to the plane bounded by the positions of the first sensor source and the second sensor source to a position of the first object, or an ambient light level of the geographic location.

3. The computer- implemented method of Claim 2, further comprising: determining a coordinate of a third position from which the second image was captured;determining a third distance between the third position and a center position on a plane in which the third position is located; determining a percentage that the third distance is of a total distance between the center position on the plane and an edge position on the plane, wherein the edge position on the plane corresponds with a position of a second object captured by a third sensor source that is at an end of a row of sensor sources that includes the first sensor source and the second sensor source; and determining the angle based on the determined percentage.

4. The computer-implemented method of Claim 1, wherein providing a third image captured by a third sensor source from a third point of view and a second distance as an input to the trained artificial intelligence model further comprises providing the third image, the second distance, and at least one of an angle, a depth, or an ambient light level.

5. The computer-implemented method of Claim 1, wherein the first sensor source, the second sensor source, and the third sensor source are aligned in a first plane, and wherein the position that is the second distance from the position of the third sensor source is aligned in the first plane.

6. The computer-implemented method of Claim 1, wherein the training data comprises a fifth image captured by a fourth sensor source within a second distance of the second sensor source.

7. The computer-implemented method of Claim 1, wherein the first sensor source comprises a camera.

8. The computer- implemented method of Claim 1, wherein the first sensor source comprises a microlens camera.

9. A system for generating a new image, the system comprising: memory that stores computer-executable instructions; and a processor in communication with the memory, wherein the computer-executable instructions, when executed by the processor, cause the processor to: receive a first image captured by a first sensor source from a first point of view and a second image captured by a second sensor source from a second point of view, wherein the first sensor source and the second sensor source are positionedwithin a first distance of each other at a geographic location, and wherein the first image and the second image each depict a first object; generate training data that includes the first image and the second image, wherein the second image is labeled with an indication that the second image is from the second point of view and is the first distance from the first image; train an artificial intelligence model using the training data; provide a third image captured by a third sensor source from a third point of view and a second distance as an input to the trained artificial intelligence model, wherein providing the third image and the second distance as the input to the trained artificial intelligence model causes the trained artificial intelligence model to output a fourth image that is generated as if the fourth image was captured at a position that is the second distance from a position of the third sensor source.

10. The system of Claim 9, wherein the second image is labeled with the indication that the second image from the second point of view, the indication that the second image is captured from the first distance from the first image, and an indication of one of a number of pixels in the second image, coordinates of the pixels in the second image, an intensity of each pixel in the second image, an angle from which the second image was captured relative to a plane bounded by positions of the first sensor source and the second sensor source, a depth represented by a distance from a position of the second sensor source relative to the plane bounded by the positions of the first sensor source and the second sensor source to a position of the first object, or an ambient light level of the geographic location.

11. The system of Claim 10, wherein the computer-executable instructions, when executed, further cause the processor to: determine a coordinate of a third position from which the second image was captured; determine a third distance between the third position and a center position on a plane in which the third position is located; determine a percentage that the third distance is of a total distance between the center position on the plane and an edge position on the plane, wherein the edge position on the plane corresponds with a position of a second object captured by a third sensorsource that is at an end of a row of sensor sources that includes the first sensor source and the second sensor source; and determine the angle based on the determined percentage.

12. The system of Claim 9, wherein the computer-executable instructions, when executed, further cause the processor to provide the third image, the second distance, and at least one of an angle, a depth, or an ambient light level.

13. The system of Claim 9, wherein the first sensor source, the second sensor source, and the third sensor source are aligned in a first plane, and wherein the position that is the second distance from the position of the third sensor source is aligned in the first plane.

14. The system of Claim 9, wherein the training data comprises a fifth image captured by a fourth sensor source within a second distance of the second sensor source.

15. The system of Claim 9, wherein the first sensor source comprises a camera.

16. The system of Claim 9, wherein the first sensor source comprises a microlens camera.

17. A non-transitory, computer-readable medium comprising computer-executable instructions for generating a new image, wherein the computer-executable instructions, when executed by a processor, cause the processor to: receive a first image captured by a first sensor source from a first point of view and a second image captured by a second sensor source from a second point of view, wherein the first sensor source and the second sensor source are positioned within a first distance of each other at a geographic location, and wherein the first image and the second image each depict a first object; generate training data that includes the first image and the second image, wherein the second image is labeled with an indication that the second image is from the second point of view and is the first distance from the first image; train an artificial intelligence model using the training data; and provide a third image captured by a third sensor source from a third point of view and a second distance as an input to the trained artificial intelligence model, wherein providing the third image and the second distance as the input to the trained artificial intelligence model causes the trained artificial intelligence model to output a fourth imagethat is generated as if the fourth image was captured at a position that is the second distance from a position of the third sensor source.

18. The non-transitory, computer-readable medium of Claim 17, wherein the second image is labeled with the indication that the second image from the second point of view, the indication that the second image is captured from the first distance from the first image, and an indication of one of a number of pixels in the second image, coordinates of the pixels in the second image, an intensity of each pixel in the second image, an angle from which the second image was captured relative to a plane bounded by positions of the first sensor source and the second sensor source, a depth represented by a distance from a position of the second sensor source relative to the plane bounded by the positions of the first sensor source and the second sensor source to a position of the first object, or an ambient light level of the geographic location.

19. The non-transitory, computer-readable medium of Claim 18, wherein the computer-executable instructions, when executed, further cause the processor to: determine a coordinate of a third position from which the second image was captured; determine a third distance between the third position and a center position on a plane in which the third position is located; determine a percentage that the third distance is of a total distance between the center position on the plane and an edge position on the plane, wherein the edge position on the plane corresponds with a position of a second object captured by a third sensor source that is at an end of a row of sensor sources that includes the first sensor source and the second sensor source; and determine the angle based on the determined percentage.

20. The non-transitory, computer-readable medium of Claim 17, wherein the computer-executable instructions, when executed, further cause the processor to provide the third image, the second distance, and at least one of an angle, a depth, or an ambient light level.

21. A cassette unit comprising: a housing, wherein the housing comprises: a first sensor source; a second sensor source;an electrical connection that couples the first sensor source to the second sensor source; a network interface coupled to the first sensor source and the second sensor source, wherein the network interface is configured to communicate with a device external to the cassette unit; and memory configured to store data captured by the first sensor source and the second sensor source; and an input / output port coupled to the housing.

22. The cassette unit of Claim 21, wherein the housing comprises a flexible tube.

23. The cassette unit of Claim 22, wherein the flexible tube comprises a Teflon tube.

24. The cassette unit of Claim 21, wherein the housing comprises a glass casing.

25. The cassette unit of Claim 21, wherein the housing is transparent.

26. The cassette unit of Claim 21 , wherein the first sensor source comprises a microlens camera.

27. The cassette unit of Claim 21 , wherein the input / output port is configured to couple the cassette unit to a second cassette unit.

28. The cassette unit of Claim 27, wherein the second cassette unit comprises a third sensor source, and wherein the first sensor source is electrically coupled to the third sensor source when the input / output port couples the cassette unit to the second cassette unit.

29. The cassette unit of Claim 21, wherein the cassette unit comprises a circular shape, and wherein the electrical connection traverses a diameter of the circular shape.

30. The cassette unit of Claim 29, wherein a location of the first sensor source is 180 degrees opposite of a location of the second sensor source.

31. The cassette unit of Claim 21 , wherein the first sensor source and the second sensor source, when the cassette unit is placed in a geographic location, are configured to capture images of the geographic location.

32. A camera strip comprising: a housing, wherein the housing comprises: a first camera; a second camera; an optical connection that couples the first camera to the second camera;a network interface coupled to the first camera and the second camera, wherein the network interface is configured to communicate with a device external to the camera strip; and memory configured to store data captured by the first camera and the second camera; and an input / output port coupled to the housing.

33. The camera strip of Claim 32, wherein the housing comprises a flexible tube.

34. The camera strip of Claim 33, wherein the flexible tube comprises a Teflon tube.

35. The camera strip of Claim 32, wherein the housing comprises a glass casing.

36. The camera strip of Claim 32, wherein the housing is transparent.

37. The camera strip of Claim 32, wherein the first camera comprises a microlens camera.

38. The camera strip of Claim 32, wherein the input / output port is configured to couple the camera strip to a second camera strip.

39. The camera strip of Claim 38, wherein the second camera strip comprises a third camera, and wherein the first camera is coupled to the third camera when the input / output port couples the camera strip to the second camera strip.

40. The camera strip of Claim 32, wherein the first camera and the second camera, when the camera strip is placed in a geographic location, are configured to capture images of the geographic location.

41. A camera comprising: an optical fiber; and a compound lens coupled to an end of the optical fiber, wherein the compound lens comprises a first lens and a second lens, wherein the first lens is positioned at a first angle and is configured to capture a first image of an object in a direction of the first angle, wherein the second lens is positioned at a second angle different than the first angle and is configured to capture a second image of the object in a direction of the second angle, and wherein the first lens is coupled to the second lens.

42. The camera of Claim 41 , wherein the first lens is coupled to the second lens via an optical connection.

43. The camera of Claim 41, wherein the first lens is coupled to the second lens via an electrical connection.

44. The camera of Claim 41, further comprising a network interface coupled to the optical fiber.

45. The camera of Claim 44, wherein the first lens is configured to transmit the first image to the network interface via the optical fiber.

46. The camera of Claim 44, wherein the network interface is coupled to a second end of the optical fiber that is different than the end of the optical fiber.

47. The camera of Claim 41, wherein the compound lens is 100 micrometers wide.

48. The camera of Claim 41, wherein the compound lens is configured to focus on an object at least a distance of 3 millimeters away from a position of the compound lens.

49. The camera of Claim 41, wherein the first lens has a first size, and wherein the second lens has a second size that is different than the first size.

50. The camera of Claim 41, wherein the first lens has a first focus length, and wherein the second lens has a second focus length that is different than the first focus length.

51. A camera comprising: an optical fiber; a first lens coupled to a first end of the optical fiber; and a second lens coupled to the first end of the optical fiber, wherein the first lens is positioned at a first angle and is configured to capture a first image of an object in a direction of the first angle, wherein the second lens is positioned at a second angle different than the first angle and is configured to capture a second image of the object in a direction of the second angle, and wherein the first lens is coupled to the second lens.

52. The camera of Claim 51, wherein the first lens is coupled to the second lens via an optical connection.

53. The camera of Claim 51, wherein the first lens is coupled to the second lens via an electrical connection.

54. The camera of Claim 51 , further comprising a network interface coupled to the optical fiber.

55. The camera of Claim 54, wherein the first lens is configured to transmit the first image to the network interface via the optical fiber.

56. The camera of Claim 54, wherein the network interface is coupled to a second end of the optical fiber that is different than the first end of the optical fiber.

57. The camera of Claim 51, wherein the compound lens is 100 micrometers wide.

58. The camera of Claim 51, wherein the compound lens is configured to focus on an object at least a distance of 3 millimeters away from a position of the compound lens.

59. The camera of Claim 51, wherein the first lens has a first size, and wherein the second lens has a second size that is different than the first size.

60. The camera of Claim 51, wherein the first lens has a first focus length, and wherein the second lens has a second focus length that is different than the first focus length.

61. A headset comprising: a screen that produces a first image; a magnifying device positioned between the screen and a wearer of the headset, wherein the magnifying device is configured to magnify the first image produced by the screen; a gyroscope that detects a direction of movement of the wearer of the headset; a network interface; and a processor configured with computer-executable instructions, wherein the computer-executable instructions, when executed by the processor, cause the processor to: determine that the wearer of the headset moved in a first direction based on a measurement captured by the gyroscope; cause the network interface to transmit an indication of the first direction to a location viewing system; and cause the screen to display a second image provided by the location viewing system in place of the first image.

62. The headset of Claim 61, further comprising a second screen and a second magnifying device.

63. The headset of Claim 62, wherein the screen and the magnifying device are positioned in front of a left eye of the wearer, and wherein the second screen and the second magnifying device are positioned in front of a right eye of the wearer.

64. The headset of Claim 61, wherein the location viewing system obtains the second image captured by a sensor source in a direction that corresponds with the first direction.

65. The headset of Claim 61, wherein the gyroscope is configured to capture the measurement in response to a movement of the wearer of the headset.

66. The headset of Claim 61, further comprising a speaker that is configured to output audio captured at a location captured by the first image.

67. The headset of Claim 61, wherein the first image is captured at a first ratio, and wherein the screen depicts the first image at a second ratio that is wider than the first ratio.

68. The headset of Claim 61, wherein the first image is captured at a first zoom level, and wherein the screen depicts the first image at a second zoom level that is more zoomed in than the first zoom level.

69. The headset of Claim 61, wherein the screen is curved at a first angle.

70. The headset of Claim 61, wherein the headset comprises a virtual reality headset.

71. A headset comprising: a projector that projects a first image; a magnifying device positioned between the projector and a wearer of the headset, wherein the projector is configured to project the first image onto a first end of the magnifying device, and wherein the magnifying device is configured to magnify the first image projected by the projector; a gyroscope that detects a direction of movement of the wearer of the headset; a network interface; and a processor configured with computer-executable instructions, wherein the computer-executable instructions, when executed by the processor, cause the processor to: determine that the wearer of the headset moved in a first direction based on a measurement captured by the gyroscope; cause the network interface to transmit an indication of the first direction to a location viewing system; andcause the projector to project a second image provided by the location viewing system in place of the first image onto the first end of the magnifying device.

72. The headset of Claim 71, further comprising a second projector and a second magnifying device.

73. The headset of Claim 72, wherein the projector and the magnifying device are positioned in front of a left eye of the wearer, and wherein the second projector and the second magnifying device are positioned in front of a right eye of the wearer.

74. The headset of Claim 71, wherein the location viewing system obtains the second image captured by a sensor source in a direction that corresponds with the first direction.

75. The headset of Claim 71, wherein the gyroscope is configured to capture the measurement in response to a movement of the wearer of the headset.

76. The headset of Claim 71, further comprising a speaker that is configured to output audio captured at a location captured by the first image.

77. The headset of Claim 71, wherein the first image is captured at a first ratio, and wherein the projector projects the first image at a second ratio that is wider than the first ratio.

78. The headset of Claim 71, wherein the first image is captured at a first zoom level, and wherein the projector projects the first image at a second zoom level that is more zoomed in than the first zoom level.

79. The headset of Claim 71, wherein an eye of the wearer of the headset is positioned in front of a second end of the magnifying device that is different than the first end of the magnifying device.

80. The headset of Claim 71, wherein the headset comprises a virtual reality headset.

81. A computer-implemented method for enabling a user to virtually navigate a geographic location, the computer-implemented method comprising: receiving an indication of a selection of the geographic location; causing a user device to display a first image of a first position at the geographic location, wherein the first image comprises an icon that represents a second user that is virtually navigating the geographic location, and wherein the icon is positioned in the first image at a location that represents a virtual location of the second user;receiving an indication of a selection of a sub-geographic location within the geographic location; receiving an indication of a selection to move in the geographic location with the second user; and causing the user device and a second user device operated by the second user to display one or more second images captured between the first position and a second position of the sub-geographic location.

82. The computer-implemented method of Claim 81, further comprising: causing the user device to display a third image captured at the second position of the sub-geographic location, wherein the user device further displays a list of items present at the sub-geographic location; receiving a selection of a first item in the list of items; and causing the user device to display a fourth image of the first item.

83. The computer-implemented method of Claim 82, wherein the fourth image comprises a stored image of the first item.

84. The computer-implemented method of Claim 82, wherein the fourth image comprises an image of an aisle in the sub-geographic location where the first item is located.

85. The computer-implemented method of Claim 82, further comprising: receiving an indication of a selection of a request to converse with an individual located at the sub-geographic location; and causing an utterance spoken by the user to be output at the sub-geographic location.

86. The computer-implemented method of Claim 85, further comprising causing the user device to output a second utterance spoken by the individual located at the sub-geographic location.

87. The computer- implemented method of Claim 81, wherein the one or more second images depict the icon that represents the second user at one or more second locations that represent the virtual location of the second user as the second user navigates from the first position to the second position.

88. The computer- implemented method of Claim 81, wherein the icon comprises an avatar of the second user.

89. The computer-implemented method of Claim 81 , wherein the user is a lead role user, and wherein the lead role user navigates for the second user within the geographic location.

90. The computer-implemented method of Claim 81, wherein the user device further displays a map of the geographic location.

91. The computer-implemented method of Claim 81 , wherein the geographic location comprises one of a store or a mall.

92. The computer-implemented method of Claim 81, wherein the geographic location comprises a mall, and wherein the sub-geographic location comprises a store in the mall.

93. A system for enabling a user to virtually navigate a geographic location, the system comprising: memory that stores computer-executable instructions; and a processor in communication with the memory, wherein the computer-executable instructions, when executed by the processor, cause the processor to: receive an indication of a selection of the geographic location; cause a user device to display a first image of a first position at the geographic location, wherein the first image comprises an icon that represents a second user that is virtually navigating the geographic location, and wherein the icon is positioned in the first image at a location that represents a virtual location of the second user; receive an indication of a selection of a sub-geographic location within the geographic location; receive an indication of a selection to move in the geographic location with the second user; and cause the user device and a second user device operated by the second user to display one or more second images captured between the first position and a second position of the sub-geographic location.

94. The system of Claim 93, wherein the computer-executable instructions, when executed, further cause the processor to: cause the user device to display a third image captured at the second position of the sub-geographic location, wherein the user device further displays a list of items present at the sub-geographic location;receive a selection of a first item in the list of items; and cause the user device to display a fourth image of the first item.

95. The system of Claim 94, wherein the computer-executable instructions, when executed, further cause the processor to: receive an indication of a selection of a request to converse with an individual located at the sub-geographic location; and cause an utterance spoken by the user to be output at the sub-geographic location.

96. The system of Claim 95, wherein the computer-executable instructions, when executed, further cause the processor to cause the user device to output a second utterance spoken by the individual located at the sub-geographic location.

97. The system of Claim 93, wherein the one or more second images depict the icon that represents the second user at one or more second locations that represent the virtual location of the second user as the second user navigates from the first position to the second position.

98. The system of Claim 93, wherein the user device further displays a map of the geographic location.

99. The system of Claim 93, wherein the geographic location comprises a mall, and wherein the sub-geographic location comprises a store in the mall.

100. A non-transitory, computer-readable medium comprising computer-executable instructions for enabling a user to virtually navigate a geographic location, wherein the computerexecutable instructions, when executed by a processor, cause the processor to: receive an indication of a selection of the geographic location; cause a user device to display a first image of a first position at the geographic location, wherein the first image comprises an icon that represents a second user that is virtually navigating the geographic location, and wherein the icon is positioned in the first image at a location that represents a virtual location of the second user; receive an indication of a selection of a sub-geographic location within the geographic location; receive an indication of a selection to move in the geographic location with the second user; andcause the user device and a second user device operated by the second user to display one or more second images captured between the first position and a second position of the sub-geographic location.

Images