Improved and immersive viewing experience

Abstract

This disclosure provides a novel stereoscopic imaging technique that enables extremely large field-of-view data sets that can be displayed in stereo, such as over 270° horizontal and 180° vertical fields of view. Therefore, it is possible to record data that is larger than the user can visualize. In this way, the user can select which subset of the data set they wish to view. Head and eye tracking then provide a natural viewing experience for the user, allowing them to view and inspect the scene as if they were naturally present and viewing it in real time.

Description

[Technical Field]

[0001] Aspects of the present disclosure generally relate to stereoscopic images on head display units. [Background technology]

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. patent application Ser. No. 17 / 225,610, filed April 8, 2021. Summary of the Invention [Problem to be solved by the invention]

[0003] Virtual reality displays provide users with 3D images.

[0004] All examples, aspects, and features described herein may be combined in any manner technically possible.

[0005] This disclosure teaches methods, software, and devices that implement a series of stereoscopic imaging systems to provide an immersive viewing experience.

[0006] A preferred embodiment of the present disclosure is the acquisition of images from a smart device. The smart device may include a laptop, phone, tablet, HDU, television, or monitor. The smart device is equipped with a left camera and a right camera. The left and right cameras are separated by a stereo distance. In a preferred embodiment, the stereo distance is equal to that of an average adult. The left camera acquires a left-eye perspective image of an area at a first time point. The right camera also acquires a right-eye perspective image of the area at the first time point. The left and right-eye perspective images comprise a stereoscopic image. The stereoscopic image is then transmitted from the smart device via a wireless connection to a head display unit (HDU). Typically, this is done via Bluetooth or the Internet. The stereoscopic image is then displayed on the head display unit (HDU), which may include a virtual reality display, an augmented reality display, or a mixed reality display. The HDU has a left-eye display and a right-eye display. The left-eye display fits over the user's left eye. The right-eye display fits over the user's right eye. The HDU is worn by a first user. The HDU displays a stereoscopic image to the first user at a second point in time in near real time (e.g., within 5 seconds of the first point in time), with a left-eye perspective image displayed on the left-eye display and a right-eye perspective image displayed on the right-eye display.

[0007] Some implementations include two-way communication and near-real-time display of stereoscopic images. (Note: This development assumes increased bandwidth with the advent of 5G communications. If 5G is unavailable for some reason, the camera resolution can be adjusted to match the locally available bandwidth.) User number 1 can view images in real time via a stereoscopic camera, and these stereoscopic images are transmitted over the Internet (or other transmission means) to user number 2, who receives the images and displays these images in near-real time on a stereoscopic display device (e.g., a virtual reality or mixed reality headset). Simultaneously, user number 2 is observed in real time via a stereoscopic camera, and these stereoscopic images are transmitted over the Internet (or other transmission means) to user number 1, who receives the images and displays these images in near-real time on a stereoscopic display device (e.g., a virtual reality or mixed reality headset).

[0008] In one embodiment, multiple users are simultaneously observed in real time by a stereoscopic camera, and these stereoscopic images are transmitted over the Internet (or other transmission means) to other users who receive the images and display these images in near real time on their respective stereoscopic display devices (e.g., virtual reality, mixed reality headsets).

[0009] In some implementations, the user can choose to point the stereo camera at a landscape scene or object of interest rather than looking through the stereo camera.

[0010] In some embodiments, there is no simultaneity between sending and displaying; rather, like current email systems, user number 1 writes and sends an email, and some time later user number 2 retrieves the email and subsequently replies. In this case, user number 1 records a series of stereo images and transmits these stereo images to user number 2, who at some point thereafter retrieves the stereo images and displays them on a stereo display device. At this point, user number 2 can respond by creating and transmitting their own set of stereo images. These implementations anticipate that users will provide audio to accompany the stereo images.

[0011] Some implementations involve one-way communication of near-real-time stereoscopic images and display by separate users. User number 1 can view images in real time with a stereoscopic camera, and these stereoscopic images are transmitted over the Internet (or other transmission means) to user number 2, who receives the images and displays them in near-real time on a stereoscopic display device (e.g., a virtual reality or mixed reality headset). In this implementation, user number 2 is not equipped with a stereoscopic camera, and user number 1 may or may not be equipped with a stereoscopic display device; in either case, user number 1 does not receive user number 2's stereoscopic images. In this implementation, there is no limit to the number of receive-only users. For example, a professor may give a lecture using a stereoscopic camera, and classmates may receive the stereoscopic images. In other implementations, user number 1 records a series of stereoscopic images and transmits these stereoscopic images to user number 2, who, at some point in the future, acquires the stereoscopic images and displays them on a stereoscopic display device. These implementations assume that users provide audio to accompany the stereoscopic images.

[0012] Some implementations include real-time recording and near real-time display / playback of stereoscopic images. A user can use a mixed reality head display unit with a stereoscopic camera (described later herein) and a recording medium connected to the stereoscopic camera. The user can choose to point the stereoscopic camera at a scene or object of interest, capture images of these areas or objects of interest using the stereoscopic camera, record and save the stereoscopic images to a recording medium, and display the recordings on the mixed reality head display unit. After the capture session, the user can choose to transfer the recorded stereoscopic images to another person equipped with a stereoscopic display device.

[0013] Some implementations record stereoscopic images and associated distances in real time using a laser range finder (LRF), also known as LIDAR. In some implementations, a user can use a mixed reality head display unit equipped with a stereoscopic camera and a laser range finder (described later herein) and a storage medium connected to the head display unit for receiving, recording, and storing data from the stereoscopic camera and the laser range finder. In some implementations, a user can accurately map an area (e.g., a room, house, or building) using a combination of rapid laser scanning with rotation of the stereoscopic camera and the laser range finder (e.g., the stereoscopic camera can be mounted on a tripod and / or gimble of the head display unit). The stereoscopic camera and the laser range finder can also be aimed at an object. In some implementations, the stereoscopically photographed and measured area can be reconstructed in virtual space.

[0014] This system has several key elements. Dual cameras can be incorporated. These cameras can be added to a laptop computer in a similar manner to current single cameras. The dual cameras are spaced apart at an interocular distance, and the viewer of the stereoscopic images collected by the cameras provides a 3D effect to the recipient. In some implementations, variable / adjustable interocular distance can be provided. Close spacing can be provided for children, moderate spacing for average-sized individuals, and wider spacing for larger individuals. Custom interocular distances can also be provided. In some implementations, the cameras are placed inside the laptop overlooking the keyboard and focused on the user's face. In other implementations, the cameras can be placed outside the laptop so that the recipient can see the environment / scenery around the user's location. Alternatively, two sets of cameras can be added to the laptop: one inside and one outside. The cameras can be controlled by keyboard input, pull-down menus and associated cursor input, voice input, etc.

[0015] In some implementations, two stereoscopic cameras can be added to a smartphone. Interocular distance can be handled in much the same way as with a laptop computer. The cameras can be on the front (i.e., the side with the bidirectional icon), the back (i.e., the side opposite the front), or both.

[0016] In some implementations, two stereoscopic cameras can be attached to a head-mounted display unit (HDU). Interocular distance can be handled in much the same way as a laptop computer. In this embodiment, the HDU can be connected wirelessly or via a cable to a recording device, or to the internet for two-way communication as described above, or viewed through the HDU display, or through a combination of these interactions. It should be noted that the HDU may operate in mixed reality mode.

[0017] In some implementations, two stereoscopic cameras can be attached to smart glasses (e.g., sunglasses). The interocular distance can be handled in much the same way as a laptop computer. In this embodiment, the smart sunglasses can be connected wirelessly or via a connecting cable to a recording device, the Internet for two-way communication as described above, or can be viewed through a smart sunglasses display, or a combination of these interactions. It is noted that the smart sunglasses may operate in a mixed reality mode.

[0018] In some implementations, two stereoscopic cameras can be attached to a smart tablet. Interocular distance can be handled in much the same way as with a laptop computer. In this embodiment, the smart tablet can be connected wirelessly or via a cable to a recording device, connected to the Internet for two-way communication as described above, or viewed via an HDU, or a combination of these interactions.

[0019] In some implementations, two commercially available stereoscopic cameras can be used. The interocular distance can be adjusted to match the specifications of the commercially available 3D camera. In this embodiment, the commercially available 3D camera can be connected to a recording device wirelessly or via a connecting cable, connected to the Internet for two-way communication as described above, or displayed via an HDU, or a combination of these interactions.

[0020] In some implementations, two small camera clusters are installed in each of the above camera configurations. Each camera is assigned a different line-of-sight angle to provide a very wide simultaneous field of view with very high resolution. Interocular distance can be handled in much the same way as a laptop computer. In this implementation, the small camera clusters are connected to a recording device wirelessly or via a connecting cable, connected to the Internet for two-way communication as described above, or displayed via an HDU, or a combination of these interactions. In this implementation, the very wide field of view nominally exceeds the field of view of the display unit. Importantly, the user (or viewer) can select a subset of the field of view to view at a particular time and change the field of view selection as desired.

[0021] In some implementations, a single ball of miniature camera pairs is designated. To provide a very wide simultaneous field of view with very high resolution, different line-of-sight angles are designated for each camera pair. Different camera pairs contribute to the very wide simultaneous field of view based on their location on the ball. The surface of the ball can be spherical and smooth, or faceted to provide a flat surface for each camera pair. Interocular distance can be treated as a standard separation for the camera members of a pair, or as a custom separation for special purposes. Note that adjacent cameras with appropriate spacing can contribute at any point in time to the stereoscopic image pair at various times. In this embodiment, the miniature camera pair can be connected wirelessly or via a connecting cable to a recording device, the Internet for the aforementioned two-way communication, or an HDU. Note that in this implementation, the very wide field of view nominally exceeds the field of view of the display device. Importantly, the user (or viewer) can select a subset of the field of view to view at a particular time and change the field of view selection as desired.

[0022] In some implementations, new functionality is incorporated into the HDU to operate with a miniature camera cluster or a single ball or pair of miniature cameras. The HDU must incorporate an inclinometer to monitor head position orientation and tilt in real time. The HDU must provide eye-tracking capabilities. An initialization process synchronizes the HDU's initial gaze angle and tilt with the stereoscopic center of view. The user can then use the control unit to select a sub-field of view to display on the HDU (Note: This can be done through various options, including, but not limited to, numbered areas available in a pull-down menu, circling the area of ​​interest, or voice command). After this sub-field is displayed, the system then works with an eye tracker to focus the foveal region of the eye on a specific area of ​​the field of interest imaged by a specific camera in the two-camera cluster or a camera in the ball. The eye tracker can map each eye's projected gaze to a spot or grid on the display and associate it with a specific camera.

[0023] In some implementations, image processing is applied to provide a virtual background for the user, which can be added to the user's display in 3D. The background can be two-dimensional, but preferably three-dimensional. The background can be chosen from a wide variety of 3D scene libraries, such as beach scenes, mountain scenes, office scenes, etc. In some embodiments, the 3D scene library can be dynamic, such as a waterfall or ocean waves.

[0024] In some implementations, image processing is applied to provide image enhancements to the user's body (e.g., acne removal, improved hair volume or style).

[0025] In some implementations, image processing is applied to provide the user with artificial appendages that can be added to the user's display and can be 3D, such as wearing a suit and tie that covers a T-shirt during a business meeting. The appendages are registered to the user's features, such as facial and body structure.

[0026] Some embodiments include using a microphone in proximity to the smart device. Audio can be recorded on the smart device. The audio can be played back on a speaker in proximity to the HDU, with the audio playback being synchronized with the stereoscopic images.

[0027] Some embodiments involve moving the smart device away from objects in the area, which makes the objects appear farther away from a user viewing stereoscopic video on the HDU.

[0028] Some embodiments involve bringing the smart device closer to an object in the area, causing the object to appear to move closer to the user viewing the stereoscopic video on the HDU.

[0029] Some of these include the tilt of the sensors on the left and right cameras over time. Specifically, initially, the direction in which the sensor corresponding to the left camera is pointing is parallel to the direction in which the sensor corresponding to the right camera is pointing. Later, the direction in which the sensor corresponding to the left camera is pointing and the direction in which the sensor corresponding to the right camera is pointing converge.

[0030] Some embodiments include a range of captures, for example, stereoscopic vision allows a user to see a stereoscopic image of themselves.

[0031] Some embodiments include using a stereoscopic camera cluster and stereoscopic LIDAR acquisition. Some embodiments include integrating an inertial measurement unit (IMU) for head-tracking and eye-tracking systems to direct the field of view of the stereoscopic images displayed on the HDU.

[0032] Some embodiments involve placing the stereoscopic imaging device on a convex surface (with or without a grid) or on a set of robotic arms.

[0033] Some of the techniques of the present disclosure are implemented using techniques described in U.S. Patent Application No. 15 / 878,463, entitled "Interactive 3D Cursor for Use in Medical Imaging," U.S. Patent Application No. 16 / 010,925, entitled "Interactive Placement of 3D Digital Representations of Surgical Devices or Anatomical Features on 3D Radiographic Images for Pre-Operative Planning," U.S. Patent Application No. 15 / 904,092, entitled "Processing 3D Medical Images for Enhanced Visualization," U.S. Patent Application No. 15 / 949,202, entitled "Smart Surgical Devices," and No. 9,615,806; Method and Apparatus for Producing and Displaying Artifact-Corrected Three-Dimensional (3D) Volume Data from Biplane Fluoroscopic Image Acquisitions; U.S. Patent No. 14 / 644,489; Method and Apparatus for Producing and Displaying Artifact-Corrected Three-Dimensional (3D) Volume Data from Biplane Fluoroscopic Image Acquisitions; U.S. Patent No. 9,980,691; Method and Apparatus for Original Display, U.S. Patent No. 9,349,183; Method and Apparatus for Three-Dimensional Display of Images, U.S. Patent Application No. 16 / 195,251; Interactive Voxel Manipulation in Volumetric Medical Images for Virtual Motion, Deformable Tissue, and Virtual Radiologic Dissection, U.S. Patent Application No. 16 / 509,592 "Implantable Markers for Assisting Surgery," U.S. Patent Application No. 16 / 524,275 "Use of Georegistered Tools for Manipulating Three-Dimensional Medical Images," P CT / US19 / 478 "Virtual Toolkit for Radiologists", U.S. Patent Application No. 16 / 563,985 "Method and Apparatus for Interaction of Virtual and Georegistered Tools", U.S. Patent Application No. 16 / 594,139 "Method and Apparatus for Performing 3D Imaging Examination of Structures Under Different Configurations and Analyzing Morphological Changes", U.S. Patent Application No. 16 / 683,256 "Method and Apparatus for Performing 3D Imaging Examination of Structures Under Different Configurations and Analyzing Morphological Changes", U.S. Patent Application No. 16 / 703,No. 629, "Radiologist-Assisted Machine Learning with Volume Subtending 3D Cursors," PCT / US19 / 239, "Radiologist-Assisted Machine Learning with Interactive Volume Subtending 3D Cursors," U.S. Provisional Application No. 62 / 843,612, "Method for Creating Computer-Generated Patient-Specific Images," U.S. Provisional Application No. 62 / 846,770, "Method for Prioritized Volume Rendering for Improved Visualization of Prioritized Items in 3D Volumes," U.S. Provisional Application No. 62 / 850,002, "Method ... Method for creating an artificial intelligence-generated differential diagnosis and management recommendation toolbox, U.S. Patent Application No. 16 / 654,047; Method for modifying imaging protocols in real time by implementing artificial intelligence; U.S. Provisional Application No. 62 / 856 / 185, "Method for image manipulation based on eye tracking," U.S. Patent Application No. 16 / 506,073, "Method for illustrating blood flow direction via a pointer," U.S. Patent Application No. 62 / 906,125, "Method and apparatus for volumetric rendering of moving fluids," and U.S. Patent Application No. 62 / 939,685, "Method and apparatus for developing organ-specific coordinate systems." [Brief explanation of the drawings]

[0034] [Figure 1] Figure 1A shows a top view of the video recording, and Figure 1B shows a displayed 2D image frame of the video recording. [Figure 2] Figure 2 shows the system hardware of the present invention. A text box is displayed. [Figure 3] Figure 3 shows the system software. [Figure 4] Figure 4 shows the mapping of the camera coordinate system to the GPS coordinate system. [Figure 5] Figure 5A shows the preferred coordinate system looking down into the xy plane from point D. Figure 5B shows the preferred coordinate system looking down into the yz plane from point C. [Figure 6] Figure 6A is a diagram plotting the locations of multiple viewers on a coordinate system, and Figure 6B is a table showing the locations of multiple viewers in Figure 6A. [Figure 7]Figure 7A shows time point #1 where three observers are looking down at a cityscape. Figure 7B shows time point #2 where the three observers from Figure 7A are still looking down at the cityscape, but at different parts of the scene. [Figure 8] Figure 8A shows a downward view from a nearly spherical stereoscopic camera system mounted on a pole in a typical urban center, and Figure 8B shows the β coordinates corresponding to the object seen in Figure 8A. [Figure 9] Figure 9A shows a natural scene, showing multiple objects at different distances from the user, and Figure 9B is a table showing the objects, ranges, camera angles, cameras used, and notes. [Figure 10] Figure 10A is a direct view (looking straight at the edge of the table) from the left camera pointed at two objects on a table (a can and a jar of honey). Figure 10B shows a second image from the left camera in the same position but tilted 20° inward (towards the can and jar of honey). Figure 10C is a direct view (looking straight at the edge of the table) from the left camera pointed at two objects on the table (a can and a jar of honey). Figure 10D shows a second image from the left camera in the same position but tilted 20° inward (towards the can and jar of honey). [Figure 11] Figure 11 shows the integration of a stereoscopic camera system into a smart device. [Figure 12] Figure 12A is an illustration of a smart device equipped with a stereo camera. Figure 12B shows a stereo camera ball. In a preferred embodiment, the stereo camera ball moves and focuses to observe an object of interest. Figure 12C shows a head display unit (HDU) with two stereo camera clusters mounted near where the wearer's eyes would be when wearing the HDU. Figure 12D shows a multi-stereo camera system / camera cluster. [Figure 13] FIG. 13 shows an example of the stereoscopic camera configuration of a smart stereo camera. [Figure 14] 14A shows an example of a smartphone stereoscopic camera configuration at a first point in time, and FIG. 14B shows an example of a smartphone stereoscopic camera configuration at a second point in time. [Figure 15] Figure 15A shows a first cant angle with a first convergence point, and Figure 15B shows a second cant angle with a second convergence point. [Figure 16] Figure 16A shows a first stereo separation with a first convergence point, and Figure 16B shows a second stereo separation with a second convergence point. [Figure 17] Figure 17A shows a first cant angle with a first convergence point and a first stereo separation, and Figure 17B shows a second cant angle with a second convergence point and a second stereo separation. [Figure 18] 18A shows the arrangement of some of the horizontal cameras of two (left and right) stereoscopic camera clusters. FIG. 18B shows the arrangement of some of the horizontal cameras of two (left and right) stereoscopic camera clusters. [Figure 19] Figure 19A shows a top view of a first head display unit with two stereoscopic camera clusters providing stereoscopic vision in the forward direction, Figure 19B shows a top view of a second head display unit with seven camera clusters providing stereoscopic vision with a horizontal field of view of approximately 180°, and Figure 19C shows a top view of a head display unit with ten stereoscopic camera clusters spaced apart to provide 360° stereoscopic vision. [Figure 20] Figure 20A is a side view of a hemispherical high-resolution camera cluster, and Figure 20B is a bottom view of the hemispherical high-resolution camera cluster. [Figure 21] Figure 21A is a bottom view of a convex surface including two high-resolution camera clusters in a first position, and Figure 21B is a bottom view of a convex surface including two high-resolution camera clusters in a second position. [Figure 22] Figure 22A is a top view of a user's eye, showing the user's eye is straight ahead. Figure 22B is a top view of two cameras on a convex surface, showing the cameras looking straight ahead. Figure 22C is a top view of a user's eye, showing the user's eye is looking to the right. Figure 22D is a top view of two cameras on a convex surface, showing the cameras looking straight ahead. [Figure 23]Figure 23 is a bottom view of a large convex surface containing three stereo camera pairs (or camera cluster pairs) shown moving over time. [Figure 24] FIG. 24 shows a rotatable stereo telephoto lens system. [Figure 25] Figure 25 is a top view of a stereoscopic LIDAR system. [Figure 26] Figure 26A shows a user viewing an image collected by the camera in Figure 26B. Figure 26B shows the positions of two moving cameras on a convex surface corresponding to the HDU direction in Figure 26A. Figure 26C shows a user viewing an image collected by the camera in Figure 26D. Figure 26D shows the positions of two moving cameras on a convex surface corresponding to the HDU direction in Figure 26C. [Figure 27] Figure 27A shows a user viewing images collected by the camera in Figure 27B. Figure 27B shows two cameras on a convex surface selected corresponding to the HDU orientation in Figure 27A. Figure 27C shows a user viewing images collected by the camera in Figure 27D. Figure 27D shows a position camera on a convex surface corresponding to the HDU orientation in Figure 27C. [Figure 28] FIG. 28 shows the grid system used by the left-eye camera to generate the corresponding left-eye view and by the right-eye camera to generate the right-eye view. [Figure 29] Fig. 29A is an external view of a head display unit with a stereoscopic camera, and Fig. 29B is a view of the inside of the head display unit with a stereoscopic camera (looking inside). [Figure 30] Figure 30A shows a head display unit with two stereoscopic camera clusters, and Figure 30B shows the camera clusters. [Figure 31] FIG. 31 shows an example of one-way stereo interaction between two or more users. [Figure 32] FIG. 32 shows an example of a two-way stereo interaction between two or more users. [Figure 33]Figure 33A illustrates stereoscopic communication between a first user wearing an HDU with a stereo display and working on a laptop with a stereo camera, and a second user wearing an HDU with a stereo display and working on a laptop with a stereo camera. Figure 33B illustrates stereoscopic communication between a first user with a laptop equipped with a stereo camera and a second user wearing an HDU with a stereo display. Figure 33C illustrates stereoscopic communication between a first user with a laptop and a second user wearing an HDU and a mobile phone with a stereo camera. Figure 33D illustrates stereoscopic communication between a first user wearing an HDU with a stereo display and working on a laptop with a stereo camera, and a second user wearing an HDU with a stereo display and working on a mobile phone with a stereo camera. [Figure 34] Figure 34A shows a user wearing an HDU holding a stereoscopic camera smartphone in a first position, Figure 34B shows a user wearing an HDU holding a stereoscopic camera smartphone in a second position, and Figure 34C shows a user wearing an HDU holding a stereoscopic camera smartphone in a third position. [Figure 35] Figure 35A is a top view of a cameraman with a wheeled camera cluster, and Figure 35B is a front view of the wheeled camera cluster looking at the cameraman. [Figure 36] Figure 36A is a top view of a soccer field showing the placement of multiple camera clusters, and Figure 36B is a side view of the soccer field showing the placement of multiple camera clusters. [Figure 37] Figure 37A shows a stereoscopic camera cluster with different lens sets. Figure 37B is a top view of cross section A of Figure 37A. Figure 37C is a top view of cross section B of Figure 37A. Figure 37D is a top view of cross section C of Figure 37A. [Figure 38] FIG. 38 shows a user walking around the city and recording stereo images. [Figure 39]Figure 39A is a top view of a stereoscopic camera cluster on a head display unit, and Figure 39B is a side view of the stereoscopic camera cluster on a head display unit. [Figure 40] Figure 40A is a top view of the instantaneous field of view (FOV) of a stereoscopic camera system. Figure 40B is a top view of the instantaneous field of view (FOV) of a stereoscopic camera system. [Figure 41] Figure 41A shows an example of a user's image and their natural background scene (left) and Figure 41B shows an example of a user's image and their simulated stereoscopic background scene (left). [Figure 42] Figure 42A shows a left view of a user's image and the implementation of the user's dynamic simulated stereoscopic background scene. Figure 42B shows a left view of a user's image and the implementation of the user's dynamic simulated stereoscopic background scene. [Figure 43] Figure 43A is a left view of a stereoscopic image of a first user. Figure 43B is a left view of a stereoscopic image of a first user. 4300b shows a first user with a tanned face. [Figure 44] Figure 44A shows the left-view perspective of a set of stereoscopic images of a user with a real dress appearance, and Figure 44B shows the left-view perspective of a set of stereoscopic images of a user with a simulated dress appearance. [Figure 45] Figure 45A shows a wide field of view stereoscopic camera system being installed at the scene of an ongoing event, Figure 45B shows a first user viewing a first view of the scene being placed in a stereoscopic HDU, and Figure 45C shows a second user viewing a second view of the scene being placed in a stereoscopic HDU. [Figure 46] FIG. 46 shows text boxes that indicate gestures or actions the user can take to modify the stereo image. [Figure 47] FIG. 47 is a flow diagram for integrating a stereo camera with an HDU. [Figure 48] Figure 48 shows an example of optimizing image acquisition for a stereo camera using LIDAR. [Figure 49]Figure 49 shows a stereoscopic LIDAR system on an HDU. [Figure 50] Figure 50A illustrates the use of a common aperture for multiple lenses to overcome the "image shift" that can be annoying to users when switching cameras. Figure 50B illustrates the use of a common aperture system with the mirror in a first position. Figure 50C illustrates the use of a common aperture system with the mirror in a second position. DETAILED DESCRIPTION OF THE INVENTION

[0035] The flow diagrams do not depict the syntax of any particular programming language. Rather, the flow diagrams indicate the functional information necessary for one skilled in the art to create circuits or generate computer software to perform the necessary processing in accordance with the present invention. It should be noted that many routine program elements, such as loops and variable initialization, the use of temporary variables, etc., are not shown. Those skilled in the art will understand that, unless otherwise indicated herein, the specific order of steps described is exemplary only and may be changed without departing from the spirit of the present invention. Thus, unless otherwise noted, the steps described below are in no particular order, implying that, where possible, the steps may be performed in any convenient or desirable order.

[0036] Figure 1A shows a top view of the video recording. 100 indicates the video camera. 101 indicates the field of view of the video camera.

[0037] 1B shows a displayed 2D image frame of a video recording, 102. This 2D image frame can be displayed on a large screen in a cinema or on a home television.

[0038] Figure 2 shows the system hardware of the present invention. A text box is displayed.

[0039] The key system hardware of this disclosure is the use of a stereoscopic camera system. Stereoscopic cameras mounted on various smart devices are envisioned, including, but not limited to, smartphones, tablets, desktop computer monitors, and laptops. Another embodiment is to mount mobile stereoscopic cameras on a convex grid. Another embodiment is to use a stereoscopic cluster of cameras. The stereoscopic camera system would generate a highly-resolution composite stereoscopic dataset of all objects in a scene. The cameras would have a variable interocular distance depending on the intended application. For example, a laptop would have stereoscopic cameras spaced approximately 2.5 inches apart, which is the standard interocular distance for humans. Other stereoscopic ball camera setups (described later in this disclosure) have stereo separation much greater than the human interocular distance, which is advantageous for providing stereoscopic viewing at long distances. Lenses can be prime lenses or zoom lenses. For example, a zoom lens could use a 24-70mm lens in a full-frame setup.

[0040] Another important system hardware aspect of this disclosure is the use of visual display options. A stereoscopic head display unit (HDU) such as an augmented reality, mixed reality, or virtual reality display can be used. Examples include, but are not limited to, HoloLens, Magic Leap, Oculus, and HTC Vive. Ideally, the stereoscopic HDU would have both head and eye tracking. Next, a large home theater television display (e.g., 180-270° horizontal viewing angle) is available as an option, and various stereoscopic displays can be used in conjunction with this to achieve a wide viewing angle stereoscopic display. Next, a large home theater screen (e.g., 180-270° horizontal viewing angle for the user) combined with a multi-projector system can be used. System interfaces, including an audio system (e.g., microphone and / or speaker) and a communication link, can also be incorporated.

[0041] Finally, the system hardware also includes supporting accessories. First, a global positioning system (GPS) can be integrated into the camera system. Second, an inertial measurement unit (IMU) system can be integrated into the camera system. Second, a graphical user interface (GUI) can be integrated, such as a keyboard, joystick, mouse, or controller. Furthermore, a lidar system can be integrated into the camera system or head display unit, as described in U.S. patent application Ser. No. 16 / 828,352, which is incorporated by reference in its entirety. Next, a computer system is required. Next, a speaker would be used to provide sound to the user. A power source is required. A microphone is required for control via voice commands. A recorder is required along with the camera system for image playback. A tripod is required. Cables are also used as needed.

[0042] Figure 3 illustrates the system software. First, system initialization software and system control software are required. Software is needed to provide communication between the major components of the system, such as the camera system, HDU, image recorder, and computer system. Software for dataset processing is also required. For example, the camera system may generate a dataset of a first size (e.g., 150 megapixels (MP) per frame) but present the data to the user at a much smaller size (e.g., 6 MP per frame). In some embodiments, large datasets are stored in the cloud and streamed (e.g., at 5G rates) during viewing. Therefore, communication software (e.g., with the Internet) is required. In some embodiments, the portion of the dataset downloaded can be based on the user's gaze angle. For example, if the user is looking forward, the front portion of the scene is streamed. And if the user is looking left, the left portion of the scene is streamed. Next, camera selection software (in multiple camera systems, as described later in this disclosure) is required. Next, software for control input to the camera system is required. For example, input could be from a rollerball mouse GUI, voice commands via the recorder, or input from HDU sensors (e.g., head and eye tracking). Next, image stabilization software is required. Next, target tracking software is required. For example, suppose the target is a defensive back on a football team. Next, scene recognition software is required. For example, a wildlife conservationist looks for animals and dangerous areas. Next, software is required to integrate target tracking with the movement of the camera on a grid. Next, image optimization software is required, which includes optimizing image acquisition from the camera system or optimizing the image after acquisition. For example, the camera system can perform tasks including, but not limited to, autofocus, convergence angle adjustment, and capturing range data for image optimization. Next, image stitching software is run.For example, in a multiple camera cluster system where the first camera in the camera cluster has an overlapping field of view with the second camera in the camera cluster, the two images can be stitched together to remove the overlap and create one large composite image. Software has been implemented to integrate multiple camera systems (e.g., dual cameras on a convex grid system using HDUs). Finally, software has been implemented for power supply monitoring and management.

[0043] FIG. 4 illustrates mapping the camera coordinate system to the GPS coordinate system. 400 illustrates a process block for determining the precise initial camera position (e.g., determining the camera's GPS coordinates at t=0). Note that calibration is also possible using an object with a known GPS location, such as a flagpole. 401 illustrates a process block for initializing the precise camera orientation (e.g., initializing an inertial measurement unit in conjunction with a GPS to determine the camera's orientation and level relative to true north and horizon, which are part of the overall camera system). For example, with respect to the coordinate system outlined in FIG. 5A, at a first point in time, the alpha angle of the overall camera setup can be 90°, and the camera beta angle can be 0°, corresponding to true north and horizon. A low-cost system could also use a carpenter's level and compass. At this point, the camera system is georegistered. Note that a head tracking system can also be incorporated. 402 illustrates a process block for initializing an eye-tracking system for any camera pair. 403 indicates a processing block that records the camera position, camera orientation (to account for head position changes if the camera is on an HDU), and eye movements (e.g., for camera pair selection) over time for each time point. Note: The coordinate system in Figure 5A is fixed relative to the camera. Note: The camera system orientation can be remapped to a GPS and north orientation system based on data from the IMU. For example, at time point #1, the camera's pointing direction (defined as an alpha angle of 90°) may coincide with true north, which corresponds to true north. At time point #2, the IMU may determine that the camera has turned northeast (45°); therefore, an alpha angle of 90° corresponds to true northeast 45°. 404 indicates a processing block that records images from the camera for each time point. Note: Image stabilization may be enabled. Note: Some cameras may not be capable of continuous recording. 405 denotes a processing block that assigns GPS locations to objects in the scene based on a combination and integration of LiDAR range, camera position, and camera orientation (which may change based on head tracking, for example. For example, this may be done automatically through scene mapping or by the user placing crosshairs on the object.The intersection / excision approach can be used by two geographically separated observers equipped with camera systems to fine-tune the position of an object. Note: Eye tracking can also be used to determine where a user looks over time. Audio recordings can also complement the images.

[0044] Figure 5A is a view looking down from point D toward the xy plane. This shows the XY plane. The camera is placed at the origin. The origin can be referenced by the GPS location. The Y axis is defined as the direction the video camera is pointing. In the case of two cameras in stereo, the Y axis is defined as the direction from the midpoint of the two cameras to the forward direction the cameras are looking. When multiple cameras are used, any camera can be defined as the Y axis. The angle, +α or -α, is used to describe the first coordinate of a point in the scene. Using this coordinate system, the α of point A is 0°, the α of point B is approximately +45°, and the α of point C is +90°.

[0045] Figure 5B shows a view of the yz plane from point C. The video camera is placed at the origin. The z axis is perpendicular (normal) to the xy plane. The angle, +β or *β, is used to describe the second coordinate of a point in the scene. Using this coordinate system, point A's β is 0°, point B's β is +45°, and point D's β is +90°. Each point also has a third coordinate, which is its distance from the origin. Thus, each point's coordinates are (α, β, r), where r is the distance from the point to the origin. Thus, point A's coordinates are (0°, 0°, 16m). Point B's coordinates are (+45°, +45°, 13m). Point C's coordinates are (+90°, 0°, 25m). Point D's coordinates are (0°, +90°, 25m). Note that the origin of this camera system is at a specific GPS location on the map. Objects in the scene can then be mapped into this preferred camera coordinate system, and because the camera coordinate system can be mapped (and remapped if the camera moves or changes orientation), objects in the camera image can also be assigned GPS coordinates.

[0046] FIG. 6A is a diagram in which the viewing positions of multiple viewers are plotted on a coordinate system.

[0047] Figure 6B is a table showing the viewing locations of multiple viewers in Figure 6A. Seven time points are shown for three viewers. Each viewer has their own viewing pattern. In this disclosure, each viewer is provided with a separate set of stereoscopic images.

[0048] FIG. 7A shows time point #1 where three observers are looking down at a cityscape. Observer #1's field of view at time point #1700a is such that observer #1 is positioned to observe a park where several people are outside moving around. Observer #1's field of view is a small oval field of view at park level. Observer #2's field of view at time point #1701a is such that observer #2 is positioned to observe traffic along a street. Observer #2's field of view is rectangular and encompasses a large portion of the street. Observer #2's field of view is a narrow trapezoidal field of view at street level. Observer #3's field of view 1702a at time point #1 is such that observer #3 is positioned to observe a flagpole on top of a building. Observer #3's field of view is narrow. Observer #3's field of view is a trapezoidal field of view above camera level.

[0049] Figure 7B shows time point #2 where the three observers from Figure 7A are still looking down at the cityscape, but at different parts of the scene. Observer #1's field of view at time point #2700b is positioned so that observer #1 is looking at the traffic pattern along the street. Observer #1's field of view at time point #2 shows an elliptical field of view at street level. Observer #2's field of view at time point #2701b is positioned so that observer #2 is looking at a park where people are moving around. Observer #2's field of view at time point #2 shows a trapezoidal field of view at park level. Observer #3's field of view at time point #2702b is positioned so that observer #3 is looking at the entire building, including the flagpole at the top. Observer #3's field of view is narrow. Observer #3's field of view has a trapezoidal field of view above and below the camera level.

[0050] Figure 8A shows a downward view from a nearly spherical stereoscopic camera system mounted on a pole in a typical city center. The black circle in the center of the figure represents the location of the nearly spherical stereoscopic camera system at the origin of this coordinate system. The camera is mounted on a pole 20 feet above the ground. The XY plane is parallel to the ground but 20 feet above it. The nearly spherical stereoscopic camera system can capture very high-resolution images anywhere there is line of sight. The camera system operates continuously, providing near, mid, and far-field coverage in any direction selected by the viewer. In this way, a user in Orlando can select a portion of the image flow from the nearly spherical stereoscopic camera system to see what is happening in the selected area of ​​interest. Suppose a user in Orlando wants to see the flag mounted atop building 800, which has coordinates (+45°, +20°, 2000 ft). Similarly, a man in London can select a portion of the image that interests him. In this example, the man in London is interested in car 802, which has coordinates (+90°, -30°, 300 ft). Similarly, a woman in Tokyo can select a portion of the image that interests her. She is interested in seeing person 801 with coordinates (-45°, -60°, 100 ft.). Dashed circle 803 indicates the area where two closely spaced (e.g., 10 inches apart) cameras from a camera system can be used together in stereo mode to produce a convergent stereoscopic image. Dashed circle 804 indicates the area where two medium-spaced (e.g., 20 inches apart) cameras from a camera system can be used together in stereo mode to produce a convergent stereoscopic image. Note that dashed circle 804 indicates the area where stereoscopic viewing is not possible. Note that multiple cameras are used to create the composite image; therefore, the images from each camera are stitched together.

[0051] FIG. 8B shows the β coordinates corresponding to the object seen in FIG. 8A.

[0052] FIG. 9A shows a natural scene, showing multiple objects at different distances from the user. The camera system functions as a near-real-time system that delivers the best possible image based on tracking the user's gaze at various moments. The image displayed at this point includes several objects that may be of interest to the person observing the scene. In this embodiment, an eye-tracking system (e.g., on a head display unit) is required. For example, if the user is looking at mountains 900 or clouds 901, the camera used (e.g., mounted on a tripod or HDU) is optimized for long-distance viewing. If the user is looking at deer 902, the camera used to deliver images to the user is optimized for medium distances. Finally, if the user is looking at nearby flowers, the camera used to deliver images to the user is optimized for closer distances. Example scenario: A user is walking around creating a travelogue of a national park. The user has a headset with two pairs of stereo cameras and one long-range camera. Recording is done in real time. The cameras can be prime or zoom lenses. For example, a first pair of zoom lenses optimized for stereo images of nearby objects (e.g., at arm's length, such as flower 902) is utilized. A second pair of zoom lenses optimized for stereoscopic images of mid-distance objects (e.g., deer 902) is utilized. A single camera is utilized for distant objects (e.g., mountains 900 or clouds 901). The headset switches between the camera pair and eye tracking depending on where the user is looking to provide dialogue. In some embodiments, capture is done with close convergence for close objects, medium convergence for mid-distance objects, and infinite convergence for long-distance objects, so that only one range of images is collected at a time. In other embodiments, the cameras capture images simultaneously, and the images can be stitched together into a single composite image.

[0053] 9B is a table showing the objects, ranges, camera angles, cameras used, and notes. Note that if desired, the stereo distance for objects in the middle distance can be wider than the stereo distance for objects in the close distance.

[0054] Figure 10A shows a direct view (looking straight at the edge of the table) from the left camera aimed at two objects on a table (a can and a honey jar), with the near object slightly overlapping the far object.

[0055] FIG. 10B shows a second image from the left camera in the same position but tilted 20° inward (toward the can and honey jar). Note that compared to FIG. 10A, the objects appear slightly further away and more of the honey jar's label is visible. Note that this 20° tilted image also includes a chair and a door (not visible in FIG. 10A). Thus, canting the camera inward improves upon the prior art by creating a more visually appealing image by implementing a physiologically correct effect of varying the overlap of two structures (the honey jar and the can) on the canted inward image. Furthermore, this technique is effective at utilizing the central (rather than peripheral) components of the image, resulting in superior image quality.

[0056] Figure 10C is a direct view (looking directly at the edge of the table) from the left camera aimed at two objects on a table (a can and a honey jar), with the near object slightly overlapping the far object.

[0057] Figure 10D shows a second image from the left camera in the same position but tilted 20° inward (towards the can and honey jar). Note that compared to Figure 10C, the objects appear slightly further away and more of the honey jar label is visible. Note that the chair and door (not visible in Figure 10A) are visible in this 20° tilted image.

[0058] Figure 11 illustrates the integration of a stereoscopic camera system into a smart device. This diagram shows various stereoscopic camera systems and stereoscopic display systems operating in conjunction with the Internet 1100, facilitating the interchange of stereoscopic cameras between users. A laptop computer is shown in 1101, depicting two solid-colored stereoscopic cameras. If these stereoscopic cameras are located inside the laptop (e.g., above the laptop's display screen) and the cant angle of the laptop's display is such that these cameras are focused on the laptop user's face, stereoscopic images can be transmitted to the intended recipient over the Internet. These stereoscopic cameras are separated by an interocular distance, which may include, but is not limited to, a standard for typical or average interactions among the adult population, adjustable to suit the recipient's facial structure, closer to suit different age groups in the youth population, wider for people with larger facial structures, etc. The gaze angles of these stereoscopic cameras can be tilted to converge at a nominal distance from the laptop user's face. This convergence point can also be adjusted. This will produce the most physiologically accurate images. A variation on the stereoscopic camera location is to place it on the outside (i.e., on the outside side of the display as seen by the laptop user). The interocular distance may be as described above. In this configuration, the stereoscopic camera points straight ahead and can provide the recipient with a stereoscopic image of the scene seen by the laptop user. The stereoscopic camera's connection to the Internet is shown as one-way communication, in that the laptop is not typically considered a stereoscopic unit. However, if the laptop user were wearing an AR / MR / VR headset 1102 or smart glasses 1103, the arrow connecting the laptop to the Internet could be bidirectional. Additionally, the stereoscopic camera could be used to record a discussion or lecture on the laptop. The discussion or lecture could then be disseminated via the Internet or a digital recording device at a later date. 1104 shows a stereoscopic camera mounted on a tablet. The discussion regarding interocular distance, convergence, and Internet connection is similar to that provided for the laptop.1105 denotes a smartphone. In this configuration, stereo cameras are located on both sides of the phone, allowing the user to hold the phone at a distance (e.g., from the face) and use the display side of the smartphone for interpersonal stereoscopic interaction. Alternatively, the back of the smartphone can be used to point the stereo cameras at an object or a general scene. Smartphones may be manufactured with a pre-set interocular distance. The convergence of the front and back stereo cameras could be handled similarly to a laptop. The connection between a smartphone and the Internet is typically one-way, with stereo images being sent from the smartphone. However, if the smartphone user were wearing an AR / MR / VR headset 1102 or smart glasses 1103, the arrow connecting the laptop to the Internet could become bidirectional. 1102 denotes an AR / MR / VR headset capable of two-way communication over the Internet. Stereo cameras can be embedded in the headset near the wearer's eyes. These stereo cameras collect stereo images of objects and scenes in front of the wearer and transmit the stereo images over the Internet. At the same time, the wearer can view the stereo images from another person equipped with a stereo camera, as shown in this figure. These stereo cameras nominally face straight ahead but can be adjusted to converge. The stereoscopic image recording capabilities of the headset are described in later figures. 1103 denotes smart glasses. The stereoscopic cameras can be embedded in the smart glasses near the wearer's eyes. Smart glasses have various modes of operation, collecting and transmitting stereoscopic images while simultaneously providing a "see-through" function, where the see-through function is opaque and the stereoscopic images are viewed. The stereoscopic image recording capabilities of the smart glasses are described in later figures. The stereoscopic image recording capabilities of the headset are described in later figures. 1106 denotes a typical desktop or deskside computer with a monitor 1107 for display. The stereoscopic cameras are attached to the monitor 1107, and the interocular distance, convergence, and communication with the Internet are similar to a laptop. However, the monitor 1107 has stereoscopic cameras attached only to the display side of the monitor.Other embodiments include a handheld stereoscopic camera 1108. In some embodiments, smart devices can connect directly (e.g., via Bluetooth) to stereoscopic devices, including smart glasses 1103 and AR / MR / VR headsets 1102.

[0059] 12A is an illustration of a smart device equipped with stereoscopic cameras. In a preferred embodiment, the stereoscopic cameras are positioned at a predetermined interocular distance on the smart device 1200. The cameras are positioned so that the areas where the stereoscopic images are generated overlap.

[0060] Figure 12B shows a stereo camera ball. In a preferred embodiment, the stereo camera ball moves and focuses to observe an object of interest. The stereo sphere camera 1201 can be a sphere or other circular or convex shape. The position of the stereo camera changes over time depending on the desired viewing direction and stereo separation, which may be from target tracking or from the user. Increasing the separation of a convex sphere system allows for stereo viewing at longer distances.

[0061] 12C shows a head display unit (HDU) with two stereoscopic camera clusters mounted near where the wearer's eyes would be when wearing the HDU. HDU 1202, which includes a first stereoscopic camera cluster and a second stereoscopic camera cluster, can be an augmented reality (AR), mixed reality (MR), or virtual reality (VR) unit.

[0062] Figure 12D shows a multi-stereo camera system / camera cluster. The cameras can be individual cameras or camera clusters. Based on where the action or subject of interest is, a camera or camera cluster will be selected to capture the subject of interest in high resolution. Different users can select different areas of interest and view different parts of a particular event, such as a football game.

[0063] FIG. 13 shows an example of a stereo camera configuration for a smart stereo camera. 1301 is a right view of a first stereo camera pair optimized for close range. 1302 is a right view of a first stereo camera pair optimized for close range. 1303 is a right view of a first stereo camera pair optimized for medium range. 1304 is a right view of a first stereo camera pair optimized for medium range. 1305 is a right view of a first stereo camera pair optimized for long range. 1306 is a right view of a first stereo camera pair optimized for long range. Note that different cant angles can be used depending on the stereo camera pair. Variable focal length zoom lenses (70-200 mm, 10-18 mm, etc.) or prime lenses (17 mm, 85 mm, etc.) can be used. For example, a 70-200 zoom range has a ratio of approximately 3:1, which is expressed as a 3x zoom. For example, with a zoom lens, you can capture a zoomed-out image (e.g., 10mm), but zooming in (e.g., 18mm) narrows the angle of view, causing objects in the image to appear closer. For example, with an 85mm prime lens, you must physically move the camera inward toward the image to change the angle of view and bring the subject closer. Note that an 85mm prime lens has a focal length of 85mm. Generally, lenses with longer focal lengths are longer, and lenses with shorter focal lengths are shorter. On a full-frame camera (no crop factor), a 50mm lens roughly corresponds to the human eye. Canon full-frame sensors measure 24 x 36mm. Zoom settings less than 50mm, such as 24mm, appear as wide-angle zooms. On the other hand, longer lens settings, such as 70mm, appear as narrow-angle telephoto sensors. A lens with a crop factor of 1.6 corresponds to a 32mm lens. Wide-angle lenses are useful for capturing landscapes or large groups of people, creating a strong sense of perspective. A long-angle telephoto setting is ideal for wildlife photography. The aperture is the opening that lets light into the camera. For example, the f-number is the size of the opening; the higher the number, the smaller the opening. Every lens has a maximum f-number. The size of the aperture determines how much light passes through the lens. This affects shutter speed and exposure. The f-number is the ratio of the aperture to the focal length of the lens you are using.For example, if the aperture is 1 inch and the lens is a 50mm lens (about 2 inches), the F-stop is F / 2. The F-stop also determines the depth of field, or the range in focus. For example, the iPhone 12 has two cameras, including an ultra-wide and a wide-angle, while the iPhone 12 Pro has an ultra-wide, wide-angle, and telephoto lens.

[0064] 14A illustrates an example stereo camera configuration of a smartphone at a first time point. 1400a illustrates the smartphone. 1401a illustrates a first position of the right stereo camera of smartphone 1400a. 1402a illustrates a first position of the left stereo camera of smartphone 1400a. Note that in some embodiments, right stereo camera 1401a and left stereo camera 1402a have a first cant angle and a first stereo separation.

[0065] FIG. 14B illustrates an example stereo camera configuration for a smartphone at a first time point. 1400b illustrates the smartphone. 1401b illustrates a second position of the right stereo camera of smartphone 1400b. 1402b illustrates a second position of the left stereo camera of smartphone 1400b. Note that in some embodiments, right stereo camera 1401b and left stereo camera 1402a have a second cant angle and a second stereo separation. The second cant angle is different from the first cant angle. The second stereo separation is different from the first stereo separation. By utilizing a narrow stereo separation (less than 1 cm) with an appropriate cant angle, stereo images of very small structures (e.g., insects such as ants) can be obtained, which can be viewed on the HDU.

[0066] 15A shows a top view in which both the left and right cameras have a first cant angle relative to a first convergence point. 1500a shows the first convergence point. 1501a shows the left camera with a first cant angle that is approximately α=0°. 1502a shows the right camera with a first cant angle that is approximately α=0°.

[0067] FIG. 15B is a top view in which both the left and right cameras have a second cant angle for the second convergence point. 1500b shows the second convergence point closer to the left camera 1501b and the right camera 1502b compared to the configuration of FIG. 15A. 1501b shows the left camera with a second cant angle, which is approximately α=+20°. 1502b shows the right camera with a second cant angle, which is approximately α=-20°. Note that the cant angle is steeper compared to FIG. 15A. Note that the closer the convergence point, the steeper the cant angle (compare to FIG. 15A).

[0068] Figure 16A shows a top view where both the left and right cameras have a first stereo separation due to a first convergence point. 1600a shows the first convergence point. 1601a shows the left camera and 1602a shows the right camera. Note the first stereo separation 1603a.

[0069] Figure 16B shows a top view in which both the left and right cameras have a first stereo separation relative to a second convergence point. 1600b shows the second convergence point. 1601b shows the left camera and 1602b shows the right camera. Note that the second stereo separation 1603b is closer than in Figure 16A.

[0070] 17A shows a top view of the left and right cameras with a first cant angle for a first stereo separation and a first convergence point. 1700a shows the first convergence point. 1701a shows the left camera with a first cant angle that is approximately α=0°. 1702a shows the right camera with a first cant angle that is approximately α=0°. 1703a shows the first stereo separation.

[0071] FIG. 17B shows a top view in which the left and right cameras have a second cant angle for a second stereo separation and a second convergence point. 1700b shows the second convergence point closer to the left camera 1701b and the right camera 1702b compared to the configuration in FIG. 17A. 1701b shows the left camera with a second cant angle of approximately α=+20°. 1702b shows the right camera with a second cant angle of approximately α=-20°. Note that the cant angle is steeper compared to FIG. 17A. Note that the cant angle is steeper as the convergence point is closer (compare to FIG. 15A). Note that the stereo separation 1703a is reduced compared to FIG. 17A. This combination produces physiologically accurate images.

[0072] Figure 18A shows the arrangement of some of the horizontal cameras in two (left and right) stereoscopic camera clusters. The two camera cluster systems (one for the left eye and one for the right eye) provide stereoscopic vision of high-resolution images of the surroundings in near real time. In the left camera cluster, the cameras are labeled (from outside to inside) as "1H," "2H," "3," "4H," and "5H." In the right camera cluster, the cameras are labeled (from outside to inside) as "5H," "4H," "3," "2H," and "1H." For example, a user can select camera 5H from the left camera cluster and camera 1H from the right camera cluster for near convergence to a near convergence point. For example, a user can select camera 4H from the left camera cluster and camera 2H from the right camera cluster for mid-distance convergence to a mid-distance convergence point.

[0073] Included is this table, "Use Dual Camera Cluster When Walking Around," which corresponds to Figure 18A. In the table below, "L" stands for left, "R" stands for right, and "H" stands for horizontal, and the numbers indicate the selected camera.

[0074] [Table 1]

[0075] Figure 18B shows the arrangement of some of the horizontal cameras of two (left and right) stereoscopic camera clusters. The two camera cluster systems (one for the left eye and one for the right eye) provide stereoscopic viewing of high-resolution images of the surroundings in near real time. For the left camera cluster, two columns are shown. The rows of the left camera cluster are labeled (from top to bottom): "1V," "2V," "3," "4V," and "5V." The inner columns of the left camera cluster are labeled (from top to bottom): "4,2," "4H," and "4,4." For the right camera cluster, two columns are shown. The rows of the right camera cluster are labeled (from top to bottom): "1V," "2V," "3V," "4V," and "5V." The inner columns of the right camera cluster are labeled (from top to bottom): "2,2," "4H," and "4,4."

[0076] Included is this table "Use Dual Camera Cluster When Walking Around" which corresponds to Figure 18B. In the table below, "L" stands for left, "R" stands for right, "V" stands for vertical, and the number stands for the selected camera.

[0077] [Table 2]

[0078] The cameras shown here are just an example, and by selecting various other cameras, a more complex camera cluster can be configured.

[0079] For example, imagine a son is touring the Sistine Chapel. Suppose the son is wearing an HDU with two camera clusters. Suppose the mother is not present but wants to join in on this spectacular tour. The mother controls the two camera clusters and can see what she wants to see. Suppose a pair of camera clusters is located in front of the user's HDU. Suppose the mother sees a beautiful painting on the right. She can use the above cameras in the camera cluster to look over to the right and see and appreciate the painting. When she looks elsewhere (by a saccadic eye movement), the HDU switches its field of view to the new location. As shown in FIG. 19, in some embodiments, she could switch to a different pair of camera clusters to get a better view of the painting on the right.

[0080] 19A is a top view of a first head display unit with two stereoscopic camera clusters that provide stereoscopic vision in the forward direction. 1900 shows the first head display unit. 1901 shows the first camera cluster for the user's left eye. 1902 shows the second camera cluster for the user's right eye. A user wearing the HDU can see images from this pair of stereoscopic camera clusters, allowing them to perceive depth and view 3D images.

[0081] FIG. 19B is a top view of a second head display unit with seven camera clusters that provides stereoscopic viewing with a horizontal field of view of approximately 180°. 1903 indicates the second head display unit. 1904 indicates the first camera cluster. 1905 indicates the second camera cluster. 1906 indicates the third camera cluster. 1907 indicates the fourth camera cluster. 1908 indicates the fifth camera cluster. 1909 indicates the sixth camera cluster. 1910 indicates the seventh camera cluster. Different pairs of camera clusters can also be used to provide stereoscopic images from different angles. For example, selecting the first camera cluster 1904 and the second camera cluster 1905 can perform stereoscopic imaging on the right side of the HDU. Furthermore, such a configuration can also provide stereoscopic images at longer distances with a larger stereo distance, such as selecting the first camera cluster 1904 and the third camera cluster 1906. Furthermore, stereoscopic viewing can be achieved over very long distances by collecting images via two HDUs spaced apart from each other, and extensions can be used to increase stereo separation.

[0082] FIG. 19C is a top view of a head display unit with ten stereoscopic camera clusters spaced apart to provide 360° stereoscopic viewing. 1911 indicates the HDU. 1912 indicates the first camera cluster. 1913 indicates the second camera cluster. 1914 indicates the third camera cluster. 1915 indicates the fourth camera cluster. 1916 indicates the fifth camera cluster. 1917 indicates the sixth camera cluster. 1918 indicates the seventh camera cluster. 1919 indicates the eighth camera cluster. 1920 indicates the ninth camera cluster. 1921 indicates the tenth camera cluster. The cameras track the eye and head movements of a user geographically distant from the cameras, and the cameras have variable zoom capabilities. For example, imagine a son standing in the center of an art gallery, and his mother chooses to view a statue behind him from a distance. She selects the eighth camera cluster 1919 and the tenth camera cluster 1921 and can view the statue in 3D using depth perception in HDU.

[0083] FIG. 20A is a side view of a hemispherical high-resolution camera cluster.

[0084] FIG. 20B is a bottom view of a hemispherical high-resolution camera cluster.

[0085] Figure 20A is a side view of a hemispherical high-resolution camera cluster. As shown, multiple cameras (represented by black circles) are positioned at various locations along the hemisphere (large gray circle). The camera characteristics (focal length, f-stop, aperture, etc.) may be the same or different. Any two cameras can be used in a stereo pair. Furthermore, the selected stereo pair may change over time. For example, the side of a hemispherical high-resolution camera cluster can be designed to provide stereoscopic images even at a distance of 50 yards. Considering that the human eye is typically 2.5 inches away, good stereoscopic images are obtained up to about 20 feet, and stereoscopic images are desired at 50 yards, the separation between the two cameras in the hemispherical high-resolution camera cluster would be 18.75 inches. For example, in the first scenario, if the first football play was made at the home team's 10-yard line, a stereo pair aimed at the home team's 10-yard line could be used. Alternatively, if the second football play was made at the 50-yard line, a stereo pair aimed at the 50-yard line could be used. In the second scenario, the quarterback is at the 30-yard line and throws a pass to a receiver in the end zone. The viewer can view images from one set of cameras at a first time epoch to focus on the quarterback, and then view images from a second set of cameras at a second time epoch to see the action in the end zone. The structures on which the cameras are mounted can be of various sizes and shapes (i.e., various curved surfaces).

[0086] Figure 20B shows a bottom view of a hemispherical high-resolution camera cluster, with multiple cameras (shown as black circles) positioned at various locations along the hemisphere (large gray circle).

[0087] FIG. 21A is a bottom view of a convex surface containing two high-resolution camera clusters in a first position. 2100a shows the convex surface. Note that the surface is the preferred embodiment; however, other methods include placing the cameras on a robotic arm to achieve the desired stereo separation, desired position, and desired cant angle. 2101a shows the first camera (or first camera cluster) in the first position. 2102a shows the second camera (or second camera cluster) in the first position. This system could be mounted on a mobile object, such as a wheeled device, a flying drone, a wire above a soccer field, or a head display unit.

[0088] Figure 21B shows a bottom view of a convex surface containing two high-resolution camera clusters in a second position. A key improvement of this invention is the ability to move positions to achieve dynamic stereo separation, dynamic cant angles, and dynamic positions. These can be controlled automatically or by user input. 2100b shows a convex surface. Note that the surface is the preferred embodiment; however, other approaches include placing the cameras on a robotic arm to achieve the desired stereo separation, desired position, and desired cant angle. 2101b shows the first camera (or first camera cluster) in a second position. 2102b shows the second camera (or second camera cluster) in a second position.

[0089] 22A is a top view of a user's eye, showing the user's eye looking straight ahead. 2200a is the user's left eye, looking straight ahead. 2201b is the user's right eye, looking straight ahead.

[0090] Figure 22B is a top view of two cameras on a convex surface, showing the cameras facing straight ahead. 2202a shows the convex surface, which in this example is a spherical surface. 2203a is the camera corresponding to the user's left eye, facing forward. 2204a is the camera corresponding to the user's right eye, facing straight ahead.

[0091] Figure 22C is a top view of the user's eyes, showing them in a position looking to the right. 2200b is the user's left eye, positioned to look to the right. 2201b is the user's right eye, positioned to look to the right.

[0092] FIG. 22D is a top view of two cameras on a convex surface, showing the cameras looking straight ahead. 2202b shows the convex surface, which in this example is a spherical surface. 2203b is the camera corresponding to the user's left eye, which is now in the correct position to look. 2204b is the camera corresponding to the user's right eye, which is now in the correct position to look. Thus, in some embodiments, the position of the cameras is determined by the position of the user's head (which is determined by a head tracking system on the HDU, e.g., an inertial measurement unit (IMU)), and the orientation of the cameras is determined by the user's gaze angle (which is determined by an eye tracking system on the HDU).

[0093] FIG. 23 shows a bottom view of a large convex surface containing three stereo camera pairs (or camera cluster pairs), shown moving over time. 2300 shows the convex surface. In this embodiment, the convex surface is positioned on the 50-yard line of a football field. 2301a shows a camera (or camera cluster) of a first stereo pair, moving over time along the dashed line. 2301b shows a camera (or camera cluster) of a first stereo pair, moving over time along the dashed line. This first stereo pair may represent camera positions for tracking a player running down the center of the field over time. 2302a shows a camera (or camera cluster) of a second stereo pair, moving over time along the dotted line. 2302b shows a camera (or camera cluster) for the second stereo pair, moving over time along the dotted line. This second stereo pair may represent camera positions for tracking a player running down the side of the field over time. 2303a shows a camera (or camera cluster) of a third stereo pair moving over time along a solid line. 2303b shows a camera (or camera cluster) of a third stereo pair moving over time along a solid line. This third stereo pair can represent camera positions for tracking a player across the center of the field over time. The lenses can be wide-angle or telephoto, as desired. A telephoto lens has a narrow FOV and therefore needs to be rotated to point relative to the target. This can be used without the need for electronic zoom. A wide-angle lens has a wide FOV and therefore does not need to be rotated to point relative to the target. This can be used in conjunction with electronic zoom.

[0094] FIG. 24 illustrates a rotatable stereo telephoto lens system for achieving long-range stereoscopic imaging. In some embodiments, using two cameras (or camera clusters) with telephoto capabilities spaced approximately 100 feet apart, stereoscopic imaging of an object 2401 can be performed at a distance of approximately two miles. Note that the cameras are mounted on a rotatable device, allowing the cameras to be pointed in a precise manner to generate the proper stereoscopic overlap. Furthermore, placing the cameras on a robotic arm can achieve the desired stereo effect depending on the distance to the target. Note that this could be implemented on the wing of an airplane for stereoscopic imaging of an object (e.g., post-hurricane damage assessment). This could also be used in conjunction with target tracking software to improve search and rescue operations (e.g., after a hurricane).

[0095] Figure 25 shows a top view of a stereoscopic LIDAR system. 2500a shows a first LIDAR device. 2500b shows a second LIDAR device, where the first LIDAR device 2500a is a distance away from the second LIDAR device 2500b. 2502 shows a first object in the area. 2503 shows a second object in the area, where the second object 2503 is hidden behind the first object 2502 from the perspective of the first LIDAR device 2500a.

[0096] Note that the second object 2503 is visible from the perspective of the second LIDAR device 2500b. In some embodiments, the first LIDAR device 2500a and the second LIDAR device 2500b cooperate to generate a stereoscopic image for the user. Note that the first 3D volume can be generated by the first LIDAR device 2500a. Note that the second 3D volume can be generated by the second LIDAR device 2500b. Note that the HDU can display the first 3D volume generated by the first LIDAR device 2500a on a first eye display (left eye display). Note that the HDU can display the 3D volume generated by the second LIDAR device 2500b on a second eye display (right eye display). This may be an improvement over existing technology because it can reveal structures that are not visible from the first perspective. Statistical and artificial intelligence analysis can be performed using the stereoscopic LIDAR system. Other features include placing the LIDAR system on a rotatable platform, or placing the LIDAR system on a device and adjusting the stereo separation to enhance visibility. For example, short-range, medium-range, or long-range stereoscopic LIDAR can be implemented using principles taught elsewhere in this disclosure. In some embodiments, the first LIDAR device 2500a is different from the second LIDAR device 2500b; for example, they may have different frequencies or different timing. In some embodiments, the LIDAR has a vertical scan pattern and / or a horizontal scan pattern. In some embodiments, the stereoscopic LIDAR system is mounted on a head display unit. In some embodiments, the left LIDAR system is mounted adjacent to the user's left eye, and the right LIDAR system is mounted adjacent to the user's right eye.

[0097] FIG. 26A shows a user viewing an image collected by the camera of FIG. 26B.

[0098] FIG. 26B shows the positions of two moving cameras on a convex surface corresponding to the HDU orientation in FIG. 26A.

[0099] FIG. 26C shows a user viewing an image collected by the camera of FIG. 26D.

[0100] FIG. 26D shows the positions of two moving cameras on a convex surface corresponding to the HDU orientation of FIG. 26C.

[0101] Figure 26A shows a user viewing an image collected by the camera in Figure 26B, with the user's head oriented horizontally.

[0102] Figure 26B shows the positions of two moving cameras on a convex surface corresponding to the HDU orientation of Figure 26A. 2600a shows the camera position corresponding to the user's right eye. 2601a shows the camera position corresponding to the user's left eye.

[0103] Figure 26C shows a user viewing an image collected by the camera of Figure 26D, with the user's head tilted, which causes the camera system of Figure 26D to move.

[0104] FIG. 26D shows the positions of two moving cameras on a convex surface corresponding to the HDU orientation of FIG. 26C. 2600b shows the camera position corresponding to the user's right eye. 2601b shows the camera position corresponding to the user's left eye. In this figure, the user is tilting their head. Note that the user can also turn their head left or right or rotate their head up or down, which will trigger a series of processes corresponding to the camera movements. Furthermore, the movement to the camera positions can be based on the user's interocular distance or can be intentionally wide to enhance stereoscopic vision at long distances. Furthermore, the cameras can be closer together and tilted inward for close-up stereoscopic vision.

[0105] Figure 27A shows a user viewing an image collected by the camera in Figure 27B, with the user's head oriented horizontally.

[0106] Figure 27B shows two cameras on a convex surface selected to correspond to the HDU orientation of Figure 27A. 2700a indicates that the selected camera corresponds to the user's right eye (see the user's head position in Figure 27A). 2701a indicates that the selected camera corresponds to the user's left eye (see the user's head position in Figure 27a).

[0107] Figure 27C shows a user viewing an image collected by the camera of Figure 27D, with the user's head tilted, and because the user's head is tilted, a different camera is selected than in the camera system of Figure 27D.

[0108] FIG. 27D shows the position cameras on the convex surface corresponding to the HDU orientation of FIG. 27C. 2700b indicates that the selected camera corresponds to the user's right eye (the user's head is tilted in FIG. 27C). 2701b indicates that the selected camera corresponds to the user's left eye (the user's head is tilted in FIG. 27B). In this figure, the user is tilting their head. Note that the user can also turn their head left and right or up and down, which triggers a corresponding process for selecting the camera system. Furthermore, the camera selection can be based on the user's interocular distance, or can be intentionally wide to enhance stereoscopic viewing at long distances. Furthermore, the cameras can be closer spaced and tilted inward (if desired) for close-up stereoscopic viewing.

[0109] Figure 28 shows the grid system used to generate corresponding left-eye views with the left-eye camera and right-eye views with the right-eye camera. When the system is initialized at t=0, the cameras are evenly spaced around the center point of the grid. The cameras then move to correspond with the left-eye grid and the right-eye grid. For example, a user can look up to 10°, and the camera can move evenly around grid position (0,5); in this way, eye coordinates can be used to guide the camera's position and pointing direction. In addition to gaze angle, the system can also track the user's eyes to generate corresponding convergence. The system can, for example, sit in the chair during a consultation through a virtual doctor scenario.

[0110] FIG. 29A is an external view of a head display unit with stereo cameras. In addition to the stereo cameras, this configuration also includes an eye-safe laser range finder (LRF). The LRF can provide a single pulse that provides range from the wearer of the HDU to a selected object or surface. Alternatively, the LRF can be a scanning laser. Stereo Camera #1 and Stereo Camera #2 are preferably spaced apart by the average interocular distance of an adult. Some embodiments include a transmission / reception recorder that can be connected to the HDU via a connecting cable. Alternatively, the transmission / reception recorder can be housed in the HDU. A bidirectional transmission signal is also illustrated as part of the HDU.

[0111] 29B is a view of the inside of a head display unit with a stereoscopic camera (looking into it), showing an internal view from the left-eye display and the right-eye display.

[0112] Figure 30A shows a head display unit with two stereoscopic camera clusters. An embodiment of the invention is to mount a high-resolution camera cluster, one for each eye.

[0113] Figure 30B shows a camera cluster. The cluster can be low density (e.g., five cameras arranged horizontally with an instantaneous field of view of 420) and cover a 210-degree field of view. Alternatively, the cluster can be denser (e.g., 30 cameras horizontally with an instantaneous field of view of 70 each) and cover a 210-degree field of view. A similar camera setup for a vertical field of view would be included in the cluster. These clusters could work in conjunction with an eye tracker to provide high-resolution or ultra-high-resolution images of wherever the eyes are in the field of view. Alternatively, all cameras could be on all the time, collecting large amounts of data. This footage could be recorded, allowing for future viewing of all data.

[0114] FIG. 31 illustrates an example of one-way stereoscopic interaction between two or more users. 3100 illustrates a step in which user 1 activates a stereo camera system (e.g., laptop, smartphone, etc.). 3101 illustrates user 2 (or an additional user) donning a stereo display unit (e.g., an augmented reality, mixed reality, or virtual reality display). 3102 illustrates utilizing a wireless connection to transmit stereo images from user 1 to user 2 (or an additional user). 3103 illustrates displaying the stereo images from user 1 on user 2's (or an additional user's) stereo display device. Note that the images may be enhanced (such as image stabilization, or other features as described in this disclosure).

[0115] FIG. 32 illustrates an example of two-way stereoscopic interaction between two or more users. 3200 indicates that user 1 activates a stereo camera system (e.g., laptop, smartphone, etc.). 3201 indicates that user 2 activates a stereo camera system (e.g., laptop, smartphone, etc.). 3202 indicates that user 1 wears a stereo display unit (e.g., an augmented reality, mixed reality, or virtual reality display). 3203 indicates that user 2 wears a stereo display unit (e.g., an augmented reality, mixed reality, or virtual reality display). 3204 indicates that wireless connectivity is utilized to transmit stereo images from user 1 to user 2 and to transmit stereo images from user 2 to user 1. 3205 indicates that user 1's stereo images are displayed on user 2's stereo display device and user 2's stereo images are displayed on user 1's stereo display device.

[0116] FIG. 33A illustrates stereoscopic communication between a first user wearing an HDU with a stereo display working on a laptop with a stereo camera and a second user wearing an HDU with a stereo display working on a laptop with a stereo camera. 3300a illustrates a wireless connection, in this case the Internet. 3302a illustrates the first user wearing HDU 3302a and working on a laptop computer with a stereo camera system 3303a. 3304a illustrates the second user wearing HDU 3305a and working on a laptop computer with a stereo camera system 3306a. First user 3301a wearing stereo HDU 3302a can see second user 3304a through images from the second user's stereo camera on laptop 3306a. A second user 3304a wearing a stereo HDU 3305a can see the first user 3301a via a stereo image from the first user's stereo camera system on the laptop 3303a.

[0117] FIG. 33B illustrates stereoscopic communication between a first user with a laptop equipped with a stereo camera and a second user wearing an HDU equipped with a stereoscopic display. 3300b illustrates a wireless connection, in this case the Internet. 3302b illustrates the first user, working on a laptop computer equipped with a stereoscopic camera system 3303b. 3304a illustrates the second user wearing an HDU 3305a. The first user 3301b cannot see the second user 3304a. However, if the second user's headset 3304a has a built-in camera (as taught in this disclosure), the first user 3301a can see an image of the surroundings in the second user's headset 3304b. The second user 3304b wearing the stereoscopic HDU 3305b can see the first user 3302b through the stereoscopic image from the first user's stereoscopic camera system on the laptop 3303b.

[0118] Figure 33C illustrates stereoscopic communication between a first user with a laptop and a second user wearing an HDU and holding a mobile phone with a stereo camera. 3300c shows a wireless connection (in this case the Internet). 3302c shows the first user working on a laptop computer equipped with a stereoscopic camera system 3303c. 3304c shows the second user wearing an HDU 3305c and holding a mobile phone equipped with a stereoscopic camera system 3306a. A first user 3301a wearing stereo HDU 3302a can see a second user 3304a through the second user's stereo camera system on laptop 3306c. A second user 3304c wearing stereo HDU 3305c can see a first user 3302c through an image from the first user's stereo camera system on laptop 3303c. The first user 3302c cannot see the second user 3304c in the stereo image.

[0119] Figure 33D illustrates stereoscopic communication between a first user wearing an HDU with a stereoscopic display working on a laptop with a stereoscopic camera and a second user wearing an HDU with a stereoscopic display working on a mobile phone with a stereoscopic camera. Reference numeral 3300d indicates a wireless connection (in this case, the Internet). Reference numeral 3302d indicates a first user wearing an HDU 3304d working on a laptop computer equipped with a stereoscopic camera system 3303d. Reference numeral 3304d indicates a second user wearing an HDU 3305d and holding a mobile phone equipped with a stereoscopic camera system 3306d. The first user 3302d wearing the stereoscopic HDU 3304d can see the second user 3304a through an image from the second user's stereoscopic camera system on the mobile phone 3306d. A second user 3304d wearing a stereoscopic HDU 3305d can see the first user 3302d through an image from the first user's stereoscopic camera system on laptop 3303d.

[0120] FIG. 34A shows a user wearing an HDU holding a smartphone with a stereoscopic camera in a first position. The user holds the smartphone in their right hand and points it toward their face. The user's head display unit displays a stereoscopic image from the user's face in near real time. For example, the left camera captures a left-eye perspective image of the face at a first time point, and the right camera captures a right-eye perspective image of the face at the first time point. The left-eye perspective image and the right-eye perspective image comprise a stereoscopic image. The smartphone is configured to transmit the stereoscopic image from the smartphone to the head display unit (HDU) via a wireless connection. The HDU is configured to display the stereoscopic image from the smart device. The HDU may be a virtual reality display, an augmented reality display, or a mixed reality display. The HDU includes a left-eye display and a right-eye display and displays the stereoscopic image to the user at a second time point. The second time point is within five seconds of the first time point. That is, this occurs in near real time. The left-eye display displays a perspective image for the left eye, and the right-eye display displays a perspective image for the right eye.

[0121] Figure 34B shows a user wearing an HDU holding a smartphone equipped with a stereoscopic camera in a second oblique position, allowing the user to view their own face from an oblique angle.

[0122] Figure 34C shows a user wearing an HDU holding the stereoscopic camera smartphone in a third position, at the back of the user's head. This allows the user to see the back of their head. Note that various zoom positions can be achieved by moving the phone closer to or further from the user's face. Eye movements can also be used to converge or change the angle of the line of sight.

[0123] FIG. 35A is a top view of a cameraman with a wheeled camera cluster.

[0124] 3500 shows a platform for the videographer. The videographer's head 3501a and shoulders 3502 are shown. The videographer is on a platform that contains a large camera cluster that provides a nearly 270° horizontal field of view. Camera 3503a is shown.

[0125] Figure 35B is a front view of the wheeled camera cluster looking at the cameraman. The videographer's head 3501b is shown above the camera cluster. Camera 3503b is shown. Platform 3500 is shown with legs 3504 connecting platform 3500 to the camera cluster frame. In some embodiments, some of the cameras have automatic target tracking (e.g., of a soccer player). In some embodiments, the videographer controls a subset of the cameras, and the remaining cameras are controlled automatically. In some embodiments, some of the cameras are fixed (e.g., prime lenses) to capture the remaining data.

[0126] Figure 36A is a top view of a football field showing the placement of multiple camera clusters. 3601 shows the football field. Camera clusters 3600a and 3600b are located at the 50-yard line and 25-yard line, respectively. Camera clusters 3600c and 3600d are located past the end zones. Camera cluster 3600e is mounted on a wire above the field and can move along the field to capture the best view of the game. For more details on example camera clusters, see U.S. Patent 10,317,218.

[0127] Figure 36B is a side view of a soccer field showing the placement of multiple camera clusters. 3601 indicates the football field. Camera clusters 3600a and 3600b are located at the 50-yard line and the 25-yard line, respectively. Camera clusters 3600c and 3600d are located past the end zones. Camera cluster 3600e is mounted on a wire above the field and can move along the field to capture the best view of the game. The field of view of the camera clusters can be predetermined. For example, sideline cameras 3600a and 3600b can have a field of view of at least 270 degrees horizontally and 90 degrees vertically.

[0128] Figure 37A shows a stereoscopic camera cluster with different lens sets. 3700 shows the housing of the camera cluster. 3701 shows the first camera in the right-eye camera cluster, which is optimized for long-distance viewing. In this example, long distance includes distances greater than 20 feet.

[0129] Reference numeral 3702 denotes a second camera in the right-eye camera cluster, optimized for a medium-distance field of view. In this example, medium-distance includes distances of 10 to 20 feet. Reference numeral 3703 denotes a third camera in the right-eye camera cluster, optimized for close-distance viewing. In this example, close-distance refers to a distance of less than 10 feet. Note that the stereo distance from the first camera pair may differ from the stereo distance from the second camera pair. Reference numeral 3704 denotes a first camera in the left-eye camera cluster, optimized for long-distance viewing. Reference numeral 3705 denotes a second camera in the left-eye camera cluster, optimized for a medium-distance field of view. Reference numeral 3706 denotes a third camera in the left-eye camera cluster, optimized for close-distance viewing. Cross-section A shows a cut through the first camera 3701 in the right-eye camera cluster and the first camera 3704 in the left-eye camera cluster. Cross-section B shows a cutaway view of the second camera 3702 in the right-eye camera cluster and the second camera 3705 in the left-eye camera cluster. Cross-section C shows a cutaway view of the third camera 3703 in the right-eye camera cluster and the third camera 3706 in the left-eye camera cluster. Note that camera clusters can have various specifications (zoom lenses, prime lenses, etc.). Zoom lenses with variable focal lengths (70-200 mm, 10-18 mm, etc.) can be used. For example, a 70-200 zoom range has a ratio of approximately 3:1, which is expressed as a 3x zoom. For example, with a zoom lens, you can capture a zoomed-out image (e.g., 10 mm), but zooming in (e.g., 18 mm) narrows the angle of view, making the objects in the image appear closer. Prime lenses (17 mm, 85 mm, etc.) can also be used. For example, with an 85 mm prime lens (focal length 85 mm), changing the angle of view and making the subject appear closer requires physically moving the camera closer to the subject. Generally, lenses with longer focal lengths are longer and lenses with shorter focal lengths are shorter. For a full-frame camera (no crop factor), a 50mm lens is roughly equivalent to the human eye. Canon's full-frame sensor measures 24x36mm. If the zoom setting is less than 50mm, such as 24mm, it will be listed as a wide-angle zoom.On the other hand, a longer lens setting, such as 70mm, will appear as a narrow-angle telephoto sensor. A lens with a crop factor of 1.6 is equivalent to a 32mm lens in the human eye. Wide-angle lenses are ideal for capturing landscapes or large groups of people, or for creating a strong sense of perspective. A long-angle telephoto setting is ideal for wildlife photography. The aperture is the opening that lets light into the camera. For example, the f-number indicates the size of the aperture; the higher the number, the smaller the opening. Every lens has a maximum f-number. The size of the aperture determines the amount of light that passes through the lens, which affects shutter speed and exposure. The f-number is the ratio of the aperture to the focal length of the lens being used. For example, if your aperture is 1 inch and your lens is a 50mm lens (approximately 2 inches), the f-number is f / 2. The f-number also determines the depth of field, or the range of objects in focus. For example, the iPhone 12 has two cameras, including an ultra-wide and a wide-angle, while the iPhone 12 Pro has an ultra-wide, wide-angle, and telephoto lens. In this disclosure, a stereoscopic camera cluster is introduced. Another novel feature is the inward cant of the stereoscopic camera pairs within the camera cluster. Three pairs are shown in Figure 37A. These can be variably tilted as shown in Figures 37B-37D.

[0130] Figure 37B is a top view of cross section A in Figure 37A. 3707 indicates a dashed line from camera 3704 (the first camera in the left-eye camera cluster, optimized for long-distance observation). Note that 3707 indicates the pointing direction of camera 3704. 3708 indicates a dashed line from camera 3701 (the first camera in the right-eye camera cluster, optimized for long-distance observation). Note that 3708 indicates the pointing direction of camera 3701. Note that pointing direction 3707 of camera 3704 and pointing direction 3708 of camera 3701 are parallel.

[0131] Figure 37C is a top view of cross section B in Figure 37A. 3709 indicates a dashed line from camera 3705 (the second camera in the left-eye camera cluster, optimized for a mid-distance field of view). Note that 3709 indicates the pointing direction of camera 3705. 3710 indicates a dashed line from camera 3702 (the second camera in the right-eye camera cluster, optimized for a mid-distance field of view). Note that 3710 indicates the pointing direction of camera 3702. Note that pointing direction 3709 of camera 3705 and pointing direction 3710 of camera 3702 slightly converge.

[0132] Figure 37D is a top view of cross section C in Figure 37A. 3711 indicates a dashed line from camera 3706 (the third camera in the left-eye camera cluster, optimized for close-range observation). Note that 3711 indicates the pointing direction of camera 3706. 3712 indicates a dashed line from camera 3703 (the third camera in the right-eye camera cluster, optimized for close-range observation). Note that 3712 indicates the pointing direction of camera 3703. Note that at intermediate distances, pointing direction 3711 of camera 3706 and pointing direction 3703 of camera 3712 converge more than pointing direction 3709 of camera 3705 and pointing direction 3710 of camera 3702. In other words, the cameras are tilted furthest inward.

[0133] FIG. 38 shows a user walking through a city and recording stereoscopic images. It is the intent of this disclosure to integrate GPS into image capture, enabling spatially accurate stereoscopic recording of the scene. A user wearing an HDU walks through a city (or scenic spot) at different times, observing and recording stereoscopic images of the scenery and points of interest. Person 3800a is walking through a scenic area at time 1 (t=1). He / she follows a path within the scenic spot, depicted by dashed line 3802. Person 3800b is again shown at time 2 (t=2), positioned at the top of the figure. He / she is equipped with a recording device 380. Shown in this figure are limited examples of what might be observed while walking through a scenic spot: an architecturally interesting building 3803, a fountain 3804, and a park bench 3805 where interesting people might sit. Audio functionality can be added to the recording device for commentary along the way. This walking can be done during the day or at night, with minimal lighting. Tourists can record in 3D where they've been and what they've seen, and travel agencies can provide previews of different locations to excite potential travelers.

[0134] FIG. 39A is a top view of a stereoscopic camera cluster on a head display unit. 3900 indicates the head display unit. 3901 indicates the right camera cluster. 3902 indicates the left camera cluster. 3903 indicates the field of view where stereoscopic vision is possible. Note that the full field of view includes areas where stereoscopic vision is not possible. Thus, the horizontal field of view of a stereoscopic image is approximately 120°. Note that this field of view captures high-resolution stereoscopic images over a much larger area than what humans can perceive. Note the previous discussion. Therefore, a stereoscopic display can also be used to retrospectively observe selected areas. Because other large FOV devices (FOVs) suffer from fisheye artifacts, users of stereoscopically spaced camera clusters experience an improvement over the prior art. As taught in this disclosure, camera clusters used in stereoscopic systems provide a wide field of view without artifacts, thereby improving over the prior art.

[0135] Figure 39B is a side view of the stereo camera cluster on the head display unit. 3900 shows the HDU. 3904 shows the vertical field of view, which is also about 120°.

[0136] Figure 40A shows a top view of the instantaneous field of view (FOV) of a stereoscopic camera system. As mentioned before, the Y-axis is defined as the direction the video camera is pointing, indicated by the arrow in this figure, at α=0°. The film crew is standing behind the camera, between α=-135° and α=+135°. 4000a shows the horizontal field of view.

[0137] Figure 40B shows a top view of the instantaneous field of view (FOV) of the stereoscopic camera system. The film crew stands behind the camera between β = -90° and β = +45°. Note that this β range allows filming of aerial activity simultaneously with ground activity. A number of cameras (not limited to 30) can be mounted on a convex surface that can be controlled by the film crew. 4000b shows the vertical field of view.

[0138] 41A is a left-hand view of an image of a user and their natural background scene. A first user 4101a is shown. The background scene includes a wall with a diploma 4100 displayed on it. Note that this scene, including both the user and the background, was captured using a stereoscopic camera as taught in this disclosure and viewed using a stereoscopic HDU as taught in this disclosure.

[0139] FIG. 41B shows an example of a user's image and a simulated stereoscopic background scene for the user. A first user 4101b is shown. The background includes mountains 4102 and clouds 4103. Note that the user's image is captured using a stereoscopic camera as taught in this disclosure. Note that the background stereoscopic scene can also display an image of a real-world scene (e.g., a stereoscopic image of Mount Everest captured with a long-distance stereoscopic telephoto lens). In this example, the first user 4101b is displayed in near real time to a second user wearing a head display unit. The stereoscopic background scene is also displayed to the second user wearing a head display unit. Nearby objects (e.g., user 4101b) are overlaid on more distant objects. That is, a composite stereoscopic image is generated in which part of the composite stereoscopic image is from the first user's near real-time video and other part of the composite stereoscopic image is from a fixed stereoscopic image. Note that in this embodiment, the stereoscopic background scene may be fixed. A dataset of simulated stereoscopic background scenes may be established, including, but not limited to, an office, a library, a mountain landscape, a garage, a gym, a living room, an oval office, etc.

[0140] FIG. 42A shows a left view of a user's image and an implementation of a dynamic simulated stereoscopic background scene for the user. A first user 4200a is shown. The background includes a mountain 4201 and clouds 4202a, which are light gray. Two birds 4203a are flying and are shown to the right of the first user 4200a. The user's image is captured using a stereoscopic camera as taught in this disclosure. In some embodiments, the stereoscopic image is from a live stereoscopic camera. In other embodiments, the stereoscopic image may be a recorded and played back stereoscopic image (e.g., recorded one month before being displayed). The stereoscopic background scene may also display an image of a real-world scene (e.g., a stereoscopic image of Mount Everest taken with a long-distance stereoscopic telephoto lens). In this example, the first user 4200a is displayed in near real time to a second user wearing a head display unit. The dynamic stereoscopic background scene is also displayed to the second user wearing a head display unit. Nearby objects (e.g., user 4200a) are superimposed on more distant objects. That is, a composite stereoscopic image is generated in which part of the composite stereoscopic image is from near real-time video of the first user and another part of the composite stereoscopic image is from dynamic stereoscopic images. Note that the image in Figure 42A is from time 1. Note that in this embodiment, the stereoscopic background scene may be dynamic.

[0141] FIG. 42B shows a left view of a user's image and an implementation of the user's dynamic simulated stereoscopic background scene. Note that the image in FIG. 42B is from time point 2, showing a first user 4200b. The background includes mountains 4201 and clouds 4202b. Note that the clouds 4202b have darkened and rain is falling from them. Note that the first user's 4200b's facial expression has changed. Note that three birds 4203b are now displayed to the left of the first user 4200b. Note that in this embodiment, the stereoscopic background scene may be dynamic. A dataset of dynamic simulated stereoscopic background scenes may be established, including, but not limited to, mountain scenes, beach scenes, basketball games, waterfalls, shopping malls, etc. Furthermore, a set of cameras located around the world (or even on Mars) may be deployed and viewed in near real time as a dynamic simulated background scene for a second user. Note that in some embodiments, a user may choose to only view the dynamic simulated stereoscopic scene.

[0142] FIG. 43A is a left diagram of a stereoscopic image of a first user. 4300a shows the first user with a white face. 4301a shows acne on the face of the first user 4300a. The first user, viewing himself using the camera system of his smart device, decides to hide his acne before going to a meeting over the Internet (e.g., a stereoscopic video conference). He then chooses to have it digitally removed using image processing techniques before being broadcast to a group of users in a Zoom meeting. He also wants to get a tan. Thus, image processing is performed on the stereoscopic image.

[0143] Figure 43B is a left view of the stereo image of the first user. 4300b shows the first user now with a tanned face. 4301a shows the facial skin area of ​​the first user 4301a is free of any pre-existing acne on the face. Additionally, the face of the first user 4300b is now tanned. The improved stereo image is sent to a conference via the Internet (e.g., a stereoscopic video conference). Various image edits are possible, such as makeup, hair color, slimming effects, hairstyles, changing eye color, etc.

[0144] 44A shows a left perspective view of a set of stereoscopic images of users with actual dress appearances. 4300a shows a first user with actual dress appearances.

[0145] Figure 44B shows a left-eye perspective view of a stereo image set of a user with a simulated dress appearance. 4300b shows a first user with a simulated (preferably realistic over cartoon) dress appearance, which is enhanced by adding a 3D model of a tie 4301 that is registered to the user and displayed in the left-eye and right-eye images of a second user viewing the first user in the HDU. A variety of images of the simulated dress appearance can be implemented, including, but not limited to, shirts, jackets, ties, jewelry, dresses, etc.

[0146] Figure 45A shows the installation of a wide field of view stereoscopic camera system at a scene of an ongoing event. 4500 shows a stereoscopic camera system mounted on a convex surface on a tripod, capturing stereoscopic images of a burning building 4501 and an emergency vehicle 4502.

[0147] Figure 45B shows a first user placed in a stereoscopic HDU viewing a first perspective of a scene. The first user is zoomed in through a window at the top to view a building. The lower part of the building and the emergency vehicle are not in the first user's field of view.

[0148] Figure 45C shows placing a second user on the stereoscopic HDU looking at a second view of the scene. The second user is looking at an emergency vehicle and tilts their head to see a slightly different perspective. This provides a new way to present the evening news.

[0149] FIG. 46 shows a text box indicating gestures or actions a user can take to modify the stereo image. Some embodiments of the disclosure allow the user to control (change camera settings). Other embodiments of the disclosure include a set of camera settings that cannot be changed, but allow the user to select which view to obtain from the camera settings (recall the discussion of various camera selections in FIG. 19C). This diagram outlines various user gestures and actions to modify the presented stereo image. Examples of ways to modify the presented stereo image include, but are not limited to, zooming, focusing, changing direction, changing brightness / color scheme, etc. The user can perform gestures or use controls to accomplish such modifications to the presented stereo image. Examples of gestures include, but are not limited to, squinting (via facial recognition analysis), moving / leaning forward (via head tracking), turning the head (via head tracking), changing the angle of gaze (via eye tracking), finger / hand / arm gestures (e.g., finger / hand / arm positions, e.g., hand movements such as clapping) determined by a camera system, voice commands (e.g., saying the word "zoom") determined by a microphone, etc. Additionally, various control devices can be used to effect modifications to the presented stereoscopic images, including, but not limited to, a standard TV controller, a joystick, a keyboard, a mouse (e.g., rollerball), etc.

[0150] Figure 47 is a flow diagram for integrating a stereoscopic camera with an HDU. 4700 shows sending stereoscopic images from the stereoscopic camera system to the head display unit. 4701 shows using HDU parameters (e.g., head tracking / head cant, eye tracking) to determine the object of interest (where the user is looking in the scene). 4702 shows the stereoscopic camera optimizing the imaging of the object of interest. 4703 shows sending the optimized stereoscopic images to the HDU.

[0151] FIG. 48 illustrates an example of using LIDAR to optimize image acquisition for a stereoscopic camera. 4800 illustrates determining an object of interest with an unknown distance within a scene (e.g., using a head display system with head and eye tracking to determine the object of interest). 4801 illustrates using LIDAR to determine the distance and angle from the camera system to the object. 4802 illustrates selecting focal length, shutter speed, aperture, ISO, stereoscopic camera convergence angle, and stereoscopic camera cant angle to optimize imaging of the object of interest. 4803 illustrates acquiring an image of the object using the selected settings. 4804 illustrates presenting the image to the user.

[0152] Figure 49 shows a stereoscopic LIDAR system on an HDU. A room 4900 is shown. A person 4901 wearing an HDU is standing at the entrance to the room 4900. A left LIDAR device (left laser beam shown as 4903) scans the area and generates a first 3D volume. The first 3D volume is displayed in near real time on the user's left-eye display. A right LIDAR device (spaced at interocular distance, right laser beam shown as 4902) scans the area and generates a second 3D volume. Note that in a preferred embodiment, the LIDAR devices are eye-safe. The second 3D volume is displayed in near real time on the user's right-eye display. Note that the first and second 3D volumes overlap at least partially. In some embodiments, the HDU includes both a stereoscopic camera system and a stereoscopic LIDAR system. This allows for more spatially accurate area analysis. Images can be acquired during head rotation, walking, and other body movements. Additionally, a stereoscopic FLIR can be incorporated, generating composite images. Simultaneously with the LRF, an L3TV camera operates, providing stereoscopic images that complement the LRF's range data. The LRF has two operating modes: a single pulse, which measures the distance to an object, and an area or volume scan. Single pulse can be used in a variety of scenarios. First, it can determine the distance to any hazard and display it on the HDU. Other applications include, but are not limited to, virtual home decor, where a set of virtual objects is inserted and how they fit within the dimensions measured by the LRF. Similarly, architects can consider how to layout virtual objects (windows, lighting, workspaces, etc.) to efficiently utilize available space. In scanning mode, the entire volume of a room can be mapped by rotating the head and scanning the laser. Voxels are created surrounding each range point, and from these voxels, the room's volume and the objects within it can be reconstructed. All of these measurements can be recorded along with the stereoscopic image for future playback. This could have important practical applications in the real estate industry.The immersive effect allows users to "walk through a virtual recreation of the property." *Viewed in life-size 3D. Outdoors, a surveyor equipped with a Global Positioning System (GPS) can measure the distance to all major features and pinpoint the location of these features from those measurements. Hikers can see the distance to major terrain features.

[0153] Figure 50A shows the use of a common aperture for multiple lenses to overcome the "image shift" that can be annoying to users when switching between cameras. 5000 teaches using an aperture to pass light to multiple (at least two) camera systems. 5001 teaches using a mirror system (e.g., a MEMS fast steering mirror) to reflect light toward a first lens. 5002 teaches using a mirror system (e.g., a MEMS fast steering mirror) to reflect light toward a second lens.

[0154] Figure 50B shows the use of a common aperture system with a fast scanning mirror in a first position. 5003 shows the aperture. 5004a shows the MEMS fast steering mirror in the first position. 5005 shows the first lens. 5006 shows the first detector. 5007 shows the second lens. 5008 shows the second detector. Note that the dashed lines show the light rays passing from the aperture 5003 through the MEMS fast steering mirror 5004a (in the first position), through the first lens 5005 and onto the first detector 5006.

[0155] FIG. 50C illustrates the use of a common aperture system with a fast scanning mirror in a first position. 5003 indicates an aperture. 5004b indicates a MEMS fast steering mirror in a first position. 5005 indicates a first lens. 5006 indicates a first detector. 5007 indicates a second lens. 5008 indicates a second detector. Note that dashed lines indicate light rays passing from aperture 5003 to MEMS fast steering mirror 5004b (in a second position), through second lens 5007, and to second detector 5008. A common inventive aspect is the use of a single aperture for multiple lenses. The arrangement of lenses and mirrors can be varied in geographical direction. Furthermore, with appropriate mirror / lens arrangement, some designs can be implemented with a single detector. In some embodiments, the user's line of sight characteristics (e.g., line of sight angle, convergence point, etc.) determine the mirror position.

Claims

1. acquiring an image from a smart device, the smart device comprises at least one selected from the group consisting of a laptop computer, a telephone, and a tablet; the smart device comprises a left camera and a right camera; the left camera and the right camera are separated by a stereo distance, the left camera acquires a left perspective image of an area at a first time point; the right camera acquires a right-view perspective image of the area at the first time point; the left-view perspective image and the right-view perspective image comprise a stereoscopic image; transmitting the stereoscopic image from the smart device to a head display unit (HDU) via a wireless connection; displaying the stereoscopic image on the head display unit (HDU), the HDU comprises at least one selected from the group consisting of a virtual reality display, an augmented reality display; and a mixed reality display; the HDU includes a left-eye display and a right-eye display; the HDU is worn by a first user; the HDU displays the stereoscopic image to the first user at a second time; the second time point is within 5 seconds of the first time point; the left-eye perspective image is displayed on the left-eye display; a step of displaying the right-eye perspective image on the right-eye display; A method comprising:

2. using a microphone in proximity to the smart device; recording a sound with the smart device; 10. The method of claim 1, further comprising the step of playing the sound on a speaker proximate to the HDU, the sound being played in synchronization with the stereoscopic images.

3. the smart device is moved away from objects in the area; the object is displayed on the HDU so as to move away from the first user; and the smart device is brought close to an object in the area, and the object is displayed on the HDU as being close to the first user; The method of claim 1 , further comprising at least one of the group consisting of:

4. At a first point in time, a direction in which a sensor corresponding to the left camera is facing is parallel to a direction in which a sensor corresponding to the right camera is facing; and At a subsequent point in time, the direction in which the sensor corresponding to the left camera is facing and the direction in which the sensor corresponding to the right camera is facing converge. The method of claim 1 further comprising:

5. The method of claim 1 , further comprising: the left camera comprising a camera cluster; and the right camera comprising a camera cluster.

6. the stereoscopic image is of the first user, the user wearing the HDU seeing themselves; and the stereoscopic image is of a second user, and the first user wearing the HDU sees the second user; The method of claim 1 , further comprising at least one of the group consisting of:

7. The method of claim 1 , further comprising the smart device having at least two pairs of stereoscopic cameras.

8. the HDU worn by the first user has an inertial measurement unit (IMU) for head tracking; and 10. The method of claim 1, further comprising: a field of view of the stereoscopic image displayed on the HDU worn by the first user changing in response to head rotation by the first user.

9. the HDU worn by the first user has an inertial measurement unit (IMU) for head tracking; and the convergence of the stereoscopic image displayed on the HDU worn by the first user varies in response to the convergence of the first user; The method of claim 1 further comprising:

10. the HDU worn by the first user has an eye tracking system; and a field of view displayed on the HDU worn by the first user that changes depending on the line of sight angle of the first user; The method of claim 1 further comprising:

11. The method of claim 1 , further comprising the smart device comprising at least one of the group consisting of an HDU, a television, and a monitor.

12. The method of claim 1 , further comprising the smart device comprising at least one of the group consisting of a convex surface and a set of robotic arms.

13. the area includes a second user and a real background; said actual background being replaced by a simulated background; the simulated background is presented on the HDU worn by the first user; The method of claim 1 further comprising:

14. The method of claim 13 , further comprising: the simulated background dynamically changing over time.

15. the area including a second user and an actual article of clothing worn by the second user; said actual clothing item being replaced by a simulated clothing item; the simulated article of clothing is presented on the HDU worn by the first user; The method of claim 1 further comprising:

16. the smart device has a field of view that is larger than the field of view of the HDU; a portion of the field of view of a smart device is selected for display on the HDU; the selection for display on the HDU is determined by the first user's left and right eye viewing angles; The method of claim 1 further comprising:

17. the smart device has a field of view that is larger than the field of view of the HDU; a portion of the field of view of the smart device is selected for display on the HDU; the selection for display on the HDU is determined by the first user's head rotation angle; The method of claim 1 further comprising:

18. the smart device has a field of view that is larger than the field of view of the HDU; a portion of the field of view of the smart device is selected for display on the HDU; The method of claim 1 , further comprising: the selection for display on the HDU being determined by congestion of the first user.

19. A smart device, the smart device comprises at least one selected from the group consisting of a laptop computer, a telephone, and a tablet; the smart device comprises a left camera and a right camera; the left camera and the right camera are separated by a stereo distance; the left camera acquires a left perspective image of an area at a first time point; the right camera acquires a right-view perspective image of the area at the first time point; the left-view perspective image and the right-view perspective image include a stereoscopic image; a smart device configured to transmit the stereoscopic images from the smart device to a head display unit (HDU) via a wireless connection; An HDU, the HDU is configured to display the stereoscopic image from the smart device; the HDU comprises at least one selected from the group consisting of a virtual reality display, an augmented reality display, and a mixed reality display; the HDU includes a left-eye display and a right-eye display; The HDU is installed by a user; the HDU displays the stereoscopic image to the user at a second time point; the second time point is within 5 seconds of the first time point; the left-eye perspective image is displayed on the left-eye display; an HDU, in which the right-eye perspective image is displayed on the right-eye display; An apparatus comprising:

20. 1. A non-transitory computer readable medium having computer readable code thereon, said medium comprising: instructions for the smart device to transmit the image; the smart device comprises at least one selected from the group consisting of a laptop computer, a mobile phone, and a tablet; the smart device comprises a left camera and a right camera; the left camera and the right camera are separated by a stereo distance, the left camera acquires a left perspective image of an area at a first time point; the right camera acquires a right-view perspective image of the area at the first time point; the left-view perspective image and the right-view perspective image comprise a stereoscopic image; the smart device is configured to transmit the stereoscopic images from the smart device to a head display unit (HDU) via a wireless connection; the HDU is configured to display the stereoscopic image from the smart device; the HDU comprises at least one selected from the group consisting of a virtual reality display, an augmented reality display, and a mixed reality display; the HDU includes a left-eye display and a right-eye display; The HDU is installed by a user; the HDU displays the stereoscopic image to the user at a second time point; the second time point is within 5 seconds of the first time point; the left-eye perspective image is displayed on the left-eye display; The right-eye perspective image is displayed on the right-eye display.

Images