Novel system and method for collecting, locating and visualizing sensor signal in extended reality

Abstract

To provide systems and methods for rendering one or more different types of datasets.SOLUTION: A method includes: obtaining three-dimensional coordinates in which each first data value within a first type of dataset is associated with one or more three-dimensional coordinates which define a location or region in real space; obtaining each second data value within a second type of dataset to be associated with one or more of the three-dimensional coordinates; spatializing the first type of dataset to create a first type of spatialized dataset; spatializing the second type of dataset to create a second type of spatialized dataset; aligning the first type of spatialized dataset with the second type of spatialized dataset to create an enhanced three-dimensional spatialized environment; and rendering the enhanced three-dimensional spatialized environment on a display component.SELECTED DRAWING: Figure 13

Description

[Technical field]

[0001] This application claims priority to Provisional Patent Application No. 62 / 846,439, filed May 10, 2019, which is incorporated herein by reference for all purposes.

[0002] The present arrangements and teachings generally relate to novel systems and methods for collecting, exploring, and visualizing sensor signal measurements in extended reality, ranging from virtual reality to augmented reality, and more particularly, the present arrangements and teachings relate to novel systems and methods using complex data spatialization, among other techniques, for collecting, exploring, and visualizing sensor signal measurements in extended reality. [Background technology]

[0003] Traditionally, virtual reality ("VR") and augmented reality ("AR") require a human interface device (commonly referred to as a "headset") to implement the full capabilities of the technology. Additionally, each of VR and AR spans a range of realities commonly referred to as "extended reality" ("XR"). For illustrative purposes, users often wear a headset to peer into an XR world, which broadly refers to a world that encompasses both VR and AR, as well as all intermediate realities.

[0004] Unfortunately, XR systems, including headsets, are typically limited in their ability to process large amounts of complex data to convey information about the real world (as opposed to the virtual world), and therefore cannot effectively inform users about the real world.

[0005] What is needed, therefore, is an XR system and method for processing large amounts of complex data to inform real spaces in the real world. Summary of the Invention [Means for solving the problem]

[0006] The systems and methods described herein relate to XR and process large amounts of complex data to inform real spaces that exist in the real world, as opposed to a virtual world.

[0007] In one aspect, the present teachings provide a method for rendering another different type of attribute value dataset and image dataset. One such exemplary method includes (i) acquiring three-dimensional pixelated or voxelized data values ​​for one or more corresponding three-dimensional position coordinates using a light sensor and a ground positioning component, (ii) acquiring one or more different types of attribute values ​​for one or more of the corresponding three-dimensional position coordinates using at least one non-light sensor, (iii) spatializing the three-dimensional pixelated or voxelized data values ​​using the corresponding three-dimensional position coordinates to produce a spatialized image dataset, (v) spatializing using the corresponding three-dimensional position coordinates and at least a particular type of attribute value to produce at least a particular type of spatialized attribute value dataset, (vi) aligning the image spatialized dataset with the spatialized attribute value dataset to produce an augmented three-dimensional spatialized environment, and (vii) rendering the augmented three-dimensional spatialized environment on a display component using a rendering engine.

[0008] In this exemplary method, each corresponding three-dimensional position coordinate is associated with at least one of three-dimensional pixelated data values, at least one of three-dimensional voxelized data values, or at least one type of attribute value.

[0009] In one preferred embodiment of the present teachings, the spatialization step to produce a spatialized image data set and / or the spatialization step to produce a spatialized attribute value data set includes (1) spatially partitioning the space into a number of subdivisions and (2) aggregating the subdivisions to produce a spatialized model for the real space, where in the preferred embodiment the real space is defined using corresponding three-dimensional position coordinates.

[0010] Additionally, the spatialization step to produce the spatialized image dataset and / or the spatialization step to produce the spatialized attribute value dataset may include using a microphone to collect audio data or information present in the real space, in this embodiment the microphone may be operable in conjunction with a filter to enhance the spatial resolution of the spatialized model for the real space.

[0011] The exemplary method described above may further include storing one or more different types of attribute values ​​and their corresponding three-dimensional coordinates in a database. Further exemplary embodiments may use multiple non-optical sensors installed on various client devices or external controllers (operating in an "edge computing" manner). Examples of client devices include smartphones or wearables. Such client devices collect intangible property data present in the real space as their users traverse various locations in the real space. In one preferred embodiment of the exemplary method, the step of obtaining multiple attribute values ​​at multiple corresponding three-dimensional position coordinates is performed such that each of the corresponding three-dimensional position coordinates is associated with at least one type of attribute value.

[0012] In one implementation of the aligning step, the processor aligns the two spatialized data sets based on one or more common intersection points present in the spatialized image data set and in the spatialized attribute value data set, which common intersection points are also in real space.

[0013] In a preferred embodiment, the rendering step of the present teachings includes (1) representing a particular type attribute value obtained at a location in real space with a virtual object, (2) comparing the particular type attribute value with a particular type threshold attribute value, (3) assigning a first color to the virtual object if the particular type attribute value is greater than the particular type threshold attribute value, (4) assigning a second color to the virtual object if the particular type attribute value is less than the particular type threshold attribute value, and (5) rendering the virtual object at an object rendering location within the augmented three-dimensional spatialized environment. In this embodiment, the object rendering location corresponds to a location of a corresponding three-dimensional position coordinate, and the corresponding three-dimensional position coordinate is associated with the particular type attribute value.

[0014] The present teachings also enable estimation and / or prediction of attribute values ​​when measured attribute values, i.e., parameter values ​​of intangible properties, are not available or obtainable. To this end, the step of spatializing to produce a spatialized attribute value data set includes interpolating using two or more of the attribute values ​​of the particular type and two or more of their corresponding three-dimensional position coordinates to calculate an intermediate attribute value of the particular type. In this embodiment, the corresponding intermediate three-dimensional position coordinates located between two or more of the corresponding three-dimensional position coordinates are associated with the intermediate attribute value of the particular type. In this embodiment, the intermediate attribute value of the particular type is an estimate of the attribute value of the particular type, and the rendering step can render an intermediate virtual object at the intermediate object position. In this rendering step, the intermediate virtual object is rendered in the augmented three-dimensional spatialized environment and at a position corresponding to the position of the corresponding intermediate three-dimensional position coordinates, and the corresponding intermediate three-dimensional position coordinates are associated with the intermediate attribute value of the particular type.

[0015] With respect to predicting attribute values, the spatialization step to produce a spatialized attribute value dataset includes inferring using two or more of the attribute values ​​of a particular type and two or more of their corresponding three-dimensional position coordinates to calculate a predicted attribute value of a particular type. In this embodiment, the corresponding external three-dimensional position coordinates are located outside the real space and positioned on a linear trajectory that intersects with two or more of the corresponding three-dimensional position coordinates. Furthermore, the corresponding external three-dimensional position coordinates are associated with the predicted attribute value of the particular type. To render the predicted attribute value of the particular type, the rendering step may further include rendering an external virtual object at an external object position within the augmented three-dimensional spatialized environment. The external object position corresponds to a position of the corresponding external three-dimensional position coordinates, and the corresponding external three-dimensional position coordinates are associated with the predicted attribute value of the particular type.

[0016] Besides interpolation and estimation, the present teachings provide other methods for determining attribute values ​​where measurements are not available or have not been made. In some embodiments, the spatialization step of producing a spatialized attribute value data set includes producing an electronic representation of a wire mesh extending in real space, the electronic representation of the wire mesh being defined using a plurality of corresponding three-dimensional position coordinates. This step can then proceed to calculate unknown attribute values ​​of at least a particular type at defined locations in real space using the electronic representation of the wire mesh.

[0017] In the exemplary embodiment described above, the rendering step includes rendering a virtual object at each of the object rendering locations within the augmented three-dimensional spatialized environment. The object rendering locations correspond to locations of corresponding three-dimensional position coordinates, and the corresponding three-dimensional position coordinates are associated with a particular type of attribute value. The exemplary method described above may further include a detection step of detecting a user's interaction with one or more of the virtual objects. This detection step is performed after the rendering step described in the exemplary method.

[0018] When the detection step detects a user interaction with one or more of the virtual objects, a visual feedback element is rendered that conveys the user interaction with the one or more of the virtual objects and / or conveys a magnitude and / or direction of a particular type of attribute value associated with the one or more of the virtual objects.

[0019] Nevertheless, the structure and method of operation of the present invention, together with additional objects and advantages thereof, will be best understood from the following description of specific embodiments when read in conjunction with the accompanying figures. [Brief description of the drawings]

[0020] [Figure 1] FIG. 2 is a block diagram of an XR system according to one embodiment of the present arrangement, the XR system including, among other things, eyewear and a processor for producing the XR. [Diagram 2] FIG. 1 illustrates eyewear according to one embodiment of the present arrangement, the eyewear including an imaging device and display for viewing XR renderings of the present teachings. [Diagram 3] FIG. 3 is a block diagram of processor-based eyewear according to one embodiment of the present arrangement, the processor-based eyewear including the imaging device of FIG. 2 integrating a non-optical sensor, a processor, and an optical sensor, such that these components work in cooperation to implement several of the methods of the present teachings described herein. [Figure 4A] 4 is a block diagram of various software modules and a rendering engine according to one embodiment of the present arrangement that may reside in a processor, such as in the processor-based eyewear of FIG. 3 or the server of FIG. 1. [Figure 4B] FIG. 11 is a block diagram of various software modules and rendering engines according to alternative embodiments of the present arrangement, which may be present in a processor, such as in the processor-based eyewear of FIG. 3 or in eyewear, such as the server of FIG. 1 and the eyewear shown in FIGS. 1, 2, and 3. [Diagram 5] FIG. 1 illustrates an XR system according to one embodiment of the present arrangement, having stored therein programmable instructions for carrying out some of the methods of the present teachings described herein. [Figure 6] FIG. 1 illustrates an XR system according to another embodiment of the present arrangement, the XR system including eyewear communicatively coupled to a smartphone having programmable instructions stored therein for performing certain methods of the present teachings described herein. [Figure 7] FIG. 1 illustrates an XR system according to yet another embodiment of the present arrangement, the XR system including eyewear communicatively coupled to a network (e.g., the Internet) having programmable instructions stored therein for performing certain methods of the present teachings described herein. [Figure 8] FIG. 1 illustrates an XR system according to yet another embodiment of the present arrangement, the XR system including eyewear communicatively coupled to a personal computer having programmable instructions stored therein for performing certain methods of the present teachings described herein. [Figure 9] FIG. 13 illustrates an XR system according to yet another embodiment of the present arrangement, the XR system having stored therein programmable instructions for performing some of the methods of the present teachings described herein in the absence of eyewear. [Figure 10] FIG. 2 is a process flow diagram for a method according to one embodiment of the present teachings for rendering two different types of data sets. [Figure 11] FIG. 2 is a process flow diagram for a method according to one embodiment of the present teachings for spatializing an image dataset. [Figure 12] 1 is a process flow diagram for a method according to one embodiment of the present teachings for spatializing one or more different types of attribute value data sets, where the attribute values ​​convey the magnitude of intangible properties present in real space. [Figure 13]FIG. 2 is a process flow diagram for a method according to one embodiment of the present teachings for rendering an image dataset and an attribute value dataset. [Figure 14] FIG. 2 illustrates a rendered 3D spatialized image according to one embodiment of the present arrangement, depicting the distribution of WiFi connection signal strength indicators within a particular real space of interest. [Figure 15] FIG. 1 shows a rendered 3D spatialized image according to one embodiment of the present arrangement, depicting "dead spots" (i.e. areas of "no connection") in the distribution of connection signal strength indicators within a particular real space of interest. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. Nevertheless, it will be apparent to one of ordinary skill in the art that the present invention may be practiced without limitation to some or all of these specific details. In other instances, well-known process steps have not been described in detail so as not to unnecessarily obscure the present invention.

[0022] The systems and methods described herein may be deployed using processor-based eyewear (e.g., headsets or glasses). In one embodiment of the arrangement, if eyewear is still not present, the smartphone renders one or more different types of data sets. As a result, the use of eyewear is not required, but represents a preferred embodiment of the arrangement.

[0023] The present arrangement provides many different configurations of systems for rendering an extended reality ("XR") of the present teachings. In one XR of the present teachings, two or more different types of datasets are rendered. In one such XR, one dataset may include image data of a real space, and another dataset may include data conveying the scale of intangible properties in the real space. In another XR of the present teachings, an image dataset is an electronic representation of the space as perceived by the user, and is spatialized and rendered. In accordance with the present teachings, an image dataset is not the only type of dataset that may be spatialized before rendering. Indeed, one or more different types of attribute value datasets present in the real space may be spatialized before rendering.

[0024] In some embodiments, the processor-based headset of the present arrangement will generally comprise a processor, the processor including one or more memory devices operable to provide machine-readable instructions to the processor and to store data. In some of these embodiments, the processor-based headset is communicatively coupled to an external device that includes one or more sensors for collecting persistent data. In other embodiments, all of the necessary sensors are integrated into the processor-based headset.

[0025] In one preferred embodiment of this arrangement, the processor-based headset can include data obtained from a remote server. The processor can also be coupled to various input / output ("I / O") devices for receiving input from a user or another system and for providing output to a user or another system. These I / O devices can include keyboards, touch screens, displays, and terminals, as well as human interaction devices such as remotely connected computer systems, modems, wireless transmitters, and handheld personal communication devices (such as cellular phones, smartphones, and digital assistants).

[0026] The processor of the arrangement may also include mass storage devices associated with disk drives and flash memory modules, as well as connections through I / O devices to a server or remote processor that houses further storage devices and peripherals.

[0027] Some embodiments may employ multiple servers and data storage devices, thus enabling operation in the cloud or drawing from multiple data sources. The present teachings and arrangements contemplate that the methods disclosed herein will also operate over a network, such as the Internet, and may be implemented using a combination of several processing devices, memory, and I / O. Moreover, any device or system that operates to perform one or more steps according to the present teachings may be considered a "server" as the term "server" is used herein, if the device or system operates to communicate all or a portion of the programmable instructions to another device, such as eyewear or a smartphone.

[0028] In some aspects of the embodiment employing multiple devices, i.e. multiple eyewear devices, the server and data storage devices may operate in the cloud or may operate to pull from multiple data sources in the absence of internet. In these configurations, the multiple devices may collectively operate as part of a peer-to-peer ad-hoc network. These configurations of the present arrangements would perform "edge computing," which is a distributed computing paradigm that brings computation and data storage closer to where it is needed to improve response times and conserve bandwidth. In some of these aspects of the present arrangements, the eyewear would communicate with each other without internet, where one of the eyewear would act as a sort of communications server, but without any connection to the others.

[0029] The processor of the system may be partially or fully integrated into a wireless device, such as a smartphone, eyewear, personal digital assistant (PDA), laptop, notebook, and tablet computing device, that operates over a wireless network. Alternatively, the entire processing system may be self-contained on a single device in some embodiments of the arrangement.

[0030] The processor of the system may be coupled to a three-dimensional ("3D") controller for interaction. The 3D controller utilizes human input to provide signals that instruct the processing system to change the display of information. A conventional 3D controller may be used to virtually move the image displayed on the screen. Several exemplary embodiments of a processor-based headset of the present arrangement are described below.

[0031] FIG. 1 illustrates a functional block diagram of an extended reality ("XR") system 100 according to one embodiment of the present arrangement. The XR system 100 includes a server 110 coupled to one or more databases 112 and a network 114 (e.g., the Internet). The network 114 may include routers, hubs, and other equipment (for implementing communication between all associated devices). A user may choose to access the server 110 by a computer 116 communicatively coupled to the network 114. The computer 116 includes, among other things, a sound capture device such as a microphone (not shown to simplify the illustration). Alternatively, a user may access the server 110 through the network 114 by using a smart device 118 such as a smartphone or PDA. The smartphone 118 may connect to the server 110 through an access point 120 coupled to the network 114. The smartphone 118, like the computer 116, may include a sound capture device such as a microphone.

[0032] The XR system 100 can include one or more user devices (e.g., XR headsets) 122, either directly coupled to the network 114 through an access point 120 or directly coupled to a remote processing device. Illustratively, an XR headset or a game controller may be coupled to the processing device to obtain user input. This coupling may be wireless using a technology such as Bluetooth, a wireless technology standard used to exchange data between fixed and mobile devices over short distances, using short-wavelength UHF radio waves in the industrial, scientific, and medical radio bands from 2.402 GHz to 2.480 GHz to create a personal area network (PAN). Regardless of how the headset is coupled to one or more different types of devices, the user can use the user device 122 to perceive the environment in which the user is located.

[0033] Traditionally, client device / server processing works by splitting processing between two devices, such as a server and a smart device (such as a smartphone or other computing device). The workload is split between the server and the client device (e.g., a smartphone or a wearable (such as eyewear)) according to predefined specifications. Illustratively, in a "light client" mode of operation, the server does the majority of the data processing, and the client device often does a minimal amount of processing, just to display and / or render the results of the processing performed on the server (e.g., server 110).

[0034] In accordance with this disclosure, a client device / server application is constructed in such a way that a server provides programmable instructions to one or more different types of client devices, which then execute those programmable instructions. The interaction between the server and the client device dictates which instructions are transmitted and executed. Additionally, the client device may at times provide programmable instructions to the server, which executes the programmable instructions. Several forms of programmable instructions are known in the art, including applets, which are written in a variety of languages, including Java and JavaScript.

[0035] A client device application running on a client device and / or a server application running on a server provide a Software as a Service (SaaS) application, where the server provides software to the client device on an as-needed basis.

[0036] In addition to transmitting instructions, client device / server applications also involve the transmission of data between the client device and the server. Often this requires data stored on the client device to be transmitted to the server for processing. The resulting data is then transmitted back to the client device for display or further processing.

[0037] The present teachings recognize that a client device (e.g., computer 116, smartphone 118, or eyewear 122) may be communicatively coupled to a variety of other devices and systems such that the client receives data directly and acts on this data before transmitting the data to another device or server. Thus, data to a client device may come from input data from a user, from memory on the device, from an external memory device coupled to the device, from a wireless receiver coupled to the device, or from a transducer or sensor coupled to the device. In this arrangement, the sensor may be an optical sensor and / or a non-optical sensor. The radio may be part of a wireless communication system such as a "WiFi" or Bluetooth receiver. "WiFi," as the term is used herein, refers to a family of wireless networking technologies based on the IEEE 802.11 family of standards commonly used for local area networking of devices and Internet access.

[0038] The sensor, communicatively coupled to either the eyewear, another client device (e.g., a smartphone or personal computer), or a server, may be any of a number of devices or instruments, and in one embodiment, a non-optical sensor in the arrangement detects the magnitude of an intangible property in real space.

[0039] A client-server system may rely on "engines" that contain processor-readable instructions (or code) for implementing various elements of a design, with which the client device and server are communicatively coupled to achieve a desired result. Each engine may be responsible for various operations and may reside in whole or in part on the client device, the server, or another device. As disclosed herein, rendering engines, display engines, data engines, execution engines, and user interface (UI) engines may be used. These engines may seek and gather information about events from remote data sources. Additionally, these engines may facilitate rendering, displaying, and / or causing the display of "processed information" generated by the processor.

[0040] The described methods and systems of the present teachings may be implemented using conventional programming tools, including database tools for collecting, storing, and searching through structured data. Additionally, web-based or other programming techniques may be employed to collect information and display or render the results. Thus, a software engine may be created to implement these methods and techniques in whole or in part, depending on the desired implementation described herein.

[0041] Figure 2 shows an example eyewear (e.g., an XR headset) 222 of this arrangement that fits around the user's eyes using straps 220. In one embodiment, the eyewear 122 of Figure 1 is eyewear 222, which includes a frame 202 equipped with three imaging devices on the user's right side (i.e., 210, 214, 218), three imaging devices on the user's left side (i.e., 208, 212, 216), and two displays (i.e., 206 and 204, one for each eye).

[0042] In the configuration of the eyewear 222, the left look-side imaging device 216 provides an image of the space to the left of the user, the left look-front imaging device 212 provides an image of the space in front of but to the left of the user, and the left look-down imaging device 208 provides an image of the space below but to the left of the user. Similarly, for the right side of the user, the right look-side imaging device 218 provides an image of the space to the right of the user, the right look-front imaging device 214 provides an image of the space in front of but to the right of the user, and the right look-down imaging device 210 provides an image of the space below but to the right of the user. As a result, these six imaging devices (i.e., cameras) capture images of the space surrounding the user. Additionally, the eyewear 222 includes a left-eye display 204 and a right-eye display 206 for rendering and / or displaying information for the user's left and right eyes, respectively.

[0043] Although eyewear 222 is not shown (to simplify the illustration and facilitate discussion) as including a non-imaging optical sensor, frame 202, in some embodiments of the arrangement incorporates one or more non-optical sensors for providing a user with non-imaging types of information regarding the real space of interest. In a preferred configuration of these embodiments, frame 202 houses a processor, which functions as a processing subsystem for processing both the imaging and non-imaging information acquired using the optical sensors (including imaging devices) and non-optical sensors described above, respectively. In this embodiment, the processor of eyewear 222 not only generates "processed information" from the imaging and non-imaging information, but also effectively renders or displays and / or causes the "processed information" to be displayed on left-eye display 204 and right-eye display 206.

[0044] 3 illustrates processor-based eyewear 322 according to one embodiment of the present arrangement, including a processing subsystem 313 configured to receive and process both imaging information obtained from light sensor device 305 and non-imaging or otherwise non-imaging device information obtained from non-light sensor device 340. Such processing of information allows processing subsystem 313 to generate “processed information” that is rendered and / or displayed or caused to be displayed on I / O interface 317 (e.g., left-eye display 204 and right-eye display 206 of FIG. 2). A network (e.g., the Internet) preferably serves as a communication bus 365 to effectively communicatively couple processing subsystem 313, light sensor device 305, non-light sensor device 340, and I / O interface 317 such that the requisite information from the sensors is received and processed, and then displayed and / or rendered on I / O interface 317.

[0045] The non-optical sensor 340 is a sensor configured to measure a magnitude of one or more intangible properties present in the real space. In a preferred embodiment of the arrangement, the non-optical sensor 340 is at least one sensor selected from the group including a sensor that measures throughput of a connection signal, a sensor that measures latency of a connection signal, a sensor that measures interference of a connection signal, a sensor that measures variability of a connection signal, a sensor that measures stability of a connection signal, a radio frequency ("RF") power output sensor, an electric field and magnetic field ("EMF") sensor, an air pressure sensor, a geomagnetic sensor, a Hall effect sensor, an ambient light level sensor, a gas concentration sensor, a smoke sensor, a sound pressure sensor, an audio harmonics sensor, a humidity sensor, a carbon dioxide emission sensor, and a temperature sensor. In one embodiment of the arrangement, the sensor that measures throughput of a connection signal measures a bandwidth of a connection signal and the sensor that measures variability of a connection signal measures stability of a connection signal.

[0046] 3 includes an optical assembly 330, an imaging device 315, an electronic display 325, an inertial measurement unit ("IMU") 342, and one or more position sensors 335. The processing subsystem includes an image processing engine 319, an application store 321, and a tracking module 323.

[0047] In the light sensor device 305, one or more imaging devices 315 (e.g., imaging devices 208, 210, 212, 214, 216, 218 of FIG. 2) capture data characterizing a scene or local area of ​​interest, typically the space surrounding a user. According to one present arrangement, the imaging device 315 includes a conventional image sensor (e.g., a camera) such that a signal captured by the imaging device 315 includes only two-dimensional image data (e.g., data without depth information). In another present arrangement, the imaging device 315 includes a projector device that enables the imaging device 315 to operate as a depth imaging system that uses the collected data to calculate depth information of the scene (e.g., by processing a portion of a structured light pattern, by time-of-flight ("ToF") imaging, or by simultaneous localization and mapping ("SLAM"), based on the captured light by one or more computer vision methods or algorithms). In this alternative arrangement, the imaging device 315 transmits corresponding data to another device, such as the processing subsystem 313, which uses the data from the imaging device 315 to determine or generate depth information.

[0048] In yet another present arrangement, imaging device 315 is a hyperspectral imaging device that represents a scene as multiple spectra of light, such that different features or objects within the scene can be visualized using specific wavelengths of light and therefore better understood, analyzed, and / or visually or quantitatively described.

[0049] In preferred embodiments of this arrangement, the electronic display 325 displays a two-dimensional or three-dimensional image based on the type of data received from the processing subsystem 313. In some of these arrangements, the optical assembly 330 can magnify the image light received from the electronic display 325, correct optical errors associated with the image light, and / or present the corrected image light to a user of the light sensor device 305.

[0050] In one embodiment of the arrangement, IMU 342 represents an electronic device that generates fast calibration data based on measurement signals received from one or more of position sensors 335. To this end, one or more position sensors 335 are configured to generate one or more measurement signals in response to movement of eyewear 322. When one or more of position sensors 335 provide information about the position at which a measurement was taken, such information is sometimes referred to herein as a “ground position component” (e.g., in connection with FIGS. 10, 11, and 13).

[0051] Examples of different types of position sensor 335 include an accelerometer, a gyroscope, a magnetometer, another suitable type of sensor that detects motion or a type of sensor used for error correction of IMU 342. Position sensor 335 may be external to IMU 342 or internal to IMU 342, or some portions may be internal to IMU 342 and other portions may be external to IMU 342. Regardless of their location, position sensor 335 may detect one or more reference points that are used to track the location of eyewear 322 within a local area, for example, by using tracking module 323 of processing subsystem 313.

[0052] In the processor subsystem 313, the image processing engine 319 can generate a three-dimensional depth mapping or multiple three-dimensional depth mappings of a space surrounding some or all of the light sensor device 305 (e.g., a "scene" or "local area" of interest) based on information received from the light sensor device 305 or components thereof (e.g., the imaging device 315). In some embodiments, the image processing engine 319 of the arrangement can generate depth information for the three-dimensional mapping of the scene based on two-dimensional or three-dimensional information received from the imaging device 315 related to the technique used in computing the depth map. The depth map can include depth scale values ​​for each of the pixels in the depth map, and the depth map can represent multiple different portions of the scene.

[0053] Continuing with the processor subsystem 313, the application store 321 may store one or more software applications or sets of programmable instructions for execution by the processing subsystem 313 or the light sensor device 305. A software application, in some instances, may represent a group of programmable instructions that, when executed by a processor (e.g., the processing subsystem 313 of FIG. 3 or the server 110 of FIG. 1), generates or renders content for presentation to a user. The content generated or rendered by a software application may be generated or rendered in response to input received from a user via the movement of the light sensor device 305 or the I / O interface 317.

[0054] Examples of software applications stored in the application store 321 include gaming applications, conferencing applications, video playback applications, or programmable instructions for implementing various methods described herein. In a preferred embodiment of the present arrangement, the application store 321 may be a non-transitory memory store that also stores data from the non-optical sensor 340 or imaging device 315, or data from other sources including the optical sensor 305, or data received from the processing subsystem 313.

[0055] The tracking module 323 can calibrate the eyewear 322 using one or more calibration parameters, and can adjust the calibration parameters to reduce errors in determining the location of the light sensor device 305 or the I / O interface 317. Additionally, the tracking module 323 can track the movement of the light sensor device 305 or the I / O interface 317 using information from the imaging device 315 to one or more position sensors 335, IMU 342, or some combination thereof. The I / O interface 317 can represent a component that allows a user to send action requests and receive responses from the processing subsystem 313. In some embodiments of the present arrangement, an external controller can send such action requests and receive such responses via the I / O interface 317. The action request can represent a request to perform a particular action in some instances. By way of example, the action request can be an instruction to start or stop capturing image or video data, or an instruction to perform a particular action within a software application. The I / O interface 317 can include one or more input devices. Exemplary input devices include a keyboard, a mouse, a handheld controller, or any other device suitable for receiving action requests and communicating the action requests to the processing subsystem 313 .

[0056] Additionally, the I / O interface 317 may enable the eyewear 322 to interact with external devices and / or system accessories, such as additional standalone sensor systems or handheld controllers, via wired or wireless channels. In a preferred embodiment of the arrangement, the light sensor device 305, the processing subsystem 313, and / or the I / O interface 317 may be integrated into a single housing or body. Other embodiments may include a distributed configuration of the eyewear 322, in which the light sensor device 305 may be in a separate housing or enclosure, but may be further coupled to the processing subsystem 313 by a wired or wireless communication channel. Illustratively, the light sensor device 305 may be internal or coupled to the processing subsystem 313 provided by an external gaming console or an external computer, such as a desktop or laptop computer. The processing subsystem 313 may also be a dedicated hardware component designed specifically to cooperate with the light sensor system 305 to perform the various operations described herein.

[0057] The eyewear 322 can use, among other things, an array of variable phase optical elements (e.g., diffractive optical elements (DOEs)) to introduce a phase delay into the wavefront of light received through a lens, thereby improving one or more of the performance of the light sensor device 305, or in particular the imaging device 315, to capture aspects of a scene. These improvements can be a function of how the light passes through the variable phase optical elements, and in some present arrangements, can also be a function of shifting the array of eyewear 322 or another optical component (e.g., a sensor or lens). By way of illustration, the phase delay introduced by the array of variable phase optical elements can enable the eyewear 322 to capture at least two different perspectives of a scene, and various embodiments of the XR system described herein can use these different perspectives to provide or increase resolution (e.g., in angular, depth, and / or spectral scales) of an output image or frame obtained from the light sensor device 305.

[0058] Various embodiments of the optical sensor devices disclosed herein (e.g., optical sensor device 305 of FIG. 3) can use variable phase optical elements to capture different perspectives of a scene in a variety of different ways and for numerous different purposes. By way of illustration, a DOE may be configured to diffuse light from a scene as an interference pattern on the optical sensor device while at a first location, and the optical sensor device may capture the interference pattern as a first perspective of the scene. The DOE may be laterally translated to a subsequent location such that the resulting interference pattern represents another perspective of the scene, and another perspective of the scene may also be captured by the optical sensor device. These two perspectives may be processed to increase angular resolution (e.g., via oversampling) or to perform depth sensing (e.g., via triangulation and / or phase discrimination).

[0059] As an example, depth values ​​of a scene may be obtained using triangulation between two perspectives by using a DOE to provide two perspectives to a single photosensor device. As another example, each element in a layer or array of variable phase optical elements may be configured to deterministically phase shift and focus light to a particular pixel (or set of pixels) of a photosensor device. These phase shifted wavefronts, which may represent different perspectives of a scene, may be captured, mixed, and compared against a reference signal to detect depth within the scene.

[0060] Also, the embodiments of the arrangements described herein may be implemented in various types of systems (e.g., conventional CMOS sensor systems, time-of-flight (ToF) systems, or hyperspectral imaging systems) having various configurations (e.g., configurations with static or movable optical components). As an example of an implementation with movable optical components (e.g., when choosing to move the optical sensor), the optical sensor device may include an array of variable phase optical elements positioned over individual pixels or voxels of the imaging device, and an actuator configured to move a component of the optical sensor device (e.g., array, sensor, or lens) to obtain two different images representing two different instantaneous fields of view (iFOV) per pixel. The system may then analyze these images to obtain or infer further spatial information about the imaged scene. In some implementations using a ToF sensor, the scene may be captured with greater spatial resolution by translating the component using a conventional large pixel ToF sensor system to oversample the image plane or a portion of the scene. In implementations using non-ToF sensors (e.g., conventional CMOS sensors), the system can perform triangulation and / or phase discrimination operations on the different iFOVs to calculate a depth map of the scene, and for both non-ToF and ToF sensors, the system can interpolate between phase-shifted iFOVs to improve the angular resolution of the image captured by the sensor.

[0061] Oversampling may also be used to increase the spatial resolution of various hyperspectral imaging systems (e.g., snapshot hyperspectral imaging systems). Conventional hyperspectral imaging can sample broadband light in the spectral domain using a hyperspectral filter (e.g., a tiled filter or a mosaic filter) placed directly on the imaging device, increasing the spectral resolution at the expense of spatial resolution. In contrast, the proposed hyperspectral imaging system can decouple the hyperspectral filter from the imaging device and position a variable phase optic between the filter and the imaging device to facilitate spatial oversampling and improved spatial resolution. For example, a scene can be captured in a hyperspectral image with greater spatial resolution by translating the variable phase optic and oversampling portions of the image plane or scene through individual windows of the hyperspectral filter.

[0062] In one embodiment of the optical sensor device, each optical component is fixed at a single location and / or movable between two or more locations in a plane perpendicular to the optical axis. For example, a system with fixed optical components can introduce two or more different phase shifts in an incident wavefront. These phase shift signals are then mixed and compared to a reference signal. As another example, a global shutter system can include an optical element that creates two phase shift paths that are captured and stored by an imaging device while the optical element is in a first location. The system can then move the optical element to a second location to create two additional phase shift paths, which may also be captured by the imaging device. As a result, the imaging device can simultaneously provide four phase shift signals to an electrical quadrature demodulation component, which may be mixed and compared to a reference signal to create a depth map of the scene.

[0063] 4A illustrates a block diagram of various software modules and engines, each of which includes programmable instructions for performing one or more steps included in various method implementations according to the present teachings. The processing subsystem 413, whether in a client device (e.g., the processing subsystem 313 of the eyewear 322 of FIG. 3) or in a server (e.g., the server 110 of FIG. 1), includes an image spatialization module 476, an attribute spatialization module 478, an alignment module 480, and a rendering engine 482.

[0064] The image spatialization module 476 includes programmable instructions for, among other things, spatializing a first type of pixel data or voxel data to produce a first type of three-dimensional spatialization data. The attribute spatialization module 478 includes programmable instructions for, among other things, spatializing a second type of three-dimensional pixel data or voxel data to produce a second type of three-dimensional spatialization data, the second type of three-dimensional spatialization data being a different type of three-dimensional spatialization data than the first type of three-dimensional spatialization data. The alignment module 480 includes programmable instructions for, among other things, aligning the first type of three-dimensional spatialization data with the second type of three-dimensional spatialization data to produce an augmented three-dimensional spatialization environment. The rendering engine 482 includes programmable instructions for, among other things, rendering the "processed information" or augmented three-dimensional spatialization environment to a display component. Although rendering engine 482 is described as a preferred embodiment, the present arrangement contemplates the use of other types of engines, such as a display engine, a data engine, an execution engine, or a user interface (UI) engine, to achieve the various results contemplated by the present teachings.

[0065] It is not necessary that each of the various modules and engines presented in Fig. 4A be located on a processor subsystem. According to this arrangement, these modules and engines may include a distributed configuration in which some modules may reside on a light sensor device (e.g., light sensor device 305 of Fig. 3) coupled to a processing subsystem (e.g., processing subsystem 313, or processing subsystem 313 located on a smartphone or personal computer) by a wired or wireless communication channel. In this configuration, the processing subsystem is preferably a dedicated hardware component designed specifically to cooperate with light sensors and / or non-light sensors to perform the various operations described herein.

[0066] FIG. 4B illustrates one exemplary distributed configuration of the modules and engines depicted in FIG. 4A. According to FIG. 4B, processor subsystem 413′ includes attribute spatialization module 478′, alignment module 480′, and rendering engine 482′, each of which are substantially similar to their counterparts depicted in FIG. 4A, namely attribute spatialization module 478, alignment module 480, and rendering engine 482. Processing subsystem 413′ of FIG. 4B is communicatively coupled to light sensor device 405 via network (e.g., Internet) 465. Light sensor device 405 is further similar to light sensor device 305 of FIG. 3, except that light sensor device 405 includes image spatialization module 476′. Programmable instructions contained within image spatialization module 476′ of FIG. 4B are substantially similar to programmable instructions found within image spatialization module 476 of FIG. 4A. In this distributed configuration, the image spatialization module in the light sensor device and the modules and rendering engine in the processor subsystem work together to perform the methods described herein.

[0067] 5 illustrates an XR system 500 according to one embodiment of the arrangement. In this embodiment, eyewear 522 has stored therein programmable instructions 200 for performing certain methods of the present teachings described herein. In one embodiment of the arrangement, eyewear 522 is substantially similar to at least one of eyewear 122 of FIG. 1, eyewear 222 of FIG. 2, or eyewear 322 of FIG. 3. In one preferred embodiment of the arrangement, programmable instructions 200 include image spatialization module 476, attribute spatialization module 478, alignment module 480, and rendering engine 482 of FIG. 4.

[0068] FIG. 6 illustrates an XR system 600 according to one embodiment of the present arrangement. In this embodiment, the eyewear 622 is substantially similar to at least one of the eyewear 122 of FIG. 1, the eyewear 222 of FIG. 2, or the eyewear 322 of FIG. 3, but is communicatively coupled to the smartphone 618. In one aspect of this embodiment, the smartphone 618 is substantially similar to the smartphone 118 of FIG. 1. Furthermore, in the XR system of FIG. 6, the smartphone 618 stores therein programmable instructions 200 for performing some methods of the present teachings described herein. In an alternative embodiment of the present arrangement, the modules and engines comprising the programmable instructions 200 are distributed between the smartphone 618 and the eyewear 622. In one implementation of this alternative embodiment, the distributed arrangement of software modules is implemented between processor subsystems in the smartphone 618 and the eyewear 622. The present arrangement recognizes that the programmable instructions in the distributed arrangement are not limited to client devices.

[0069] To this end, Figure 7 illustrates an XR system 700 according to an alternative embodiment of the present arrangement. In this embodiment, eyewear 722 is communicatively coupled to a network (e.g., the Internet) 714, with eyewear 722 having stored therein all or at least some of the modules or engines comprising programmable instructions 200. Eyewear 722 is substantially similar to at least one of eyewear 122 of Figure 1, eyewear 222 of Figure 2, or eyewear 322 of Figure 3, and network 714 is substantially similar to at least one of network 114 of Figure 1, communication bus 365 of Figure 3, or network 465 of Figure 4B.

[0070] 8 illustrates an XR system 800 according to yet another embodiment of the present arrangement. In this embodiment, eyewear 822 is communicatively coupled to a personal computer 816, which can function as a client device and / or a server. In one embodiment, the processing subsystem is in the personal computer 816, which stores programmable instructions 200 for performing some of the methods of the present teachings described herein. In an alternative implementation of the XR system illustrated in FIG. 8, the modules and engines comprising the programmable instructions 200 are distributed between the personal computer 816 and the eyewear 822.

[0071] 9 illustrates an XR system 900 according to another embodiment of the present arrangement, which is a smartphone 918 having programmable instructions 200 stored therein. In this embodiment, the smartphone 918 includes a display on which "processed information" resulting from the analysis of various types of data sets is rendered. As a result, the XR system of FIG. 9 does not require eyewear to perform some of the methods of the present teachings described herein.

[0072] In one aspect, the present teachings provide a method for rendering two or more different types of datasets. Figure 10 shows a flowchart of a method 1000 according to some embodiments of the present teachings for rendering two different types of datasets. The method 1000 preferably begins with step 1002, which includes obtaining a first type of dataset. The first type of dataset includes a plurality of first data values. Each first data value in the first type of dataset is associated with one or more three-dimensional coordinates. The three-dimensional coordinates further define a particular location or region in real space.

[0073] In one embodiment of the present teachings, a ground positioning component (e.g., position sensor 335 of FIG. 3) and a first sensor provide or facilitate the provision of a first type of data set. The ground positioning component informs the location of measurements made when acquiring the first data set described in step 1002. The ground positioning component, in preferred embodiments of the arrangement, uses relative positioning techniques to arrive at three-dimensional coordinates, as opposed to making absolute position measurements such as those encountered when using a Global Positioning System ("GPS") sensor. Furthermore, the three-dimensional coordinates are obtained in these preferred embodiments using positioning measurements relative to one or more reference points or coordinates. The ground positioning component may be located in eyewear (e.g., eyewear 222 of FIG. 2 or eyewear 322 of FIG. 3).

[0074] The first sensor may be an optical sensor (e.g., optical sensor 305 in FIG. 3) or a non-optical sensor (e.g., non-optical sensor 340 in FIG. 3) and may be disposed within the eyewear, such as a ground positioning component. Ultimately, the arrangement relies on obtaining the first dataset from a storage medium, such as a first database, which may then be within or communicatively coupled to the eyewear, such that the first dataset information is available for processing as described below.

[0075] Method 1000 also includes step 1004, which includes obtaining a second type of data set, the second type of data set being different from the first type of data set. In one embodiment of the present teachings, one or both of the first type of data set and the second type of data set still represent a measured parameter, such as a magnitude, of one or more intangible properties present in the real space. Examples of such intangible properties include throughput of a connection signal, latency of a connection signal, interference of a connection signal, variability of a connection signal, stability of a connection signal, RF power output, EMF, air pressure, geomagnetic field, Hall effect, ambient light level, gas concentration, smoke, sound pressure, audio harmonics, humidity, carbon dioxide emission, or temperature.

[0076] Moreover, the second data set, like the first data set, is associated with one or more of the three-dimensional coordinates. As a result, when the first data set and the second data set are acquired, location information regarding the data values ​​in each data set is known. To this end, the disclosure of the nature and function of the ground positioning component, the first sensor, and the first database, as described above in connection with the first data set in step 1002, applies equally to acquiring the second data set in step 1004. The second data set may also be acquired using a second sensor and a second database. Like the first sensor, the second sensor may be an optical sensor (e.g., optical sensor 305 in FIG. 3) or a non-optical sensor (e.g., non-optical sensor 340 in FIG. 3) and may be located within the eyewear or in an external device outside the eyewear. The second database may be within the eyewear or communicatively coupled to the eyewear.

[0077] After the requisite number of first data values ​​are obtained from step 1002, the method 1000 may then proceed to step 1006, which includes spatializing the first type of data set using the plurality of three-dimensional coordinates to create a first type of spatialized data set. In a preferred embodiment of the present teachings, an image spatialization module (e.g., image spatialization module 476 of FIG. 4A or image spatialization module 476' of FIG. 4B) is used to create the first type of spatialized data set. The image spatialization module may be on the eyewear or on an external processor. If the image spatialization module is on an external processor, the external processor, external to the eyewear, is communicatively coupled to the eyewear to convey "processed information" resulting from processing the first type of spatialized data set to the eyewear.

[0078] In embodiments where an image spatialization module is used in step 1006, this module preferably begins the spatialization step by spatially partitioning the real space into a plurality of subdivisions. The image spatialization module then aggregates the subdivisions to create a spatialized model of the real space. In the spatialization step 1006, the first data values ​​may be distributed based on the spatialization model to create a first type of spatialized data set. Furthermore, the "real space" is electronically represented in the method using a plurality of three-dimensional position coordinates, which may have been retrieved from steps 1002 and / or 1004.

[0079] At any time after the second data set is obtained from step 1004, the method 1000 can then proceed to step 1008, which includes spatializing the second type data set using the plurality of three-dimensional coordinates to produce a second type spatialized data set. In a preferred embodiment of the present teachings, an attribute spatialization module (e.g., attribute spatialization module 478 of FIG. 4A or attribute spatialization module 478' of FIG. 4B) is used to produce the second type spatialized data set. The attribute spatialization module may be on the eyewear or on an external processor. If the attribute spatialization module is on an external processor, the external processor, external to the eyewear, is communicatively coupled to the eyewear to convey "processed information" resulting from processing the second type spatialized data set to the eyewear.

[0080] In embodiments where an attribute spatialization module is used in step 1008, this module preferably begins the spatialization step by spatially partitioning the real space into a number of subdivisions, as does the image spatialization module in step 1006. The attribute spatialization module then aggregates the subdivisions to produce a spatialized model of the real space. In the spatialization step 1008, the second data values ​​may be distributed based on the spatialization model to produce a second type of spatialized data set.

[0081] In one implementation of method 1000, when a spatialization model is generated from one of the image spatialization module or the attribute spatialization module, the other spatialization module is free to use the generated spatialization model and does not need to generate a separate model or recreate the spatialization model to implement the distribution of the first or second data values, as the case may be.

[0082] Once the two data sets have been spatialized to form the first and second types of spatialized data sets, the method 1000 can proceed to step 1010, which includes aligning the first type of spatialized data set with the second type of spatialized data set to create an augmented three-dimensional spatialized environment. In a preferred embodiment of the present teachings, an alignment module (e.g., alignment module 480 of FIG. 4A or alignment module 480′ of FIG. 4B) is used to perform the alignment step 1010. The alignment module can reside on the eyewear or on an external processor, which is located outside the eyewear and communicatively coupled to the eyewear. In the method 1000, the augmented three-dimensional spatialized environment represents the “processed information” that is then rendered on a display component of the eyewear or smartphone.

[0083] Regardless of where they are, the analysis module can use one or more common spatial features present in both the first type of spatialization data set and the second type of spatialization data set to perform the alignment of the two spatialization data sets. Examples of such common spatial features include intersections, longitudes, or latitudes. Nevertheless, using one or more common intersections found in both the first type of spatialization data set and the second type of spatialization data set to perform the alignment represents a preferred embodiment of step 1010.

[0084] After aligning the two spatialized data sets, method 1000 proceeds to step 1012, which includes rendering the augmented three-dimensional spatialized environment using a rendering engine (e.g., rendering engine 482 of FIG. 4A or rendering engine 482 of FIG. 4B). In some embodiments of this arrangement, the rendering engine may reside on the eyewear or on an external processor that is located external to and communicatively coupled to the eyewear.

[0085] The rendering step can render the augmented three-dimensional spatialized environment on a display component (e.g., a display component embedded in frame 202 of FIG. 2 or I / O interface 317 of FIG. 3). In some exemplary embodiments, the display component is integrated into a client device, where the client device is at least one device selected from the group including a wearable, a smartphone, and a computer. In a preferred embodiment of the present teachings, the wearable is eyewear (e.g., eyewear 222 of FIG. 2 or eyewear 322 of FIG. 3).

[0086] The present teachings provide a novel method for rendering an image dataset. Figure 11 shows a flow chart for one exemplary method 1100, which preferably begins with step 1102 of obtaining boundary data of a real space. In some embodiments, step 1102 of the present teachings includes obtaining three-dimensional coordinates that define, i.e., electronically represent, the boundary of a scene. The scene typically exists in a real space in front of a user of the eyewear.

[0087] Method 1100 then includes step 1104, which calls for subdividing (also referred to as subdivision or subpartitioning) the boundary data of the real space into a plurality of subdivisions. In some embodiments, step 1104 of the present teachings includes subdividing the electronically represented boundary obtained in step 1102 such that each of the resulting subdivisions is electronically represented by a three-dimensional subdivision boundary coordinate. In other embodiments of the present teachings, the subdivision described in step 1104 includes spatially partitioning the real space into a plurality of subdivisions. In these embodiments, the plurality of corresponding three-dimensional position coordinates collectively form an electronic representation of the real space, which is spatially partitioned. The word "corresponeing" is used in conjunction with the term "position coordinate" as it appears in method 1100, to convey that "position coordinate" is the coordinate from which at least one pixelated value or at least one voxelized value is obtained. Thus, a "position coordinate" "corresponds to" or is associated with a pixelated or voxelized value obtained at the "position coordinate."

[0088] The method 1100 includes a step 1106, which includes obtaining three-dimensional pixelated or voxelized data values ​​for corresponding position coordinates, where each of the corresponding position coordinates is "associated" with at least one of the associated three-dimensional pixelated or voxelized data values. The word "associated" used in connection with the terms "three-dimensional pixelated data values" and "three-dimensional voxelized data values" conveys that a pixelated or voxelized value obtained at a particular position coordinate is "associated" with this position coordinate.

[0089] In one example of step 1106, obtaining the three-dimensional pixelated data values ​​or three-dimensional voxelized data values ​​includes measuring the light intensity or color intensity present in a unit pixel area or unit voxel (volume) space of a light sensor (e.g., light sensor 305 in FIG. 3), respectively.

[0090] According to one embodiment of the present teachings, step 1106 is performed substantially similarly to step 1002 of Figure 10. Additionally, various embodiments that obtain the necessary data values ​​described in connection with step 1002 of Figure 10, for example using an optical sensor or a database, may be similarly performed in step 1106 of Figure 11.

[0091] Once the subdivisions have been obtained, method 1100 may proceed to step 1108, which includes identifying one or more of the subdivisions that contain one or more of the corresponding position coordinates. Such subdivisions are preferably referred to as "selected subdivisions," and step 1108 generates one or more of such selected subdivisions. In other words, a selected subdivision simply refers to a subdivision (i.e., a subdivision resulting from step 1104) that includes at least one location for which at least one three-dimensional position coordinate, i.e., a measurement of a pixelated or voxelized value, was obtained.

[0092] The method 1100 then proceeds to step 1110, which includes assigning at least one of the associated three-dimensional pixelated data values ​​or the associated three-dimensional voxelized data values ​​to one or more of the selected subdivisions to define one or more assigned subdivisions. In this step, each of the three-dimensional pixelated data values ​​or the associated three-dimensional voxelized data values ​​associated with at least one of the corresponding position coordinates is assigned to the selected subdivisions. Furthermore, after the assignment, these selected subdivisions are referred to as "assigned subdivisions". In other words, each assigned subdivision is assigned at least one of the associated three-dimensional pixelated data values ​​or at least one of the associated three-dimensional voxelized data values.

[0093] In one preferred embodiment, step 1110 includes assigning at least one of the associated three-dimensional pixelated data values ​​or at least one of the associated three-dimensional voxelized data values ​​to all portions of the selected subdivision. In another preferred embodiment, where more than one pixelated or voxelized value is available, step 1110 includes assigning the sum of two or more weighted values ​​of the associated three-dimensional pixelated data values ​​or the sum of two or more weighted values ​​of the associated three-dimensional voxelized data values ​​to the selected subdivision.

[0094] After step 1110, the method 1100 preferably proceeds to step 1112, which includes aggregating the assigned subdivisions to form a spatialized image dataset. In some preferred embodiments, step 1112 of the present teachings includes aggregating a plurality of three-dimensional subdivision boundary coordinates to form a spatialized image dataset, the three-dimensional subdivision boundary coordinates defining one or more subdivisions. In other preferred embodiments, step 1112 of the present teachings includes aggregating a plurality of subdivisions to create a spatialized model for the real space. In these embodiments, the associated three-dimensional pixelated data values ​​or the associated three-dimensional voxelized data values ​​are distributed based on the spatialized model to create a spatialized image dataset.

[0095] Regardless of how it is formed, the spatialized image dataset of the present teachings may be further used in other ways described herein to achieve other novel results, with the spatialized image dataset acquired in method 1100 representing the “processed information” that is rendered on a display component of eyewear or a smartphone in a subsequent step.

[0096] To this end, in an optional implementation, the method 1100 may proceed to a rendering step, which uses a rendering engine to render the spatialized image dataset on a display component of a client device. In one embodiment of the present teachings, the optional rendering step of the method 1100 is performed in a manner substantially similar to the rendering step 1012 of Figure 10. Moreover, various embodiments described with respect to the rendering step 1012 of Figure 10 may similarly be performed in the optional rendering step of the method 1100.

[0097] In another aspect, the present teachings provide a novel method for rendering an attribute value data set. Figure 12 shows a flow chart for one exemplary method 1200, which preferably begins with step 1202 of obtaining real space boundary data. In some embodiments, step 1202 of the present teachings includes obtaining three-dimensional coordinates that define, i.e., electronically represent, a boundary of the scene. Step 1202 is substantially similar to step 1102 of Figure 11, and various embodiments described for step 1102 of Figure 11 may be similarly implemented to perform step 1202.

[0098] Method 1200 then includes step 1204, which calls for subdividing the boundary data of the real space into a plurality of subdivisions. Step 1204 is substantially similar to step 1104 of FIG. 11, and various embodiments described for step 1104 of FIG. 11 may be similarly implemented to perform step 1204. As an example, step 1204 of the present teachings may include subdividing the electronically represented boundary obtained in step 1202, such that each of the resulting subdivisions is electronically represented by a three-dimensional subdivision boundary coordinate. As another example, the subdivision of step 1204 may include spatially partitioning the real space into a plurality of subdivisions. In this example, the plurality of corresponding three-dimensional position coordinates collectively form an electronic representation of the spatially partitioned real space. The word "corresponding" used in conjunction with the term "position coordinate" as it appears in method 1200 conveys that the "position coordinate" is a coordinate for which at least one attribute value of a particular type has been obtained. Thus, the "location coordinates" "correspond to" or relate to the attribute values ​​captured at the "location coordinates."

[0099] Method 1200 also includes step 1206, which entails obtaining one or more different types of attribute values ​​for the corresponding location coordinates. In this step, each of the corresponding location coordinates is "associated" with at least one type of attribute value. The word "associated" used in conjunction with the term "attribute value" conveys that one or more attribute values ​​obtained at a particular location coordinate are "associated" with that location coordinate.

[0100] In one example of step 1206, a magnitude of an intangible property of the real space is obtained. In one implementation of this example, step 1206 includes detecting, in the real space, a value, e.g., a magnitude, of a parameter of at least one type of attribute selected from the group including: connection signal throughput, connection signal latency, connection signal interference, connection signal variability, connection signal stability, RF power output, EMF, air pressure, geomagnetic field, Hall effect, ambient light level, gas concentration, smoke, sound pressure, audio harmonics, humidity, carbon dioxide emission, and temperature.

[0101] According to one embodiment of the present teachings, step 1206 is performed in a manner substantially similar to step 1004 of Figure 10. Further, for example, various embodiments described in connection with step 1004 of Figure 10 that use non-optical sensors or databases to obtain necessary data values ​​may similarly be performed in step 1206 of Figure 12.

[0102] Once the subdivisions have been obtained, method 1200 may proceed to step 1208, which includes identifying one or more of the subdivisions that contain one or more of the corresponding position coordinates. Such subdivisions are preferably referred to as "selected subdivisions," and step 1208 produces one or more of such selected subdivisions. In other words, a selected subdivision simply refers to a subdivision (i.e., a subdivision resulting from step 1204) that contains at least one three-dimensional position coordinate, i.e., at least one location, for which a measurement of at least one attribute value of a particular type has been obtained.

[0103] Method 1200 then proceeds to step 1210, which includes assigning at least one of at least one type of associated attribute value to one or more of the selected subdivisions to define one or more assigned subdivisions. In this step, each of the associated attribute values ​​of a particular type associated with at least one of the corresponding location coordinates is assigned to the selected subdivisions. Further, after assignment, these selected subdivisions are referred to as "assigned subdivisions." In other words, each assigned subdivision is assigned at least one of the associated attribute values ​​of a particular type.

[0104] In one preferred embodiment, step 1210 includes assigning at least one of the associated attribute values ​​of a particular type to all portions of the selected subdivision. In another preferred embodiment, where more than one attribute value of a particular type is available, step 1210 includes assigning a sum of two or more weight values ​​of the associated attribute values ​​of a particular type to the selected subdivision.

[0105] After step 1210, method 1100 preferably proceeds to step 1212, which includes aggregating the assigned subdivisions to form a spatialized attribute value data set. In some preferred embodiments, step 1212 of the present teachings includes aggregating a plurality of three-dimensional subdivision boundary coordinates to form a spatialized attribute value data set, the three-dimensional subdivision boundary coordinates defining one or more subdivisions. In other preferred embodiments, step 1212 of the present teachings includes aggregating a plurality of subdivisions to create a spatialized model for the real space. In these embodiments, at least one type of associated attribute value is distributed based on the spatialized model to create a spatialized attribute value data set.

[0106] Regardless of how it is formed, the spatialized attribute value dataset of the present teachings may be further used in other ways described herein to achieve other novel results, with the spatialized attribute value dataset representing the “processed information” of method 1200 that is subsequently rendered on a display component of the eyewear or smartphone.

[0107] To this end, in an optional implementation, method 1200 may proceed to a rendering step, which uses a rendering engine to render the spatialized attribute value dataset on a display component of a client device. In one embodiment of the present teachings, the optional rendering step of method 1200 is performed in a manner substantially similar to rendering step 1012 of Figure 10. Moreover, various embodiments described with respect to rendering step 1012 of Figure 10 may similarly be performed in the optional rendering step of method 1200.

[0108] In yet another aspect, the present teachings provide a method for rendering an image dataset and one or more attribute value datasets. Figure 13 shows a flowchart for a method 1300 according to one embodiment of the present teachings for rendering an image dataset and one or more attribute value datasets.

[0109] In a preferred embodiment, method 1300 begins with step 1302, which includes obtaining three-dimensional pixelated or voxelized data values ​​for one or more corresponding three-dimensional position coordinates, where each of the corresponding three-dimensional position coordinates is associated with at least one associated three-dimensional pixelated or voxelized data value. Step 1302 is preferably performed using an optical sensor and a ground positioning component. Additionally, the various embodiments described for step 1002 of FIG. 10 may be similarly performed to perform step 1302.

[0110] Regardless of the method and components used, step 1302 provides images of a real space (e.g., inside four corners of a wall in a room). In some embodiments of the present teachings, when such images are acquired using an XR system of the present arrangement, metadata fields may be attached to the images. These metadata fields may include at least one information selected from the group including the model of the imaging device, the time the image was acquired, whether a flash was used during image capture, shutter speed, focal length, light value, and location information provided by a geographical positioning component. The collection of images may thus be used to map a real space, such as an office environment (see, e.g., FIG. 13). As described below, data relating to the real space is processed in accordance with the present teachings to visualize the real space from a number of perspectives.

[0111] In many aspects, a user or viewer is not only enabled to visualize a three-dimensional replica of the real space, but one or more users or viewers traverse the space using conventional controls such as joysticks, 3D controllers, and keyboards to collect the associated three-dimensional pixelated data values ​​or associated three-dimensional voxelized data values ​​described in step 1302. To this end, as described above in the context of "edge computing", a number of client devices and controllers in the possession and / or control of the users or viewers preferably collect the associated three-dimensional pixelated data values ​​or associated three-dimensional voxelized data values ​​and corresponding position coordinates described in step 1302.

[0112] Method 1300 also includes step 1304, which involves obtaining one or more different types of attribute values ​​for one or more of the corresponding three-dimensional position coordinates using at least one non-optical sensor. In this step, each of the corresponding three-dimensional position coordinates is associated with at least one type of associated attribute value. Various embodiments described for step 1004 of FIG. 10 may be similarly implemented to perform step 1304.

[0113] The present teachings recognize that step 1302 and / or step 1304 may include many novel implementation features. Illustratively, in cases where multiple client devices and / or external controllers such as positioning controllers may be employed to collect various types of data values ​​(e.g., image data values ​​or attribute values) from a user or viewer, the client devices or external controllers know their relative positions with respect to one or more reference points or coordinates using a geolocation component or a positioning sensor disposed on the geolocation component. Moreover, the client devices or external controllers detect the location of the client devices or external controllers when within view on the mobile application and adjust their position or fuzziness or possibly extend virtual control based on their position or based on their state. As one or more users or viewers move the client device or external controller away from the reference point or coordinates, the display may ultimately present the same image data and / or intangible property data of the real space, but the "processed information" may be rendered from a different view point and / or may be rendered from a perspective that is even further away.

[0114] Conventional 3D mapping techniques may be employed to implement some embodiments disclosed herein in steps 1302 and 1304, and possibly in some subsequent steps described in Figure 13. By way of illustration, different types of datasets may employ different modeling techniques to optimize visualization and 3D interactivity of the data. Additionally, some dataset animations may be simulated by rapidly plotting the datasets in succession.

[0115] As another example, a graphical user interface element may be supported with a three-dimensional visualization or rendering that presents the state of a selected data analysis. For example, and without limitation, a slider may be mapped to the fuzziness of a data layer (i.e., a data layer corresponding to a particular type of data set). Moving the slider up may change the particular type of data layer threshold level, making the data layer more obscure, and conversely, moving the slider down may make the particular type of data layer more transparent. These graphical elements may be animated for better usability, such as fading text, or moving arrows and / or lines. In a slider layer example, the actual fuzziness value and / or percentage may be drawn on the slider knob itself while the slider knob is being slid, and then disappear after a predetermined period of time, such as about 2 seconds, from when the user finishes moving the slider. In this manner, the graphical interface elements of the present teachings may convey the state of one or more different types of data, as well as the state of buttons and associated components of a client device or external controller.

[0116] The present teachings recognize that by correlating image data with intangible property data, an XR system of this arrangement can present multiple variable views of a physical space in a single rendering.

[0117] After step 1302 obtains the requisite number of pixelated or voxelized values, method 1300 can proceed to step 1306, which calls for spatializing the three-dimensional pixelated or voxelized data values ​​to produce a spatialized image dataset. In a preferred embodiment, step 1306 relies on a plurality of corresponding three-dimensional position coordinates to arrive at the spatialized image dataset. By way of illustration, one or more steps described in method 1100 of FIG. 11 may be performed to produce the spatialized image dataset. Additionally, various embodiments described with respect to one or more steps of method 1100 of FIG. 11 may be similarly performed to perform step 1306.

[0118] After step 1304 obtains the requisite number of attribute values, method 1300 may proceed to step 1308, which calls for spatializing at least the associated attribute values ​​of at least one particular type to produce a spatialized attribute value dataset of this type. Step 1308, like step 1306, may rely on a number of corresponding three-dimensional position coordinates to arrive at the spatialized attribute value dataset. By way of illustration, one or more steps described in method 1200 of FIG. 12 may be performed to produce the spatialized attribute value dataset. Additionally, various embodiments described with respect to one or more steps of method 1200 of FIG. 12 may be similarly performed to perform step 1308.

[0119] After the spatialized image dataset and the spatialized attribute value dataset are acquired, the method 1300 proceeds to step 1310, which includes aligning the image spatialized dataset with the attribute value spatialized dataset to create an augmented three-dimensional spatialized environment. According to one embodiment, step 1310 of the present teachings is substantially similar to step 1010 of FIG. 10, and various embodiments described with respect to step 1010 of FIG. 10 may be similarly performed to perform step 1310. Illustratively, one or more common features present in both the image spatialized dataset and the spatialized attribute value dataset, such as one or more intersections, longitudes or latitudes, are used to perform the alignment of step 1310. The augmented three-dimensional spatialized environment of step 1310 represents the "processed information" of the method 1300, and is rendered in a subsequent step on the display components of the eyewear and the smartphone.

[0120] As a result, the method 1300 then proceeds to step 1312, which includes rendering the augmented three-dimensional spatialized environment on the display component using a rendering engine. According to one embodiment, step 1312 of the present teachings is substantially similar to step 1012 of FIG. 10, and the various embodiments described with respect to step 1012 of FIG. 10 may be similarly implemented to perform step 1312.

[0121] Each of method 1000 of Figure 10, method 1100 of Figure 11, method 1200 of Figure 12, and method 1300 of Figure 13 represent various types of programmable instructions of the present teachings that reside in the XR system of the present arrangement. While use of the XR system of the present arrangement represents a preferred mode of implementation, the present teachings recognize that these programmable instructions may be executed using other structural details not described herein.

[0122] FIG. 14 shows a rendering of an office environment 1400 according to one embodiment of the present teachings. In this rendering of the office environment 1400, a three-dimensional image 1410 of the actual office is overlaid with a wireless connection signal strength indicator ("RSSI") 1412, represented as a virtual bar (dotted line). The raw data, RSSI 1412, obtained from step 1304 of FIG. 13, is integrated into the image of the office 1410, the raw data, obtained from step 1302 of FIG. 13. The "processed information" underlying the rendered office environment 1400 is developed using step 1410. In the final rendered office environment 1400, the rendered image data present in the real space is shown with a solid line border, and the rendered intangible property data, i.e., RSSI 1412, is shown with a dashed line border. As a result, the rendered office environment 1400 clearly conveys the distribution information of RSSI at various locations in the office space.

[0123] In one embodiment of the present teachings, animating the various renderings present in FIG. 14 allows a user or viewer to visualize the signal strength distribution from various perspectives, thereby allowing the viewer or user to identify areas of sufficient signal strength and / or areas of weak signal strength.

[0124] In some embodiments of the present teachings, the different colors of the RSSI 1412 are used to convey various signal strength values ​​obtained, for example, from various communication systems. In one embodiment, a virtual bar color representation, representing the position of the non-photosensor signal measurement, i.e., the RSSI, may be presented in various colors that vary along the wavelength spectrum, preferably ranging from a green color representation to a red color representation. In one preferred embodiment, a green color representation conveys that a particular type of measured attribute value is greater than a particular type of attribute threshold (e.g., RSSI), and a red color representation conveys that a particular type of attribute value is less than a particular type of attribute threshold. The present teachings recognize that this color-coded virtual bar representation easily conveys RSSI distribution information, representing one of the many advantages of the present teachings that are not available using conventional graphical techniques.

[0125] Additionally, when a user selects or compresses one selected RSSI 1416, the selected RSSI 1416 may be highlighted and a numerical value 1414 associated with the selected RSSI 1416 may be presented on the display to provide more specific information about the magnitude of the measured RSSI. The present teachings recognize that the virtual bar of the RSSI 1412 may not only provide actual measured information, but may also provide an estimate or prediction of the magnitude of the RSSI.

[0126] FIG. 15 illustrates a rendering of another environment 1500 according to another embodiment of the present teachings. In the rendering 1500, the presence of physical objects in the real world is indicated with a solid border, and the magnitude of intangible properties in the real space is indicated with a dashed border. Additionally, the rendering 1500 illustrates a virtual bar representing the RSSI 1512 and the event history location 1514. The event history location 1514 includes information about the cellular signal strength, and the interrupted call. This information may be collected from the user whenever a call is interrupted by an application that allows the user to detect the interrupted call and send the information from the geolocation component to a data store. The indicia representing the location of the interrupted call may be spherical, with a size proportional to the amount of the interrupted call and / or the duration of the interrupted call. In some embodiments of the present teachings, the indicia may be a graphical representation of the type of information being displayed. For example, and without limitation, the indicia may be an image of a cell phone. As a result, there are three layers of data: a first data layer devoted to presenting the physical objects present in the real space, a second data layer devoted to representing the distribution of one or more intangible properties in the real space (i.e. including the magnitude of the intangible properties), and a third layer devoted to providing and / or representing some analytical information derived from one or more measurements of the intangible properties present in the real space. Thus, the three layers of data allow a greater degree of visualization of the real space.

[0127] In one embodiment, the programmable instructions of the present teachings rely on complex data spatialization, sampling algorithms, and machine learning. When the various methods described herein are implemented as software applications, such applications discover signals originating from nearby communication devices that rely on radio waves for wireless connectivity, such as WiFi routers and Bluetooth devices. This information is preferably displayed back in a fully interactive, fully immersive overlay shown through the display of the present XR system, such as a headset or smartphone.

[0128] The present teachings recognize that a distinction can be made between signal collection devices and signal visualization devices, even though they may be the same device in some embodiments of the present teachings. Signal collection devices are generally called sensors and are classified as sensors that possess a measurement device. In the context of radio frequency data, a signal collection device would be a device that houses an RF receiver and an antenna that operates with protocols such as WiFi, Bluetooth, ZigBee, Z-Wave, 3G, 4G, LTE, and 5G. Signal visualization devices, on the other hand, are typically characterized by a display that can show objects mapped in real space. The visualization can include the use of holographic elements to create the perception that objects are floating in front of the user, such as in an AR headset or a smartphone case. A VR headset can use a simulated environment that allows remote viewing of a real space. Mixed reality ("MR") devices and / or XR devices are typically a combination of both technologies, i.e., AR and VR, allowing for either type of experience.

[0129] Examples of devices that include both signal collection and signal visualization components include the HoloLens available from Microsoft, the Magic Leap One available from Magic Leap, and smartphones such as the iPhone® X available from Apple, Inc., and the Galaxy S10 available from Samsung. Examples of devices that include only signal collection components include a dedicated RF antenna, microphone, and Geiger counter. Examples of devices that include only signal visualization components include the Meta2 (i.e., an AR headset), Oculus Rift (i.e., a VR headset), and HTC Vive (i.e., a mixed reality headset).

[0130] In one embodiment of this arrangement, signal measurements are accomplished by mounting a number of types of sensors to the signal collection device, which are discussed in connection with Figures 2 and 3 described herein.

[0131] The present teachings recognize that using sensor fusion and potential directionality from eyewear allows for triangulation of signals. Additionally, by using headset-mounted or stand-alone microphones, advanced audio profiles can be constructed from noise sources and rooms where noise propagation characteristics (such as material reverberation factors) can be modeled. The present teachings further recognize that using special high-pass and low-pass filters allows for triangulation of some types of equipment such as low-humming industrial generators or high-pitched transformers. Even in a typical home, devices such as televisions, computers, and wireless routers emit distinct patterns of noise that are beyond the range of the human ear but within the range of a microphone. With such recognition, the present teachings can achieve audio fingerprinting. Exemplary devices used for audio fingerprinting include at least one device selected from the group including a thermometer, a hygrometer, a barometer, and an air flow sensor. The present teachings recognize that, in some embodiments, by using a simple weather station, measurement of signals in 3D and eventual visualization of the signals can accomplish such things as HVAC mapping, microclimate analysis, chemical tracing, insulation planning, and food storage inspection.

[0132] The sensors used in this arrangement may be used to detect radiation, both heat and infrared. To this end, examples of such sensors include Geiger counters and specific isotope detectors used in laboratory or nuclear power plant environments.

[0133] In other embodiments, the sensors of the present arrangement include sensors that detect several harmful gases (e.g., carbon monoxide, methane, and natural gas), as well as harmful molds that occur naturally but are not always easily detected. In this context, the present teachings recognize that sensors that detect smoke can be useful when implementing methods to inform about several conditions. Smoke detection sensors are useful for emergency scenarios such as fire fighting and industrial safety, or ventilation planning. Similarly, light detection sensors used in the present arrangement may be used in the form of a light meter to gather accurate color intensity and temperature values ​​from the environment. This information can be useful for the film, lighting, and interior design industries to map the reflectance of a scanned scape and the overall photo atmosphere.

[0134] In instances where vibration information is deemed useful or necessary, the sensors of the present arrangement are sensors that detect vibration and seismic activity. For home automation applications, the sensors of the present arrangement detect overlay positions of home appliances, HVAC controls, light switches, and power outlets. Additionally, the present teachings are applied to achieve industrial automation based on image recognition of objects, manual input, or beacon triangulation.

[0135] Whether or not it needs to be automated, the arrangement preferably includes components that facilitate spatial data collection. To this end, the collection process preferably requires one or more users to make sensor signal measurements from various positions in the space. More preferably, these sensor signal measurements are uniformly distributed in the target space. By way of illustration, when the present teachings are implemented as a software application on a mobile device (e.g., eyewear or a smartphone), the sensor signal measurements may be made automatically as one or more users walk around the space. These users may mount sensors to a headset or AR or XR enabled display, hold a signal collection device within the view of the headset or AR or XR enabled display, or both.

[0136] Upon activating the signal collection mode, one or more of such users follow on-screen prompts that prompt the user to walk around an area and have the system scan the area. This may also be referred to as an active mode of operation. Information resulting from an active mode of operation may be stored and / or processed in the cloud or on a local cluster of graphics processing units ("GPUs").

[0137] Before, during, or after the collection process, a calibration procedure may be initiated that requires collection of several types of sensor positions. By way of illustration, for calibration of a radio frequency (RF) antenna, a user may be instructed through a headset (e.g., an AR or XR headset) to precisely locate a known signal emitter, such as a WiFi router, in a room. The user may then be instructed to walk clockwise around the known signal emitter, keeping the signal emitter at approximately arm's length, and then walk counterclockwise around the same emitter.

[0138] In this procedure, the visualization system or spatialization subsystem collects hundreds or thousands of sensor data samples at several locations in space while visually tracking the emitter. The system then performs mathematical analysis on the measurements and the exact locations of these measurements to model and build a 3D representation of the sensor's signal measurements. By identifying the emitter's location, the visualization system or spatialization subsystem can lock onto the emitter's exact location and then measure the variation in the sensor's signal measurement readings during the user's circular movement. Assuming a constant emitter output, and that the emitter's signal output should remain relatively constant along the circular path around the emitter, any variation in signal collection can be attributed with a high degree of certainty to the sensor's performance and dynamics.

[0139] A cellular antenna, for example, does not radiate radiation in a perfect sphere, but rather in an oval shape with many overlaps that may not be very symmetrical. In this case, the present teachings recognize that carrying a protective case around the signal collection device, or holding the protective case in some manner, may also change the performance of the antenna.

[0140] The present arrangement recognizes that more than one antenna may be used during collection of sensor signal measurements. The collection process of the present teachings supports a large number of simultaneous users, taking into account that in some cases users have networked connections that allow many users' applications (software that embodies the present teachings and is installed on a client device) to communicate. When operating in a multi-user collection mode, the room is subdivided into zones and each user is responsible for scanning their respective area contained within a particular zone. In cases where two areas overlap, measurements from these overlapping areas are closer to the sensor readings and can be used to improve calibration and accuracy.

[0141] There are two modes of collection of sensor signal measurements: active mode of collection and passive mode of collection. In the active mode of collection, the user is shown a scanned outline of the space and is visually instructed (e.g., by presenting arrows in the user's AR or XR headset) to walk to various points to take sensor signal measurements. Feedback representing sensor signal measurements is shown in the headset when collection from various areas in the space is complete. In the passive mode of collection, on the other hand, an application (software embodying the present teachings and installed on a client device) can run as a background process and "listen" for sensor signals while the user is performing another task or perhaps playing a game. Most such applications installed on spatial computing devices such as AR or XR headsets utilize these sensor signal scanning capabilities continuously. In the passive mode of collection, sensor signal scanning for collection purposes is done in the background for later processing while other tasks are performed in the foreground. Regardless of whether an active or passive mode of collection is performed, the collection step or process collects not only the signal measurement data but also the coordinates of the location where the signal measurement was taken. This spatial data may be stored in memory (e.g., on the local device with a database or log) from which it is later retrieved for processing, if necessary in an encrypted format for security. Spatial data may alternatively be stored on a remote server or use cloud-based synchronization. As yet another example, the present teachings may rely on a local cache of cloud data. The present teachings recognize that existing room measurements may be retrieved from cloud storage, originating from other users or from public or private building plans.

[0142] The signal measurement data and its corresponding position coordinates may be used to perform edge and cloud GPU signal analysis, interpolation between one or more sensor signal measurements, and inference of sensor signal measurement data using algorithms and calculations per signal type to predict sensor signal values ​​outside of the space. In some cases, the arrangement involves employing on-device hardware dedicated to single pass filtering and live feedback, synchronization to the cloud for supercomputer-class multi-pass filtering, and much more advanced algorithm processing. As mentioned above, cloud storage can also aggregate multiple users' sensor signal measurements to fill in the gaps. To process the sensor signal data, the present teachings may rely on either CPU-based ray tracing or traditional approaches of ray tracing. In one embodiment, the present teachings use multiple cores by distributing equal portions of the scene or space being perceived by the user's AR or VR headset. In an alternative embodiment, the present teachings use the relatively new technique of GPU-based ray tracing to perform ray tracing like CPU ray tracing in traditional approaches. As an illustration, the present teachings recognize that when live GPU ray tracing is performed, a specialized GPU (adapted from gaming technology) may be implemented. Nvidia's RTX hardware, which includes built-in special ray tracing instructions, works well for this. An application (software that embodies the present teachings and is installed on a client device) can convert sensor signal measurements into a format that the GPU can recognize and understand, effectively fooling the GPU into thinking it is rendering a video game scene.

[0143] The present teachings may also rely on mobile ray tracing, which should enable highly accurate results even when operating on battery power in the field. During such processing, the present teachings may perform steps such as storing the processed data, rendering the VR, AR, or XR signals, and / or providing feedback during the sensor signal measurement collection process.

[0144] In instances where the present teachings rely on one or more users walking around a room, the general shape of the measured sensor signals may be recorded and mapped into a spatial model by the AR / VR / XR operating system underlying the arrangement. This three-dimensional or spatialized model may be made available to the system to build an approximate virtual representation of the room's layout and contents. This is electronically represented to the user as a "mesh" that is overlaid on top of the detected shapes. The present teachings recognize that as one or more people walk through many of the various locations in a space to collect attribute values, the greater number of values ​​at the various locations results in higher resolution three-dimensional spatialized attribute value data that is continually updated as more people walk to cover more area (in other words, the parties contribute to the higher resolution of the three-dimensional spatialized attribute value data).

[0145] In one aspect of the present teachings, unscanned unmeshed areas are represented by a dim semi-transparent grid of dots / intersections that may project into the still unmeshed elements of the room. In another aspect of the present teachings, unscanned meshed areas are represented by a dim semi-transparent grid of dashed or continuous lines that roughly approximate the shape of the room and its contents. In yet another aspect of the present teachings, scanned meshed areas are shown with thick colored vertical or horizontal lines whose brightness corresponds to the confidence of the signal measurements. In some of these visual configurations, the arrangement can display areas that have never been scanned, or display areas that have been scanned but are out of date or have low confidence, or display areas that have never been scanned but do not need to be scanned as they may be interpolated or inferred.

[0146] In some embodiments of the present teachings, the raw signal may be painted on a background or in the form of "fog" and the processed signal may be represented on a background or in the form of "rain". When rendering an augmented 3D spatialized environment, the present teachings can show metadata such as protocol version, device address, and frequency. Other examples presented during rendering include display techniques for extreme scale including instancing, point clouds, and frustum culling. In the context of the rendered augmented three-dimensional spatialized environment, the arrangement may enable at least one of the following: interaction with the signal representation by the user, switching between signal layers, enabling / disabling multiple signal overlays using a graphical user interface, proximity based control, activating whichever signal layer is strongest or has the most accurate data, hand interaction for precise measurements in space, lending a hand to get an estimated measurement at this exact point, touching the interactive signal in the form of a "raindrop" to get more information, time control for historical analysis, performing an overlay of transmit and receive activity, displaying emitters, animating 3D icons placed at the location of detected transmitters, displaying sensors and signal collectors, displaying the signal passing between devices, animating various styles of lines including dotted and dashed lines drawn between transmitters and receivers, and / or displaying the direction and overall path of this signal through the air. In cases where the system detects physical barriers and signal reflections, the lines shown will be based on a model to create the most accurate representation. Illustratively, the line segments use a combination of animation color, size, and frequency to indicate a magnitude of signal activity, such as bandwidth or latency of network communications.

[0147] In embodiments where the present teachings rely on multi-user operation, fusion of sensor signal measurements obtained from multiple signal collectors for multiple users is performed. When merging or fusing sensor signal measurements obtained from multiple signal collectors, the present teachings preferably use sensor proximity as a factor in the merging to fuse sensor measurement readings into the collection process. To this end, the position calibration process described above facilitates accurate fusion of sensor measurement data. By way of illustration, fusion or merging according to the present teachings relies on a process similar to that performed by a differential GPS system, which uses known points to perform robust corrections. The present teachings may also use AI / ML for transceiver placement and room-scale optimization when performing fusion or merging. In other embodiments, the present teachings may perform macro-scale aggregation such that carrier-level optimizations achieve fusion or merging as described herein.

[0148] Although the present invention has been shown and described herein as embodied in one or more embodiments, it is not intended to be limited to the details shown, since various modifications and structural changes can be made therein without departing from the spirit of the invention and within the scope and range of equivalents of the claims. Accordingly, it is appropriate that the appended claims be construed broadly and in a manner consistent with the scope of the invention, as set forth in the following claims.

Claims

1. 1. A method for rendering two different types of data sets, comprising: acquiring a first type of data set, each first data value in said first type of data set being associated with one or more three-dimensional coordinates, said one or more three-dimensional coordinates defining a location or region in real space; acquiring a second type of dataset, wherein each second data value in the second type of dataset is associated with one or more of the three-dimensional coordinates, a type of data in the second type of dataset differs from that of the first type of dataset, and the at least one of the first type of dataset or the second type of dataset represents an intangible property of the real space; spatializing said first type dataset using a plurality of said three-dimensional coordinates to produce a first type spatialized dataset; spatializing said second type dataset using said plurality of said three-dimensional coordinates to produce a second type spatialized dataset; aligning said first type of spatialized data set with said second type of spatialized data set to create an augmented three-dimensional spatialized environment; Rendering the augmented three-dimensional spatialized environment on a display component using a rendering engine; A method comprising:

2. 2. The method for rendering two different types of datasets as described in claim 1, wherein the step of acquiring the first type of dataset is performed using a geographic positioning component and a first sensor and / or a first database, the geographic positioning component and the first sensor being disposed on eyewear, and the first database being within or communicatively coupled to the eyewear.

3. 3. The method for rendering two different types of datasets as described in claim 2, wherein the step of acquiring a second type of dataset is performed using the first sensor or a second sensor and / or the first database or a second database, the second sensor being located in at least one of eyewear or an external device outside the eyewear, and the second database being within the eyewear or communicatively coupled to the eyewear.

4. The step of spatializing the first type of data set is performed using an image spatialization module resident on an eyewear or on an external processor external to the eyewear, the external processor being communicatively coupled to the eyewear, and the method further comprising: spatially partitioning the real space into a plurality of subdivisions using the image spatialization module, the space being defined using the plurality of three-dimensional position coordinates; aggregating the subdivisions to produce a spatialized model of the real space, the first data values ​​being distributed based on the spatialized model to produce the first type of spatialized data set; The method for rendering two different types of data sets according to claim 1 , further comprising:

5. The step of spatializing the second type of data set is performed using an attribute spatialization module resident on an external processor located on or outside the eyewear, the external processor being communicatively coupled to the eyewear, and the method further comprising: spatially partitioning the real space into a plurality of discrete subdivisions using the attribute spatialization module, the space being defined using the plurality of three-dimensional position coordinates; aggregating the individual subdivisions to produce a spatialized model of the real space, the second data values ​​being distributed based on the spatialized model to produce the second type of spatialized data set; The method for rendering two different types of data sets according to claim 1 , further comprising:

6. 2. The method for rendering two different types of data sets as described in claim 1, wherein the step of aligning the first type of spatialization data with the second type of spatialization data is performed using an alignment module resident on an eyewear or on an external processor located outside the eyewear, the external processor being communicatively coupled to the eyewear.

7. 2. The method for rendering two different types of datasets as described in claim 1, wherein the aligning step includes using one or more common spatial features present in the first type of spatialized dataset and the second type of spatialized dataset to create the augmented three-dimensional spatialized environment.

8. 2. The method for rendering two different types of datasets as described in claim 1, wherein in the rendering step, the rendering engine resides on an external processor located on the eyewear or outside the eyewear, the external processor being communicatively coupled to the eyewear.

9. 2. The method for rendering two different types of datasets as described in claim 1, wherein the display component is integrated into a client device, the client device being at least one device selected from the group including eyewear, a smartphone, and a computer.

10. 1. A method for rendering an image dataset, comprising: acquiring boundary data of a real space; subdividing the boundary data of the real space into a plurality of subdivisions; obtaining three-dimensional pixelated data values ​​or three-dimensional voxelized data values ​​for corresponding position coordinates, each of the corresponding position coordinates being associated with at least one of an associated three-dimensional pixelated data value or an associated three-dimensional voxelized data value; identifying one or more of the subdivisions each containing one or more corresponding position coordinates to define one or more selected subdivisions; assigning at least one of the associated three-dimensional pixelated data values ​​or the associated three-dimensional voxelized data values ​​associated with one of the corresponding position coordinates to one or more of the selected subdivisions to define one or more of the assigned subdivisions such that each assigned subdivision is assigned at least one of the associated three-dimensional pixelated data values ​​or the associated three-dimensional voxelized data values; aggregating the assigned subdivisions to form a spatialized image data set; A method comprising:

11. The method for rendering the image dataset of claim 10 , further comprising the step of rendering the spatialized image dataset on a display component of a client device using a rendering engine.

12. 11. The method for rendering the image dataset of claim 10, wherein the step of obtaining boundary data comprises obtaining three-dimensional coordinates defining a boundary of a scene, and the step of obtaining the three-dimensional pixelated data values ​​or the three-dimensional voxelized data values ​​comprises measuring light intensity values ​​or color intensity at pixels or voxels of a light sensor, respectively.

13. The method for rendering said image dataset according to claim 10 , wherein in said step of subdividing, each of said subdivisions is defined by three-dimensional subdivision boundary coordinates.

14. 14. The method for rendering the image dataset of claim 13, wherein the step of identifying one or more of the subdivisions comprises determining whether one of the corresponding position coordinates is within one of the subdivisions defined by the three-dimensional subdivision boundary coordinates.

15. 11. The method for rendering the image dataset of claim 10, wherein the combining step comprises combining a plurality of the three-dimensional subdivision boundary coordinates to form the spatialized image dataset, the three-dimensional subdivision boundary coordinates defining a plurality of the subdivisions.

16. 11. The method for rendering the image dataset according to claim 10, wherein in the assigning step at least one or a sum of two or more weighted values ​​of the associated three-dimensional pixelated data values ​​or at least one or a sum of two or more weighted values ​​of the associated three-dimensional voxelized data values ​​are assigned to the selected subdivision.

17. In said step of subdivision, spatially partitioning the real space into the plurality of subdivisions, the real space being defined using a plurality of corresponding three-dimensional position coordinates; aggregating the subdivisions to produce a spatialized model for the real space, the associated three-dimensional pixelated data values ​​or the associated three-dimensional voxelized data values ​​being distributed based on the spatialized model to produce the spatialized image data set; The method for rendering the image dataset of claim 10, comprising:

18. 1. A method for rendering an attribute value dataset, comprising: acquiring boundary data of a real space; subdividing the boundary data of the real space into a plurality of subdivisions; obtaining one or more different types of attribute values ​​of corresponding position coordinates, each of the corresponding coordinates being associated with at least one type of associated attribute value, the attribute value representing a measure of an intangible property of the real space; identifying one or more of the subdivisions each containing one or more of the corresponding position coordinates to define one or more selected subdivisions; assigning at least one type of associated attribute value associated with one of said corresponding position coordinates to one or more of said selected subdivisions to define one or more of said assigned subdivisions, such that each of said assigned subdivisions is assigned at least one type of said associated attribute value; aggregating the assigned subdivisions based on the associated attribute values ​​of a particular type to form a spatialized attribute value data set for the particular type; A method comprising:

19. 20. The method for rendering the attribute value dataset of claim 18, further comprising the step of rendering the spatialized attribute value dataset on a display component using a rendering engine of a client device.

20. 20. The method for rendering the attribute value dataset of claim 18, wherein the step of obtaining one or more different types of attribute values ​​comprises detecting in the real space a magnitude of at least one type of attribute selected from the group consisting of: throughput of a connection signal, latency of the connection signal, interference of the connection signal, variability of the connection signal, stability of the connection signal, RF power output, EMF, air pressure, geomagnetic field, Hall effect, ambient light level, gas concentration, smoke, sound pressure, audio harmonics, humidity, carbon dioxide emission, and temperature.

21. 20. The method for rendering the attribute value dataset of claim 18, wherein in the step of assigning, at least one of the associated attribute values ​​is assigned to all portions of the selected subdivision.

22. 20. The method for rendering the attribute value dataset of claim 18, wherein in the assigning step, a sum of weight values ​​of two or more associated attribute values ​​of a particular type is assigned to all portions of the selected subdivision.

23. said step of subdividing said boundary data of said real space into said plurality of subdivisions further comprising: spatially partitioning the real space into the plurality of subdivisions, the real space being defined using a plurality of corresponding three-dimensional position coordinates; aggregating the plurality of subdivisions to produce a spatialization model for the space, the associated attribute values ​​of a particular type being distributed based on the spatialization model to produce the spatialized attribute value data set of the particular type; 20. The method for rendering the attribute value dataset of claim 18, comprising:

24. 1. A method for rendering an attribute value dataset and an image dataset, comprising: obtaining, using an optical sensor and a ground positioning component, three-dimensional pixelated or voxelized data values ​​for one or more corresponding three-dimensional position coordinates, each of the corresponding three-dimensional position coordinates being associated with at least one associated three-dimensional pixelated or voxelized data value; acquiring, using at least one non-optical sensor, one or more different types of attribute values ​​for one or more of the corresponding three-dimensional position coordinates, each of the corresponding three-dimensional position coordinates being associated with at least one type of associated attribute value; spatializing the associated three-dimensional pixelated data values ​​or the associated three-dimensional voxelized data values ​​using a plurality of the corresponding three-dimensional position coordinates to produce a spatialized image data set; spatializing the associated attribute values ​​of at least said type using said plurality of said corresponding three-dimensional position coordinates to produce a spatialized attribute value data set of at least said type; aligning the image spatialization data set with the spatialization attribute value data set to create an augmented three-dimensional spatialized environment; Rendering the augmented three-dimensional spatialized environment on a display component using a rendering engine; A method comprising:

25. the step of spatializing to produce the spatialized image dataset and / or the step of spatializing to produce the spatialized attribute value dataset further comprising: spatially partitioning a space into a plurality of subdivisions, the space being defined using the corresponding three-dimensional position coordinates; combining the subdivisions to produce a spatialized model for the space; 25. A method for rendering the attribute value dataset and the image dataset of claim 24, comprising:

26. 26. The method for rendering the attribute value dataset and the image dataset of claim 25, wherein the step of spatializing to produce the spatialized image dataset and / or the step of spatializing to produce the spatialized attribute value dataset comprises the steps of using a microphone to collect audio data present in a real space and operating the microphone in conjunction with a filter to increase the spatial resolution of the spatialized model for the space.

27. 25. The method for rendering the attribute value dataset and the image dataset of claim 24, further comprising the step of storing the associated attribute values ​​of one or more different types and the corresponding three-dimensional coordinates in a database.

28. 25. The method for rendering the attribute value dataset and the image dataset of claim 24, wherein using at least one non-light sensor comprises using a plurality of non-light sensors installed on various client devices including smartphones or wearables.

29. 29. The method for rendering the attribute value dataset and the image dataset of claim 28, wherein the step of obtaining one or more different types of attribute values ​​comprises obtaining the attribute values ​​of at least one type at the plurality of corresponding three-dimensional position coordinates such that each of the corresponding three-dimensional position coordinates is associated with at least one attribute value of the type.

30. 25. The method for rendering the attribute value dataset and the image dataset of claim 24, wherein the aligning step is performed using a processor based on one or more common intersection points that exist in the spatialized image dataset and in the spatialized attribute value dataset and that are in a real space defined by the plurality of corresponding three-dimensional position coordinates.

31. said rendering step further comprising: representing the associated attribute values ​​of a particular type obtained at a location in real space with a virtual object; comparing the associated attribute value of the particular type with a threshold attribute value of the particular type; assigning a first color to the virtual object if the associated attribute value of the particular type is greater than the threshold attribute value of the particular type; assigning a second color to the virtual object if the associated attribute value of the particular type is less than the threshold attribute value of the particular type, the second color being different from the first color; rendering the virtual object at an object rendering location within the augmented three-dimensional spatialized environment that corresponds to the location of the corresponding three-dimensional position coordinate associated with the associated attribute value of the particular type; 25. A method for rendering the attribute value dataset and the image dataset of claim 24, comprising:

32. 32. The method for rendering the attribute value dataset and the image dataset of claim 31 , wherein the step of spatializing to produce the spatialized attribute value dataset includes a step of interpolating using two or more of the associated attribute values ​​of the particular type and two or more of the corresponding three-dimensional position coordinates to calculate intermediate associated attribute values ​​of the particular type, wherein corresponding intermediate three-dimensional position coordinates located between two or more of the corresponding three-dimensional position coordinates are associated with the intermediate associated attribute values ​​of the particular type.

33. 33. The method for rendering the attribute value dataset and the image dataset of claim 32, further comprising the step of rendering an intermediate virtual object at an intermediate object position within the augmented three-dimensional spatialized environment that corresponds to the location of the corresponding intermediate three-dimensional position coordinate associated with the intermediate associated attribute value of the particular type.

34. 32. The method for rendering the attribute value dataset and the image dataset of claim 31 , wherein the step of spatializing to produce the spatialized attribute value dataset includes a step of inferring using two or more of the associated attribute values ​​of a particular type and two or more of the corresponding three-dimensional position coordinates to calculate a predicted associated attribute value of the particular type, wherein corresponding external three-dimensional position coordinates located outside the real space and positioned on a linear trajectory intersecting two or more of the corresponding three-dimensional position coordinates are associated with the predicted associated attribute value of the particular type, and the real space is defined using the plurality of the corresponding three-dimensional position coordinates.

35. 35. The method for rendering the attribute value dataset and the image dataset of claim 34, further comprising, in the rendering step, rendering an external virtual object at an external object position within the augmented three-dimensional spatialized environment that corresponds to a position of the corresponding external three-dimensional position coordinate associated with the predicted associated attribute value of the particular type.

36. 25. The method for rendering said attribute value dataset and said image dataset of claim 24, wherein said step of spatializing to produce a spatialized attribute value dataset of at least said type comprises the step of producing an electronic representation of a wire mesh that extends a real space defined using said plurality of said corresponding three-dimensional position coordinates, said method further comprising the step of using said electronic representation of said wire mesh to calculate unknown attribute values ​​of at least said type at defined positions in said real space.

37. 25. The method for rendering the attribute value dataset and the image dataset of claim 24, wherein the rendering step includes rendering a virtual object at each of the object rendering locations in the augmented three-dimensional spatialized environment that correspond to the corresponding three-dimensional position coordinate locations associated with the associated attribute values ​​of the particular type, the method further including detecting user interaction with one or more of the virtual objects, the detecting step being performed after the rendering step.

38. 38. The method for rendering the attribute value dataset and the image dataset of claim 37, wherein if the step of detecting an interaction of the user with one or more of the virtual objects detects an interaction of the user with one or more of the virtual objects, the method further comprises the step of rendering a visual feedback element conveying the interaction of the user with one or more of the virtual objects and / or the step of rendering a magnitude and / or a direction of the associated attribute value of the particular type associated with one or more of the virtual objects.

39. 25. The method for rendering the attribute value dataset and the image dataset of claim 24, further comprising the step of receiving an instruction to move from a first mode of operation displaying attribute values ​​of a first type to a second mode of operation displaying attribute values ​​of a second type, the attribute values ​​of the first type being different from the attribute values ​​of the second type.

40. rendering within the augmented three-dimensional spatialized environment maximum and / or minimum scale positions and / or maximum and / or minimum scale directions of any of the attribute values ​​of the particular type, at least one intermediate attribute value of the particular type, and at least one predicted attribute value of the particular type; the intermediate attribute value is calculated from an interpolation step using two or more of the associated attribute values ​​of the particular type and two or more of the corresponding three-dimensional position coordinates; the external attribute values ​​are calculated from an inference step using two or more of the associated attribute values ​​of the particular type and two or more of the corresponding three-dimensional position coordinates.

25. A method for rendering the attribute value dataset and the image dataset of claim 24.

41. 25. The method for rendering the attribute value dataset and the image dataset of claim 24, wherein a plurality of types of attribute values ​​convey a strength of a wireless connection signal between two devices, each type of attribute value conveying the strength of a different type of the wireless connection signal rendered with a virtual object of a different color.

42. 42. The method for rendering the attribute value dataset and the image dataset of claim 41, wherein a first type of wireless connection that uses an industry standard family based wireless networking technology is color coded a first color, a second type of wireless connection that is performed over short distances using short wavelength UHF radio waves is color coded a second color, and a third type of wireless connection that uses multiple small interconnected transmitters that rely on high frequency radio waves for connectivity is color coded a third color, each of the first color, the second color, and the third color being different from each other.

43. said step of spatializing to produce said spatialized attribute value data set of at least said type, analyzing at least the attribute values ​​of the type relative to one or more of the corresponding three-dimensional position coordinates to develop analytical data conveying directional and / or location information regarding at least the attribute values ​​of the type, wherein one or more of the non-optical sensors include a temperature sensor and / or a humidity sensor, and the associated attribute values ​​are attribute values ​​of a type conveying information regarding air flow rate and / or energy loss; 25. A method for rendering the attribute value dataset and the image dataset of claim 24, comprising:

44. 44. A method for rendering the attribute value dataset and the image dataset as described in claim 43, wherein the associated attribute value of air flow provides information regarding an air flow direction and / or a location experiencing an air flow problem, and the associated attribute value of energy loss provides information regarding a location experiencing an energy loss.

45. 1. An extended reality system for rendering a user interface, comprising: an optical sensor for acquiring three-dimensional pixelated or voxelized data values ​​at corresponding three-dimensional position coordinates in real space; a non-optical sensor for acquiring one or more different types of attribute values ​​at the corresponding three-dimensional position coordinates, the attribute values ​​representing a magnitude of an intangible property of the real space, the intangible property being at least one property selected from the group including throughput of a connection signal, latency of the connection signal, interference of the connection signal, variability of the connection signal, stability of the connection signal, RF power output, EMF, air pressure, geomagnetic field, Hall effect, ambient light level, gas concentration, smoke, sound pressure, audio harmonics, humidity, carbon dioxide emission, and temperature; a surface location component for obtaining the corresponding three-dimensional position coordinates in the real space and associated with the three-dimensional pixelated data values ​​or the three-dimensional voxelized data values; a display component for displaying the rendered information; a processor communicatively coupled to the ground positioning component, the optical sensor, the non-optical sensor, and the display component; spatializing the three-dimensional pixelated or voxelized data values ​​using the plurality of the corresponding three-dimensional position coordinates to produce a spatialized image data set; spatializing at least one type of said associated attribute values ​​using said plurality of said corresponding three-dimensional position coordinates to produce a spatialized attribute value data set; aligning the image spatialization data set with the spatialization attribute value data set to create an augmented three-dimensional spatialized environment; and Rendering the augmented three-dimensional spatialized environment on the display component using a rendering engine. a processor operable to execute instructions to A system comprising:

46. 46. ​​The extended reality system for rendering the user interface of claim 45, wherein the processor and the sensor are bidirectionally communicatively coupled to enable the processor to instruct the sensor to collect additional data and discard originally collected data.

47. 46. ​​The extended reality system for rendering the user interface of claim 45, wherein the processor and the non-optical sensor are located on different devices.

48. 46. ​​The extended reality system for rendering the user interface of claim 45, wherein the processor is located on eyewear and the non-optical sensor is located on at least one device selected from the group including a smartphone, a wearable, or an external controller.

49. 46. ​​An extended reality system for rendering the user interface of claim 45, further comprising an image spatialization module disposed in the processor or the light sensor device and configured to spatialize the three-dimensional pixelated data values ​​or the three-dimensional voxelized data values ​​to produce the spatialized image dataset.

50. 46. ​​The extended reality system for rendering the user interface of claim 45, further comprising an attribute value spatialization module disposed within the processor and configured to spatialize at least one type of the associated attribute values ​​to produce the spatialized attribute value data set.

51. 46. ​​The extended reality system for rendering the user interface of claim 45, further comprising an alignment module disposed in the processor and configured to align the image spatialized dataset with the spatialized attribute value dataset.

52. 46. ​​An extended reality system for rendering the user interface of claim 45, wherein the rendering engine is located within the processor.

53. 46. ​​An extended reality system for rendering the user interface of claim 45, wherein the processor is located in one of a client device and / or a computer network, the client device being a smartphone and / or eyewear.

54. 1. An extended reality system for rendering a user interface, comprising: an optical sensor for acquiring three-dimensional pixelated or voxelized data values ​​at the corresponding position coordinates; a ground positioning component for acquiring boundary data of a real space and for acquiring the corresponding position coordinates within said real space and associated with said three-dimensional pixelated data values ​​or said three-dimensional voxelized data values; a display component for displaying the rendered information; a processor communicatively coupled to the ground positioning component, the light sensor, and the display component; subdividing the boundary data of the real space into a plurality of subdivisions; identifying one or more of the subdivisions each containing one or more of the corresponding position coordinates to define one or more selected subdivisions; assigning at least one of the three-dimensional pixelated data values ​​or the three-dimensional voxelized data values ​​associated with one of the corresponding position coordinates to one or more of the selected subdivisions to define one or more assigned subdivisions, such that each of the assigned subdivisions is assigned at least one of the three-dimensional pixelated data values ​​or the three-dimensional voxelized data values; and aggregating the assigned subdivisions based on the three-dimensional pixelated data values ​​or the three-dimensional voxelized data values ​​to form a spatialized image data set. a processor operable to execute instructions to A system comprising:

55. 1. An extended reality system for rendering a user interface, comprising: a non-optical sensor for acquiring one or more different types of attribute values ​​at the corresponding three-dimensional position coordinates, the attribute values ​​representing a magnitude of an intangible property of a real space, the intangible property being at least one property selected from the group including throughput of a connection signal, latency of the connection signal, interference of the connection signal, variability of the connection signal, stability of the connection signal, RF power output, EMF, air pressure, geomagnetic field, Hall effect, ambient light level, gas concentration, smoke, sound pressure, audio harmonics, humidity, carbon dioxide emission, and temperature; a ground positioning component for obtaining boundary data of a real space and for obtaining the corresponding location coordinates within the real space and associated with one or more different types of the attribute values; a display component for displaying the rendered information; a processor communicatively coupled to the ground positioning component, the non-optical sensor, and the display component; subdividing the boundary data of the real space into a plurality of subdivisions; identifying one or more of the subdivisions each containing one or more of the corresponding position coordinates to define one or more selected subdivisions; assigning at least one type of associated attribute value associated with one of the corresponding position coordinates to one or more of the selected subdivisions to define one or more of the assigned subdivisions, such that each assigned subdivision is assigned at least one type of associated attribute value; and aggregating the assigned subdivisions based on the associated attribute values ​​of at least one type to form a spatialized attribute value data set; a processor operable to execute instructions to A system comprising:

56. 1. An extended reality system for rendering a user interface, comprising: an optical sensor for acquiring three-dimensional pixelated or voxelized data values ​​at corresponding three-dimensional position coordinates; a surface location component for obtaining corresponding three-dimensional position coordinates in space and associated with said three-dimensional pixelated data values ​​or said three-dimensional voxelized data values; a non-optical sensor for acquiring one or more different types of attribute values ​​at the corresponding three-dimensional position coordinates, the attribute values ​​representing a magnitude of an intangible property of the real space, the intangible property being at least one property selected from the group including throughput of a connection signal, latency of the connection signal, interference of the connection signal, variability of the connection signal, stability of the connection signal, RF power output, EMF, air pressure, geomagnetic field, Hall effect, ambient light level, gas concentration, smoke, sound pressure, audio harmonics, humidity, carbon dioxide emission, and temperature; a display component for displaying the rendered information; a processor communicatively coupled to the ground positioning component, the optical sensor, the non-optical sensor, and the display component; spatializing the three-dimensional pixelated or voxelized data values ​​using the plurality of the corresponding three-dimensional position coordinates to produce a spatialized image data set; spatializing at least one type of said associated attribute values ​​using said plurality of said corresponding three-dimensional position coordinates to produce a spatialized attribute value data set; aligning the image spatialization data set with the spatialization attribute value data set to create an augmented three-dimensional spatialized environment; and Rendering the augmented three-dimensional spatialized environment on the display component using a rendering engine. a processor operable to execute instructions to A system comprising:

Images