Secure display of content for confidential viewing

Abstract

Systems and methods for securely displaying a document or other content for viewing by a user. The system may use an AR / VR headset or other display device. The method carried out by the system includes: obtaining a document to be viewed by a user of the display device of a display device system; capturing sensor data using at least one sensor of the display device system; determining sensor-based user identification data based on the sensor data; authenticating the user using the sensor-based user identification data to determine identity information of the user; determining whether to display the document and / or determining display attributes for display of the document; and displaying the document for viewing by the user and in accordance with the display attributes, if any.

Description

SECURE DISPLAY OF CONTENT FOR CONFIDENTIAL VIEWINGTECHNICAL FIELD

[0001] This invention relates to methods, systems, and equipment for authenticating a user and securely displaying content to the user in a manner that avoids copying or access to the content by others.BACKGROUND

[0002] The evolution of technology has greatly enhanced the efficiency and accessibility of information processing across various sectors, but it has also introduced significant risks, particularly in the area of information security. A major concern is the theft of confidential information through Optical Character Recognition (OCR) and related digital imaging technologies. Originally designed to aid in the digitization and management of documents, these technologies can also be misused to covertly access and extract sensitive information without leaving any traceable evidence. The risk of information theft using OCR is alarmingly high, as unauthorized text extraction from images and documents has become a common covert tactic that does not require the consent of the information owner. This stealthy nature of the technology poses a severe threat to both individual and corporate privacy.

[0003] With the rise of remote work, the security of internal documents is increasingly compromised. Employees handling sensitive data on personal or less-secured devices may not be adequately protected against unauthorized surveillance technologies, such as security cameras capable of capturing computer screens, especially when working in public places where the risk of information theft is significantly higher. The consequences of such information theft are profound and far-reaching, raising serious concerns among data security and digital ethics experts about the legal and ethical challenges that could arise. These breaches can lead to severe consequences, including the compromise of personal privacy, corporate espionage, and even threats to national security, underscoring the critical need for improved measures in information handling and security protocols. This necessity drives the development of advanced solutions to counteract the sophisticated methods used in the unauthorized extraction of confidential information, as highlighted by the proposed patent, which aims to introduce an innovative approach to safeguard sensitive data against the vulnerabilities posed by OCR and similar technologies.SUMMARY

[0004] In accordance with an aspect of the invention, there is provided a method of securely displaying a document or other content for viewing by a user. The method includes:

[0005] obtaining a document to be viewed by a user of a display device of a display device system;

[0006] capturing sensor data using at least one sensor of the display device system;

[0007] determining sensor-based user identification data based on the sensor data;

[0008] authenticating the user using the sensor-based user identification data to determine identity information of the user;

[0009] determining whether to display the document and / or determining display attributes for display of the document; and

[0010] displaying the document for viewing by the user and in accordance with the display attributes, if any.

[0011] Various embodiments of this method may include any of the following additional steps or features, either alone or in any feasible combination.

[0012] the at least one sensor is or includes a biometric sensor, and the display device is an AR / VR device that includes the biometric sensor.

[0013] the biometric sensor is a microphone, a fingerprint sensor, a photoplethysmography (PPG) sensor, a near infrared (NIR) sensor, other infrared (IR) sensor, or a camera.

[0014] the document is displayed as a graphic in an augmented reality (AR) or virtual reality (VR) fashion and in a manner in which the document as displayed is only viewable by the user.

[0015] the AR / VR device is used to sign the document in order to produce an electronic signature indicating an identity of the user.

[0016] the AR / VR device includes a computer application configured to obtain and display the document in a confidential manner so that the user is able to sign the document to produce the electronic signature without the document being viewable by others during signing of the document.

[0017] the computer application is configured to prevent copying, downloading, screenshotting, and screen recording while the document is being viewed by the user.

[0018] the method further comprises: tracking a line of sight of the user through use of eye tracking data captured by an eye tracking sensor to determine a line of sight area of the document within the line of sight of the user; wherein displaying the document includes displaying the document so that a non-line of sight area is obfuscated and content of the document in the line of sight area remains interpretable.

[0019] the eye tracking sensor is a near infrared (NIR) sensor.

[0020] the display device is a smartphone, television, laptop screen, desktop or computer monitor, tablet, infotainment unit display of a vehicle, or other open-view display device.

[0021] the content of the document within the line of sight area includes text that is readable and the non-line of sight area includes other text that is blurred or otherwise obfuscated so as to be unreadable.

[0022] the authenticating the user includes generating biometric profile data of the user as the sensor-based user identification data and determining the identity information of the user based on the biometric profile data.

[0023] the at least one sensor used for obtaining the sensor data is or includes an accelerometer, gyroscope, magnetometer, camera, proximity sensor, or an optical sensor.

[0024] the at least one sensor includes a first sensor and a second sensor, wherein the sensor data includes first sensor data obtained from the first sensor and second sensor data obtained from the second sensor, and wherein a context-aware biometric profile of the user is generated based on the first sensor data and the second sensor data.

[0025] a machine learning (ML) technique is used to generate the identity information of the user based on the first sensor data and the second sensor data.

[0026] In accordance with a second aspect of the invention, there is provided a method of securely displaying a document or other content for viewing by a user. This method includes:

[0027] obtaining a document to be viewed by a user of a display device of a display device system;

[0028] capturing sensor data from an eye tracking sensor of the display device system;

[0029] determining a line of sight direction of the user;

[0030] determining a line of sight area of the document based on the line of sight direction; and

[0031] displaying the document on the display device for viewing of the line of sight area by the user whereby adjacent portions of the document outside of the line of sight area are hidden or prevented from being displayed on the display device.

[0032] Various embodiments of this second aspect of the invention may include any of the following additional steps or features, either alone or in any feasible combination.

[0033] the display device is a smartphone, television, laptop screen, desktop or computer monitor, tablet, infotainment unit display of a vehicle, or other open-view display device.

[0034] the eye tracking sensor is a near infrared (NIR) sensor.

[0035] the content of the document within the line of sight area includes text that is readable and the non-line of sight area includes other text that is blurred or otherwise obfuscated so as to be unreadable.

[0036] displaying the document includes displaying the document so that content of the document in a non-line of sight area is obfuscated and content of the document in the line of sight area remains interpretable.

[0037] the content of the document within the line of sight area includes text that is readable and the non-line of sight area includes other text that is blurred or otherwise obfuscated so as to be unreadable.

[0038] the method further comprises: authenticating the user to determine identity information of the user; wherein the authenticating the user includes generating a context-aware biometric profile of the user based on sensor data obtained from at least one sensor and determining the identity information of the user based on the context-aware biometric profile.

[0039] the at least one sensor used for obtaining the sensor data is or includes an accelerometer, gyroscope, magnetometer, camera, proximity sensor, or an optical sensor.

[0040] the at least one sensor includes a first sensor and a second sensor, wherein the sensor data includes first sensor data obtained from the first sensor and second sensor data obtained from the second sensor, and wherein a context-aware biometric profile of the user is generated based on the first sensor data and the second sensor data.

[0041] a machine learning (ML) technique is used to generate the identity information of the user based on the first sensor data and the second sensor data.BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Preferred exemplary embodiments will hereinafter be described in conjunction with the appended drawings, wherein like designations denote like elements, and wherein:

[0043] FIG. 1 diagrammatically depicts a first embodiment of a secure document viewing display system;

[0044] FIG. 2 diagrammatically depicts a second embodiment of a secure document viewing display system;

[0045] FIG. 3 diagrammatically depicts a third embodiment of a secure document viewing display system;

[0046] FIG. 4 is a flowchart that depicts one embodiment of a method for securely displaying a document or other content for viewing by a user; and

[0047] FIG. 5 is a flowchart that depicts another embodiment of a method for securely displaying a document or other content for viewing by a user.DETAILED DESCRIPTION

[0048] The system and method described herein enables enhanced privacy and security when viewing documents or other content on an electronic display of a display device. According to at least some embodiments, the electronic display (referred to also as “display”) is a display of an augmented reality or virtual reality device (referred to as “AR / VR display device”) and the display device is the AR / VR display device; here, the display may be a projector of the AR / VR display device, for example. As discussed below, a document viewing display system having the display device is configured to perform various functionality discussed herein, including a method of securely displaying a document or other content for viewing by a user, at least in embodiments.

[0049] According to one embodiment, the method of securely displaying a document or other content for viewing by a user includes: obtaining a document to be viewed by a user of the display device, for example, the AR / VR display device; capturing sensor data from at least one sensor of the display device system; determining sensor-based user identificationdata based on the sensor data; authenticating the user using the sensor-based user identification data to determine identity information of the user; determining whether to display the document and / or determining display attributes for display of the document; and displaying the document for viewing by the user and in accordance with the display attributes, if any. Here, the user is authenticated using the captured sensor data, which may be biometric sensor data used to generate captured biometric profile data that is then compared with user predetermined user profile data to determine identity information of the user. The identity information is information indicating an identity of the user, such as a username, for example.

[0050] According to one embodiment, the method of securely displaying a document or other content for viewing by a user includes: obtaining a document to be viewed by a user of a display device of a display device system; capturing sensor data from an eye tracking sensor of the display device system; determining a line of sight direction of the user; determining a line of sight area of the document based on the line of sight direction; and displaying the document on the display device for viewing of the line of sight area by the user whereby adjacent portions of the document outside of the line of sight area are hidden or prevented from being displayed on the display device. This method is employed for displaying content on an open-view display device, which is a display device that displays content as a part of its standard operation for a user in a manner that is viewable by other individuals behind, to the side, above, below, in front of, or otherwise nearby the user. For example, televisions, computer monitors, tablets, and even smartphones and other small handheld devices generally have a planar display screen that is viewable by individuals in the surrounding area. On the other hand, AR / VR display devices, such as Google Glass™, Meta Quest™, and Microsoft HoloLens™, often permit only the user (here, the wearer) to view the content displayed, and these devices are referred to as private-view display devices.

[0051] With reference to FIGS. 1-3, there are shown various embodiments of a system 10, 110, 210, each of which may be referred to also as a document viewing display system 10, 110, 210, for securely displaying a document or other content for viewing by a user. Like components, specifically those separated by a different of 100 such as 10 and 110 or 112 and 212, refer to like component and discussion of such like components is hereby incorporated and attributed amongst said like components, as will be appreciated by those skilled in the art in light of the following discussion.

[0052] Referring now to FIG. 1, the system 10 includes a processing subsystem 12, a display 14, and at least one sensor 16, with the display 14 and the at least one sensor 16 each being communicatively coupled to the processing subsystem 12, as shown in FIG. 1. Further, FIG. 1 shows the document viewing display system 10 having a display device 20, which includes the display 14 and, in some embodiments, includes the processing subsystem 12 and / or portions thereof. Moreover, the document viewing display system 10 may include one or more other components not shown or described, such as other sensors, output devices, communication equipment, etc.

[0053] The processing subsystem 12 is for performing various data processing functionality described herein, including performing the method, such as the method 400 (FIG. 4) and / or the method 500 (FIG. 5). The processing subsystem 12 includes at least one processor 18 and memory 20 storing computer instructions that, when executed by the at least one processor 18, causes the functionality attributed to the processing subsystem 12 to be performed. In embodiments, the processing subsystem 12 is an on-device processing subsystem that is included as a part of the display device 20 that includes the display 14 and that may also include one or more of the at least one sensor 16. In other embodiments, the processing subsystem 12 is a remote processing subsystem and / or portions of the processing subsystem 12 are performed remotely; for example, a variety of cloud-based services may be used for implementing the functionality described herein and / or attributed to the processing subsystem 12.

[0054] Various data stores may be used for implementing the memory of the processing subsystem 12. A “data store” or “electronic data store” refers to any data storage platform, such as cloud storage services like Amazon S3™ or Google Drive™, relational databases like MySQL™ or PostgreSQL™, and NoSQL databases like MongoDB™ or Cassandra™, that is designed for storing data in a manner accessible by electronic processing systems. This includes any repository that allows for the organization, management, and retrieval of data through digital means, enabling efficient data handling, analysis, and processing by computers and other electronic devices. This electronic data store could take the form of a cloud database, such as Amazon’s DynamoDB™, which offers scalability and performance. Alternatively, for structured data and frequent queries, a relational database like MySQL™ or PostgreSQL™ could be employed. If the data is unstructured or of transitory structure, NoSQL databases such as MongoDB™ or Apache Cassandra™ might be more suitable. For data that is time-series in nature, time-series databases like InfluxDB™ or Timeseal eDB™ could provide efficient storage and query capabilities. For large-scale dataanalysis, data warehousing solutions such as Google BigQuery™, Amazon Redshift™, or Snowflake™ can be used. In cases where fast data access is paramount, in-memory databases like Redis™ or Memcached™ could be employed. Lastly, for storing large volumes of raw data, distributed file systems like Hadoop HDFS™ or cloud-based solutions like Amazon S3™ could be used. The choice of data store would ultimately depend on the specific requirements of the system handling the particular data.

[0055] The processor 18 and memory 20 are shown in the illustrated embodiments as being included as a part of a desktop computer, but it will be appreciated that the system and method apply to a variety of computers, such as mobile computers (e.g., smartphones, tablets, laptops), among others, and, in embodiments, may be integrated with the display device.

[0056] The processor 18 may be any suitable electronic processor, such as: x86 / x86-64 processors, such as Intel Core series (e.g., Intel™ Core i3, i5, i7, i9) and AMD Ryzen™ series (e.g., AMD Ryzen™ 3, 5, 7, 9), widely adopted in the personal computing domain; ARM processors, such as Qualcomm Snapdragon™ (e.g., Snapdragon ARI™) and Apple Ml™ processors, which are popular for mobile and embedded systems; power architecture processors, like IBM Power9, which are oftentimes tailored for server and high-performance computing scenarios; SPARC™ (Oracle™) processors (e.g., SPARC M7, T7), which are oftentimes used for high-end servers; z / Architecture (IBM™) processors (e.g., IBM zl5), oftentimes used in mainframe systems; MIPS processors (e.g., MIPS32, MIPS64), which are generally characterized by a reduced instruction set architecture, and are commonly used to address the requirements of embedded systems and networking devices; RISC-V processors (e.g., SiFive™, HiFive™), which is an open-source instruction set architecture, that is commonly used for research, embedded systems, and specialized computing environments; and other like electronic processors, such as graphics processing units (GPUs).

[0057] The memory 20 is a non-transitory, computer-readable memory that is implemented as non-volatile computer data storage devices, such as ROM (read-only memory), solid- state drives (SSDs) (including other solid-state storage such as solid-state hybrid drives (SSHDs)), other types of flash memory, hard disk drives (HDDs), magnetic or optical disc drives, non-volatile random access memory (NVRAM), etc. The memory 20 is used to store data files that are to be accessed by the processor 18, such as the computer instructions used by the at least one processor 18 for performing the method.

[0058] The processing subsystem 12 may use secure processor or transitory memory for secure storage specifically designed for handling personally-identifying information (PII). In such scenarios, this memory may include secure RAM, such as Dynamic RAM with encryption capabilities (e.g., LPDDR5 with integrated encryption) or Trusted Execution Environment (TEE) memory that isolates and secures data during processing. Additionally, the system may utilize hardware security modules (HSMs), which could include devices like Trusted Platform Modules (TPMs) or dedicated secure processors like Apple™’ s Secure Enclave or Google™’ s Titan M. These HSMs are designed to manage and store cryptographic keys securely, ensuring that sensitive information, including PII, is protected from unauthorized access or tampering. By incorporating these secure memory options, the system ensures that PII is stored and processed in a highly secure environment, meeting the requirements for data protection and compliance with security standards. Cryptographic keys and information may be stored in secure data storage for long-term storage.

[0059] The display 14 is included as a part of the display device 20, which is a device having an electronic display used for displaying documents to a user and here is the display 14. The display 14 may be any of a variety of suitable electronic projectors or other displays, such as, for example, projectors used for displaying information as a part of an AR / VR display device, a smartphone touchscreen display, a laptop computer monitor, etc. The display 14 includes appropriate drivers and processing capabilities for communication and display of information, particularly one or more computer documents, files, or other collections of information, generally referred to as documents. The display 14 is communicatively coupled to the processing subsystem 12, as shown in FIG. 1.

[0060] The at least one sensor 16 is used for determining information about the user, such as, for example, biometric information. In embodiments, the at least one sensor 16 is or includes a biometric sensor, such as, for example, a microphone, a fingerprint sensor, a photoplethysmography (PPG) sensor, a near infrared (NIR) sensor, other infrared (IR) sensor, or a camera. Further, the processing subsystem 12 is provisioned with corresponding sensor data processing capabilities in order to process the captured sensor data in order to generate information suitable for identifying and / or authenticating the user as a particular individual or being of a particular identity. In embodiments, multiple biometric sensors are employed as the at least one sensor 16; for example, a first biometric sensor captures first sensor data and a second biometric sensor captures second sensor data, and this first and second sensor data is used to determine biometric profile data. The biometric profile data represents a profile generated from captured sensor data from the at least one sensor 16 andis comparable to predetermined biometric profile data so that, through the comparison, the user may be identified.

[0061] The system 10, which may be in the form of an augmented reality (AR) device, at least in some embodiments, may include one or more auxiliary or debug ports (referred to as “debug ports”), such as a universal serial bus (USB) type C port, such as, for example, in the case of using a Meta Quest™ as the AR device. If these debug ports are not disabled, they should adhere to strict security standards, such as, for example, implementing authentication mechanisms to control access, isolating the ports from the network to prevent unauthorized access, and ensuring that secure communication protocols are followed. Additionally, debugging should be completely disabled if not required. To further enhance security, access control measures and logs should be captured to monitor activity. It is also beneficial to address physical port attacks by incorporating relevant security concepts.

[0062] Additionally, over-the-air firmware authentication or authenticated software update capabilities is put into in place to ensure that security patches and software updates follow security standards. At least in embodiments, these updates should be authenticated before being applied to the device, and flash memory should not store software in raw format to prevent tampering or unauthorized access. Furthermore, hardware tampering-proof printed circuit boards (PCBs) and housing should be considered. Desirably, the device should be tamper-proof, avoiding the use of exposed screws, and connections, wiring, or communication channels should be placed in deeper layers of the PCB instead of on the top layer, at least in embodiments. Any test points, debugging symbols, or packaging names should also be avoided to prevent reverse engineering or tampering.

[0063] With reference to FIG. 2, there is shown another embodiment of a document viewing display system 110. The system 110 includes an augmented reality / virtual reality (AR / VR) device 120 and a remote processing subsystem 104 having at least one processor 106 and memory 108. The AR / VR display device 120 includes a processing subsystem 112, a display 114, a first sensor 116a, and a second sensor 116b. The processing subsystem 112 includes a computer program or application 124 executed on the AR / VR display device 120 for performing various steps of the methods discussed herein, including each step of those methods, at least in embodiments. The computer application 124 is comprised of computer instructions that are stored on the memory of the processing subsystem 112 and executed by the at least one processor of the processing subsystem 112.

[0064] With reference to FIG. 3, there is shown yet another embodiment of a document viewing display system 210. The document viewing display system 210 includes an open-view display device 220 having a processing subsystem 212, display 214, and an eye tracking sensor 216 installed thereon. However, in other embodiments, the eye tracking sensor 216 may be separate from the open-view display device 220 and communicatively coupled thereto. The document viewing display system 210 further comprises a remote processing subsystem 222, which has its own processor(s) 106 and memory 108. The remote processing subsystem 222 is used for supporting a computer program or application 224 executed on the open-view display device 220 for performing various steps of the methods discussed herein, including each step of those methods, at least in embodiments. In one embodiment, the remote processing subsystem 222 is used for performing one or more steps of the methods discussed herein and / or otherwise is used to support the computer application 224 so as to facilitate performance of the method. The processing subsystem 212 is a local processing subsystem because it is local to the display device 220 whereas the remote processing subsystem 222 is remotely located from the display device 220. In embodiments, the processing subsystem 212 is considered to include the at least one processor of the display device 220 and at least one processor of the remote processing subsystem 222.

[0065] The eye tracking sensor 216 is a device comprising one or more sensing elements, such as near-infrared (NIR) emitters and / or cameras, that monitor the user’s eye position and movement. In one embodiment, the process begins with the NIR emitters emitting nearinfrared light towards the user’ s eyes. This light is invisible to the human eye but effectively captures detailed reflections from the eye’s surface without causing discomfort. The cameras positioned around the display device then capture these reflections, focusing on the positions of the pupils and corneal reflections, which are crucial for determining eye orientation and movement. The reflections are converted into digital signals by the cameras, resulting in raw data consisting of images and reflection points representing the eye’s position over time. This raw data undergoes initial processing by the display device’s onboard processors to filter out noise and enhance the clarity and accuracy of the captured eye features. Once processed, the data is transmitted to the central processing unit (CPU) or a dedicated eye-tracking module within the display device system via high-speed data buses to ensure minimal latency. Finally, the processed data is stored in the device’s memory, which may include volatile memory (RAM) for immediate processing needs or non-volatile memory (such as an SSD) for longer-term storage. This stored data includes detailed logs of eye position coordinates, timestamps, and initial processing results, providing a foundation for subsequent analysis and utilization in the AR / VR environment.

[0066] With reference now to FIG. 4, there is shown a method 400 of securely displaying a document or other content for viewing by a user. The method 400 is carried out by a document viewing display system, such as the document viewing display system 10, 110, 210. In particular, in at least some embodiments, the method 400 is configured for use on an AR / VR display device, such as the AR / VR display device 120 of the document viewing display system 110. Further, according to embodiments, the method 400 is used to present the user with content of a secure document without other individuals (or “eavesdroppers”) viewing the content while the user does. In some embodiments, a computer application, such as the computer application 124 of the AR / VR display device. Although the discussion of the method 400 discussed the steps as being performed in a particular order, the steps of the method 400 may be performed in any technically-feasible order, as will be appreciated by those skilled in the art in view of the following discussion; for example, in embodiments, step 420 is performed prior to or concurrently with step 410.

[0067] The method 400 begin with step 410, wherein a document to be viewed by a user of a display device of a display device system is obtained. In embodiments, the document is obtained through accessing a secure memory device, such a secure cloud-data storage or secure local storage. The document may be comprised of a computer file and may include textual content, images, and other graphics, for example. In some embodiments, the document is a to-be-executed document, meaning the document is to be provided to a user for signature. The method 400 continues to step 420.

[0068] In step 420, sensor data is captured using at least one sensor of the display device system. The at least one sensor 16 may be, for example, any of a variety of sensors installed on the display device 20 and / or in communication therewith. For example, the document viewing display system 110 includes two sensors 116a,b that are each installed as a part of the AR / VR display device 120 and that each provides sensor data to the processing subsystem 112. The types of sensors used for the at least one sensor 16 may vary, such as where the first sensor 116a is of a first sensor type (e.g., camera) and the second sensor 116b is of a second sensor type (e.g., microphone, accelerometer). First sensor data from the first sensor 116a and second sensor data from the second sensor 116b is provided to the processing subsystem 112. In some embodiments, the sensor data is stored in memory of the processing subsystem 112 and / or stored remotely, such as in the memory 108 of the remote processing subsystem 104.

[0069] In at least some embodiments, the at least one sensor is a biometric sensor that is used for sensing characteristics of a user of the document viewing display system, such as awearer of the AR / VR display device 120. The biometric sensors that may be used for capturing the sensor data include, for example, any one or more of the following: accelerometers, gyroscopes, magnetometers, inside-out tracking cameras, proximity sensors, and optical sensors. Accelerometers and gyroscopes capture unique patterns of head and body movements, creating a biometric signature for each user. This data can be compared during authentication to verify the user’s identity. In some embodiments, the biometric sensors may include iris recognition sensors that capture and analyze unique iris patterns for user authentication. This iris data provides an additional layer of security, leveraging the distinctiveness of the user’s iris to ensure precise and reliable identification within the AR / VR environment. Magnetometers assist in positional tracking and orientation, providing additional data points for user authentication. Changes in magnetic field orientation can be used as part of the authentication process. Inside-out tracking cameras capture the user’s facial features or other identifying characteristics for biometric authentication, with facial recognition algorithms applied to authenticate users based on captured images. Proximity sensors detect when the headset is worn or removed, triggering authentication prompts or locking the device when not in use to prevent unauthorized access. Optical sensors, combined with computer vision algorithms, recognize hand gestures or other unique movements as part of a multi-factor authentication process. By combining data from these sensors, AR / VR platforms can implement robust user authentication systems that adapt to individual user behaviors and characteristics. This multi-sensor approach enhances security while maintaining a seamless user experience in virtual reality environments, at least in embodiments. The method 400 continues to step 430.

[0070] In step 430, sensor-based user identification data is determined based on the sensor data. The sensor-based user identification data is data that is indicative of unique characteristics of the user of the document viewing display system. The sensor-based user identification data is determined based on the sensor data. This sensor-based user identification data is indicative of the unique characteristics of an individual. By fusing data from accelerometers, gyroscopes, magnetometers, inside-out tracking cameras, proximity sensors, and optical sensors, a comprehensive and unique profile of the current user is created. In embodiments, this fusion process involves integrating these diverse data points using machine learning techniques. Initially, data preprocessing may be performed to clean and normalize the sensor data, ensuring consistency and accuracy. Feature extraction techniques can then be applied to identify key attributes from each sensor’s data stream, such as movement patterns from accelerometers and gyroscopes, positional changes frommagnetometers, facial features from cameras, and gestures from optical sensors. Once the features are extracted, machine learning (ML) algorithms such as support vector machines (SVM), neural networks, or ensemble methods like random forests can be employed to combine these features into a cohesive user profile representing the current user detected at the document viewing display system. These algorithms may be trained on a dataset of sensor readings paired with known user identities, allowing them to learn the unique patterns and correlations between different sensor data points that characterize each individual. In some embodiments, the sensor-based user identification data is latent feature data generated by a ML model, such as a neural network, that takes input from multiple different sensors, for example. The method 400 continues to step 440.

[0071] In step 440, the user is authenticated using the sensor-based user identification data to determine identity information of the user. In at least one embodiment, this involves leveraging ML techniques to compare real-time sensor data with pre-existing user profiles. The obtained identification data undergoes feature extraction to identify key attributes from each sensor's data stream. These features, forming unique biometric signatures, are input into a pre-trained machine learning model. For authentication, real-time sensor data is compared against stored profiles using algorithms like support vector machines (SVM), neural networks, or ensemble methods. The model analyzes patterns and relationships in the data to determine the likelihood of a match. Probabilistic models or deep learning techniques handle complex, non-linear relationships, enhancing accuracy. In one embodiment, the ML model generates a confidence score for each potential match, and if this score exceeds a predefined threshold, the user is authenticated. This process grants access to the VR environment or specific functionalities based on the authenticated identity. At least according to implementations, this multi-sensor, machine learning-driven approach ensures robust, accurate, and adaptive authentication, enhancing security and user experience in AR / VR environments. The method 400 continues to step 450.

[0072] In step 450, it is determined whether to display the document and / or display attributes for display of the document are determined. When the user is authenticated, identity information is determined as a result, and this identity information indicates an identity of the user. Using the determined identity, user authorization data for the user is obtained, and this user authorization data indicates whether the user has access to the document and, in embodiments, what type of access (e.g., read only, write access). In some embodiments, in addition to determining whether to display the document, display attributes for use in displaying the document may be used, such as, for example, access to specificsections of the document based on the user’s role and / or signature capabilities. These display attributes ensure that the document is presented in a manner tailored to the user’s needs and access rights, enhancing both usability and security. The method 400 continues to step 460.

[0073] In step 460, the document is displayed for viewing by the user and in accordance with the display attributes, if any. In embodiments, the document is displayed for viewing by the user on an AR / VR display device 120, such as display 114. The display of the document is configured in accordance with the determined display attributes, if any. These attributes may include personalized layout settings, font sizes, color schemes, and access to specific sections of the document based on the user's role or preferences. By leveraging the capabilities of the AR / VR display device, the document can be presented in an immersive and interactive format, enhancing the user experience. The system ensures that the display of the document is both secure and tailored to the individual user, aligning with their authorization level and personal settings. This approach not only improves usability but also maintains the integrity and confidentiality of the displayed content in a virtual environment. The method 400 then ends.

[0074] In some embodiments, the method 400 further includes a step of receiving a signature from the user at the display device after displaying the document for viewing in step 450. In embodiments where the display device is an AR / VR display device, the signature may be provided by the user through, for example, gesture recognition, where the user signs in the air using hand movements tracked by the device’s sensors, or by using a virtual keyboard or stylus within the AR / VR interface. This signature input method ensures that the process remains intuitive and seamless, leveraging the capabilities of the AR / VR technology to authenticate the user’s approval or consent in a secure and user-friendly manner.

[0075] With reference to FIG. 5, there is shown a method 500 of securely displaying a document or other content for viewing by a user. The method 500 is carried out by a document viewing display system, such as the document viewing display system 10, 110, 210. In particular, in at least some embodiments, the method 500 is configured for use on an open-view display device, such as the open-view display device 220 of the document viewing display system 210. Further, according to embodiments, the method 500 is used to present the user with content of a secure document without other individuals (or “eavesdroppers”) viewing the content while the user does. In some embodiments, a computer application, such as the computer application 224 of the open-view display device220. Although the discussion of the method 500 discussed the steps as being performed in a particular order, the steps of the method 500 may be performed in any technically-feasible order, as will be appreciated by those skilled in the art in view of the following discussion; for example, in embodiments, step 520 is performed prior to or concurrently with step 510.

[0076] The method 500 begin with step 510, wherein a document to be viewed by a user of a display device of a display device system is obtained. This step is analogous to step 410 of the method 400 and that discussion of the step 410 is hereby incorporated and attributed to the step 510 to the extent that discussion is not inconsistent with the express discussion of the step 510. The method 500 continues to step 520.

[0077] In step 520, sensor data is captured using an eye tracking sensor of the display device system. In one embodiment, the eye tracking sensor 216 captures sensor data (referred to as eye tracking sensor data) by emitting near-infrared (NIR) light towards the user’ s eyes so as to illuminate the user’s eyes without causing discomfort as the light is invisible to the human eye. This illumination creates reflections on the surface of the eyes, including the pupil and cornea. Cameras and other sensor components within the eye tracking sensor 216 then capture these reflections in real-time. The captured sensor data includes high- resolution images and reflection points that detail the positions and movements of the user’s eyes. This raw data is fundamental, providing the basis for further processing to determine gaze information, but at this stage, it solely represents the immediate optical reflections recorded by the eye tracking sensor 216. The method 500 continues to step 530.

[0078] In step 530, a line of sight direction of the user is determined. As used herein, the term “line of sight direction,” when used in conjunction with a user viewing a display, refers to the direction the user is looking or direction in which the user’s eyes are focused. This line of sight direction may be determined by analyzing the relative positions of key eye features, such as the pupil center and corneal reflections, captured by the eye tracking sensor 216. The eye tracking sensor 216, which may include NIR emitters and cameras, illuminates the eyes and captures the necessary reflections to calculate this direction, at least in some embodiments.

[0079] The eye tracking sensor data captured by the eye tracking sensor 216 is used to extract or otherwise determine key features such as the pupil center and corneal reflections (often referred to as glints). Advanced image processing algorithms analyze these features, calculating the relative positions and angles to determine the gaze or line of sight direction, which represents the direction in which the user is looking. Calibration data, specific to theuser and the display setup, may be applied to refine the accuracy of this gaze direction determination. The method 500 continues to step 540.

[0080] In step 540, a line of sight area of the document is determined based on the line of sight direction. As used herein, the term “line of sight area,” when used in conjunction with a user viewing a display screen or other display area of a display, refers to a specific region on the display area or document that corresponds to the user’s line of sight direction, corresponding to an area of the display screen that the user is looking and / or focused. In embodiments, this area is identified by mapping the calculated gaze direction onto the display coordinates, taking into account the screen’s geometry and the user’s position relative to the display. The line of sight area indicates the portion of the content that the user is looking at, which could include various types of information such as text, images, or interactive elements.

[0081] Once the line of sight direction is established, it is translated into specific coordinates on the display screen in order to determine an anchor point (or starting or reference point) for determining the line of sight area. This translation considers the display’s geometry, including screen or display area size and resolution, as well as the user’s position relative to the screen or display area. In embodiments, gaze coordinates refer to one or more coordinates indicating where the user is looking on the document or display area. The system identifies the corresponding area of interest (the area of focus or attention), which could be a specific section of text, an image, or an interactive graphical element. This may involves using the gaze coordinates as a center of a circular area having a predefined radius of a particular resolution or display area size; or may include identifying a line of text withing a text document that the user is looking at and determining the line of sight area to be the area having the preceding N words and following M words, where N and M are positive integers, such as 3 or 6. In embodiments, advanced algorithms may be used to handle certain common document structures or types of content being displayed. The method 500 continues to step 550.

[0082] In step 550, the document is displayed on the display device for viewing of the line of sight area by the user whereby adjacent portions of the document outside of the line of sight area are hidden or prevented from being displayed on the display device. For example, the open-view display device 220 of the document viewing display system 210 is used to display the content in the line of sight area, while obfuscating or preventing other content of the document from being displayed. At least in embodiments, the method ensures that only the specific region corresponding to the user’s line of sight direction is visible on thescreen, while adjacent portions of the document outside of this line of sight area are hidden or prevented from being displayed. This selective display technique not only allows the user to focus exclusively on the content they are currently looking at, providing a streamlined and distraction-free viewing experience, but also enhances security, as discussed below.

[0083] By obscuring content outside the line of sight area, it prevents Optical Character Recognition (OCR) software and unauthorized images of the screen from capturing the full or large portions of the document content. Such unauthorized images may be obtained by closed circuit television (CCTV) or other monitoring cameras, user mobile devices (e.g., smartphones) including wearable devices (e.g., Ray -Ban™ Meta™ smart glasses, smart contact lenses). These emerging technologies present future attack paths where humans can wear smart glasses or use contact lenses equipped with cameras to discreetly record confidential data within a AR / VR display device. Another potential threat is Neuralink™ or similar brain-computer interfaces, which could allow users to directly capture or memorize confidential data without any physical traceability. It is advantageous for AR / VR display systems to identify and mitigate these potential attack vectors to reduce security risks. Implementing detection mechanisms and security measures tailored to these new technologies enables safeguarding sensitive information in AR / VR display environments, ensuring that unauthorized access and replication of data are minimized. More generally, according to implementations, this approach safeguards sensitive information by ensuring that only the visible, line of sight area is exposed, thus reducing the risk of the entire document being accessed or replicated through unauthorized means. The method 500 ends.

[0084] It is to be understood that the foregoing description is of one or more embodiments of the invention. The invention is not limited to the particular embodiment(s) disclosed herein, but rather is defined solely by the claims below. Furthermore, the statements contained in the foregoing description relate to the disclosed embodiment(s) and are not to be construed as limitations on the scope of the invention or on the definition of terms used in the claims, except where a term or phrase is expressly defined above. Various other embodiments and various changes and modifications to the disclosed embodiment s) will become apparent to those skilled in the art.

[0085] As used in this specification and claims, the terms “e.g.,” “for example,” “for instance,” “such as,” and “like,” and the verbs “comprising,” “having,” “including,” and their other verb forms, when used in conjunction with a listing of one or more components or other items, are each to be construed as open-ended, meaning that the listing is not to beconsidered as excluding other, additional components or items. Other terms are to be construed using their broadest reasonable meaning unless they are used in a context that requires a different interpretation. In addition, the term “and / or” is to be construed as an inclusive OR. Therefore, for example, the phrase “A, B, and / or C” is to be interpreted as covering all of the following: “A”; “B”; “C”; “A and B”; “A and C”; “B and C”; and “A, B, and C ”

Claims

CLAIMS1. A method of securely displaying a document or other content for viewing by a user, comprising: obtaining a document to be viewed by a user of a display device of a display device system; capturing sensor data using at least one sensor of the display device system; determining sensor-based user identification data based on the sensor data; authenticating the user using the sensor-based user identification data to determine identity information of the user; determining whether to display the document and / or determining display attributes for display of the document; and displaying the document for viewing by the user and in accordance with the display attributes, if any.

2. The method of claim 1, wherein the at least one sensor is or includes a biometric sensor, and wherein the display device is an AR / VR device that includes the biometric sensor.

3. The method of claim 2, wherein the biometric sensor is a microphone, a fingerprint sensor, a photoplethysmography (PPG) sensor, a near infrared (NIR) sensor, other infrared (IR) sensor, or a camera.

4. The method of claim 2, wherein the document is displayed as a graphic in an augmented reality (AR) or virtual reality (VR) fashion and in a manner in which the document as displayed is only viewable by the user.

5. The method of claim 2, wherein the AR / VR device is used to sign the document in order to produce an electronic signature indicating an identity of the user.

6. The method of claim 5, wherein the AR / VR device includes a computer application configured to obtain and display the document in a confidential manner so that the user isable to sign the document to produce the electronic signature without the document being viewable by others during signing of the document.

7. The method of claim 6, wherein the computer application is configured to prevent copying, downloading, screenshotting, and screen recording while the document is being viewed by the user.

8. The method of claim 1, further comprising: tracking a line of sight of the user through use of eye tracking data captured by an eye tracking sensor to determine a line of sight area of the document within the line of sight of the user; wherein displaying the document includes displaying the document so that a non-line of sight area is obfuscated and content of the document in the line of sight area remains interpretable.

9. The method of claim 8, wherein the eye tracking sensor is a near infrared (NIR) sensor.

10. The method of claim 8, wherein the display device is a smartphone, television, laptop screen, desktop or computer monitor, tablet, infotainment unit display of a vehicle, or other open-view display device.

11. The method of claim 8, wherein the content of the document within the line of sight area includes text that is readable and the non-line of sight area includes other text that is blurred or otherwise obfuscated so as to be unreadable.

12. The method of claim 1, wherein the authenticating the user includes generating biometric profile data of the user as the sensor-based user identification data and determining the identity information of the user based on the biometric profile data.

13. The method of claim 12, wherein the at least one sensor used for obtaining the sensor data is or includes an accelerometer, gyroscope, magnetometer, camera, proximity sensor, or an optical sensor.

14. The method of claim 12, wherein the at least one sensor includes a first sensor and a second sensor, wherein the sensor data includes first sensor data obtained from the first sensor and second sensor data obtained from the second sensor, and wherein a context-awarebiometric profile of the user is generated based on the first sensor data and the second sensor data.

15. The method of claim 14, wherein a machine learning (ML) technique is used to generate the identity information of the user based on the first sensor data and the second sensor data.

16. A method of securely displaying a document or other content for viewing by a user, comprising: obtaining a document to be viewed by a user of a display device of a display device system; capturing sensor data from an eye tracking sensor of the display device system; determining a line of sight direction of the user; determining a line of sight area of the document based on the line of sight direction; and displaying the document on the display device for viewing of the line of sight area by the user whereby adjacent portions of the document outside of the line of sight area are hidden or prevented from being displayed on the display device.

17. The method of claim 16, wherein the display device is a smartphone, television, laptop screen, desktop or computer monitor, tablet, infotainment unit display of a vehicle, or other open-view display device.

18. The method of claim 16, wherein the eye tracking sensor is a near infrared (NIR) sensor.

19. The method of claim 16, wherein the content of the document within the line of sight area includes text that is readable and the non-line of sight area includes other text that is blurred or otherwise obfuscated so as to be unreadable.

20. The method of claim 16, wherein displaying the document includes displaying the document so that content of the document in a non-line of sight area is obfuscated and content of the document in the line of sight area remains interpretable.

21. The method of claim 20, wherein the content of the document within the line of sight area includes text that is readable and the non-line of sight area includes other text that is blurred or otherwise obfuscated so as to be unreadable.

22. The method of claim 16, further comprising: authenticating the user to determine identity information of the user; wherein the authenticating the user includes generating a context-aware biometric profile of the user based on sensor data obtained from at least one sensor and determining the identity information of the user based on the context-aware biometric profile.

23. The method of claim 22, wherein the at least one sensor used for obtaining the sensor data is or includes an accelerometer, gyroscope, magnetometer, camera, proximity sensor, or an optical sensor.

24. The method of claim 22, wherein the at least one sensor includes a first sensor and a second sensor, wherein the sensor data includes first sensor data obtained from the first sensor and second sensor data obtained from the second sensor, and wherein context-aware biometric profile of the user is generated based on the first sensor data and the second sensor data.

25. The method of claim 24, a machine learning (ML) technique is used to generate the identity information of the user based on the first sensor data and the second sensor data.

Images