System and method for transmitting and using image data

Abstract

A system includes a first integrated circuit (52) for obtaining first image data from a first camera (16), obtaining second image data from a second camera (16), compressing the first image data, and combining one or more frames of the second image data into a single frame (100). A first integrated circuit (52) transmits first image data in a compressed form (66) and second image data in an uncompressed form via a single frame via an image data delivery protocol connection. The system includes a second integrated circuit (54) configured to receive first image data in a compressed form from the first integrated circuit (52), decrypt the first image data, use the first image data for display on an electronic display coupled to the second integrated circuit (54), and display the first image data on the electronic display. Receiving second image data via the single frame via the image data delivery protocol connection, and performing user authentication using the second image data.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to U.S. Application No. 63 / 586,367, filed September 28, 2023, entitled “System and Methods for Transmission and Consumption of Image Data,” the entire contents of which are incorporated herein by reference for all purposes. Background Technology

[0003] This disclosure relates generally to image processing, and more specifically to transmitting compressed and uncompressed image data between different devices for display and / or authentication.

[0004] Electronic devices typically employ one or more authentication methods to ensure authorized users have access to the device and / or sensitive information it may contain. For example, these authentication methods may include passwords or biometric sensors (e.g., fingerprint sensors, facial recognition sensors) to authenticate the user of the electronic device. Electronic devices may include multiple integrated circuit systems that can operate in combination to provide a realistic experience for the user. However, it may be difficult to provide a separate authentication system for each of these multiple integrated circuit systems. Additionally, compressing secure data before transmission can reduce latency. However, compression can lead to loss of confidentiality, increased vulnerabilities, and reduced data integrity, among other things. Summary of the Invention

[0005] This disclosure provides systems and methods for transmitting and using image data (such as biometric data) to authenticate users of electronic devices. For example, an electronic device may include two integrated circuits that can transmit biometric image data from one integrated circuit to the other without compression, while simultaneously transmitting other image data in a compressed format. A first integrated circuit of the two integrated circuits can obtain the biometric data, and a second integrated circuit of the two integrated circuits can use the biometric data to authenticate a user. The first integrated circuit may include one or more cameras that can capture biometric images (e.g., images of human eyes) and other images (e.g., environmental background images). Additionally or alternatively, the first integrated circuit may capture biometric data via one or more biometric sensors. The first integrated circuit may compress image data other than the biometric image data via streaming encoding or decoding circuitry and transmit the image data to the second integrated circuit. Thus, the first integrated circuit can avoid compressing the biometric data (e.g., biometric images, data from other biometric sensors). Instead, the first integrated circuit can provide the biometric data to the second integrated circuit by transmitting uncompressed frames of the biometric data. The first integrated circuit can combine multiple frames of the biometric data into a single frame (e.g., a superframe) and transmit the biometric image data via the single frame using an image data delivery protocol (e.g., DisplayPort (DP), Low Power DisplayPort (LPDP)).

[0006] The second integrated circuit can receive compressed image data and uncompressed biometric data. The second integrated circuit can then decompress the image data for display. Furthermore, the authentication system of the second integrated circuit can perform authentication based on the uncompressed biometric data received via the image data delivery protocol. Sending the biometric data in a superframe (rather than via a separate stream) via the image data delivery protocol reduces latency and enables improved security of the biometric data within the superframe. Attached Figure Description

[0007] A better understanding of the various aspects of this disclosure can be achieved by reading the following detailed description and referring to the accompanying drawings, in which:

[0008] Figure 1 It is a block diagram of an electronic device including an electronic display according to the implementation plan;

[0009] Figure 2 It is based on the implementation plan. Figure 1 A front view of a mobile phone, an example of an electronic device;

[0010] Figure 3It is based on the implementation plan. Figure 1 A front view of a tablet device, an example of an electronic device;

[0011] Figure 4 It is based on the implementation plan. Figure 1 A front view of a laptop computer, an example of an electronic device;

[0012] Figure 5 It is based on the implementation plan. Figure 1 Front and side views of an example watch, an electronic device;

[0013] Figure 6 It is based on the implementation plan. Figure 1 A front view of a computer showing examples of electronic devices;

[0014] Figure 7 It is based on the implementation plan. Figure 1 A block diagram of an electronic device system, the system including a first integrated circuit communicating with a second integrated circuit;

[0015] Figure 8 This is a block diagram illustrating the transmission of image data between a first integrated circuit and a second integrated circuit according to an implementation scheme;

[0016] Figure 9 This is a block diagram illustrating the placement of biometric data in a superframe according to an implementation scheme;

[0017] Figure 10 This is a block diagram illustrating the transmission of compressed and uncompressed image data according to an implementation scheme;

[0018] Figure 11 This is a block diagram illustrating the various data types included in a superframe according to the implementation scheme; and

[0019] Figure 12 This is a flowchart illustrating the transmission and use of image data between the first integrated circuit and the second integrated circuit, according to the implementation plan. Detailed Implementation

[0020] One or more specific embodiments of this disclosure will now be described. These described embodiments are merely examples of the technology currently disclosed. Additionally, in an attempt to provide a brief description of these embodiments, not all characteristics of an actual embodiment may be described in this specification. It should be understood that in the development of any such actual embodiment, as in any engineering or design project, decisions specific to many embodiments must be made to achieve the developer’s specific objectives, such as compliance with system-related and business-related constraints that may vary from one embodiment to another. Furthermore, it should be understood that such development efforts can be complex and time-consuming, but for those skilled in the art who benefit from this disclosure, such development efforts may remain routine tasks of design, preparation, and manufacture.

[0021] When describing elements of various embodiments of this disclosure, the articles “a,” “an,” and “the” are intended to mean one or more elements present among the elements. The terms “comprising,” “including,” and “having” are intended to be inclusive and mean that additional elements may exist in addition to those listed. Additionally, it should be understood that references to “an embodiment” or “an embodiment” of this disclosure are not intended to be construed as excluding the existence of additional embodiments that also incorporate the described features. Furthermore, the phrase A “based on” B is intended to mean that A is at least partially based on B. Moreover, the term “or” is intended to be inclusive (e.g., logical OR) and not exclusive (e.g., logical XOR). In other words, the phrase A “or” B is intended to mean A, B, or both A and B.

[0022] This disclosure relates to transmitting and using image data and / or biometric data (e.g., security data) between a first integrated circuit and a second integrated circuit. The second integrated circuit may include an authentication system that can authenticate a user based on biometric data received from the first integrated circuit. Additionally, the first integrated circuit may include one or more cameras that can capture one or more images of the user and / or the environment. The first integrated circuit can receive image data associated with the one or more images from the one or more cameras and capture the biometric data.

[0023] The first integrated circuit may further include streaming encoding or decoding circuitry that implements a compression protocol to compress the image data. The first integrated circuit can then transmit the image data (or any other suitable data, such as audio and / or video data) to the second integrated circuit via the streaming encoding or decoding circuitry. Furthermore, the first integrated circuit can combine (e.g., stitch together) multiple frames of the image data from the one or more cameras into a superframe. The first integrated circuit can then transmit (according to an image data protocol) uncompressed image data corresponding to the biometric data (e.g., as a superframe). The image data protocol is one that enables the delivery of the biometric data between the first and second integrated circuits.

[0024] The second integrated circuit can receive compressed image data and the biometric data. The second integrated circuit may also include streaming encoding or decoding circuitry that can decompress the image data so that the second integrated circuit can use the image data for display. Additionally, the authentication system of the second integrated circuit can perform authentication based on the biometric data received via the image data delivery protocol (e.g., as multiple uncompressed frames; as a superframe combining biometric data from multiple sources, such as two image frames from two eye-facing cameras). Therefore, sending the biometric data via a superframe (rather than via a separate stream) reduces latency in the transmission of the biometric data and enables improved secure transmission of the biometric data within the superframe. Furthermore, sending both the compressed and uncompressed image data reduces latency in the transmission of both the image data and the biometric data.

[0025] In view of the above, Figure 1 Example electronic device 10 can be used to authenticate users using such a system. As described in more detail below, electronic device 10 can be any suitable electronic device, such as a computer, mobile phone, portable media device, tablet computer, television, virtual reality headset, wearable device (e.g., watch), vehicle dashboard, etc. Therefore, it should be noted that... Figure 1 This is merely an example of a specific implementation and is intended to illustrate the types of components that may exist in electronic device 10.

[0026] Electronic device 10 may include one or more electronic displays 12, input devices 14, eye trackers 15, one or more cameras 16, input / output (I / O) ports 17, processor core complexes 18 having one or more processors or processor cores, local memory 20, main memory storage devices 22, network interfaces 24, power supplies 26, and image processing circuitry 28. Figure 1The various components described herein may include hardware elements (e.g., circuitry), software elements (e.g., a tangible, non-transitory computer-readable medium storing instructions), or combinations of hardware and software elements. It should be understood that the various components may be combined into fewer components or separated into additional components. For example, local memory 20 and main memory storage device 22 may be included in a single component. Furthermore, image processing circuitry 28 (e.g., a graphics processing unit, a display image processing pipeline, etc.) may be included in the processor core complex 18 or implemented separately.

[0027] The processor core complex 18 is operatively coupled to local memory 20 and main memory storage device 22. Therefore, the processor core complex 18 can execute instructions stored in local memory 20 or main memory storage device 22 to perform operations such as generating and / or sending image data for display on electronic display 12. Thus, the processor core complex 18 may include one or more general-purpose microprocessors (such as Reduced Instruction Set Computing (RISC) processors), one or more application-specific integrated circuits (ASICs), one or more field-programmable gate arrays (FPGAs), or any combination thereof.

[0028] In addition to program instructions, local memory 20 or main memory storage device 22 may store data to be processed by processor core complex 18. Therefore, local memory 20 and / or main memory storage device 22 may include one or more tangible, non-transitory computer-readable media. For example, local memory 20 may include random access memory (RAM), and main memory storage device 22 may include read-only memory (ROM), rewritable non-volatile memory such as flash memory, hard disk drive, optical disk, etc.

[0029] Network interface 24 can transmit data with another electronic device or network. For example, network interface 24 (e.g., radio frequency system) can enable electronic device 10 to communicatively couple to a personal area network (PAN) (such as a Bluetooth network), a local area network (LAN) (such as an 802.11x Wi-Fi network), and / or a wide area network (WAN) (such as a 4G, LTE, or 5G cellular network).

[0030] Power source 26 can provide power to operate the processor core complex 18 and / or other components in the electronic device 10. Therefore, power source 26 may include any suitable energy source, such as a rechargeable lithium polymer (Li-poly) battery and / or an alternating current (AC) power converter.

[0031] I / O port 17 enables electronic device 10 to interact with various other electronic devices. For example, when a portable storage device is connected, I / O port 17 enables processor core complex 18 to transmit data to the portable storage device. Furthermore, input device 14 enables a user to interact with electronic device 10. For example, input device 14 may include buttons, keyboard, mouse, touchpad, etc. Additionally or alternatively, electronic display 12 may include touch sensing components that enable user input to electronic device 10 by detecting the occurrence and / or location of an object touching its screen (e.g., the surface of electronic display 12).

[0032] Additionally, the electronic display 12 may be a display panel having one or more display pixels. For example, the electronic display 12 may include a self-emitting pixel array having an array of one or more self-emitting pixels or liquid crystal pixels. The electronic display 12 may include any suitable circuitry (e.g., display driver circuitry) to drive the self-emitting pixels, including, for example, row drivers and / or column drivers (e.g., display drivers). Each self-emitting pixel may include any suitable light-emitting element, such as an LED (e.g., an OLED or microLED). However, any other suitable type of pixel may also be used, including non-self-emitting pixels (e.g., liquid crystals used in liquid crystal displays (LCDs), digital micromirror devices (DMDs) used in DMD displays). The electronic display 12 may control the light emission from the display pixels to present a visual representation of information, such as a graphical user interface (GUI) of an operating system, an application interface, still images, or video content, by displaying image data frames. For displaying images, the electronic display 12 may include display pixels implemented on the display panel. The display pixel can represent a subpixel, each subpixel controlling the brightness value of a color component (e.g., red, green, or blue for an RGB pixel arrangement, or red, green, blue, or white for an RGBW arrangement). As used herein, a display pixel can refer to a set of subpixels (e.g., red, green, and blue subpixels) or a single subpixel.

[0033] Additionally, the electronic display 12 can control light emission from display pixels to present a visual representation of information based on image data corresponding to the visual representation of the information. For example, the electronic display 12 can present graphics, including a graphical user interface (GUI) of an operating system, an application interface, still images, or video content, by at least partially based on image data display frames. Specifically, the electronic display 12 is operatively coupled to a processor core complex 18 and an image processing circuit 28 such that the electronic display 12 can display frames based on image data generated by the processor core complex 18 or the image processing circuit 28. As will be described herein, the electronic display 12 can receive image data frames, for example, captured by one or more cameras 16, via a network interface 24, an input device 14, and / or an I / O port 17. In some embodiments, the electronic display 12 may represent multiple displays that can display image data perceived as a single frame corresponding to the left and right eyes.

[0034] One or more cameras 16 may be positioned facing a viewer (e.g., one or both eyes of the viewer, the viewer's face, one or both hands of the viewer, and the viewer's body, etc.) and / or facing the environment surrounding the viewer. For example, one or more cameras 16 may include four cameras facing one or both eyes of the viewer (e.g., an upper left camera, an upper right camera, a lower left camera, and a lower right camera). As another example, in some embodiments, one or more cameras 16 may include six additional cameras facing outward and downward, outward and sideways, and / or facing the viewer's chin (e.g., mouth). As another example, in some embodiments, one or more cameras 16 may include two additional cameras, each facing outward to the general environment. As yet another example, in some embodiments, one or more cameras 16 may include two additional cameras, which include a short-range depth camera (e.g., a short-range depth sensor) and a long-range depth camera (e.g., a long-range depth sensor). Image data captured by the short-range depth camera and the long-range depth camera may be combined with image data captured from the two additional cameras facing the general environment to enable the creation of a three-dimensional scene of the environment. In this way, one or more cameras 16 can capture image data associated with the viewer and / or the environment.

[0035] Eye tracker 15 measures the position and movement of one or both eyes of a person viewing the electronic display 12 of electronic device 10. For example, eye tracker 15 may include a camera (e.g., camera 16) that records the movement of the viewer's eyes as the viewer (e.g., user) views the electronic display 12. However, several different operations can be employed to track the movement of the viewer's eyes. For example, different types of infrared / near-infrared eye-tracking techniques, such as bright pupil tracking and dark pupil tracking, can be used. In both types of eye tracking, infrared or near-infrared light is reflected from one or both eyes of the viewer to produce a corneal reflection. The vector between the center of the pupil and the corneal reflection can be used to determine the point on the electronic display 12 that the viewer is looking at. When generating / processing image data for display on the electronic display 12, processor core complex 18 can use the viewing angle of the viewer's eyes. Furthermore, processor core complex 18 can use iris or retinal data from one or both eyes of the viewer to capture biometric data and perform authentication.

[0036] As described above, the electronic display 12 can display an image by controlling the brightness output (e.g., light emission) of subpixels based on corresponding image data. In some embodiments, the pixel or image data may be generated by an image source (such as a processor core complex 18, a graphics processing unit (GPU), or an image sensor (e.g., a camera 16)). Additionally, in some embodiments, image data may be received from another electronic device 10, for example, via a network interface 24 and / or I / O port 17. Furthermore, in some embodiments, the electronic device 10 may include multiple electronic displays 12 and / or may perform image processing (e.g., via image processing circuitry 28) on one or more external electronic displays 12, such as those connected via network interface 24 and / or I / O port 17.

[0037] Electronic device 10 can be any suitable electronic device. For ease of illustration, an example of a suitable electronic device 10, particularly a handheld device 10A, is shown below. Figure 2 In some implementations, the handheld device 10A can be a portable phone, media player, personal data manager, handheld gaming platform, etc. For illustrative purposes, the handheld device 10A can be a smartphone, such as any iPhone available from Apple Inc. ® model.

[0038] The handheld device 10A may include a housing 30 (e.g., a casing) to protect internal components from physical damage and / or shield them from electromagnetic interference. The housing 30 may at least partially enclose the electronic display 12. In the depicted embodiment, the electronic display 12 displays a graphical user interface (GUI) 32 with an array of icons 34. By way of example, an application can be launched when an icon 34 is selected via an input device 14 or a touch-sensing component of the electronic display 12.

[0039] The input device 14 can be accessed through an opening in the housing 30. Furthermore, the input device 14 enables the user to interact with the handheld device 10A. For example, the input device 14 allows the user to activate or deactivate the handheld device 10A, navigate the user interface to the home screen, navigate the user interface to a user-configurable application screen, activate voice recognition features, provide volume control, and / or switch between vibration and ring modes. Additionally, the I / O port 17 can also be opened through the housing 30. Furthermore, the electronic device may include one or more cameras 16 to capture images or videos. In some embodiments, the camera 16 may be used in conjunction with virtual reality or augmented reality visualizations on the electronic display 12.

[0040] Figure 3 Another example of a suitable electronic device 10 is shown, specifically a tablet device 10B. For illustrative purposes, the tablet device 10B may be any iPad available from Apple Inc. ® model. Figure 4 Another example of a suitable electronic device 10 is shown, specifically a computer 10C. For illustrative purposes, computer 10C could be any MacBook available from Apple Inc. ® or IMAC ® model. Figure 5 Another example of a suitable electronic device 10 is shown, specifically a watch 10D. For illustrative purposes, the watch 10D can be any Apple Watch available from Apple Inc. ® Model. As depicted, the tablet device 10B, computer 10C, and watch 10D each also include an electronic display 12, an input device 14, an I / O port 17, and a housing 30. The electronic display 12 can display a GUI 32. Here, the GUI 32 displays a visualization of a clock. When the visualization is selected via the input device 14 or the touch-sensing component of the electronic display 12, an application can be launched, such as transforming the GUI 32 into a presentation. Figure 2 and Figure 3 Icon 34 is discussed in the text.

[0041] See Figure 6Computer 10E can represent Figure 1 Another embodiment of the electronic device 10. The computer 10E can be any suitable computer, such as a desktop computer, server, or laptop, but can also be a standalone media player or video game console. For example, the computer 10E could be an iMac from Apple Inc. in Cupertino, California. ® MACBOOK ® Or other similar devices. It should be noted that computer 10E may also refer to a personal computer (PC) from another manufacturer. A similar housing 30 may be provided to protect and enclose the internal components of computer 10E, such as electronic display 12. In some embodiments, the user of computer 10E may interact with computer 10E using various peripheral input devices 14, such as keyboard 14A or mouse 14B that can be connected to computer 10E.

[0042] As described above, the electronic display 12 can display images, at least in part, based on image data. The image data can be processed, for example, via image processing circuitry 28, before being used to display the corresponding image on the electronic display 12. Furthermore, image processing circuitry 28 can process image data for display on one or more electronic displays 12. For example, image processing circuitry 28 may include a display pipeline, memory-to-memory scaler and rotator (MSR) circuitry, warp compensation circuitry, or additional hardware or software components for processing image data. Image data can be processed by image processing circuitry 28 to reduce or eliminate image artifacts, compensate for one or more different software or hardware-related effects, and / or format image data for display on one or more electronic displays 12. It should be understood that this technology can be implemented in separate circuitry, software, and / or firmware, and can be considered as part of, separate from, and / or in parallel with the display pipeline or MSR circuitry.

[0043] Figure 7 It is based on the implementation plan. Figure 1 A block diagram of a system 50 of an electronic device, the system including a first integrated circuit 52 (e.g., a first integrated circuit, a first integrated circuit package formed by a plurality of integrated circuits in a package, a first portion of the first integrated circuit), the first integrated circuit communicating with a second integrated circuit 54 (e.g., a second integrated circuit, a second integrated circuit package formed by a plurality of integrated circuits in a package, a second portion of the first integrated circuit). Figure 7As illustrated, the first integrated circuit 52 can send (e.g., transmit) compressed image data to the second integrated circuit 54. For example, the compressed image data may include image data of the user's surrounding environment, image data associated with eye tracking (e.g., gaze tracking, facial expression tracking), or any other suitable image data unrelated to biometric data (e.g., security data) captured by one or more cameras 16.

[0044] The first integrated circuit 52 can also transmit uncompressed superframes, which may include biometric data for authentication. That is, the first integrated circuit 52 can capture biometric data from image data provided by one or more cameras 16. The first integrated circuit 52 can combine multiple frames of image data associated with the biometric data captured by the one or more cameras 16. For example, the first integrated circuit 52 can receive a first frame of image data from a first camera and a second frame of image data from a second camera. The first integrated circuit 52 can combine the first frame and the second frame of image data into a single superframe (uncompressed). This single superframe can be a frame structure for accommodating multiple smaller frames and data. That is, each smaller frame can be combined or stitched together to form a single frame (e.g., a single large frame). The first integrated circuit 52 can then send this single superframe to the second integrated circuit 54.

[0045] In some embodiments, the second integrated circuit 54 may capture image data from one or more cameras 16 and send compressed image data to the first integrated circuit. Additionally or alternatively, the second integrated circuit 54 may combine multiple frames of image data associated with biometric data into a single superframe and send that single superframe to the first integrated circuit 52. In this way, the first integrated circuit 52 and the second integrated circuit 54 may each send and / or receive compressed image data and uncompressed superframes. Additional details regarding the transmission and use of image data between the first integrated circuit 52 and the second integrated circuit 54 will be discussed below.

[0046] In view of the above, Figure 8This is a block diagram illustrating the transmission of image data between a first integrated circuit 52 and a second integrated circuit 54. The first integrated circuit 52 may include one or more cameras 16 that capture image data and provide the image data to the first integrated circuit 52 via a link 60 using an image data delivery protocol such as DisplayPort (DP), Low Power DisplayPort (LPDP), or Mobile Interface Processor Interface (MIPI). For the purposes of discussion, link 60 may be referred to as an LPDP link, but may take any other suitable form. The image data may be associated with eye-tracking data, continuity data, iris identification data, and / or anti-spoofing data (e.g., data that enables prevention and / or detection of attempts to spoof the system with false or forged information). The continuity data, iris identification data, and / or anti-spoofing data may contain biometric data that can be used by the second integrated circuit 54. In some embodiments, the second integrated circuit 54 may also use other biometric data, such as facial data, fingerprint data, ear shape data, motion data (e.g., gait data), and audio data (e.g., voice data).

[0047] It should be noted that each of the first integrated circuit 52 and the second integrated circuit 54 may be included in a single integrated circuit package, or may each be integrated into a separate integrated circuit package. In other words, the first integrated circuit 52 and the second integrated circuit 54 may be separate devices, or may be components of a single device. Furthermore, either the first integrated circuit 52 or the second integrated circuit 54 may receive captured image data from the camera 16 and transmit compressed image data and / or uncompressed biometric data to each other. It should also be noted that each of the first integrated circuit 52 and the second integrated circuit 54 may be included in a single integrated circuit package, or may each be integrated into a separate integrated circuit package. In other words, the first integrated circuit 52 and the second integrated circuit 54 may be separate devices, or may be components of a single device.

[0048] Image data may include multiple frames, each associated with a corresponding camera 16. For example, a first camera may capture and transmit frames of first image data, a second camera may capture and transmit frames of second image data, and a third camera may capture and transmit frames of third image data, and so on. In some embodiments, at least some of the one or more cameras 16 may be configured to operate in a secure mode. Thus, one or more cameras 16 configured in secure mode may each output a secure image including a secure signature.

[0049] LPDP link 60 may include corresponding LPDP receive (RX) circuitry and / or transmit (TX) circuitry. Additionally, LPDP link 60 may tunnel via wired communication protocols such as Universal Serial Bus (USB) 2, USB 3, and USB 4. In some embodiments, image data may be transmitted between one or more cameras 16 and the first integrated circuit 52 via a wireless link. It should be noted that although communication is discussed as using USB and LPDP link 60 via a wired connection, any suitable communication method, protocol, and / or standard may be used. Furthermore, it should be noted that any additional number of LPDP links may be included in the first integrated circuit 52 and / or the second integrated circuit 54. For example, the first integrated circuit 52 and / or the second integrated circuit 54 may include 3 LPDP links and 5 LPDP links, etc. Each LPDP link may transmit and / or receive a corresponding superframe.

[0050] The first integrated circuit 52 may include a sensor interface 62 (e.g., a component or subsystem, an image data delivery interface) that facilitates connectivity and communication between one or more cameras 16, a processing system (e.g., a processor core complex 18, an image signal processor 64), memory 20, and / or a superframe synthesizer 68. For example, the first integrated circuit 52 may receive image data and transmit it to the sensor interface 62 via an LPDP link 60. The sensor interface 62 may deliver (e.g., send, transmit) image data to the processing system, memory 20, and / or superframe synthesizer 68. In some embodiments, each of the one or more cameras 16 may be associated with a corresponding sensor interface 62. Thus, each sensor interface 62 may handle synchronization with each other so that image data from each of the one or more cameras 16 can be received and / or delivered in a coordinated manner.

[0051] The superframe synthesizer 68 can receive at least two frames of biometric data. Additionally, the superframe synthesizer 68 can arrange or place each of the received frames of biometric data into a superframe (as a single frame). Each of the received frames of biometric data can be arranged or placed without compression (e.g., uncompressed). The superframe synthesizer 68 can then transmit (e.g., transfer) the superframe as an uncompressed superframe to the second integrated circuit 54 via the LPDP link 72. This uncompressed superframe may include biometric data for authentication. In some embodiments, the size of the superframe may be fixed for each LPDP link 72.

[0052] In some embodiments, the first integrated circuit 52 may transmit image data to an image signal processor 64 (e.g., an image signal processor). The image signal processor 64 may include hardware components and / or software algorithms that enable the processing of image data received by one or more cameras 16. The image data may be processed and converted into a usable image or video format for storage in memory and / or for display. Furthermore, the image signal processor 64 may enable the enhancement and / or improvement of the quality of image data captured by one or more cameras 16. In some embodiments, the image signal processor 64 may store and / or deliver the processed image data (or a portion of the image data) to memory 20 for further processing, editing, display on electronic display 12, and / or storage.

[0053] As described herein, image data can be delivered from sensor interface 62 to memory 20. First integrated circuit 52 can then capture biometric data. For example, first integrated circuit 52 can receive a first frame from a first camera, a second frame from a second camera, and a third frame from a third camera. First integrated circuit 52 can identify the association of the first frame with general image data (of the viewer and / or environment). Additionally, first integrated circuit 52 can identify the association of the second and third frames with the viewer's biometric data (e.g., security data). Therefore, first integrated circuit 52 can compress the first frame via streaming encoding or decoding circuitry 66. Streaming encoding or decoding circuitry 66 can implement image compression protocols (e.g., Apple Professional Quality Intermediate codec, ProRES codec) to compress the image data. Furthermore, first integrated circuit 52 can combine the second and third frames (without compression) in a superframe.

[0054] The first integrated circuit 52 can then transmit the compressed first frame of image data to the streaming encoding or decoding circuit 70 of the second integrated circuit 54 via streaming encoding or decoding circuit 66. Additionally, the first integrated circuit 52 can transmit second and third frames of image data via superframes. The first integrated circuit 52 and the second integrated circuit 54 may each include LPDP links 72, 74 to enable communication (e.g., transmission and reception) between them. LPDP links 72, 74 may be similar to and / or identical to LPDP link 60. In some embodiments, uncompressed biometric data may be transmitted frame by frame.

[0055] The second integrated circuit 54 may receive the compressed first frame via streaming encoding or decoding circuitry 70. Furthermore, streaming encoding or decoding circuitry 70 may decompress the first frame using an image compression protocol. Streaming encoding or decoding circuitry 70 may transmit (e.g., deliver) the first frame to image signal processor 76 of the second integrated circuit 54 or write the first frame directly to memory 78 of the second integrated circuit 54. Image signal processor 76 may be similar to and / or identical to image signal processor 64. If the first frame is transmitted to image signal processor 76, the first frame may be processed and written to memory 78. The first frame may be written to memory 78 to enable the display of the first frame of image data on a display (e.g., display 12, or any other suitable display communicating with and / or associated with the second integrated circuit 54).

[0056] The second integrated circuit 54 can receive superframes containing biometric data via the LPDP link 74. Furthermore, the second integrated circuit 54 may include a sensor interface 80 that facilitates connection and communication between the LPDP link 74, the image signal processor 76, the memory 78, and / or the authentication system 82. In some embodiments, such as when the superframe contains encrypted biometric data, where each frame includes a secure signature, the sensor interface 80 can transmit the superframe to the authentication system 82. The authentication system 82 can decrypt the secure signature and enable or perform verification or authentication.

[0057] In other embodiments, sensor interface 80 may write the superframe directly to memory 78 (without modifying the superframe). Authentication system 82 may include a system designed to verify the identity of a user before enabling access to the device. Authentication system 82 may use biometric data received in the superframe to perform authentication (e.g., verification) of the user. As described herein, biometric data may be based on iris identification data, continuity data, and / or anti-spoofing data.

[0058] In view of the above, Figure 9 This is a block diagram illustrating the placement of biometric data in superframe 100 according to an embodiment. During authentication or initialization, a second integrated circuit 54 may use the biometric data to perform authentication or initialization, which may include iris identification data and / or anti-spoofing data. Iris identification data enables the identification of a user of electronic device 10 (or any other suitable device). That is, the user's iris enables the identification of the user and the determination of the association between the user and electronic device 10. As an example, electronic device 10 may include a head-mounted device. The user can power on the head-mounted device, and the head-mounted device can retrieve iris identification data to verify the user's identity.

[0059] Therefore, as Figure 9As illustrated, the first integrated circuit 52 may include a first camera 16A, a second camera 16B, a third camera 16C, and a fourth camera 16D. Each of the cameras 16 may be positioned on the top, bottom, left, and / or right portion of the electronic device 10, facing one or both eyes of the user. For example, the first camera 16A may be an upper left camera, the second camera 16B may be an upper right camera, the third camera 16C may be a lower left camera, and the fourth camera 16D may be a lower right camera. The first camera 16A may capture a first frame 102 of image data, the second camera 16B may capture a second frame 104 of image data, the third camera 16C may capture a third frame 106 of image data, and the fourth camera 16D may capture a fourth frame 108 of image data.

[0060] In some implementations, each of the cameras 16 can be set to a secure mode, so that each of the first frame 102, the second frame 104, the third frame 106, and the fourth frame 108 can be a secure frame and can each include a secure signature (e.g., a digital signature). In some implementations, frames associated with the left eye and frames associated with the right eye can be captured and received simultaneously to achieve proper authentication.

[0061] The first integrated circuit 52 can receive a first frame 102 at a first sensor interface 62A (e.g., via a first LPDP link 60A), a second frame 104 at a second sensor interface 62B (e.g., via a second LPDP link 60B), a third frame 106 at a third sensor interface 62C (e.g., via a third LPDP link 60C), and a fourth frame 108 at a fourth sensor interface 62D (e.g., via a fourth LPDP link 60D). Furthermore, the first sensor interface 62A can deliver the first frame 102 to the superframe synthesizer 68, the second sensor interface 62B can deliver the second frame 104 to the superframe synthesizer 68, the third sensor interface 62C can deliver the third frame 106 to the superframe synthesizer 68, and the fourth sensor interface 62D can deliver the fourth frame 108 to the superframe synthesizer 68. Each sensor interface in sensor interface 62 can handle synchronization with each other so that the first frame 102, the second frame 104, the third frame 106 and / or the fourth frame 108 can be received and transmitted in a coordinated manner at the superframe synthesizer 68, for example, simultaneously.

[0062] As described herein, the superframe synthesizer 68 can place or arrange the first frame 102, the second frame 104, the third frame 106, and the fourth frame 108 within the superframe 100 without compression. For example, the first frame 102 may include an upper-left camera frame and may be placed in the upper-left corner of the superframe 100, and the second frame 104 may include an upper-right camera frame and may be placed in the upper-right corner of the superframe 100. As another example, the third frame 106 may include a lower-left camera frame and may be placed in the lower-left corner of the superframe 100, and the fourth frame 108 may include a lower-right camera frame and may be placed in the lower-right corner of the superframe 100. The superframe synthesizer 68 can then transmit the superframe 100 to the second integrated circuit 54 via the LPDP link 72. The superframe 100 can be transmitted in grid scan order. It should be noted that, although the above describes... Figure 9 Four cameras are described, but any suitable number of cameras can be implemented when capturing biometric data. For example, the illustrated implementation may include two cameras, six cameras, eight cameras, and so on.

[0063] In some implementations, the iris identification frame may include a full-resolution frame (e.g., 1280×1280 10bpp) and may operate for multiple frames at a display rate of frames per second (e.g., 90, 100, 120). Metadata for the iris identification data may be generated by camera 16 or image signal processor 64. Furthermore, the anti-spoofing data may also be encrypted and include a secure signature (e.g., digitally signed). The anti-spoofing data may also include a full-resolution frame and may operate at a display rate of frames per second. Metadata for the anti-spoofing data may also be generated by the sensor. Additionally, supplementary metadata associated with the iris identification data and the anti-spoofing data may be provided by image signal processors (e.g., 64, 76). Furthermore, external general-purpose input / output (GPIO) ports (e.g., I / O port 17) may drive camera 16 to implement unencrypted or encrypted modes. For example, a GPIO high may indicate an unencrypted mode, and a GPIO low may indicate an encrypted mode (e.g., anti-spoofing mode).

[0064] Therefore, transmitting secure frames (e.g., first frame 102, second frame 104, third frame 106, and fourth frame 108) via superframe 100 without compression can minimize or reduce the loss of data confidentiality, vulnerabilities, and / or reduced data integrity. Furthermore, transmitting secure frames via superframe 100 without compression can improve latency in communication.

[0065] Sometimes, ownership of electronic device 10 may change. That is, a user may stop using electronic device 10 (and / or remove electronic device 10), and a different or additional user may begin using electronic device 10. Therefore, electronic device 10 can identify the continuity of use of electronic device 10 to determine whether a user (e.g., the current user) has completed authentication, or whether an additional user is currently initializing electronic device 10 without completing authentication.

[0066] In view of the above, Figure 10 This is a block diagram illustrating the transmission of compressed image data and superframe 100 (e.g., an uncompressed superframe) according to an embodiment. In some embodiments, camera 16 may enable eye tracking of a user. First integrated circuit 52 may transmit eye-tracking data to second integrated circuit 54 via streaming encoding or decoding circuitry 66. However, iris identification data may continue to be transmitted to second integrated circuit 54 via superframe 100. Furthermore, first integrated circuit 52 may support the identification or monitoring of the continuity of use of electronic device 10.

[0067] The use of continuous identification or monitoring can be a security mechanism that enables user detection without performing full authentication. Continuous data may include frames with secure signatures, which may be periodically transmitted along with eye-tracking data over time periods (e.g., every second and every two seconds, etc.). Furthermore, continuous data allows electronic device 10 to determine whether authentication can be repeated for additional users or whether authentication can be omitted for the same user. Therefore, the first integrated circuit 52 can transmit continuous data in parallel with eye-tracking data transmitted via superframe 100 and via streaming encoding or decoding circuitry 66. The following will discuss… Figure 10 Further details regarding parallel transmission will be discussed.

[0068] As illustrated, the first integrated circuit 52 may include a first camera 16A and a second camera 16B. The first camera 16A may capture a first set of images, and the second camera 16B may capture a second set of images. The first set of images and / or the second set of images may include image data associated with eye-tracking data and / or continuity data. The first set of images (e.g., first image data) may be received by a first sensor interface 62A and may be split (e.g., divided) into first eye-tracking data and first continuity data. Additionally, the second set of images may be received by a second sensor interface 62B and may be split into second eye-tracking data and second continuity data.

[0069] The first gaze tracking data can be transmitted to the streaming encoding or decoding circuit 66, and the first continuity data can be transmitted to the superframe synthesizer 68. The streaming encoding or decoding circuit 66 can implement an image compression protocol to compress the first gaze tracking data and send the first gaze tracking data to the second integrated circuit 54. In addition, the superframe synthesizer 68 can receive the first continuity data and arrange one or more frames of the first continuity data in a superframe 100.

[0070] The second integrated circuit 54 can receive the compressed first gaze tracking data via the streaming encoding or decoding circuit 70. Furthermore, the second integrated circuit 54 can also receive superframe 100 via an LPDP link (e.g., 72, 74). The streaming encoding or decoding circuit 70 can decompress the first gaze tracking data using an image compression protocol and transmit the first gaze tracking data to the image signal processor 76 and / or memory 78.

[0071] In parallel with the above (e.g., at the same or similar times), second gaze-tracking data may be transmitted to streaming encoding or decoding circuitry 66, and second continuity data may be transmitted to superframe synthesizer 68. Streaming encoding or decoding circuitry 66 may implement an image compression protocol to compress the second gaze-tracking data and transmit it to second integrated circuit 54. Additionally, superframe synthesizer 68 may receive the second continuity data and arrange one or more frames of the second continuity data together with the first continuity data. For example, superframe synthesizer 68 may arrange the first continuity data (e.g., first frame 102) on the left portion of superframe 100 and arrange the second continuity data (e.g., second frame 104) on the right portion of superframe 100. It should be noted that one or more frames of the first continuity data and one or more frames of the second continuity data may be arranged simultaneously (at the same or similar times) by superframe synthesizer 68. Superframe synthesizer 68 may then transmit the second continuity data to second integrated circuit 54 via superframe 100.

[0072] The second integrated circuit 54 can receive compressed second gaze-tracking data via streaming encoding or decoding circuit 70. Streaming encoding or decoding circuit 70 can decompress the second gaze-tracking data using an image compression protocol and transmit the second gaze-tracking data to image signal processor 76 and / or memory 78. Furthermore, the second integrated circuit 54 can receive superframe 100 via an LPDP link (e.g., 72, 74) and can transmit superframe 100 to image signal processor 76 for decryption of secure signatures. It should be noted that although the first camera 16A and the second camera 16B have been described above, any suitable number of cameras can be used when capturing gaze-tracking data and / or continuity data.

[0073] Therefore, continuous data allows the second integrated circuit 54 to determine whether authentication can be repeated for additional users or whether authentication can be omitted for the same user. Thus, the reduced or minimized authentication process improves or increases the operating power of the second integrated circuit 54. Furthermore, transmitting continuous data via superframe 100 reduces latency by transmitting continuous data via a dedicated channel.

[0074] Superframe 100 can include various data types associated with image data. Figure 11 This is a block diagram illustrating various data types that may be included in a superframe 100 according to an embodiment. Each type of data illustrated may be associated with either the left camera (e.g., in camera 16) or the right camera (e.g., in camera 16). Figure 11 As illustrated, superframe 100 may include left metadata 130, right metadata 132, left grid data 134, right grid data 136, left keying data 138, right keying data 140, gaps 142, left red, green, or blue (RGB) pixel arrangement data 144, and / or right RGB pixel arrangement data 146. Left metadata 130 and / or right metadata 132 may include camera settings and status, statistics calculated by camera 16, image signal processing settings and status, statistics calculated by image signal processors (e.g., 64, 76), timestamps, and / or settings and status of auxiliary devices (e.g., eye blinking LED on or off).

[0075] Figure 12 This is a flowchart 160 illustrating the transfer and use of image data and biometric data between a first integrated circuit 52 and a second integrated circuit 54, according to an embodiment. The transfer and use of image data can be performed by any suitable device (e.g., a controller) of a component of the controllable electronic device 10 (or an attached electronic device), such as a processor core complex 18 (or an attached processor associated with the attached electronic device) . In some embodiments, flowchart 160 can be implemented by using the processor core complex 18 to execute instructions stored in a tangible, non-transitory computer-readable medium (such as memory (e.g., 20, 78)). For example, flowchart 160 can be executed at least in part by one or more software components (such as an operating system of the electronic device 10 and one or more software applications of the electronic device 10). Although flowchart 160 is described using a specific order of steps, it should be understood that the steps described herein are contemplated to be performed in a different order than illustrated, and some described steps may be skipped or not performed at all. Furthermore, it should be noted that the first integrated circuit 52 and / or the second integrated circuit 54 can perform each of the steps described below.

[0076] At block 162, the processor core complex 18 may receive image data from the camera 16 at the first integrated circuit 52. For example, the processor core complex 18 may receive image data including an image of the user's iris. At block 164, the processor core complex 18 may capture biometric data from the image data at the first integrated circuit 52. That is, the processor core complex 18 may extract features associated with biometric features from the image data. As an example, the processor core complex may extract features associated with unique features of the user's iris (e.g., texture or unique pattern). Image data not associated with biometric data may be separate image data. That is, a portion of the image data may be biometric data, and another portion of the image data may be separate image data.

[0077] At block 166, processor core complex 18 may compress image data at first integrated circuit 52 and transmit it to second integrated circuit 54. As described herein, processor core complex 18 may compress and transmit image data via streaming encoding or decoding circuitry 66 using a compression protocol. Additionally, at block 168, processor core complex 18 may transmit biometric data via superframe 100 (e.g., a single frame). In practice, superframe synthesizer 68 may arrange or place each received frame of biometric data within superframe 100 without compressing each received frame. Superframe 100 may be transmitted as an uncompressed superframe to second integrated circuit 54 via LPDP link 60.

[0078] At block 170, processor core complex 18 may receive image data and biometric data at second integrated circuit 54. As described herein, the image data may be compressed before transmission. Therefore, at block 172, processor core complex 18 may decompress the image data via streaming encoding or decoding circuitry 70 for use by image signal processor 76. Additionally, at block 174, processor core complex 18 may perform authentication based on biometric data. For example, processor core complex 18 may compare biometric data with biometric data stored in memory (e.g., 20, 78) to determine whether a similarity score exceeds a predefined threshold. Based on authentication, access to electronic device 10 (or any other suitable device) may be permitted or denied.

[0079] Therefore, the techniques described herein for transmitting biometric data via superframe 100 without compression minimize or reduce loss of data confidentiality, vulnerabilities, and / or reduced data integrity. Furthermore, transmitting biometric data via superframe 100 without compression improves latency in communication. Additionally, the techniques described herein for transmitting biometric data via superframe 100 while simultaneously transmitting other image data in compressed format improve latency in communication while maintaining the integrity of the biometric data.

[0080] The specific embodiments described above have been shown by way of example, and it should be understood that various modifications and alternatives are permissible. It should also be understood that the claims are not intended to be limited to the specific forms disclosed, but rather to cover all modifications, equivalents, and alternatives falling within the substance and scope of this disclosure.

[0081] The techniques described herein and protected by the claims are referenced and applied to specific examples of physical and practical nature that significantly improve the technical field and are therefore not abstract, intangible, or purely theoretical. Furthermore, if any claim appended to this specification contains one or more elements designated as “component for [performing] [function]…” or “step for [performing] [function]…”, it is intended that such elements should be interpreted in accordance with 35 USC 112(f). However, for any claim containing elements designated in any other manner, it is intended that such elements should not be interpreted in accordance with 35 USC 112(f).

[0082] As is widely recognized, the use of personally identifiable information should comply with privacy policies and practices that are generally accepted to meet or exceed industry or governmental requirements for protecting user privacy. Specifically, personally identifiable information data should be managed and processed to minimize the risk of unintentional or unauthorized access or use, and the nature of authorized use should be clearly explained to users.

Claims

1. A system comprising: A first integrated circuit, the first integrated circuit being configured to: First image data is obtained from the first camera; Second image data is obtained from the second camera; Compress the first image data; Combine one or more frames of the second image data into a single frame; The first image data is sent in compressed form; as well as The second image data is transmitted in uncompressed form via the single frame through the image data delivery protocol connection; and A second integrated circuit, the second integrated circuit being configured to: Receive the compressed form of the first image data from the first integrated circuit; Decrypt the first image data; The first image data is used for display on an electronic display coupled to the second integrated circuit; The second image data is received via the single frame through the image data delivery protocol connection; as well as The second image data is used to perform user authentication.

2. The system of claim 1, wherein the image data delivery protocol connection includes a low-power DisplayPort (LPDP) connection.

3. The system according to claim 1, wherein the second image data corresponds to biometric data.

4. The system of claim 3, wherein the biometric data includes iris identification data, continuity data, anti-spoofing data, or any combination thereof.

5. The system of claim 1, wherein the first image data includes eye-tracking data.

6. The system of claim 1, wherein the first integrated circuit is configured to combine one or more frames of the second image data into the single frame via a synthesizer.

7. The system of claim 1, wherein each frame of the one or more frames of the second image data is encrypted to include a secure signature.

8. The system of claim 7, the system comprising the second integrated circuit configured to decrypt the secure signature and perform authentication based on the second image data.

9. The system of claim 1, wherein the single frame includes metadata, grid data, matting data, gaps, red, green, or blue (RGB) pixel arrangement data, or any combination thereof, associated with the second image data.

10. The system of claim 1, wherein the second camera is configured to operate in a secure mode, the secure mode causing the second camera to output a secure frame of the second image data including a secure signature.

11. The system of claim 1, wherein the second image data is associated with one or both eyes.

12. A first integrated circuit, the first integrated circuit comprising: A first camera, configured to capture first image data; A second camera, configured to capture second image data; A first image data delivery interface is configured to: receive the first image data, send a first portion of the first image data to a streaming encoding or decoding circuit, and send a second portion of the first image data to a synthesizer; and A second image data delivery interface is configured to receive the second image data, send a first portion of the second image data to the streaming encoding or decoding circuit, and send a second portion of the second image data to the synthesizer.

13. The first integrated circuit of claim 12, wherein the streaming encoding or decoding circuit is configured to compress the first portion of the first image data and the first portion of the second image data.

14. The first integrated circuit of claim 13, wherein the streaming encoding or decoding circuit is configured to send a compressed first portion of the first image data and a compressed first portion of the second image data to the second integrated circuit.

15. The first integrated circuit of claim 12, wherein the synthesizer is configured to arrange one or more frames of the second portion of the first image data and one or more frames of the second portion of the second image data in a single frame.

16. The first integrated circuit of claim 15, wherein the synthesizer is configured to transmit the single frame to the second integrated circuit via a low-power DisplayPort (LPDP) connection.

17. The first integrated circuit of claim 12, wherein the first portion of the first image data and the first portion of the second image data include gaze tracking data.

18. The first integrated circuit of claim 12, wherein the second portion of the first image data and the second portion of the second image data comprise continuous data.

19. A system comprising: A first integrated circuit, the first integrated circuit being configured to: Receive image data; Biometric data is obtained from the image data; The image data is transmitted in compressed form via streaming encoding or decoding circuitry; and The biometric data is transmitted in uncompressed form via a single frame.

20. The system of claim 19, wherein the single frame comprises two or more frames of the biometric data.

Images